silent semantic changes with reiser4

silent semantic changes with reiser4

[Christoph Hellwig]

After looking trough the code and mailinglists I'm quite unhappy with
a bunch of user-visible changes that Hans sneaked in and make reiser4
incompatible with other filesystems and have a slight potential to break
even in the kernel.

 o files as directories
   - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
     .htaccess handling in apache and glibc compilation this also renders
     this flag entirely useless and opens up the races it tries to
     prevent against cmpletely useless
   - meaning of the -x permission.  This one has different meanings on
     directories vs files on UNIX systems.  If we want to support
     directories as files we'll probably have to find a way to work
     around this.
   - dentry aliasing.  I can't find a formal guarantee in the code this
     can't happen
 
 o metafiles - ..metas as a magic name that's just taken out of the
   namespace doesn't sound like a good idea.  If we want this it should
   be a VFS-level option and there should be a translation-layer to
   xattrs.  Not doing this will again confuse applications greatly that
   expect uniform filesystem behaviour.

Given these problems I request that these interfaces are removed from
reiser4 for the kernel merge, and if added later at the proper VFS level
after discussion on linux-kernel and linux-fsdevel, like we did for
xattrs.

Re: silent semantic changes with reiser4

[Lee Revell]

On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote:
>    - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
>      .htaccess handling in apache and glibc compilation this also renders
>      this flag entirely useless and opens up the races it tries to
>      prevent against cmpletely useless

So `find -type d' would list every file on the system?

Lee

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Tue, Aug 24, 2004 at 04:35:18PM -0400, Lee Revell wrote:
> On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote:
> >    - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
> >      .htaccess handling in apache and glibc compilation this also renders
> >      this flag entirely useless and opens up the races it tries to
> >      prevent against cmpletely useless
> 
> So `find -type d' would list every file on the system?

the find I have here is using lstat and not open with O_DIRECTORY, so
no.

Re: silent semantic changes with reiser4

[Lee Revell]

On Tue, 2004-08-24 at 16:38, Christoph Hellwig wrote:
> On Tue, Aug 24, 2004 at 04:35:18PM -0400, Lee Revell wrote:
> > On Tue, 2004-08-24 at 16:25, Christoph Hellwig wrote:
> > >    - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
> > >      .htaccess handling in apache and glibc compilation this also renders
> > >      this flag entirely useless and opens up the races it tries to
> > >      prevent against cmpletely useless
> > 
> > So `find -type d' would list every file on the system?
> 
> the find I have here is using lstat and not open with O_DIRECTORY, so
> no.
> 

Ugh, how embarrassing, I completely forgot about stat().

Lee

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> > >    - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
> > >      .htaccess handling in apache and glibc compilation this also renders
> > >      this flag entirely useless and opens up the races it tries to
> > >      prevent against cmpletely useless
> > 
> > So `find -type d' would list every file on the system?
> 
> the find I have here is using lstat and not open with O_DIRECTORY, so
> no.

The find-like program I use (called treescan) uses O_DIRECTORY as an
optimisation.  It assumes that O_DIRECTORY will only open objects
which are directories and can be read using readdir().

However, if reiser4 returns d_type values, then it won't even attempt
an open on non-DT_DIR objects, and that's a better optimisation.
(reiserfs doesn't return d_type values, unfortunately).

So the list of files that treescan finds depends on whether reiser4
implements d_type.

This is nothing like a POSIX filesystem.  You untar a tree, and then
listing it recursively shows extra things created by reiser4.

I quite like the principle, but because it's not like POSIX and
doesn't match some program's expectations, it's a problem in its
present form.

xattrs aren't a complete solution as you can't store structured data
in an xattr.  For example, with reiser4's model, you can cd into a
.tar, .zip, .mp3 or .xml file and list the internal structure along
with the file's metadata.  You can't do that with xattrs.

Programs exist which quite reasonably assume that when you create a
file, you can't opendir() the file, and recursive listings (like find,
ls -R et al.) won't automatically traverse into every file.

On the other hand, being able to enter a file in a directory-like way
allows structured representations of the contents to be accessed in
the very useful "everything's a file" way -- i.e. ordinary tools.

So here's a semantic proposal:

   1. O_DIRECTORY won't open an ordinary file.
      Corollary: opendir("file") won't open an ordinary file.

   2. An ordinary file path followed by "/" won't open an ordinary file.
      Corollary: opendir("file/") won't open an ordinary file.

      This is because appending a trailing slash is an alternate
      way for userspace to get the same results as O_DIRECTORY.

   3. An ordinary file path followed by "/" _and_ one or more path
      components will open the file as a directory and enter it.
      Corollary: opendir("file/.") will open an ordinary file.

   4. The type of "file/." shall be S_IFDIR, _not_ S_IFREG.
      Corollary: stat("file/.") will return that it's a directory.

      The intention here is that explicit requests to examine the
      metadata or alternate structure representations of a file will
      create such a view, but the view is only available if requested
      explicitly.

      When such a view is created, the results of stat(), O_DIRECTORY
      and opendir() are absolutely consistent.  This will minimise
      confusion.  Programs which recurse over a directory tree won't
      look inside any of the files.  However they can be explicitly
      asked to recurse starting from a path inside a file: then
      they'll recurse over a single file's metadata and structured data.

Regarding the problems of safe locking in the VFS.  The VFS assumes
that directories are not hard linked: i.e. that they cannot appear at
more than one path in a filesystem.  Files-as-directories breaks that.

However, VFS does support directories on multiple paths, using bind mounts.

So it wouldn't be out of the question if entering a file (as described
above) effectively auto-mounted a bind mount at that point.


-- Jamie

Re: silent semantic changes with reiser4

[Jeff Garzik]

Christoph Hellwig wrote:
>  o files as directories
>    - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
>      .htaccess handling in apache and glibc compilation this also renders
>      this flag entirely useless and opens up the races it tries to
>      prevent against cmpletely useless


Ouch.

I would definitely classify this as a security hole, since userland 
definitely uses O_DIRECTORY to avoid races.

	Jeff

Re: silent semantic changes with reiser4

[viro]

On Tue, Aug 24, 2004 at 04:38:25PM -0400, Jeff Garzik wrote:
> Christoph Hellwig wrote:
> > o files as directories
> >   - O_DIRECTORY opens succeed on all files on reiser4.  Besides breaking
> >     .htaccess handling in apache and glibc compilation this also renders
> >     this flag entirely useless and opens up the races it tries to
> >     prevent against cmpletely useless
> 
> 
> Ouch.
> 
> I would definitely classify this as a security hole, since userland 
> definitely uses O_DIRECTORY to avoid races.

Feh.  That's far from the worst parts of the mess introduced by "hybrid"
crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit
higher on the suckitude scale, IMO.

Re: silent semantic changes with reiser4

[viro]

On Tue, Aug 24, 2004 at 09:53:44PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
 
> Feh.  That's far from the worst parts of the mess introduced by "hybrid"
> crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit
> higher on the suckitude scale, IMO.

While we are at it - consider these hybrids vetoed until
	a) sys_link()/sys_link() deadlock is fixed
	b) sys_link()/sys_rename() deadlock is fixed
	c) correctness proof of the locking scheme (in
Documentation/filesystems/directory-locking) is updated to match the
presense of the file/directory hybrids.

Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince
us that there's no more crap of that kind".  IMO a reasonable request, seeing
that the first look at the patches in -mm4 had turned up two exploits in
that area, despite the *YEARS* of warnings about potential trouble and need
to be careful there (actually, I've given Hans too much credit and assumed
that link/link never happens since nobody would be dumb enough to provide
->link() method for non-directory inodes; turns out that somebody is dumb
enough and link/link is as exploitable as link/rename).

Re: silent semantic changes with reiser4

[Hans Reiser]

I allowed myself to get talked out of a final top to bottom code audit, 
and obviously that was a mistake.

It will probably take about 6 weeks.  Apologies for wasting your time 
before that was done.

Hans


viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Tue, Aug 24, 2004 at 09:53:44PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
>  
>
>>Feh.  That's far from the worst parts of the mess introduced by "hybrid"
>>crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit
>>higher on the suckitude scale, IMO.
>>    
>>
>
>While we are at it - consider these hybrids vetoed until
>	a) sys_link()/sys_link() deadlock is fixed
>	b) sys_link()/sys_rename() deadlock is fixed
>	c) correctness proof of the locking scheme (in
>Documentation/filesystems/directory-locking) is updated to match the
>presense of the file/directory hybrids.
>
>Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince
>us that there's no more crap of that kind".  IMO a reasonable request, seeing
>that the first look at the patches in -mm4 had turned up two exploits in
>that area, despite the *YEARS* of warnings about potential trouble and need
>to be careful there (actually, I've given Hans too much credit and assumed
>that link/link never happens since nobody would be dumb enough to provide
>->link() method for non-directory inodes; turns out that somebody is dumb
>enough and link/link is as exploitable as link/rename).
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Wed, Aug 25, 2004 at 11:28:56AM -0700, Hans Reiser wrote:
> I allowed myself to get talked out of a final top to bottom code audit, 
> and obviously that was a mistake.
> 
> It will probably take about 6 weeks.  Apologies for wasting your time 
> before that was done.

I don't think you'll get anywhere with auditing.  We need to write down
the semantics you want, define them at the VFS level and make sure
they're not conflicting with defined userspace semantics or kernel
assumptions.

I think you need to learn the basic distinction between the VFS layer
and a lowlevel filesystem driver.

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>
>
>I don't think you'll get anywhere with auditing.  We need to write down
>the semantics you want, define them at the VFS level and make sure
>they're not conflicting with defined userspace semantics or kernel
>assumptions.
>
>I think you need to learn the basic distinction between the VFS layer
>and a lowlevel filesystem driver.
>
>
>
>  
>
How old are you?  I thought you were the guy at Linux Tag with fashion 
oriented hair who gave a talk on his XFS work?  Did I confuse you with 
someone else?


Hans

Re: silent semantic changes with reiser4

[William Lee Irwin III]

On Thu, Aug 26, 2004 at 02:02:29AM -0700, Hans Reiser wrote:
> How old are you?  I thought you were the guy at Linux Tag with fashion 
> oriented hair who gave a talk on his XFS work?  Did I confuse you with 
> someone else?

Index: mm1-2.6.9-rc1/CREDITS
===================================================================
--- mm1-2.6.9-rc1.orig/CREDITS	2004-08-26 10:37:57.424567967 -0700
+++ mm1-2.6.9-rc1/CREDITS	2004-08-26 11:35:24.977260109 -0700
@@ -2677,6 +2677,14 @@
 S: 79098 Freiburg
 S: Germany
 
+N: Hans Reiser
+E: reiser@namesys.com
+W: http://www.namesys.com/
+D: official Linux kernel hairstyle critic
+S: 6979 Exeter Dr
+S: Oakland, CA 94611
+S: USA
+
 N: Joerg Reuter
 E: jreuter@yaina.de
 W: http://yaina.de/jreuter/

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, William Lee Irwin III wrote:
>  
> +N: Hans Reiser
> +E: reiser@namesys.com
> +W: http://www.namesys.com/
> +D: official Linux kernel hairstyle critic
> +S: 6979 Exeter Dr
> +S: Oakland, CA 94611
> +S: USA
> +

LOL.

In all fairness, how _would_ you have described it?

Besides, I don't think this should go in the CREDITS file, since hair 
styling criticism is clearly an ongoing MAINTAINERS issue, no?

		Linus

Re: silent semantic changes with reiser4

[William Lee Irwin III]

On Thu, 26 Aug 2004, William Lee Irwin III wrote:
>> +N: Hans Reiser
>> +E: reiser@namesys.com
>> +W: http://www.namesys.com/
>> +D: official Linux kernel hairstyle critic
>> +S: 6979 Exeter Dr
>> +S: Oakland, CA 94611
>> +S: USA

On Thu, Aug 26, 2004 at 11:49:19AM -0700, Linus Torvalds wrote:
> LOL.
> In all fairness, how _would_ you have described it?
> Besides, I don't think this should go in the CREDITS file, since hair 
> styling criticism is clearly an ongoing MAINTAINERS issue, no?

As far as maintenance goes, absolutely. As far as describing it goes,
I probably would have said it was short, red, and spiky.


Index: mm1-2.6.9-rc1/MAINTAINERS
===================================================================
--- mm1-2.6.9-rc1.orig/MAINTAINERS	2004-08-26 10:38:09.867927190 -0700
+++ mm1-2.6.9-rc1/MAINTAINERS	2004-08-26 11:58:27.759469733 -0700
@@ -1352,6 +1352,12 @@
 L:	linuxppc64-dev@lists.linuxppc.org
 S:	Supported
 
+LINUX KERNEL HAIRSTYLE CRITICISM
+P: Hans Reiser
+M: reiser@namesys.com
+W: http://www.namesys.com/
+S: Maintained
+
 LINUX SECURITY MODULE (LSM) FRAMEWORK
 P:	Chris Wright
 M:	chrisw@osdl.org

Re: silent semantic changes with reiser4

[Hans Reiser]

William Lee Irwin III wrote:

>
>As far as maintenance goes, absolutely. As far as describing it goes,
>I probably would have said it was short, red, and spiky.
>
>  
>
Ah, so it was him, is he disowning XFS these days or?

Re: silent semantic changes with reiser4

[William Lee Irwin III]

William Lee Irwin III wrote:
>> +LINUX KERNEL HAIRSTYLE CRITICISM

On Fri, Aug 27, 2004 at 12:10:12PM +0200, Giuseppe Bilotta wrote:
> Hm. He doesn't critic the Linux Kernel hairstyle, but the Linux 
> Kernel Developers' ...

Absolutely right.


Index: mm1-2.6.9-rc1/MAINTAINERS
===================================================================
--- mm1-2.6.9-rc1.orig/MAINTAINERS	2004-08-26 10:38:09.000000000 -0700
+++ mm1-2.6.9-rc1/MAINTAINERS	2004-08-27 08:27:42.738139199 -0700
@@ -1352,6 +1352,12 @@
 L:	linuxppc64-dev@lists.linuxppc.org
 S:	Supported
 
+LINUX KERNEL DEVELOPER'S HAIRSTYLE CRITICISM
+P: Hans Reiser
+M: reiser@namesys.com
+W: http://www.namesys.com/
+S: Maintained
+
 LINUX SECURITY MODULE (LSM) FRAMEWORK
 P:	Chris Wright
 M:	chrisw@osdl.org

Re: silent semantic changes with reiser4

[William Lee Irwin III]

On Fri, Aug 27, 2004 at 08:25:39AM -0700, William Lee Irwin III wrote:
>> +LINUX KERNEL DEVELOPER'S HAIRSTYLE CRITICISM

> Aren't there more than one Linux Kernel Developer?  Or is this to
> indicate that he indeed only criticised the hairstyle of one developer?

Best I repair that, then.


Index: mm1-2.6.9-rc1/MAINTAINERS
===================================================================
--- mm1-2.6.9-rc1.orig/MAINTAINERS	2004-08-26 10:38:09.000000000 -0700
+++ mm1-2.6.9-rc1/MAINTAINERS	2004-08-27 09:18:59.234195262 -0700
@@ -1352,6 +1352,12 @@
 L:	linuxppc64-dev@lists.linuxppc.org
 S:	Supported
 
+LINUX KERNEL DEVELOPERS' HAIRSTYLE CRITICISM
+P: Hans Reiser
+M: reiser@namesys.com
+W: http://www.namesys.com/
+S: Maintained
+
 LINUX SECURITY MODULE (LSM) FRAMEWORK
 P:	Chris Wright
 M:	chrisw@osdl.org

Re: silent semantic changes with reiser4

[Hans Reiser]

I had not intended to respond to this because I have nothing positive to 
say, but Andrew said I needed to respond and suggested I should copy 
Linus. Sigh.

Dear Christoph,

Let me see if I can summarize what you and your contingent are saying, 
and if I misconstrue anything, let me know.;-)

You ignored everything I said during the discussion of xattrs about how 
there is no need to have attributes when you can just have files and 
directories, and that xattrs reflected a complete ignorance of name 
space design principles.  When I said we should just add some nice 
optional features to files and directories so that they can do 
everything that attributes can do if they are used that way, you just 
didn't get it.  You instead went for the quick ugly hack called xattrs.  
You then got that ugly hack done first, because quick hacks are, well, 
quick.  I then went about doing it the right way for Reiser4, and got 
DARPA to fund doing it.  I was never silent about it.

Making files into directories caused only two applications out of the 
entire OS to notice the change, and that was because of a bug in what 
error code we returned that we are going to fix.  You think that was a 
disaster; I think it was a triumph.

Now a cleanly architected filesystem with no attributes and just files 
and directories that can do everything attributes are used for exists.  
You don't want it to have the competitive advantage.  Instead, you want 
it to have its clean design excised until you have something that 
duplicates it ready to go, and only then should it be allowed that users 
will use the features of your competitor's filesystem which you 
disdained implementing for so long.

Since you never studied or understood namespace design principles (or 
you would not have created and supported xattrs), you want to rename it 
to be called VFS, rewrite what we have done, and take over as the 
maintainer, mangling its design in a committee clusterfuck as you go. 

We have just implemented very trivial semantic enhancements of the FS 
namespace, nothing like as ambitious as www.namesys.com/whitepaper.html 
or WinFS, and you are already pissing your pants.

Is that a fair summary?

Eat my dust. 

Hans

PS

I should of course qualify what I have said.  The use of files and 
directories in place of attributes is not a finished work.  It has bugs, 
sys_reiser4() does not yet work, and there are little features still 
missing like having files readdir ignores.

Still, except for the bugs, what we have is usable, and there are a lot 
of happy reiser4 users right now even with the bugs. It will need a 
little bit more time, and then all the pieces will be in place.

PPS

If you implement your filesystems as reiser4 plugins, and rename 
reiser4's plugin code to be called "vfs", your filesystems will go 
faster.  Not as fast as reiser4 though, because it has a better layout 
and that affects performance a lot, but faster is faster....  See 
www.namesys.com/benchmarks.html for details.

PPPS

 Since we have such a performance lead, Namesys is about to change its 
focus from the storage layer to semantics, look at 
www.namesys.com/whitepaper.html for details.  Semantic enhancements are 
the important stuff, and finally Namesys is where we have all the 
storage layer prerequisites done right, and the real work can begin.  
The gap between us is about to widen further.



Christoph Hellwig wrote:

>After looking trough the code and mailinglists I'm quite unhappy with
>a bunch of user-visible changes that Hans sneaked in and make reiser4
>incompatible with other filesystems 
>
if we leave you in the dust, run faster.... not my problem....


>Given these problems I request that these interfaces are removed from
>reiser4 for the kernel merge, and if added later at the proper VFS level
>after discussion on linux-kernel and linux-fsdevel, like we did for
>xattrs.
>
>
>  
>
If you can't help fight WinFS, then get out of the way.  Namesys is on 
the march.  Read www.namesys.com/whitepaper.html.

Or, be smart, recognize that reiser4 is faster and more flexible than 
your storage layers because we are older and wiser and worked harder at 
it, join the team, and start contributing plugins that tap into the 
higher performance it offers.

Microsoft tried to build a storage layer that could handle small objects 
without losing performance, failed, and gave up at considerable cost to 
their architecture and pocketbook.

We just broke a hole in the enemy line.  You could come swarming through 
it with us, but it sounds like you prefer complaining to HQ that we are 
getting too far in front of you.

Re: silent semantic changes with reiser4

[Matthew Wilcox]

That's a nice marketing talk.  Get back to us when you have some technical
contribution to make.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: silent semantic changes with reiser4

[Hans Reiser]

Matthew Wilcox wrote:

>That's a nice marketing talk.  Get back to us when you have some technical
>contribution to make.
>
>  
>
www.namesys.com/download.html

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote:
> You ignored everything I said during the discussion of xattrs about how 
> there is no need to have attributes when you can just have files and 
> directories, and that xattrs reflected a complete ignorance of name 
> space design principles.

Actually in most of the discussion you simply didn't participate.  While
xattrs might not be the nicest interface they have the advantag of not
breaking the SuS assumption of what directories vs files are, and they
do not break the Linux O_DIRECTORY semantics that are defined and need
to solve real-world races either.

> When I said we should just add some nice 
> optional features to files and directories so that they can do 
> everything that attributes can do if they are used that way, you just 
> didn't get it.  You instead went for the quick ugly hack called xattrs.  
> You then got that ugly hack done first, because quick hacks are, well, 
> quick.  I then went about doing it the right way for Reiser4, and got 
> DARPA to fund doing it.  I was never silent about it.

For one thing _I_ didn't decide about xattrs anyway.  And I still
haven't seen a design from you on -fsdevel how you try to solve the
problems with files as directories.

> Now a cleanly architected filesystem with no attributes and just files 
> and directories that can do everything attributes are used for exists.  
> You don't want it to have the competitive advantage.  Instead, you want 
> it to have its clean design excised until you have something that 
> duplicates it ready to go, and only then should it be allowed that users 
> will use the features of your competitor's filesystem which you 
> disdained implementing for so long.

My competitors filesystem?  If you look at MAINTAINERS I maintain only
vxfs and sysvfs, neither of which I'd suggest anyone to run their system
on.

> Since you never studied or understood namespace design principles (or 
> you would not have created and supported xattrs), you want to rename it 
> to be called VFS, rewrite what we have done, and take over as the 
> maintainer, mangling its design in a committee clusterfuck as you go. 

Hans, please stop the personal crap or the black helicopters will kidnap
you.   When was the last time you actually worked on kernel namespace
code instead of talking marketing bullshit and ignoring all real world
problems.

> If you implement your filesystems as reiser4 plugins, and rename 
> reiser4's plugin code to be called "vfs", your filesystems will go 
> faster.  Not as fast as reiser4 though, because it has a better layout 
> and that affects performance a lot, but faster is faster....  See 
> www.namesys.com/benchmarks.html for details.

Could you pass on that crack pipe please?

Re: silent semantic changes with reiser4

[Christoph Hellwig]

Btw, I just got reminded you might take my saying as an "piss of you're
idea stinks" or similar things.  So let me clarify again the actual
technical and project managment issues another time before we start
getting really personal :)

Over the last at least five years we've taken as much as possible
semantics out of the filesystems and into the VFS layer, thus having
a separation between the semantical layer (VFS) and the low level
filesystem.  Your attributes are absoultely a VFS thing and as such
should not happen at the filesystem layer, and no, that doesn't mean
they're bad per se, I just think they are a rather bad fit for Linux.

So now go on and try to work together with the other peope doing VFS
level work instead of hiding, or if you think you can't work together
with us search a nice research OS where you can take over the VFS layer,
if your ideas prove to be good I'm sure Linux will pick them up sooner
or later.

	Christoph

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> 
> Over the last at least five years we've taken as much as possible
> semantics out of the filesystems and into the VFS layer, thus having
> a separation between the semantical layer (VFS) and the low level
> filesystem.  Your attributes are absoultely a VFS thing and as such
> should not happen at the filesystem layer, and no, that doesn't mean
> they're bad per se, I just think they are a rather bad fit for Linux.

Now this I agree with, in the sense that I think that if we want to 
support this, it should be supported at a VFS layer. 

On the other hand, I think doing it inside the filesystem with ugly hacks 
is an acceptable way to prototype the idea before it's been proven to 
really be workable. Maybe it has more problems with legacy apps than we'd 
expect..

		Linus

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Wed, Aug 25, 2004 at 01:24:36PM -0700, Linus Torvalds wrote:
> Now this I agree with, in the sense that I think that if we want to 
> support this, it should be supported at a VFS layer. 
> 
> On the other hand, I think doing it inside the filesystem with ugly hacks 
> is an acceptable way to prototype the idea before it's been proven to 
> really be workable. Maybe it has more problems with legacy apps than we'd 
> expect..

Oh, I'm the last person to tell anyone how to prototype things.  I just
don't want such inconsistancies in the mainline kernel.

Re: silent semantic changes with reiser4

[viro]

On Wed, Aug 25, 2004 at 01:24:36PM -0700, Linus Torvalds wrote:
> 
> 
> On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> > 
> > Over the last at least five years we've taken as much as possible
> > semantics out of the filesystems and into the VFS layer, thus having
> > a separation between the semantical layer (VFS) and the low level
> > filesystem.  Your attributes are absoultely a VFS thing and as such
> > should not happen at the filesystem layer, and no, that doesn't mean
> > they're bad per se, I just think they are a rather bad fit for Linux.
> 
> Now this I agree with, in the sense that I think that if we want to 
> support this, it should be supported at a VFS layer. 

ACK.  However, I'm still not seeing *ANYTHING* that would look like a workable
scheme in presense of hardlinks.  Show me how to make that deadlock- and
race-free and we might very well do it in VFS.

_That_ is what's missing and it's needed no matter where it's implemented.
You want hybrid objects - you want to solve that one.  So far I've seen
nothing workable.

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Wed, 25 Aug 2004, Christoph Hellwig wrote:
>  
>
>>Over the last at least five years we've taken as much as possible
>>semantics out of the filesystems and into the VFS layer, thus having
>>a separation between the semantical layer (VFS) and the low level
>>filesystem.  Your attributes are absoultely a VFS thing and as such
>>should not happen at the filesystem layer, and no, that doesn't mean
>>they're bad per se, I just think they are a rather bad fit for Linux.
>>    
>>
>
>Now this I agree with, in the sense that I think that if we want to 
>support this, it should be supported at a VFS layer. 
>
>On the other hand, I think doing it inside the filesystem with ugly hacks 
>  
>
what is ugly? ;-/

>is an acceptable way to prototype the idea before it's been proven to 
>really be workable. Maybe it has more problems with legacy apps than we'd 
>expect..
>
>		Linus
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>
>Over the last at least five years we've taken as much as possible
>semantics out of the filesystems and into the VFS layer, thus having
>a separation between the semantical layer (VFS) and the low level
>filesystem.
>
VFS is the common filesystem layer.  The only reason you think semantics 
belong in the common filesystem layer is that you are not innovating in 
your semantics, and feel content with stasis.

I don't.  I expect that semantics will get radically changed over the 
next few years as we compete with Giampaolo and whatever lesser lights 
are working at Microsoft.

>  Your attributes are absoultely a VFS thing and as such
>should not happen at the filesystem layer, and no, that doesn't mean
>they're bad per se, I just think they are a rather bad fit for Linux.
>
>So now go on and try to work together with the other peope doing VFS
>level work instead of hiding, or if you think you can't work together
>with us search a nice research OS where you can take over the VFS layer,
>if your ideas prove to be good I'm sure Linux will pick them up sooner
>or later.
>
>	Christoph
>
>
>  
>
I tell you what, use xattrs for all the half speed filesystems, and the 
users and I will use metafiles.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 01:42:06AM -0700, Hans Reiser wrote:
> Christoph Hellwig wrote:
> 
> >
> >Over the last at least five years we've taken as much as possible
> >semantics out of the filesystems and into the VFS layer, thus having
> >a separation between the semantical layer (VFS) and the low level
> >filesystem.
> >
> VFS is the common filesystem layer.  The only reason you think semantics 
> belong in the common filesystem layer is that you are not innovating in 
> your semantics, and feel content with stasis.
> 
> I don't.  I expect that semantics will get radically changed over the 
> next few years as we compete with Giampaolo and whatever lesser lights 
> are working at Microsoft.

Hans, please stop the gooddamn personal attack bullshit.

How do you for example suggestion exporting your semantics over the
network if they're not done at the VFS level?  How do you want some
clusterfilesystem support them or tmpfs?

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>
>
>How do you for example suggestion exporting your semantics over the
>network if they're not done at the VFS level?  How do you want some
>clusterfilesystem support them or tmpfs?
>
>
>  
>
by accesses to filename/metas.....

That's the beauty of simplicity, not much is needed.  xattrs on the 
other hand would be a lot of work to integrate into a cluster filesystem.

Hans

Re: silent semantic changes with reiser4

[Chris Mason]

On Thu, 2004-08-26 at 04:42, Hans Reiser wrote:
> Christoph Hellwig wrote:
> 
> >
> >Over the last at least five years we've taken as much as possible
> >semantics out of the filesystems and into the VFS layer, thus having
> >a separation between the semantical layer (VFS) and the low level
> >filesystem.
> >
> VFS is the common filesystem layer.  The only reason you think semantics 
> belong in the common filesystem layer is that you are not innovating in 
> your semantics, and feel content with stasis.

This misunderstanding is the heart of the problem really.  In many ways,
the VFS is the semantic layer, it defines the picture of the directory
tree and the locking rules the applications see.

There's no rule that says you can't innovate in the VFS, in fact it is
where the semantic innovations belong.  Putting them there is what
allows other filesystems to implement your semantics, and this is
exactly what linux needs if these new semantics are going to be
standardized.

In other words, writing a reiser4 plugin is one way to get to the new
semantics, but it must not be the only way.  Keeping them entirely
within reiser4 is good for the research stage, but for actual deployment
they need to be finalized and pushed up.

If the early linux filesystems had taken the same attitude you have
(don't write new filesystems, only write plugins), there would be no
framework allowing the wealth of filesystems we do have, including
reiser4.

-chris

Re: silent semantic changes with reiser4

[Hans Reiser]

Chris Mason wrote:

>
>
>If the early linux filesystems had taken the same attitude you have
>(don't write new filesystems, only write plugins), there would be no
>framework allowing the wealth of filesystems we do have, including
>reiser4.
>
>  
>
Au contraire, with my approach, there would be 10x as many filesystems, 
because people would be easily able to mix-and-match different methods 
and plugins from different filesystems together, and if you wanted the 
MySQL semantics, you would mount mysql, and if you wanted the zope 
semantics you would mount zope.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 04:53:51PM -0700, Hans Reiser wrote:
> >If the early linux filesystems had taken the same attitude you have
> >(don't write new filesystems, only write plugins), there would be no
> >framework allowing the wealth of filesystems we do have, including
> >reiser4.
> >
> >  
> >
> Au contraire, with my approach, there would be 10x as many filesystems, 
> because people would be easily able to mix-and-match different methods 
> and plugins from different filesystems together, and if you wanted the 
> MySQL semantics, you would mount mysql, and if you wanted the zope 
> semantics you would mount zope.

And if you wanted to make free love you'd make free love you'd make free love
and there would be piece on all earth..

You seem to have lost contact to reality a little.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> 
> For one thing _I_ didn't decide about xattrs anyway.  And I still
> haven't seen a design from you on -fsdevel how you try to solve the
> problems with files as directories.

Hey, files-as-directories are one of my pet things, so I have to side with 
Hans on this one. I think it just makes sense. A hell of a lot more sense 
than xattrs, anyway, since it allows scripts etc standard tools to touch 
the attributes.

It's the UNIX way.

And yes, the semantics can _easily_ be solved in very unixy ways.

One way to solve it is to just realize that a final slash at the end 
implies pretty strongly that you want to treat it as a directory. So what 
you do is:

 - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
 - with the slash, it won't open _without_ O_DIRECTORY (EISDIR)

Problem solved. Very user-friendly, and very intuitive.

Will it potentially break something? Sure. Do we care? Me, I'll take that 
kind of extension _any_ day over xattrs, that are fundamentally flawed in 
my opinion and totally useless. The argument that applications like "tar" 
won't understand the file-as-directory thing is _flawed_, since legacy 
apps won't understand xattrs either.

Oh, add a O_NOXATTRS flag to force a path lookup to only use regular
directories, the same way we have O_NOFOLLOW and friends. That allows
people to see the difference, if they care (ie a file server might decide
that it doesn't want to expose things like this).

I never liked the xattr stuff. It makes little sense, and is totally 
useless for 99.9999% of everything. I still don't see the point of it, 
except for samba. Ugly.

		Linus

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
> And yes, the semantics can _easily_ be solved in very unixy ways.
> 
> One way to solve it is to just realize that a final slash at the end 
> implies pretty strongly that you want to treat it as a directory. So what 
> you do is:
> 
>  - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
>  - with the slash, it won't open _without_ O_DIRECTORY (EISDIR)
> 
> Problem solved. Very user-friendly, and very intuitive.

That would solve the O_DIRECTORY issue, the dentry aliasing still needs
work though with the semantics for link/unlink/rename.

Maybe Hans & you should start 2.7 to work this out? :)

Re: silent semantic changes with reiser4

[Hans Reiser]

I just want to add that I AM capable of working with the other 
filesystem developers in a team-player way, and I am happy to cooperate 
with making portions more reusable where there is serious interest from 
other filesystems in that, but Christoph is a puppy who has never 
written or designed a major filesystem from scratch, and Nikita is a big 
dog who has written stuff very few projects are lucky enough to see the 
likes of, and when Christoph insults Nikita's code, or my design 
guidance for that code, it is not going to bring out my good side.  The 
plugin and metafiles code needs many improvements, but Christoph does 
not have the expertise to understand what those needed improvements are 
because he hasn't invested the work into understanding the code.

Christoph is a bright and clever young fellow, who just hasn't had the 
years of study of the field yet.  I wish him well, and away.;-)

Hans

Re: silent semantic changes with reiser4

[Chris Mason]

On Wed, 2004-08-25 at 16:41, Hans Reiser wrote:
> I just want to add that I AM capable of working with the other 
> filesystem developers in a team-player way, and I am happy to cooperate 
> with making portions more reusable where there is serious interest from 
> other filesystems in that, 

Prove it.  Stop replying for today and come back tomorrow with some
useful discussions.  Christoph suggested that some of the v4 semantics
belong in the VFS and therefore linux as a whole.  He's helping you to
make sure the semantics and fit nicely with the rest of kernel
interfaces and are race free.

Take him up on the offer.

-chris

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Wed, Aug 25, 2004 at 01:41:14PM -0700, Hans Reiser wrote:

>Christoph is a bright and clever young fellow, who just hasn't had the 
>years of study of the field yet.  I wish him well, and away.;-)

I see this is as an opportunity where you can share some of your
experience to Cristoph and many others and work to get the semantics
into VFS.

Please make this work :)

-- 
mjt

Re: silent semantic changes with reiser4

[Rik van Riel]

On Wed, 25 Aug 2004, Hans Reiser wrote:

> other filesystems in that, but Christoph is a puppy who has never 

It's not Christoph who's shown more bark than bite in this thread.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>On Wed, 25 Aug 2004, Hans Reiser wrote:
>
>  
>
>>other filesystems in that, but Christoph is a puppy who has never 
>>    
>>
>
>It's not Christoph who's shown more bark than bite in this thread.
>
>  
>
The bite is at www.namesys.com/download.html


Hans

Re: silent semantic changes with reiser4

[viro]

On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
> 
> 
> On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> > 
> > For one thing _I_ didn't decide about xattrs anyway.  And I still
> > haven't seen a design from you on -fsdevel how you try to solve the
> > problems with files as directories.
> 
> Hey, files-as-directories are one of my pet things, so I have to side with 
> Hans on this one. I think it just makes sense. A hell of a lot more sense 
> than xattrs, anyway, since it allows scripts etc standard tools to touch 
> the attributes.
> 
> It's the UNIX way.

Not if you allow link(2) on them.  And not if you design and market your
stuff as a general-purpose backdoor into kernel.  Note how *EVERY* *DAMN*
*OPERATION* is made possible to override by "plugins".  Which is the reason
for deadlocks in question, BTW.

Don't fool yourself - that's what Hans is selling.  Target market: ISV.
Marketed product: a set of hooks, the wider the better, no matter how
little sense it makes.  The reason for doing that outside of core kernel:
bypassing any review and being able to control the product being sold (see
above).

Shame that it got an actual filesystem mixed in with the marketing plans
and general insanity...

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2418 bytes --]

Am Mittwoch, den 25.08.2004, 21:42 +0100 schrieb :

> > > For one thing _I_ didn't decide about xattrs anyway.  And I still
> > > haven't seen a design from you on -fsdevel how you try to solve the
> > > problems with files as directories.
> > 
> > Hey, files-as-directories are one of my pet things, so I have to side with 
> > Hans on this one. I think it just makes sense. A hell of a lot more sense 
> > than xattrs, anyway, since it allows scripts etc standard tools to touch 
> > the attributes.
> > 
> > It's the UNIX way.
> 
> Not if you allow link(2) on them.

That doesn't make sense anyway. (actually, I tried what happens and the
result was an Oops ;))

It should be completely forbidden to link into a meta-directory or out
of such a directory. You could think of those meta-directory as a sysfs
for that inode. Of course it's not an own filesystem and that means that
there need to be a lot of security precautions in the VFS layer. Where
something like that belongs anyway, if done correctly.

>   And not if you design and market your
> stuff as a general-purpose backdoor into kernel.  Note how *EVERY* *DAMN*
> *OPERATION* is made possible to override by "plugins".  Which is the reason
> for deadlocks in question, BTW.

What do you mean? If you tell that file that you want it to be
compressed or encrypted or modify some attributes (like ACLs) this isn't
necessarily a backdoor.

> Don't fool yourself - that's what Hans is selling.  Target market: ISV.
> Marketed product: a set of hooks, the wider the better, no matter how
> little sense it makes.  The reason for doing that outside of core kernel:
> bypassing any review and being able to control the product being sold (see
> above).

Yes, I don't think it was a good idea either. Probably someone should
remove these features and make it a "normal" filesystem. The people who
need it now can turn it on again and a real solution could be worked out
in Linux 2.7.

I wouldn't use it on a public server anyway now because I'm not
convinced some malicious guy could find a way to exploit that. What if
you changed into a meta directory using ftp and some manage to break
things? This might be very dangerous.

I personally think that the idea of doing something like this (I'm not
speaking of the current implementation which I think is really bad) is
the right way to go in the long term.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Andrea Arcangeli]

On Wed, Aug 25, 2004 at 11:00:00PM +0200, Christophe Saout wrote:
> It should be completely forbidden to link into a meta-directory or out
> of such a directory. [..]

agreed.

> Yes, I don't think it was a good idea either. Probably someone should

I personally would like plugins only if the API they use wouldn't allow
them to corrupt the underlying fs, I'm not sure if this is the case with
reiserfs4.

About the backwards compatibility, another option is to add a O_FILEDIR
and have bash learn about it when you cd into a file. No magic with the
slashes then.

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrea Arcangeli wrote:

>On Wed, Aug 25, 2004 at 11:00:00PM +0200, Christophe Saout wrote:
>  
>
>>It should be completely forbidden to link into a meta-directory or out
>>of such a directory. [..]
>>    
>>
>
>agreed.
>
>  
>
>>Yes, I don't think it was a good idea either. Probably someone should
>>    
>>
>
>I personally would like plugins only if the API they use wouldn't allow
>them to corrupt the underlying fs, I'm not sure if this is the case with
>reiserfs4.
>  
>
You compile crap into the kernel, you are screwed.  We have not and will 
not change that.

Reiser4 plugins are not for end users to download from amazon.com, they 
are for weekend hackers to send me a cool plugin for me to review, 
assign a plugin id to, and send to Linus in the next release.  Sometimes 
being less ambitious works better, and dynamically loaded plugins would 
have made the infrastructure too bulky to be fun and fast.

Re: silent semantic changes with reiser4

[Andrea Arcangeli]

On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote:
> Reiser4 plugins are not for end users to download from amazon.com, they 
> are for weekend hackers to send me a cool plugin for me to review, 
> assign a plugin id to, and send to Linus in the next release.  Sometimes 

then what's the difference in having the plugin fixed in stone into
reiserfs? That's my whole point. Get the patch from the weekend hacker,
check it, send the patch to Linus to add the new feature to reiser4,
just call it "feature" not plugin. That's how it works normally for
everything. Many fs have many features, many of them optional.  you
wouldn't need to build any hook infrastructure either that way. Hooks
would be needed if this wasn't open source me thinks. Or if you want
people to fetch the module from amazon without your review.

Another reason I could see the modularization/hooking useful is if those
feature would take lots of kernel space, but this sure isn't the case
for reiserfs, infact having modules would waste _more_ ram due the half
wasted page allocation for the module text. The only single reason we
use modules is to avoid wasting tons of ram by loading every possible
device driver on earth, it's not that we use modules because they're
more flexible, if something they're more fragile. I never use modules in
my test kernel just to go fast (because I self compile them). The last
good thing of the modules is during development you don't need to reboot
the machine to test a kernel change in a driver, but you don't need that
with reiserfs since you can make the fs itself a module (just change the
name of the fs, AFIK you do that all the time already).

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

Am Donnerstag, den 26.08.2004, 13:28 +0200 schrieb Andrea Arcangeli:

> then what's the difference in having the plugin fixed in stone into
> reiserfs? That's my whole point.

I think you're not exactly understanding what a reiser4 "plugin" is.

Currently plugins are not modules. It just means that there's a defined
interface between the reiser4 core and the plugins. Just like you could
see filesystems as "VFS plugins".

In fact, reiser4 plugins are
- users of the reiser4 core API
- some of them are implementing Linux VFS methods (thus being some sort
of glue code between the Reiser4 storage tree and the Linux VFS)


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Andrea Arcangeli]

On Thu, Aug 26, 2004 at 01:44:59PM +0200, Christophe Saout wrote:
> I think you're not exactly understanding what a reiser4 "plugin" is.
> 
> Currently plugins are not modules. It just means that there's a defined
> interface between the reiser4 core and the plugins. Just like you could
> see filesystems as "VFS plugins".

but filesystems are exactly always modules (in precompiled kernels at
least).

Anyways I don't see any visible EXPORT_SYMBOL in reiser4.

> 
> In fact, reiser4 plugins are
> - users of the reiser4 core API
> - some of them are implementing Linux VFS methods (thus being some sort
> of glue code between the Reiser4 storage tree and the Linux VFS)

then it seems a bit misleading to call those things plugins.

As you wrote those are "users of the reiser4 core API", with plugins I
always meant dynamically loadable things, like plugins for web browers
(hence kernel modules in kernel space). While here apparently you can't
plugin anything at runtime, it's just code that uses the reiser4 core
API that can be modified with a kernel patch as usual.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1469 bytes --]

Am Donnerstag, den 26.08.2004, 14:16 +0200 schrieb Andrea Arcangeli:

> > Currently plugins are not modules. It just means that there's a defined
> > interface between the reiser4 core and the plugins. Just like you could
> > see filesystems as "VFS plugins".
> 
> but filesystems are exactly always modules (in precompiled kernels at
> least).

Yes, of cousre.

> Anyways I don't see any visible EXPORT_SYMBOL in reiser4.

Yes, there's no module infrastructure there. These "plugins" need to be
linked at compile time. Some plugins need to be anyway since the the
filesystem is useless without them. Others whould be optional features.

> > In fact, reiser4 plugins are
> > - users of the reiser4 core API
> > - some of them are implementing Linux VFS methods (thus being some sort
> > of glue code between the Reiser4 storage tree and the Linux VFS)
> 
> then it seems a bit misleading to call those things plugins.
> 
> As you wrote those are "users of the reiser4 core API", with plugins I
> always meant dynamically loadable things, like plugins for web browers
> (hence kernel modules in kernel space). While here apparently you can't
> plugin anything at runtime, it's just code that uses the reiser4 core
> API that can be modified with a kernel patch as usual.

Right. The name is probably at bit misleading. But assuming the
compression "plugin" whould be dynamically loadable at some later point
it would be a plugin then?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Thu, Aug 26, 2004 at 02:16:30PM +0200, Andrea Arcangeli wrote:
> On Thu, Aug 26, 2004 at 01:44:59PM +0200, Christophe Saout wrote:
> > I think you're not exactly understanding what a reiser4 "plugin" is.
> > 
> > Currently plugins are not modules. It just means that there's a defined
> > interface between the reiser4 core and the plugins. Just like you could
> > see filesystems as "VFS plugins".
> 
> but filesystems are exactly always modules (in precompiled kernels at
> least).
> 
> Anyways I don't see any visible EXPORT_SYMBOL in reiser4.

they would be there, I think.
> 
> > 
> > In fact, reiser4 plugins are
> > - users of the reiser4 core API
> > - some of them are implementing Linux VFS methods (thus being some sort
> > of glue code between the Reiser4 storage tree and the Linux VFS)
> 
> then it seems a bit misleading to call those things plugins.

There should be infracture to change plugin on fly, and choosing object's
plugins at object's creation time.  Currently only certain object's plugins
can be changed through <object>/metas/plugin interface. 

> 
> As you wrote those are "users of the reiser4 core API", with plugins I
> always meant dynamically loadable things, like plugins for web browers
> (hence kernel modules in kernel space). While here apparently you can't
> plugin anything at runtime, it's just code that uses the reiser4 core
> API that can be modified with a kernel patch as usual.

-- 
Alex.

Re: silent semantic changes with reiser4

[Hans Reiser]

Alex Zarochentsev wrote:

>
>
>There should be infracture to change plugin on fly, and choosing object's
>plugins at object's creation time.  Currently only certain object's plugins
>can be changed through <object>/metas/plugin interface. 
>  
>
To be more clear for those not familiar with our project, there is much 
more work to be done, and one of the areas that (extremely) needs more 
work is changing plugins on the fly and clean interfaces for specifying 
them at creation time.

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrea Arcangeli wrote:

>On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote:
>  
>
>>Reiser4 plugins are not for end users to download from amazon.com, they 
>>are for weekend hackers to send me a cool plugin for me to review, 
>>assign a plugin id to, and send to Linus in the next release.  Sometimes 
>>    
>>
>
>then what's the difference in having the plugin fixed in stone into
>reiserfs? That's my whole point. Get the patch from the weekend hacker,
>check it, send the patch to Linus to add the new feature to reiser4,
>just call it "feature" not plugin. 
>
Think of it as being like VFS, where you can plugin new filesystems.  
Only in this case, you can plugin new kinds of files, and everything you 
need to implement those new kinds of files (items, nodes, keys, 
processing before flushing to disk, etc.) also gets implemented as a 
plugin.  It is an Uber-VFS.

> The only single reason we
>use modules is to avoid wasting tons of ram by loading every possible
>device driver on earth,
>
I am not concerned about ram in this design, I want nifty new kinds of 
files easy crafted over a weekend by sysadmins working in Canada and 
Guatemala.  Software engineering cost is what matters to me ( I turned 
40, so I think different now....;-) )

Re: silent semantic changes with reiser4

[Andrea Arcangeli]

On Thu, Aug 26, 2004 at 04:53:23PM -0700, Hans Reiser wrote:
> Think of it as being like VFS, where you can plugin new filesystems.  
> Only in this case, you can plugin new kinds of files, and everything you 
> need to implement those new kinds of files (items, nodes, keys, 
> processing before flushing to disk, etc.) also gets implemented as a 
> plugin.  It is an Uber-VFS.

I'm just confused by the naming, following your comparison with it being
like VFS, I don't call reiser4 a "plugin" of the VFS.

The vfs offers an interface for the fs to register into
it. Like the scsi highlevel code also offers an interface for the
lowlevel driver to register into it, but we don't call scsi lowlevel
drivers "plugins".

the ext3 directory code calling ext3_getblk would be also a plugin in
your vocabulary, and the ext3_getblk call would be plugin-hook.

> I am not concerned about ram in this design, I want nifty new kinds of 
> files easy crafted over a weekend by sysadmins working in Canada and 
> Guatemala.  Software engineering cost is what matters to me ( I turned 
> 40, so I think different now....;-) )

Adding proper layering and abstractions is sure a good thing, if you can
turn the street into a crowd of reiser4 kernel hackers because your API
is so clean that any sysadmin can craft new kind of files over the
weekend you're welcome to go ahead ;). (I just doubt the "plugin"
terminology can make a difference by the time people start scratching
their head into the code, the design and the clean abstraction of the
code will make a difference instead)

Re: silent semantic changes with reiser4

[Yury Umanets]

Andrea Arcangeli wrote:

>On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote:
>  
>
>>Reiser4 plugins are not for end users to download from amazon.com, they 
>>are for weekend hackers to send me a cool plugin for me to review, 
>>assign a plugin id to, and send to Linus in the next release.  Sometimes 
>>    
>>
>
>then what's the difference in having the plugin fixed in stone into
>reiserfs? That's my whole point. Get the patch from the weekend hacker,
>check it, send the patch to Linus to add the new feature to reiser4,
>just call it "feature" not plugin. That's how it works normally for
>everything. Many fs have many features, many of them optional.  you
>wouldn't need to build any hook infrastructure either that way. Hooks
>would be needed if this wasn't open source me thinks. Or if you want
>people to fetch the module from amazon without your review.
>
>Another reason I could see the modularization/hooking useful is if those
>feature would take lots of kernel space, but this sure isn't the case
>for reiserfs, infact having modules would waste _more_ ram due the half
>wasted page allocation for the module text. The only single reason we
>use modules is to avoid wasting tons of ram by loading every possible
>device driver on earth, it's not that we use modules because they're
>more flexible, if something they're more fragile. I never use modules in
>my test kernel just to go fast (because I self compile them). The last
>good thing of the modules is during development you don't need to reboot
>the machine to test a kernel change in a driver, but you don't need that
>with reiserfs since you can make the fs itself a module (just change the
>name of the fs, AFIK you do that all the time already).
>  
>
There lots of good things about plugins as they are in reiser4. And 
probably the best one (as for me) is that, reiser4 core code (quite 
complex thing) does not need to be changed after adding new plugins 
(read new features).

It is written once and provides some basic functionality (balancing, 
atoms, etc.), which works well with all kinds of plugins if they 
implement required functions from plugin interface.

As for me, this looks like Linux's VFS. Sure, if I want to add some very 
new filesystem with features which were not foreseen,  and VFS does not 
provide needed interface, it will need some changes. But in simple case, 
when I just want to develop once more disk filesystem, which say will be 
different from another disk filesystem in only that respect it manages 
its metadata in different manner and aims to achieve better performance 
this way. In this case I will not need to chnage VFS at all. The same 
about reiser4 core and new plugins.

>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>


-- 
umka

Re: silent semantic changes with reiser4

[Yury Umanets]

Andrea Arcangeli wrote:

>On Thu, Aug 26, 2004 at 01:35:01AM -0700, Hans Reiser wrote:
>  
>
>>Reiser4 plugins are not for end users to download from amazon.com, they 
>>are for weekend hackers to send me a cool plugin for me to review, 
>>assign a plugin id to, and send to Linus in the next release.  Sometimes 
>>    
>>
>
>then what's the difference in having the plugin fixed in stone into
>reiserfs? That's my whole point. Get the patch from the weekend hacker,
>check it, send the patch to Linus to add the new feature to reiser4,
>just call it "feature" not plugin. That's how it works normally for
>everything. Many fs have many features, many of them optional.  you
>wouldn't need to build any hook infrastructure either that way. Hooks
>would be needed if this wasn't open source me thinks. Or if you want
>people to fetch the module from amazon without your review.
>
>Another reason I could see the modularization/hooking useful is if those
>feature would take lots of kernel space, but this sure isn't the case
>for reiserfs, infact having modules would waste _more_ ram due the half
>wasted page allocation for the module text. The only single reason we
>use modules is to avoid wasting tons of ram by loading every possible
>device driver on earth, it's not that we use modules because they're
>more flexible, if something they're more fragile. I never use modules in
>my test kernel just to go fast (because I self compile them). The last
>good thing of the modules is during development you don't need to reboot
>the machine to test a kernel change in a driver, but you don't need that
>with reiserfs since you can make the fs itself a module (just change the
>name of the fs, AFIK you do that all the time already).
>  
>
There lots of good things about plugins as they are in reiser4. And
probably the best one (as for me) is that, reiser4 core code (quite
complex thing) does not need to be changed after adding new plugins
(read new features).

It is written once and provides some basic functionality (balancing,
atoms, etc.), which works well with all kinds of plugins if they
implement required functions from plugin interface.

As for me, this looks like Linux's VFS. Sure, if I want to add some very
new filesystem with features which were not foreseen,  and VFS does not
provide needed interface, it will need some changes. But in simple case,
when I just want to develop once more disk filesystem, which say will be
different from another disk filesystem in only that respect it manages
its metadata in different manner and aims to achieve better performance
this way. In this case I will not need to chnage VFS at all. The same
about reiser4 core and new plugins.

>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>


-- 
umka

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Wed, 25 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
>
> On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
> > 
> > It's the UNIX way.
> 
> Not if you allow link(2) on them.

Heh. I don't think that's a very strong argument against being "unixy", 
considering how traditional unix _used_ to handle directories.

mkdir/rmdir/rename only came later. Now, obviously they did come later for 
a good reason, but still..

The interesting part is that thanks to the dcache, we should be perfectly
able to actually _see_ circular links etc, so some of the problems with 
linking directories should actually be quite solvable - something that is 
_not_ true for a traditional UNIX VFS layer.

Of course, the dcache introduces some new problems of its own wrt
directory aliasing, but I don't actually think that should be fundamental
either. Treating them more as a "static mountpoint" from an FS angle and
less as a traditional Unix hardlink should be doable, I'd have thought.

(Also, it's entirely possible that the filesystem may not support some of
the more esoteric linking/renaming operations. For example, in a
traditional xattrs setup where the xattr is linked on-disk with the file
it is associated with, you simply _can't_ link it somewhere else, or
rename it to any other directory. That's not a VFS layer issue, obviously,
but I thought I'd bring up the point that file-as-dir cases may have
limitations that normal files don't have).

>  And not if you design and market your stuff as a general-purpose
> backdoor into kernel.

Now that's a separate argument, and not one I'm personally interested in
arguing at least right now. I haven't actually looked at the reiser4 code,
so I'm really _only_ arguing against special-case attributes.

		Linus

Re: silent semantic changes with reiser4

[viro]

On Wed, Aug 25, 2004 at 02:00:01PM -0700, Linus Torvalds wrote:
> Of course, the dcache introduces some new problems of its own wrt
> directory aliasing, but I don't actually think that should be fundamental
> either. Treating them more as a "static mountpoint" from an FS angle and
> less as a traditional Unix hardlink should be doable, I'd have thought.

Yeah, if we ditch the "mountpoints are busy and untouchable" stuff.  Which
I'd love to, but it's a hell of a visible (and admin-visible) change.

FWIW, current deadlocks are unrelated to actual operation succeeding.
Look: we have sys_link() making sure that parent of target is a directory
(PATH_LOOKUP, in a "it has ->lookup()" sense), then locking target's parent,
then checking that it has ->link() (everyone on reiser4 does) and then
checking that source (old link to file) is *not* a directory (in S_ISDIR
sense).  Then we lock source.

Note that currently it's OK - we get "all non-directories are always locked
after all directories".  With filesystem that provides hybrid objects with
non-NULL ->link() it's not true and we are in deadlock country.  Before
we get anywhere near fs code.

I'm not saying that this particular instance is hard to fix, but it wasn't
even looked at.  All it would take is checking the description of current
locking scheme and looking through the proof of correctness (present in the
tree).  That's the first point where said proof breaks if we have hybrids.
And it's what, about 4 screenfuls of text?

I have no problems with discussing such stuff and no problems with having it
merged if it actually works.  But let's start with something better than
"let's hope nothing breaks if we just add such objects and do nothing else,
'cause hybridi files/directories are good, mmmkay?"

Re: silent semantic changes with reiser4

[Jamie Lokier]

This message suggests a way to extend the VFS safe locking rules to
include files-as-directories.

viro@parcelfarce.linux.theplanet.co.uk wrote:
> Note that currently it's OK - we get "all non-directories are always locked
> after all directories".  With filesystem that provides hybrid objects with
> non-NULL ->link() it's not true and we are in deadlock country.  Before
> we get anywhere near fs code.

Is this a problem if we treat entering a file-as-directory as crossing
a mount point (i.e. like auto-mounting)?

Simply doing a path walk would lock the file and then cross the mount
point to a directory.

A way to ensure that preserves the lock order is to require that the
metadata is in a different filesystem to its file (i.e. not crossing a
bind mount to the same filesystem).

That has the side effect of preventing hard links between metadata
files and non-metadata, which in my opinion is fine.

Path walking will lock the file, and then lock the directory on a
different filesystem.  Lock order is still safe, provided a strict
order is maintained between the two filesystems.

The strict order is ensured by preventing bind mounts which create a
path cycle containing a file->metadata edge.  One way to ensure that
is to prevent mounts on the metadata filesystems, but the rule doesn't
have to be that strict.  This condition only needs to be checked in
the mount() syscall.

-- Jamie

Re: silent semantic changes with reiser4

[viro]

On Thu, Aug 26, 2004 at 01:11:52AM +0100, Jamie Lokier wrote:
> Is this a problem if we treat entering a file-as-directory as crossing
> a mount point (i.e. like auto-mounting)?

Yes - mountpoints can't be e.g. unlinked.  Moreover, having directory
mounted on non-directory is also an interesting situation.

> Simply doing a path walk would lock the file and then cross the mount
> point to a directory.

*Ugh*

What would happen if you open that directory or chdir there?  If it's
"underlying file stays locked" - we are in even more obvious deadlocks.

> A way to ensure that preserves the lock order is to require that the
> metadata is in a different filesystem to its file (i.e. not crossing a
> bind mount to the same filesystem).
> 
> That has the side effect of preventing hard links between metadata
> files and non-metadata, which in my opinion is fine.

We don't actually need a different fs - different vfsmount will do just fine.

> The strict order is ensured by preventing bind mounts which create a
> path cycle containing a file->metadata edge.  One way to ensure that
> is to prevent mounts on the metadata filesystems, but the rule doesn't
> have to be that strict.  This condition only needs to be checked in
> the mount() syscall.

You really don't want to lock mountpoint on path lookup, so I don't see
how that would be relevant - it's a hell to clean up, for one thing
(I've crossed ten mountpoints on the way, when do I unlock them and
how do I prevent deadlocks from that?)  Besides, different namespaces
can have completely different mount trees, so tracking down all that
stuff would be hell in its own right.

The main issue I see with all schemes in that direction (and something
like that could be made workable) is the semantics of unlink() on
mountpoints.  *Especially* with users being able to see attributes of
files they do not own (e.g. reiser4 mode/uid/gid stuff).  Ability to
pin down any damn file on the system and make it impossible to replace
is not something you want to give to any user.

Re: silent semantic changes with reiser4

[Jamie Lokier]

viro@parcelfarce.linux.theplanet.co.uk wrote:
> > Is this a problem if we treat entering a file-as-directory as crossing
> > a mount point (i.e. like auto-mounting)?
> 
> Yes - mountpoints can't be e.g. unlinked.  Moreover, having directory
> mounted on non-directory is also an interesting situation.

Ok, so can we make it so mountpoints can be unlinked? :)

The mount would continue to exist, but with no name, until its last
user disappears.

> > Simply doing a path walk would lock the file and then cross the mount
> > point to a directory.
> 
> *Ugh*
> 
> What would happen if you open that directory or chdir there?  If it's
> "underlying file stays locked" - we are in even more obvious deadlocks.

I think the underlying file does not stay locked, and once you've
entered it as a directory, it can be unlinked.

If you have the directory open or chdir into it, then it _may_ have
the effect of keeping the file's storage allocated when you unlink it
-- just like when a file is unlinked while opened.  As that is not a
user-visible property, it's a filesystem-specific implementation
detail as to whether it keeps the file's data in existence while the
metadata directories and/or files are open.

> > A way to ensure that preserves the lock order is to require that the
> > metadata is in a different filesystem to its file (i.e. not crossing a
> > bind mount to the same filesystem).
> > 
> > That has the side effect of preventing hard links between metadata
> > files and non-metadata, which in my opinion is fine.
> 
> We don't actually need a different fs - different vfsmount will do just fine.

> > The strict order is ensured by preventing bind mounts which create a
> > path cycle containing a file->metadata edge.  One way to ensure that
> > is to prevent mounts on the metadata filesystems, but the rule doesn't
> > have to be that strict.  This condition only needs to be checked in
> > the mount() syscall.
> 
> You really don't want to lock mountpoint on path lookup, so I don't see
> how that would be relevant - it's a hell to clean up, for one thing
> (I've crossed ten mountpoints on the way, when do I unlock them and
> how do I prevent deadlocks from that?)  Besides, different namespaces
> can have completely different mount trees, so tracking down all that
> stuff would be hell in its own right.

I didn't mean locking a chain of mountpoints, I meant the temporary
state where two dentries and/or inodes are locked, parent and child,
during a path walk.  However I'm not very familiar with that part of
the VFS and I see that the current RCU dcache might not lock that much
during a path walk.

> The main issue I see with all schemes in that direction (and something
> like that could be made workable) is the semantics of unlink() on
> mountpoints.  *Especially* with users being able to see attributes of
> files they do not own (e.g. reiser4 mode/uid/gid stuff).  Ability to
> pin down any damn file on the system and make it impossible to replace
> is not something you want to give to any user.

I agree, users shouldn't be able to pin down a file.

I think unlink() should succeed on a file while something is visiting
inside its metadata directory.

It's a filesystem quality-of-implementation feature whether that
actually releases the file's data.  It's a desirable feature because
one user shouldn't be able to pin another user's quota'd data if they
don't have permission to open the file, but if it's not implemented by
a filesystem then it doesn't break anything fundamental.

It's a semantics question whether unlinking a file makes the metadata
(i.e. "uid", "mode", "content-type" etc.) disappear at the same time,
or if the metadata stays around until the last visitor leaves it.  A
filesystem might be able to keep the metadata in existence even if it
deletes the file's storage on unlink(), but it would be nice for the
VFS to declare which semantic is preferred.

One of the big potential uses for file-as-directory is to go inside
archive files, ELF files, .iso files and so on in a convenient way.
In those cases, if you open one of the virtually generated "archive
content" files, then you might expect the data to continue to exist
after the underlying file is unlinked.  I think that's reasonable:
being inside an archive is very similar to having it open.  There is
no quota pinning problem with this, because "archive content" files
should inherit permission restrictions from the underlying file.  If
you can't read the file, then you can't read it's unpacked contents.

(reiser4 doesn't offer that last feature, but any of the myriad
userspace filesystem hooks could offer it if the VFS has approprate
auto-mounting file-as-directory hooks).

-- Jamie

Re: silent semantic changes with reiser4

[viro]

On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote:
> viro@parcelfarce.linux.theplanet.co.uk wrote:
> > > Is this a problem if we treat entering a file-as-directory as crossing
> > > a mount point (i.e. like auto-mounting)?
> > 
> > Yes - mountpoints can't be e.g. unlinked.  Moreover, having directory
> > mounted on non-directory is also an interesting situation.
> 
> Ok, so can we make it so mountpoints can be unlinked? :)

User-visible change of behaviour and IIRC a SuS violation on top of that.
 
> I think the underlying file does not stay locked, and once you've
> entered it as a directory, it can be unlinked.

So why lock it at all in that case?

> I didn't mean locking a chain of mountpoints, I meant the temporary
> state where two dentries and/or inodes are locked, parent and child,
> during a path walk.  However I'm not very familiar with that part of
> the VFS and I see that the current RCU dcache might not lock that much
> during a path walk.

Never had been needed on crossing mountpoints, actually.

> I agree, users shouldn't be able to pin down a file.
> 
> I think unlink() should succeed on a file while something is visiting
> inside its metadata directory.

See above.  Again, the fundamental problem with that is allowing unlink
and friends on a mountpoint.  I would love to do that, but it always
generated -EBUSY on all Unices.  Linux got a bit more users and userland
code than Plan 9 - they can afford such changes, but...

And yes, from the kernel POV it's trivial to do - witness the MNT_DETACH
codepath in umount - it's much simpler than "normal" umount exactly because
it doesn't try to emulate old "it's busy, can't umount" behaviour.

With umount we could introduce "don't bother with that shit" flag.  With
unlink() we would have to make that default behaviour to be useful.

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>
>
>One of the big potential uses for file-as-directory is to go inside
>archive files, ELF files, .iso files and so on in a convenient way.
>  
>
Yes, this was part of the plan, tar file-directory plugins would be cute.

Re: silent semantic changes with reiser4

[Adrian Bunk]

On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote:
> Jamie Lokier wrote:
> 
> >One of the big potential uses for file-as-directory is to go inside
> >archive files, ELF files, .iso files and so on in a convenient way.
> > 
> >
> Yes, this was part of the plan, tar file-directory plugins would be cute.

Silly question:

GNU Midnight Commander allows for ages to go into e.g. tar files, so I 
know the benefits of this. Additionally, in GNU Midnight Commander, this 
works no matter which file system I use (e.g. it works on iso9660), and 
it even works the same way on other OS's like e.g. Solaris and NetBSD.

What is the technical reason why a tar plugin should be reiser4 
specific, instead of a generic VFS or userspace solution that would 
allow the same also on other fs like e.g. iso9660?

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1316 bytes --]

Am Donnerstag, den 26.08.2004, 16:05 +0200 schrieb Adrian Bunk:

> > Yes, this was part of the plan, tar file-directory plugins would be cute.
> 
> Silly question:
> 
> GNU Midnight Commander allows for ages to go into e.g. tar files, so I 
> know the benefits of this. Additionally, in GNU Midnight Commander, this 
> works no matter which file system I use (e.g. it works on iso9660), and 
> it even works the same way on other OS's like e.g. Solaris and NetBSD.
> 
> What is the technical reason why a tar plugin should be reiser4 
> specific, instead of a generic VFS or userspace solution that would 
> allow the same also on other fs like e.g. iso9660?

I don't think that this should be done inside the kernel. tar is a
perfectly valid userspace application.

What reiser4 can do, but the VFS can't is to insert or remove data in
the middle of a file. Adding this above the page cache would probably be
almost impossible (truncate seems already complicated enough).

But below page-cache and thus invisible from the VFS or the applications
it is extremely useful for the compression plugin. When changing some
data in the middle of the file the compressed data might grow in size.
reiser4 can handle this inside the storage layer and doesn't mess with
the rest of the filesystem.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Matthew Wilcox]

On Thu, Aug 26, 2004 at 04:25:13PM +0200, Christophe Saout wrote:
> What reiser4 can do, but the VFS can't is to insert or remove data in
> the middle of a file. Adding this above the page cache would probably be
> almost impossible (truncate seems already complicated enough).
> 
> But below page-cache and thus invisible from the VFS or the applications
> it is extremely useful for the compression plugin. When changing some
> data in the middle of the file the compressed data might grow in size.
> reiser4 can handle this inside the storage layer and doesn't mess with
> the rest of the filesystem.

Never say never ... regular readers of -fsdevel already know what I'm
going to say so can skip this post.

When compressing data on an ext2-like filesystem, you can use the
property of "holes" to deallocate disc blocks in the middle of a file.
It works like this:

Split the file into 64k chunks
Compress each chunk individually, then write them to disc on 64k boundaries,
leaving gaps.
Now you can quickly seek to the appropriate 64k block for the data you're
looking for and decompress just that 64k block.

This is how RISCiX managed to fit an 80MB distro onto a 50MB disc.
Someone has written an equivalent filesystem for Linux called SquashFS.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christophe Saout wrote:
> What reiser4 can do, but the VFS can't is to insert or remove data in
> the middle of a file. Adding this above the page cache would probably be
> almost impossible (truncate seems already complicated enough).

That would be one of those "special features" that a
VFS-plus-userspace implementation of archive views could take
advantage of on reiser4, while using a slower method (sometimes much
slower) on all other filesystems.

By the way, can reiser4 share parts of files between different files?

-- Jamie

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1479 bytes --]

Am Donnerstag, den 26.08.2004, 16:04 +0100 schrieb Jamie Lokier:

> Christophe Saout wrote:
> > What reiser4 can do, but the VFS can't is to insert or remove data in
> > the middle of a file. Adding this above the page cache would probably be
> > almost impossible (truncate seems already complicated enough).
> 
> That would be one of those "special features" that a
> VFS-plus-userspace implementation of archive views could take
> advantage of on reiser4, while using a slower method (sometimes much
> slower) on all other filesystems.

I'm just thinking about something. While you can't cut bytes out of unix
iles a lot of filesystems can do this (holes). Most of them only on a
block boundary, reiser4 on a byte boundary. If the filesystems could
export this functionality using an enhanced API we could implement a
compression plugin and other things on the VFS level that works with
every filesystem supporting the required mechanisms, not only reiser4.
And those features would take advantage of reiser4's storage mechanisms.
I think Hans made the plugins reiser4-only because only reiser4 has a
similar API at that time (and obviously because he didn't even think
about doing it otherwise).

> By the way, can reiser4 share parts of files between different files?

At the moment a file is exactly one object. But someone could write
another file plugin that spans a file across multiple objects, then yes,
multiple files could share parts.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christophe Saout wrote:
> I'm just thinking about something. While you can't cut bytes out of unix
> iles a lot of filesystems can do this (holes). Most of them only on a
> block boundary, reiser4 on a byte boundary. If the filesystems could
> export this functionality using an enhanced API we could implement a
> compression plugin and other things on the VFS level that works with
> every filesystem supporting the required mechanisms, not only reiser4.
> And those features would take advantage of reiser4's storage mechanisms.
> I think Hans made the plugins reiser4-only because only reiser4 has a
> similar API at that time (and obviously because he didn't even think
> about doing it otherwise).

Yes.  In some cases the compression or re-archiving algorithm would
choose different parameters, depending on the alignment capabilities
of the underlying filesystem.  In some cases, that alignment is
"offset 0 is the only one you can have", which is a severe special
case where truncate and rewrite is required for changes.

> > By the way, can reiser4 share parts of files between different files?
> 
> At the moment a file is exactly one object. But someone could write
> another file plugin that spans a file across multiple objects, then yes,
> multiple files could share parts.

Nice.  That's useful for files which appear as themselves and also as
part of a container.

And also for revision managed filesystems :)

-- Jamie

Re: silent semantic changes with reiser4

[Nikita Danilov]

Jamie Lokier writes:
 > Christophe Saout wrote:
 > > What reiser4 can do, but the VFS can't is to insert or remove data in
 > > the middle of a file. Adding this above the page cache would probably be
 > > almost impossible (truncate seems already complicated enough).
 > 
 > That would be one of those "special features" that a
 > VFS-plus-userspace implementation of archive views could take
 > advantage of on reiser4, while using a slower method (sometimes much
 > slower) on all other filesystems.
 > 
 > By the way, can reiser4 share parts of files between different files?

It can, but fsck is supposed to cope with this. :)

 > 
 > -- Jamie

Nikita.

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Thu, Aug 26, 2004 at 04:04:34PM +0100, Jamie Lokier wrote:
> Christophe Saout wrote:
> > What reiser4 can do, but the VFS can't is to insert or remove data in
> > the middle of a file. Adding this above the page cache would probably be
> > almost impossible (truncate seems already complicated enough).
> 
> That would be one of those "special features" that a
> VFS-plus-userspace implementation of archive views could take
> advantage of on reiser4, while using a slower method (sometimes much
> slower) on all other filesystems.
> 
> By the way, can reiser4 share parts of files between different files?

no, those file plugins are not written yet :)

Do you mean COW files or some another thing?  For COW files, the reiser4
support is not enough, we need to teach cp(1) or another utility to inform the
fs layer that copying can be done by creation of COW files.

> -- Jamie

-- 
Alex.

Re: silent semantic changes with reiser4

[Shaya Potter]

On Sun, 2004-08-29 at 16:34 +0400, Alex Zarochentsev wrote:

> Do you mean COW files or some another thing?  For COW files, the reiser4
> support is not enough, we need to teach cp(1) or another utility to inform the
> fs layer that copying can be done by creation of COW files.

it's easy enough to add that via an fs ioctl(), I did the equivalent
last semester for some research I'm doing while using ext3cow (which
unfortunately was put on hold for the summer), it's just a take off of
sys_link(), a sys_cowpy() of sorts.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Alex Zarochentsev wrote:
> > > What reiser4 can do, but the VFS can't is to insert or remove data in
> > > the middle of a file. Adding this above the page cache would probably be
> > > almost impossible (truncate seems already complicated enough).
> > 
> > That would be one of those "special features" that a
> > VFS-plus-userspace implementation of archive views could take
> > advantage of on reiser4, while using a slower method (sometimes much
> > slower) on all other filesystems.
> > 
> > By the way, can reiser4 share parts of files between different files?
> 
> no, those file plugins are not written yet :)
> 
> Do you mean COW files or some another thing?

COW files would be nice as well, but I meant another thing: for files
which are parts of an uncompressed archive, it makes sense, if the fs
offers it efficiently, to share the storage.

Doing it properly requires that memory is also shared, for example
when COW files are used to isolate trees in virtual server jails, you
still want the mapped executables to share memory.

You can't do that just by writing a plugin.

> For COW files, the reiser4 support is not enough, we need to teach
> cp(1) or another utility to inform the fs layer that copying can be
> done by creation of COW files.

Actually we don't.  The VM and VFS layers could deduce it with no
changes to cp(1), by tracking which pages are identical due to read(3)
and write(3).  However, changing cp(1) is easier.

reiser4 support is obviously the first prerequisite...

-- Jamie

Re: silent semantic changes with reiser4

[Jamie Lokier]

Adrian Bunk wrote:
> What is the technical reason why a tar plugin should be reiser4 
> specific, instead of a generic VFS or userspace solution that would 
> allow the same also on other fs like e.g. iso9660?

It should be a generic VFS plugin, not reiser4 or userspace.  The VFS
plugin should call out to userspace for most actions (except handling
cached data), and it should take advantage of special reiser4 features
for storage and performance optimisations.  But it should still work
over a standard filesystem, when those special features aren't
available.  I guess FUSE and many earlier projects are heading in this
direction.

A generic userspace solution doesn't let you "cd" into a tar file from
all programs like you can inside Midnight Commander.

Gnome and KDE take the approach that every userspace system call
should be intercepted and filtered, to create the illusion of virtual
data.  As a result, different programs see different virtual data: you
can't just cut and paste a path from Gnome or KDE into any other
program.  It's not just a "social problem of libraries" thing:
sometimes I have programs which don't link to libc.  Sometimes I have
programs which mustn't link with anything that calls malloc().  It'd
be silly for them to have a different view of the filesystem just
because they can't link with some userspace library.

The Gnome/KDE/Midnight Commander pure userspace solution is silly: if
_every_ program in the system should get the same view, it makes much
more sense for the kernel to filter the system calls and redirect the
virtual accesses to a userspace daemon, while keeping the real
accesses at full speed.

Furthermore is makes much more sense for the kernel's page cache to
hold those uncompressed pages, than for every userspace application to
try and cooperatively manage a cache of uncompressed fragments in the
most inefficient way.

There's another problem with the Midnight Commander approach.  If I
"cd" into a tar file, and then a program writes to the tar file, I
don't always see the changes straight away. The two views aren't
coherent.  This isn't an easy problem to solve, but it should be
solved.

When a simple "cd" into .tar.gz or .iso is implemented properly, it
will have _no_ performance penalty after you have first looked in the
file, so long as it remains in the on-disk cache.  And, the filesystem
will manage that cache intelligently.

Imagine: for looking at source files and such, you probably won't
bother untarring in future, and you won't bother keeping untarred
source trees in your home directory for easy access to things you look
at often.  Why waste the space?  You could install whole applications
as a .tar and run them from within it, with no performance penalty.

Similarly, the filesystem will be able to archive directories
automatically that haven't been touched in a long time, with no
visible change except increased storage space.  "grep" will be a bit
slower, but you'll have a useful search tool by then (using coherent
indexes) which will be more useful than grep, and much faster.

-- Jamie

Re: silent semantic changes with reiser4

[Horst von Brand]

Jamie Lokier <jamie@shareable.org> said:

[...]

> When a simple "cd" into .tar.gz or .iso is implemented properly, it
> will have _no_ performance penalty after you have first looked in the
> file, so long as it remains in the on-disk cache.  And, the filesystem
> will manage that cache intelligently.

Nonsense. The .iso or .tar or whatever would have to be kept un-isoed or
un-tarred in memory (or on disk cache) for this to be true, and that takes
quite a long time. Each time you want to peek anew at linux/Makefile, the
whole tarfile will have to be read and stored somewhere, and that is just
too slow for my taste. The .tar format is optimized for compact storage,
the on-disk format of a filesystem is optimized for fast access and
modifiability. Now go ahead and enlarge a file on your .iso/.tar a bit...it
will take ages to rebuild the whole thing. There is a _reason_ why there
are filesystems and archives, and they use different formats. If it weren't
so, everybody and Aunt Tillie would just carry .ext3's around, and would
wonder what the heck all this fuss is about.

> Imagine: for looking at source files and such, you probably won't
> bother untarring in future, and you won't bother keeping untarred
> source trees in your home directory for easy access to things you look
> at often.  Why waste the space?  You could install whole applications
> as a .tar and run them from within it, with no performance penalty.

Perpetuum mobile is a nice idea too.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Jamie Lokier]

I'm going to explain why filesystem support for .tar.gz or other
"document container"-like formats is useful.  This does _not_ mean tar
in the kernel (I know someone who can't read will think that if I
don't say it isn't); it does mean hooks in the kernel for maintaining
coherency between different views and filesystem support for cacheing.

The vision I'm going for here is:

  1. OpenOffice and similar programs store compound documents in
     standard file formats, such as .tar.gz, compressed XML and such.

     Fs support can reduce CPU time handling these sorts of files, as
     I explain below, while still working with the standard file formats.

     With appropriate userspace support, programs can be written which
     have access to the capabilities on all platforms, but reduced CPU
     time occurs only on platforms with the fs support.

  2. Real-time indexing and local search engine tools.  This isn't
     just things like local Google; it's also your MP3 player scanning
     for titles & artists, your email program scanning for subject
     lines to display the summary fast, your blog server caching built
     pages, your programming environment scanning for tags, and your
     file transfer program scanning for shared deltas to reduce bandwidth.

     I won't explain how these work as it would make this mail too
     long.  It should be clear to anyone who thinks about it why the
     coherency mechanism is essential for real-time, and a consistent
     interface to container internals helps with performance.

Horst von Brand wrote:
> Jamie Lokier <jamie@shareable.org> said:
> > When a simple "cd" into .tar.gz or .iso is implemented properly, it
> > will have _no_ performance penalty after you have first looked in the
> > file, so long as it remains in the on-disk cache.  And, the filesystem
> > will manage that cache intelligently.
> 
> Nonsense. The .iso or .tar or whatever would have to be kept un-isoed or
> un-tarred in memory (or on disk cache) for this to be true, and that takes
> quite a long time. Each time you want to peek anew at linux/Makefile, the
> whole tarfile will have to be read and stored somewhere,

Wrong.  "So long as it remains in the on-disk cache" means each time
you peek at linux/Makefile, the tarfile is _not_ read.

For a tarfile it's slow the first time, and when it falls out of the
on-disk cache, otherwise, for component files you are using regularly
(even over a long time) it's as fast as reading a plain file.

You obviously know this, as you mentioned on-disk cache in the reply,
so I infer from the rest of your mail that what you're trying to say
is more about modifications than reading archives.  That it would be
silly to keep working data in .tar.gz files, because working inside
them regularly would be slow.

Which means you must be assuming, incorrectly, that these .tar.gz
files are really kept up to date on disk with every component file
modification.

Which is silly.  .tar.gz files are suitable for *transport* and
*archival*, not regular random access; it's almost rude of you to
suggest I didn't know that.

The proposal is that .tar.gz files (and others) are analysed on demand
and content cached on disk as it is read.  Then subsequent reads will
be as fast as if you had unpacked the archives by hand, manually using
the tar command.  This is obviously exactly the same as you do now,
with a small bit of added convenience.

The other part of the proposal is that when you modify a component,
the modifications are stored on disk in the same way as ordinary
files, using the regular high performance random access disk
structures.  Nothing is done to recreate the archive at this point; I
think this is where you misunderstood and thus flamed.

_If_ after modifying components, you then read the .tar.gz as a file,
then (and only then) is it recreated, taking in the worst case the
same time as running the tar command.

The _only_ times when that occurs are precisely those times when you
would have run the tar command manually: because you only read the
.tar.gz file when you need the flat file for some purpose, such as
attaching it to an email, transferring by FTP or HTTP, or reading it
into a program that needs it in that format.

If there is anything about that strategy that doesn't make sense, then
I suggest I have failed to explain it properly, and you're welcome to
demand a clearer explanation.

> the .tar format is optimized for compact storage, the on-disk format
> of a filesystem is optimized for fast access and modifiability.

Actually no, .tar is not compact at all.  It's also not optimised for
random read access, but after an index is build it is very fast for that.

.tar.gz is compact.  Although that is not especially fast for random
read access, you can build a "compression dictionary index" which
optimises random read access even in a .tar.gz, without ever unpacking
the whole thing.

Some formats like .iso, .zip and .jar are optimised for compact
storage _and_ fast random access.  They come with an index, and don't
need one to be built and cached.

These are not filesystem-like formats, obviously, but they are the
formats you need to pack and unpack when exchanging data with other
people.  That's the _only_ reason they're on your disk (virtually;
they may not really exist some of the time).

> Now go ahead and enlarge a file on your .iso/.tar a bit...it
> will take ages to rebuild the whole thing. There is a _reason_ why there
> are filesystems and archives, and they use different formats. If it weren't
> so, everybody and Aunt Tillie would just carry .ext3's around, and would
> wonder what the heck all this fuss is about.

If you enlarge a file in your .iso/.tar subdirectory a bit... nothing
happens.  Why would a smart programmer do anything so silly as rebuild
the archive at that point?

_If_ you subsequently read the .iso/.tar _file_, then and only then
does it rebuild.  Once, after lots of component writes.  The only time
you would ever do that is if you are specifically reading the archive
file, which means you actually want to use the repacked file at that
point, for example to FTP it somewhere or use as an email attachment.

If the filesystem does not do that on demand, then you would have run
the tar command manually at that point, precisely because that's a
point where you need the repacked archive.  So in case where the
filesystem repacks the archive, it takes exactly the same time as you
would have taken anyway; it's just automatic instead of manual.  (As a
handy side effect, the automatic method offers lower latency for
transmissions).

Now, why would we bother with all this?

I see three reasons: convenience, time efficiency, and storage efficiency.

Convenience is simply that it is handy to be able to look inside
archive files, in those situations where we _currently_ use them, without
having to manually untar when we need to, and without having to
remember to clean up old directories when we discover we aren't using
those often any more.  This is _not_ an argument for using .tar.gz
files in place of ordinary directories!  Convenience applies to doing
the things you do now.

Time efficiency has two angles.  A simple one is that accessing
.tar.gz contents through any kind of filesystem interface, even pure
userspace, can be faster than unpacking whole files, simply because
there are ways to decode parts without unpacking it all.

However, the main time efficiency that I see comes from the increasing
number of applications where the "Open" and "Save" operations store
data in *.gz files (e.g. OpenOffice compressed XML documents), or
*.tar.gz files (some compound document formats), or other things like
that.  (If you think about it, quite a lot of things are like that).

With these, every "Open" currently has to decompress and maybe unpack
an archive format.  Every "Save" currently has to pack and then
compress.  This is done so the user sees a single flat file containing
a complex document, but it is a waste of CPU time until the user
actually transports the flat file.

The lazy proposal, as described earlier in this mail, _removes_ these
decompression, unpacking, packing and compression CPU-intensive steps
when they are unnecessary.  The experience of a single file containing
a complex document is maintained, but the CPU time is reduced in many
typical operations.  "Open" gets faster after you first look at a
file, "Save" gets a lot faster for large documents, and the equivalent
of grep (or later, real-time local search engine) gets a lot faster
too.  There is no operation where CPU time is overall increased.

This is what I've meant throughout this thread, when I say containers:
document files of the kind used to hold text, figures, etc. that are
typically transported as a unit, and edited as a unit, but nonetheless
at the moment they're stored in somewhat CPU intensive formats, for
compactness.  That's fine for a 1 page letter, but think of the
OpenOffice 500-page book containing a large number of diagrams.

However, even simple programs that read & write compressed XML benefit.

The proposal allows that sort of thing to be handled more time
efficiently that it is today, and in a way that is very practical to use.

(It's unthinkable that OpenOffice and similar programs would have a
lot of code which stored data in a special way just for Linux, just
for these performance benefits which are otherwise user-invisible, but
it's thinkable that a general purpose userspace library which is
portable to all platforms could be written, which takes advantage of
the facility when it's available and does the equivalent of today's
"compress on save" when the filesystem facility isn't available).

Finally, storage efficiency comes from simply allowing the filesystem
and supporting tools to decide when it is best to store data in
unpacked, packed & compressed, both at the same time, or another
archival form.  The filesystem has comparativaly good knowledge of
which data to archive and when, but it can only maintain the illusion
if there's a mechanism to make archived forms and unpacked forms
coherent.

Now, I'm sure there is a way to implement this on top of a neat and
simple kernel feature involving weird bind mounts, leases, dnotifies
and FUSE.  But those kernel offering are quite a mess at the moment
and don't fit together in a way which can usefully create this effect.

Auto-mounting uservfs directories over file-as-directory, using
moveable bind mounts _nearly_ offers the kernel primitives we need to
build this in userspace and get the all the efficiencies.  But not quite.

(We could obviously do it all in userspace by putting _everything_ in
a userspace filesystem, but that would be silly as it would throw away
all of the performance of having a threaded filesystem in the kernel.
It might do as a proof of concept though).

-- Jamie

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Wed, 1 Sep 2004, Jamie Lokier wrote:
>
> I'm going to explain why filesystem support for .tar.gz or other
> "document container"-like formats is useful.  This does _not_ mean tar
> in the kernel (I know someone who can't read will think that if I
> don't say it isn't); it does mean hooks in the kernel for maintaining
> coherency between different views and filesystem support for cacheing.

I think that's a valid thing, but there are some fundamental problems with 
it if you expect it to work on a normal filesystem (ie something that 
isn't fundamentally designed as a database).

For example, _what_ kind of coherency do you think is acceptable? Quite
frankly, using standard UNIX interfaces, absolute coherency just isn't an
option, because it's just not possible to try to atomically update a view
at the same time somebody else is writing to the "main file". "mmap()" is
the most obvious example of this, but even the _basic_ notion of multiple
"read" calls is not atomic without locking that is _way_ too expensive.

A "read()" on a file is not atomic even on the _plain_ file: if somebody 
does a concurrent "write()", the reader may see a partial update. This 
becomes a million times more confusing if the reader is seeing a 
structured view of the file the writer is modifying.

Also, it's likely impossible to write() to the view-file, again unless you 
expect all the underlying filesystems to be something really special.

So from a _practical_ standpoint, I suspect that the best you can really 
do pretty cheaply (and which gets you 90% of what you probably want) is:

 - open-close consistency: the "validity" of the cache is checked at 
   _open_ time, and no guarantees are given about the cache being 
   updated afterwards.
 - read-only access to the cache (ie you can only read the view, not write 
   to it).

and quite frankly, I think you can do the above pretty much totally in
user space with a small library and a daemon (in fact, ignoring security
issues you probably don't even need the daemon). And if you can prototype
it like that, and people actually find it useful, I suspect kernel support
for better performance might be possible.

Suggested interface:

	int open_cached_view(int base_fd, char *type, char *subname);

where "type" would be the type of the view (ie "tar" for a tar-file view,
"idtag" for a mp3 ID tag, or NULL for "autodetect default view") and
"subname" would be the cache entry name (ie the tar-file filename, or the
tag type to open).

I bet you could write a small library to test this out for a few types.  
See if it's useful to you. And only if it's useful (and would make a huge
performance difference) would it be worth putting in the kernel.

Implementation of the _user_space_ library would be something like this:

	#define MAXNAME 1024
	int open_cached_view(int base_fd, char *type, char *subname)
	{
	        struct stat st;
	        char filename[PATH_MAX];
	        char name[MAXNAME];
	        int len, cachefd;
	
	        if (fstat(base_fd, &st) < 0)
	                return -1;
	        sprintf(name, "/proc/self/fd/%d", base_fd);
	        len = readlink(name, filename, sizeof(filename)-1);
	        if (len < 0)
	                return -1;
	        filename[len] = 0;
	
	        /* FIXME! Replace '/' with '#' in "type" and "subname" */
	        len = snprintf(name, sizeof(name),
	                        "%04llx/%04llx/%s/%s/%s",
	                        (unsigned long long) st.st_dev,
	                        (unsigned long long) st.st_ino,
	                        type ? : "default",
	                        subname,
	                        filename);
	        errno = ENAMETOOLONG;
	        if (len >= sizeof(name))
	                return -1;
	        cachefd = open(name, O_RDONLY);
	        if (cachefd >= 0) {
	                /* Check mtime here - maybe we could have kernel support */
	                return cachefd;
	        }
	        if (errno != ENOENT)
	                return -1;
	        /*
	        .. try to generate cache file here ..
	         */

see what I'm aiming at? You start out with a generic "attribute cache" 
library that does some hacky things (like depending on "mtime" for 
coherency) and then if that works out you can see if it's useful.

			Linus

Re: The argument for fs assistance in handling archives

[Hans Reiser]

Linus Torvalds wrote:

>On Wed, 1 Sep 2004, David Masover wrote:
>  
>
>>And that is the right solution.  Not the only one, but the right one.
>>Caching isn't the only thing sorely in need of transaction support right
>>now.  Actually, I find it hard to think of anything on Linux which
>>shouldn't have transactions -- why should /etc/fstab or
>>/home/david/homework be more fragile than /var/lib/mysql?
>>    
>>
>
>It's easy to talk big. 
>
>It's damn hard to _implement_ a complex system, and make it stable and 
>bug-free, and support legacy applications.
>
>There is a reason why we do only what _must_ be done in kernel space.
>
>		Linus
>
>
>  
>
Filesystems need transactions. Given that they need it, it was very 
tempting to build an infrastructure offering some of the functionality 
to user space, and so we did in reiser4. What we have is very 
rudimentary, but I think I am prepared to say that important aspects of 
high performance transactions are better done in the kernel than in user 
space. It will be interesting to see what aspects of transaction 
functionality belong in user space as our transactions API comes into 
existence. Clearly there are some things about isolation that are better 
implemented by user space because user space can make optimizations the 
kernel doesn't know enough to make. Still, it will be interesting to see 
how low we can make the overhead of isolation. Right now reiser4 
supports atomicity but not isolation, which is to say we offer to 
guarantee that a set of operations will either all survive a crash or 
none of them will, but we don't let you roll them back without a crash. 
(Originally I called this a transcrash rather than atomicity, but nobody 
liked the term.) Being modest in our ambition was important to 
performance, but maybe in time we will find a high performance way to be 
more ambitious.

Re: The argument for fs assistance in handling archives

[Linus Torvalds]

On Wed, 1 Sep 2004, David Masover wrote:
> 
> I think transactions should be done in kernel space.  It's the only way
> to "enforce" them sanely.

Well, they clearly can't, since it depends on the filesystem. Also, 
transactions have a tendency to be expensive, and people have a tendency 
to ask for more than you give them. Doing transactions on one file is 
only the beginning - you'll find people who want transactions across file 
boundaries etc.

That's a basic fact when it comes to pretty much anything: you can always 
find people who want something better. You can't be all things to all 
people, so what you actually _want_ to do is to expose the _minimal_ set 
of capabilities that people can build on.

So I definitely don't want this discussion to degenerate (any more than it
lng since has ;) into what people _wish_ for. No, it should be a "what is
the _least_ we can absolutely do" that solves real problems.

		Linus

Re: The argument for fs assistance in handling archives

[Hans Reiser]

Linus Torvalds wrote:

>
>. Doing transactions on one file is 
>only the beginning - you'll find people who want transactions across file 
>boundaries etc.
>  
>
Yup, that's what reiser4 is aiming at and where things get exciting for 
version control systems and the like.  We are just missing the api code 
for it at the moment, all the infrastructure is there, and the api is in 
progress in sys_reiser4.

Re: The argument for fs assistance in handling archives

[Linus Torvalds]

On Thu, 2 Sep 2004, Hans Reiser wrote:
> Linus Torvalds wrote:
> >
> >. Doing transactions on one file is 
> >only the beginning - you'll find people who want transactions across file 
> >boundaries etc.
>
> Yup, that's what reiser4 is aiming at and where things get exciting for 
> version control systems and the like.  We are just missing the api code 
> for it at the moment, all the infrastructure is there, and the api is in 
> progress in sys_reiser4.

But _my_ point is, no user program is going to take _advantage_ of
anything that only one filesystem on one system offers.

So there's no point.

It's much saner (from _any_ app standpoint) to roll their own database in 
user space - that way it just works.

In other words, nobody is really ever going to take advantage of this. 
This is _not_ how technical advanncement happens. The way you get people 
to take advantage of something is to have a nice gradual ramp-up, not a 
sudden new feature that they can't realistically use.

So before you do transactions in the filesystem, you have to be able to do 
them in user space - and then make the filesystem version be _compatible_ 
with that user space library. That way you can get people to use the 
library ("hey, it works anywhere") and then you can get them to use your 
filesystem ("hey, if you use our super-duper filesystem, you can do what 
you are already doing five times faster").

See? You're starting at the wrong end. Give me a portable interface to use 
_first_. Then do transactions in the filesystem.

(Now, as an academic exercise and as a learnign experience and prototyping 
work it migt make sense to do the filesystem first. But then you have to 
really _consider_ it a prototype, and realize that to make it useful you'd 
still have to do the user thing before you can "sell" people on the idea).

		Linus

Re: The argument for fs assistance in handling archives

[Hans Reiser]

Linus Torvalds wrote:

> But _my_ point is, no user program is going to take _advantage_ of
>
>anything that only one filesystem on one system offers.
>  
>
Apple does not have this problem....

and yes, the apps will take advantage of it, which is different from 
depending on it.  If you use the wrong fs you will lose some of the 
features of the app.

For 30 years nothing much has happened in Unix filesystem semantics 
because of sheer cowardice (excepting Clearcase, which priced itself 
into a niche market).   It is 25 years past time for someone to change 
things.  That someone will have first mover advantage, and the more 
little semantic features possessed the more lure there will be to use it 
which will increase market share which will lure more apps into 
depending on it and in a few years the other filesystems will 
(deservedly) have only a small market share because the apps won't all 
work on them.

Besides, there are enhancements which are simply compelling.  You can 
write a dramatically better performance version control system with a 
much simpler design if the FS is atomic.    Our transaction manager 
first draft was written by a version control guy, and he would probably 
be happy to tell you how  lack of atomicity other than rename makes 
version control software design hideous.

We have the performance lead.  By next year we will be stable enough for 
mission critical servers, and then we start the serious semantic 
enhancements.

If you don't embrace progress, then you doom Linux to following behind, 
because the guys at Apple are pretty aggressive now that Jobs is back, 
and they WILL change the semantics, and they will do so in compelling 
ways, and Linux will be reduced to aping them when it should be leading 
them.

Hans

Re: The argument for fs assistance in handling archives

[Oliver Hunt]

How would we go about finding out how many data forks were in a file? 
Because in order to be able to retrieve data from a fork we would need
to know that the fork were there.  Currently this would imply that we
go looking through mtab or some such to find out what fs we're running
on, which seems ugly.

Alternatively we go through the _exciting_ task of making every other
fs (with the exceptions of ntfs, and whatever it is that macs use,
which would need there own custom code) and add code that effectively
goes

getNumForks(fileref){ return 1;} 

or add a new open call that can take a fork number...

either way we have to add a new syscall, that doesn't conform to any
real standard(though i suppose it would be possible to use
macOS/windows style fork iopening interface)

I personally like the concept of having multiple forks in a file, but
in this case I'm inclined towards usermode first, then if it takes off
add kernel level support.

--Oliver Hunt

On Thu, 02 Sep 2004 01:58:30 -0700, Hans Reiser <reiser@namesys.com> wrote:
> Linus Torvalds wrote:
> 
> > But _my_ point is, no user program is going to take _advantage_ of
> >
> >anything that only one filesystem on one system offers.
> >
> >
> Apple does not have this problem....
> 
> and yes, the apps will take advantage of it, which is different from
> depending on it.  If you use the wrong fs you will lose some of the
> features of the app.
> 
> For 30 years nothing much has happened in Unix filesystem semantics
> because of sheer cowardice (excepting Clearcase, which priced itself
> into a niche market).   It is 25 years past time for someone to change
> things.  That someone will have first mover advantage, and the more
> little semantic features possessed the more lure there will be to use it
> which will increase market share which will lure more apps into
> depending on it and in a few years the other filesystems will
> (deservedly) have only a small market share because the apps won't all
> work on them.
> 
> Besides, there are enhancements which are simply compelling.  You can
> write a dramatically better performance version control system with a
> much simpler design if the FS is atomic.    Our transaction manager
> first draft was written by a version control guy, and he would probably
> be happy to tell you how  lack of atomicity other than rename makes
> version control software design hideous.
> 
> We have the performance lead.  By next year we will be stable enough for
> mission critical servers, and then we start the serious semantic
> enhancements.
> 
> If you don't embrace progress, then you doom Linux to following behind,
> because the guys at Apple are pretty aggressive now that Jobs is back,
> and they WILL change the semantics, and they will do so in compelling
> ways, and Linux will be reduced to aping them when it should be leading
> them.
> 
> Hans
> 
> 
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: The argument for fs assistance in handling archives

[Helge Hafting]

Oliver Hunt wrote:

>How would we go about finding out how many data forks were in a file? 
>Because in order to be able to retrieve data from a fork we would need
>to know that the fork were there.  Currently this would imply that we
>go looking through mtab or some such to find out what fs we're running
>on, which seems ugly.
>
>  
>
Depends on how the forks eventually get implemented.
With the file-as-directory concept, all you need is to
look at the file's directory part to see what is there.  (The forks,
implemented as files in a subdirectory.)  It is done the same way
as for an ordinary directory, so nothing "new".

>Alternatively we go through the _exciting_ task of making every other
>fs (with the exceptions of ntfs, and whatever it is that macs use,
>which would need there own custom code) and add code that effectively
>goes
>
>getNumForks(fileref){ return 1;} 
>
>  
>
Necessary if some other mechanism is used, sure.
Helge Hafting

Re: The argument for fs assistance in handling archives

[Oliver Hunt]

> Depends on how the forks eventually get implemented.
> With the file-as-directory concept, all you need is to
> look at the file's directory part to see what is there.  (The forks,
> implemented as files in a subdirectory.)  It is done the same way
> as for an ordinary directory, so nothing "new".
> 
This still doesn't solve the problem of how we distinguish between a
multi-fork file
and a directory, for those old programs(ie. almost all that currently
exist), to be able to access meaningful data we would need to either
return just the primary fork, a serialized version of all forks in the
file, or make the file look more or less identical to a directory.

The first option could cause problems when transfering files between
different filesystems,
the second results in programs getting metadata they can't handle, and
the third option results in few of the advantages over, well,
directories...  And even those applications that could handle the fork
information nicely would need to fs type to find out whether they were
handling a directory or a multi-forked file...

As I say I like the idea, but I can't see anyway of implementing it in
a way that is useful without first putting considerable effort into at
least the VFS if not all the actual fs drivers.

--Oliver Hunt

Re: The argument for fs assistance in handling archives

[Spam]

  

>> Depends on how the forks eventually get implemented.
>> With the file-as-directory concept, all you need is to
>> look at the file's directory part to see what is there.  (The forks,
>> implemented as files in a subdirectory.)  It is done the same way
>> as for an ordinary directory, so nothing "new".
>> 
> This still doesn't solve the problem of how we distinguish between a
> multi-fork file
> and a directory, for those old programs(ie. almost all that currently
> exist), to be able to access meaningful data we would need to either
> return just the primary fork, a serialized version of all forks in the
> file, or make the file look more or less identical to a directory.

  Well. wasn't the idea that unless programs specifically tried to
  open the file-as-dir as a directory it would look like a file?

  ls -F would show it as file. Or have I understood wrong?

> The first option could cause problems when transfering files between
> different filesystems,

  The meta-data should be deleted if it the file is copied or moved to
  a medium that doesn't support it. However a _warning_ may be shown
  to the user if there is risk to loose data.

  I do not think we should not implement something like this because
  most other filesystems in Linux doesn't support it. Think of other
  limitations there is in various ones. Large file support. Limited
  amount of files, length of file names, etc.

> the second results in programs getting metadata they can't handle, and
> the third option results in few of the advantages over, well,
> directories...  And even those applications that could handle the fork
> information nicely would need to fs type to find out whether they were
> handling a directory or a multi-forked file...

  No, if an application supported the file-as-dir then it could simply
  try to open the file as a directory. It would fail on systems that
  didn't support it.

  The meta-data and file streams would be seen as ordinary files. If
  applications support the contents or not isn't really relevant, is
  it?.

> As I say I like the idea, but I can't see anyway of implementing it in
> a way that is useful without first putting considerable effort into at
> least the VFS if not all the actual fs drivers.

  Indeed. It is important that something like this gets implemented as
  a transparent way as possible. If it could be done in a general way
  so other filesystems like ext3/4 can eventually support it then that
  would be wonderful. I do not, however, think that we should block it
  in reiser4 because no other filesystems support it.

  ~S

> --Oliver Hunt

Re: The argument for fs assistance in handling archives

[Giuliano Pochini]

On Thu, 2 Sep 2004, Spam wrote:

>   Well. wasn't the idea that unless programs specifically tried to
>   open the file-as-dir as a directory it would look like a file?
>
>   ls -F would show it as file. Or have I understood wrong?

Yes, otherwise apps that do

if (S_ISDIR()) {
	..
} else if (S_ISREG()) {
	..
}

would behave differently from apps that check the file in
different order, and they would probably break because in
a regular fs that order is not important. The goal is to
keep the semantic changes as hidden as possible for apps
that don't know about the new features.


--
Giuliano.

Re: The argument for fs assistance in handling archives

[Christian Mayrhuber]

On Thursday 02 September 2004 12:02, Spam wrote:
> > As I say I like the idea, but I can't see anyway of implementing it in
> > a way that is useful without first putting considerable effort into at
> > least the VFS if not all the actual fs drivers.
> 
>   Indeed. It is important that something like this gets implemented as
>   a transparent way as possible. If it could be done in a general way
>   so other filesystems like ext3/4 can eventually support it then that
>   would be wonderful. I do not, however, think that we should block it
>   in reiser4 because no other filesystems support it.

What about extending the namespace with leading ".." and "...".

In Unix names starting with a  "." already have the meaning of being a 
hidden/config entry.

A name starting ".." means streams/metainformation, etc. something belonging 
to the file/directory that should get backed up, copied, etc.
There could be a "..streams", "..metas" or a "..acl" entry.

If it starts with "..." it means some system specific information, like the
name of the hash algorithm used for the current directory. It's basically 
information that is not portable and not required by applications, like
"...fsplugins".

There is still the big problem of how to copy files with their associated 
streams or meta information onto a standard unix filesystem as the 
file/directory duality cannot be expressed. (It's forbidden to have
a directory + file with the same name in the same directory)
Maybe copy could create something like a ".#filename" directory for this kind
of information if the advanced features are not supported on the target 
filesystem. This is neither nice nor clean, but at least you don't loose 
information. I do not suggest that the kernel should simulate the advanced 
features with  ".#filename" directories, it's more a backup/restore thing 
that could work if it's written down properly.

-- 
lg, Chris

Re: The argument for fs assistance in handling archives

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 390 bytes --]

On Thu, 02 Sep 2004 12:02:59 +0200, Spam said:

>   The meta-data should be deleted if it the file is copied or moved to
>   a medium that doesn't support it. However a _warning_ may be shown
>   to the user if there is risk to loose data.

OK... I'll bite.  How do you report the warning to the user if you're using
an unenhanced utility to copy a file to a file system that may be lossy?

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: The argument for fs assistance in handling archives

[Spam]

> On Thu, 02 Sep 2004 12:02:59 +0200, Spam said:

>>   The meta-data should be deleted if it the file is copied or moved to
>>   a medium that doesn't support it. However a _warning_ may be shown
>>   to the user if there is risk to loose data.

> OK... I'll bite.  How do you report the warning to the user if you're using
> an unenhanced utility to copy a file to a file system that may be lossy?

  That I do not really know. But I think this is somewhat the
  discussion here - how to make the support for metas and streams
  as good as possible.

  I suppose it would be difficult to actually query the user about
  something like this unless the applications themselves do. Perhaps
  warnings in the syslog is good for now?

  Already  now  things  like advanced attributes get lost when copying
  data from a supported fs to a non-supported one.

  Perhaps documentation around the ways to use streams and meta data
  should mention that only the supported fs will keep the extra info.

  As long as the user is aware that whatever information he puts in
  the meta or streams will be lost then it is ok, isn't it?

  ~S

  
  
  

Re: The argument for fs assistance in handling archives

[Helge Hafting]

Valdis.Kletnieks@vt.edu wrote:

>On Thu, 02 Sep 2004 12:02:59 +0200, Spam said:
>
>  
>
>>  The meta-data should be deleted if it the file is copied or moved to
>>  a medium that doesn't support it. However a _warning_ may be shown
>>  to the user if there is risk to loose data.
>>    
>>
>
>OK... I'll bite.  How do you report the warning to the user if you're using
>an unenhanced utility to copy a file to a file system that may be lossy?
>  
>
Well, _if_ you're using my sort of file-as-dir then you'd expect a
plain "cp" to only copy the file.  You'd use cp -a to copy the
file's subdirectory too.  If you're copying to anther filesystem
that doesn't support file-as-directory then "cp -a" won't be able
to create a directory with the same name as the file, and
will do whatever cp does in such a situation.  I.e. something like
"NAME exists but is not a directory".

Helge Hafting

Re: The argument for fs assistance in handling archives

[Helge Hafting]

Oliver Hunt wrote:

>>Depends on how the forks eventually get implemented.
>>With the file-as-directory concept, all you need is to
>>look at the file's directory part to see what is there.  (The forks,
>>implemented as files in a subdirectory.)  It is done the same way
>>as for an ordinary directory, so nothing "new".
>>
>>    
>>
>This still doesn't solve the problem of how we distinguish between a
>multi-fork file
>and a directory, for those old programs(ie. almost all that currently
>exist), to be able to access meaningful data we would need to either
>return just the primary fork, a serialized version of all forks in the
>file, or make the file look more or less identical to a directory.
>  
>
If they open "filename" then they get what you calls the primary fork.
If they open "filename/someforkname" then they get some other fork, using
exactly the same mechanism as when they opens "directoryname/somefilename"
And if you want to use the old tools to look at some forks, just do a
$ cd filename
It will work if "filename" happens to be a file-as-dir thing, then do
$ ls
And you'll see the forks by name.  Because the "forks" are merely
plain files in a subdirectory. 

The only new thing needed is the ability for something to be both
file and directory at the same time.  Some tools will need
a update - usually only because they blindly assume that a directory
isn't a file too, or that a file can't be a directory too.  Remove the 
mistaken
assumption and things will work because the underlying system
calls (chdir or open) _will_ work.



>The first option could cause problems when transfering files between
>different filesystems,
>the second results in programs getting metadata they can't handle, and
>the third option results in few of the advantages over, well,
>directories...  
>
Few advantages over directories perhaps, but other implementatins
don't have more advantages over directories than this one. 
Requiring another syscall to open forks other than the primary
breaks _all_ existing software because it obviously don't use that
syscall yet.  And then it doesn't provide any advantages over the
file-as-plain-directory way. . .

>And even those applications that could handle the fork
>information nicely would need to fs type to find out whether they were
>handling a directory or a multi-forked file...
>
>  
>
The distinction might not be important.  A directory can be seen
as a multi-stream file without the main stream.  And there's always
"stat" which should be able to tell what you're looking at.

>As I say I like the idea, but I can't see anyway of implementing it in
>a way that is useful without first putting considerable effort into at
>least the VFS if not all the actual fs drivers.
>  
>
Sure, implementing it well will take effort, wich is why it is important
to find a good way of doing it so the effort won't be wasted.

Helge Hafting

Re: The argument for fs assistance in handling archives

[Horst von Brand]

Helge Hafting <helge.hafting@hist.no> said:

[...]

> The only new thing needed is the ability for something to be both
> file and directory at the same time.

Then why have files and directories in the first place?

>                                       Some tools will need
> a update - usually only because they blindly assume that a directory
> isn't a file too, or that a file can't be a directory too.  Remove the
> mistaken assumption and things will work because the underlying system
> calls (chdir or open) _will_ work.

But with some weird restrictions: No moving stuff around between files, no
linking, some "files" can't be deleted (how would you handle removing the
principal stream of a file?). Some stuff you'd love to do (is, in fact, the
reason for this all) just can't be allowed (i.e., J. Random Luser setting
his own icon for system-wide emacs). So the tools/scripts/users/sysadmins
will have to be painfully aware that some of the files aren't, and some of
the directories aren't either. Major pain in the neck to use, if you look
closer.  Add extra kernel complexity. For little (if any) gain.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: The argument for fs assistance in handling archives

[Spam]

  

> Helge Hafting <helge.hafting@hist.no> said:

> [...]

>> The only new thing needed is the ability for something to be both
>> file and directory at the same time.

> Then why have files and directories in the first place?

  Good point, we don't need them :) Directories are just a visible
  grouping of files to make it easier for the user to manage. But some
  things aren't really that intuitive with todays layout - especially
  for non-unix users.

  Just an example where the user needs to edit config file for some
  program. Where should he look?
  /etc/app.conf ?
  /etc/app/app.conf ?
  /etc/conf.d/app.conf ?
  /var/lib/app/app.conf ?
  and so on....

  Using file-as-dir isn't really that much of a change. It isn't those
  that will confuse people anyway.

>>                                       Some tools will need
>> a update - usually only because they blindly assume that a directory
>> isn't a file too, or that a file can't be a directory too.  Remove the
>> mistaken assumption and things will work because the underlying system
>> calls (chdir or open) _will_ work.

> But with some weird restrictions: No moving stuff around between files, no
> linking, some "files" can't be deleted (how would you handle removing the
> principal stream of a file?).

  Well. there are read-only files. And if you remove the main stream
  (which is the file, really) then it will all be gone =)

>  Some stuff you'd love to do (is, in fact, the
> reason for this all) just can't be allowed (i.e., J. Random Luser setting
> his own icon for system-wide emacs).

  Users do not normally have write permissions to system-wide
  applications. Why would it be any different now?

>  So the tools/scripts/users/sysadmins
> will have to be painfully aware that some of the files aren't, and some of
> the directories aren't either. Major pain in the neck to use, if you look
> closer.  Add extra kernel complexity. For little (if any) gain.

  Not sure what you mean "aren't". Things shouldn't be that much
  different to administer. Normal permissions should still apply. Sure
  extra complexity comes _if_ you want to use extended features that
  are using meta-info. But there is where we need some patches to tar
  and other backup tools.

  One other way would be to enter a specific mode (chroot, a bash
  flag --show-metas, etc) that would allow all streams and metas to be
  seen as files in directories. Then tar and other tools would back it
  all up. Restoring will be a little trickier as we don't know if
  stuff was files or folders before. But I am sure that would be
  solvable. Perhaps a tool to convert them back to normal files with
  meta-data and streams.

  ~S

Re: The argument for fs assistance in handling archives

[Chris Dukes]

On Fri, Sep 03, 2004 at 09:30:38PM +0200, Spam wrote:
> 
>   
> 
> > Helge Hafting <helge.hafting@hist.no> said:
> 
> > [...]
> 
> >> The only new thing needed is the ability for something to be both
> >> file and directory at the same time.
> 
> > Then why have files and directories in the first place?
> 
>   Good point, we don't need them :) Directories are just a visible
>   grouping of files to make it easier for the user to manage. But some
>   things aren't really that intuitive with todays layout - especially
>   for non-unix users.
> 
>   Just an example where the user needs to edit config file for some
>   program. Where should he look?
>   /etc/app.conf ?
>   /etc/app/app.conf ?
>   /etc/conf.d/app.conf ?
>   /var/lib/app/app.conf ?
>   and so on....
> 
>   Using file-as-dir isn't really that much of a change. It isn't those
>   that will confuse people anyway.
> 

If the non-computer literate user is expected to edit the config file,
a pretty interface that hides the internals from the user should be provided.
For the sake of power users, users that believe themselves to be power users,
and configuration changes on multiple systems.
1) Track the location of the config file with the native package management
system.
2) Keep a human readable/editable format.
3) Provide APIs or schema so that the files may be parsed for correctness,
or edited or created programatically.
4) Provide a tool that not only checks for a parseable file, but does
sanity checks such as (Does file exist).

I'm at a loss as to how any of this has anything to do with filesystems.
The closest I see is one implementation of the above (gconf2) happens to
use the filesystem for it's default hierarchical database backend.

As for developers that are too stupid/lazy to provide the GUI and meet
the other criteria...  I suggest extensive use of cattle prods on
them as they attend an automata class.

-- 
Chris Dukes
Warning: Do not use the reflow toaster oven to prepare foods after
it has been used for solder paste reflow. 
http://www.stencilsunlimited.com/stencil_article_page5.htm

Re: The argument for fs assistance in handling archives

[Spam]

  

> On Fri, Sep 03, 2004 at 09:30:38PM +0200, Spam wrote:
>> 
>>   
>> 
>> > Helge Hafting <helge.hafting@hist.no> said:
>> 
>> > [...]
>> 
>> >> The only new thing needed is the ability for something to be both
>> >> file and directory at the same time.
>> 
>> > Then why have files and directories in the first place?
>> 
>>   Good point, we don't need them :) Directories are just a visible
>>   grouping of files to make it easier for the user to manage. But some
>>   things aren't really that intuitive with todays layout - especially
>>   for non-unix users.
>> 
>>   Just an example where the user needs to edit config file for some
>>   program. Where should he look?
>>   /etc/app.conf ?
>>   /etc/app/app.conf ?
>>   /etc/conf.d/app.conf ?
>>   /var/lib/app/app.conf ?
>>   and so on....
>> 
>>   Using file-as-dir isn't really that much of a change. It isn't those
>>   that will confuse people anyway.
>> 

> If the non-computer literate user is expected to edit the config file,
> a pretty interface that hides the internals from the user should be provided.
> For the sake of power users, users that believe themselves to be power users,
> and configuration changes on multiple systems.
> 1) Track the location of the config file with the native package management
> system.
> 2) Keep a human readable/editable format.
> 3) Provide APIs or schema so that the files may be parsed for correctness,
> or edited or created programatically.
> 4) Provide a tool that not only checks for a parseable file, but does
> sanity checks such as (Does file exist).

> I'm at a loss as to how any of this has anything to do with filesystems.
> The closest I see is one implementation of the above (gconf2) happens to
> use the filesystem for it's default hierarchical database backend.

  The relevance was as to the presentation of meta-data, streams and
  such and their implication on users.

  I just used the example of config files because there are not so
  much of a standard among them. Administrators still need to learn
  and remember where and what to backup. This wouldn't be so much of a
  change if they decide to introduce some usage for meta-data or named
  streams.

  Remember, even if the functionality exist you do not have to use
  everything of it.
  
> As for developers that are too stupid/lazy to provide the GUI and meet
> the other criteria...  I suggest extensive use of cattle prods on
> them as they attend an automata class.

  Lets all start with the bash(ing) ;).

  But really. What you say is true. The users do not care so much
  about what is going on below the curtains. Here is where the clash
  between different groups of Linux users and developers appear most
  vivid.

  Sure KDE and Gnome are on their way, but their efforts still have a
  long way to go. They still don't talk to eachother to strive for a
  comon goal - but competes instead (which may even be a good thing).

  ~S

Re: The argument for fs assistance in handling archives

[Horst von Brand]

Spam <spam@tnonline.net> said:
> Horst von Brand <vonbrand@inf.utfsm.cl> said:
> > Helge Hafting <helge.hafting@hist.no> said:
> 
> > [...]
> 
> >> The only new thing needed is the ability for something to be both
> >> file and directory at the same time.
> 
> > Then why have files and directories in the first place?

>   Good point, we don't need them :)

Great. Then everything is a firectory (or dile?). And a firectory points at
other firectories and contains data. I just don't see how you are supposed
to distinguish the data from further firectories...

>                                     Directories are just a visible
>   grouping of files to make it easier for the user to manage. But some
>   things aren't really that intuitive with todays layout - especially
>   for non-unix users.

No operating system is "intuitive". A person is able to glance over some 7
items, a filesystem handles tens of thousands of objects. Nothing
"intuitive" possible in the layout, everybody has to _learn_ how to handle
this.

>   Just an example where the user needs to edit config file for some
>   program. Where should he look?

Another example: User looks for the sysadmin. Where does she look? Answers
to both questions depend on extra data that can't possibly be deduced just
from the statement.

[...]

>   Using file-as-dir isn't really that much of a change. It isn't those
>   that will confuse people anyway.

See above.

> >>                                       Some tools will need
> >> a update - usually only because they blindly assume that a directory
> >> isn't a file too, or that a file can't be a directory too.  Remove the
> >> mistaken assumption and things will work because the underlying system
> >> calls (chdir or open) _will_ work.
> 
> > But with some weird restrictions: No moving stuff around between files, no
> > linking, some "files" can't be deleted (how would you handle removing the
> > principal stream of a file?).
> 
>   Well. there are read-only files. And if you remove the main stream
>   (which is the file, really) then it will all be gone =)

While some other process is messing with another stream. Right, all goes
away when the last reference goes away. But it stays iff some other process
recreates the data stream. Now what if another process goes ahead and
creates a new file with the same name in the meanwhile... but this is very
diferent from "real" directories, if you delete some particular member it
doesn't go away. Explain that to the poor user that can't be trusted to
keep the distinction between directories and files straight...

> >  Some stuff you'd love to do (is, in fact, the
> > reason for this all) just can't be allowed (i.e., J. Random Luser setting
> > his own icon for system-wide emacs).

>   Users do not normally have write permissions to system-wide
>   applications.

Exactly.

>                 Why would it be any different now?

Because the user wants to be able to associate an
icon/description/comment/... with system files too. This is supposed to be
one of the key reasons for the feature...

> >  So the tools/scripts/users/sysadmins
> > will have to be painfully aware that some of the files aren't, and some of
> > the directories aren't either. Major pain in the neck to use, if you look
> > closer.  Add extra kernel complexity. For little (if any) gain.

>   Not sure what you mean "aren't". Things shouldn't be that much
>   different to administer.

They aren't. Some directories are files, and different rules apply to
them. Some files have structure, and you have to exercise extra care not to
loose that. Very different indeed in my book.

>                            Normal permissions should still apply.

Even streams inside a file have permissions? What does it mean if a file is
"searchable" (new permission needed to handle "x" on directories vs files!)
but not writeable, and the default stream is writeable? Or do they just
have the permissions of the container? In that case, the file must be
readable so the user can have a nice icon for it. Consider /etc/shadow...
And if somebody tries to change permissions on one of the streams?

>                                                                   Sure
>   extra complexity comes _if_ you want to use extended features that
>   are using meta-info. But there is where we need some patches to tar
>   and other backup tools.

And new options (where people already get confused by current options).

>   One other way would be to enter a specific mode (chroot, a bash
>   flag --show-metas, etc) that would allow all streams and metas to be
>   seen as files in directories. Then tar and other tools would back it
>   all up. Restoring will be a little trickier as we don't know if
>   stuff was files or folders before. But I am sure that would be
>   solvable.

"I'm sure it can be solved somehow" isn't enough.

>             Perhaps a tool to convert them back to normal files with
>   meta-data and streams.

Yikes!
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: The argument for fs assistance in handling archives

[Alexander G. M. Smith]

Horst von Brand wrote on Fri, 03 Sep 2004 22:27:07 -0400:
> Great. Then everything is a firectory (or dile?). And a firectory points at
> other firectories and contains data. I just don't see how you are supposed
> to distinguish the data from further firectories...

I like to call them fildirutes (file/directory/attribute).  A file
typing system would tell you the intended purpose of a particular
fildurute.  So if a fildurute called X has X/..metas/mimefiletype
containing "application/x-directory" then you know that it should be
treated as being primarily a container for other fildirutes and shown
to the user as a folder in a GUI view.  If it said it was
"application/x-text-document" then the GUI system would default to
opening it in a word processor.  Either way, that's only a hint about
how it should be presented to the user, not something the kernel
enforces.

For efficiency, the file type might just be stored in the fildirute's
inode as a code number (since most things would have a file type) that
the file system exposes as a file called ..metas/mimefiletype.

- Alex

Re: The argument for fs assistance in handling archives

[Helge Hafting]

Horst von Brand wrote:

>Spam <spam@tnonline.net> said:
>  
>
>>Horst von Brand <vonbrand@inf.utfsm.cl> said:
>>    
>>
>>>Helge Hafting <helge.hafting@hist.no> said:
>>>      
>>>
>>>[...]
>>>      
>>>
>>>>The only new thing needed is the ability for something to be both
>>>>file and directory at the same time.
>>>>        
>>>>
>>>Then why have files and directories in the first place?
>>>      
>>>
>
>  
>
>>  Good point, we don't need them :)
>>    
>>
>
>Great. Then everything is a firectory (or dile?). And a firectory points at
>other firectories and contains data. I just don't see how you are supposed
>to distinguish the data from further firectories...
>  
>
By name.
fopen("somefile", "r");  //opens somefile data
fopen("somefile/substream", "r");  //opens a substream of somefile. 

Helge Hafting

Re: The argument for fs assistance in handling archives

[Helge Hafting]

Horst von Brand wrote:

>Helge Hafting <helge.hafting@hist.no> said:
>
>[...]
>
>  
>
>>The only new thing needed is the ability for something to be both
>>file and directory at the same time.
>>    
>>
>
>Then why have files and directories in the first place?
>
>  
>
>>                                      Some tools will need
>>a update - usually only because they blindly assume that a directory
>>isn't a file too, or that a file can't be a directory too.  Remove the
>>mistaken assumption and things will work because the underlying system
>>calls (chdir or open) _will_ work.
>>    
>>
>
>But with some weird restrictions: No moving stuff around between files, no
>  
>
I see no need for such a weird restriction.  If it is "weird", why have it?
If someone want to move stuff from one subdirectory to another, let them.
I see no need to stop that just because a file-as-directory was involved.

>linking, some "files" can't be deleted (how would you handle removing the
>  
>
linking is a problem of course.  We can't have hardlinks to directories,
so no hardlinks to file-as-dir either unless the general problems with
directory links are solved.

>principal stream of a file?). 
>
A design decision.  Removing the principal stream may delete
the entire tree.  Or it may delete the principal stream only,
turning that file-as-dir back into a plain directory.

>Some stuff you'd love to do (is, in fact, the
>reason for this all) just can't be allowed (i.e., J. Random Luser setting
>his own icon for system-wide emacs). So the tools/scripts/users/sysadmins
>will have to be painfully aware that some of the files aren't, and some of
>the directories aren't either. Major pain in the neck to use, if you look
>closer.  Add extra kernel complexity. For little (if any) gain.
>  
>
The gain is indeed the interesting question here.  It isn't something
I desperately need.  But if file-as-directory _happens_, then I hope
it'll happen in a good way.  No unnecessary complications or restrictions,
and allowing existing stuff to work as well as possible.  There will 
surely be some
tool problems - but we certainly don't need to break every existing
tool the way a new form of "open" will. . .

Helge Hafting

Re: The argument for fs assistance in handling archives

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 597 bytes --]

Salut,

On Fri, Sep 03, 2004 at 10:22:55AM +0200, Helge Hafting wrote:
> Requiring another syscall to open forks other than the primary
> breaks _all_ existing software because it obviously don't use that
> syscall yet.  And then it doesn't provide any advantages over the
> file-as-plain-directory way. . .

Actually...

We might tune the sys_open()  call to take an additional argument (the
stream ID),  and introduce a compatibility interface  into *libc which
chooses strid=0 by default if the plain old open call is being used.

Maybe this can be handled transparently by cpp..

				Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives

[Spam]

  

> Salut,

> On Fri, Sep 03, 2004 at 10:22:55AM +0200, Helge Hafting wrote:
>> Requiring another syscall to open forks other than the primary
>> breaks _all_ existing software because it obviously don't use that
>> syscall yet.  And then it doesn't provide any advantages over the
>> file-as-plain-directory way. . .

> Actually...

> We might tune the sys_open()  call to take an additional argument (the
> stream ID),  and introduce a compatibility interface  into *libc which
> chooses strid=0 by default if the plain old open call is being used.

> Maybe this can be handled transparently by cpp..

  How are things like copy handled today? Is it just some code linked
  in during compilation (ie, the application handles the copy itself)
  or is it a library function in libc? If it is in libc then it ought
  to be enough if libc is patched to support the new semantics with
  file-as-dir etc.

  ~S


> 		Tonnerre

Re: The argument for fs assistance in handling archives

[Helge Hafting]

Tonnerre wrote:

>Salut,
>
>On Fri, Sep 03, 2004 at 10:22:55AM +0200, Helge Hafting wrote:
>  
>
>>Requiring another syscall to open forks other than the primary
>>breaks _all_ existing software because it obviously don't use that
>>syscall yet.  And then it doesn't provide any advantages over the
>>file-as-plain-directory way. . .
>>    
>>
>
>Actually...
>
>We might tune the sys_open()  call to take an additional argument (the
>stream ID),  and introduce a compatibility interface  into *libc which
>chooses strid=0 by default if the plain old open call is being used.
>  
>
But this isn't necessary, as I pointed out above.
There is no _need_ for a new kind of open(), so
why make one in libc?

If you can open a fork/substream/whatever by issuing
open("filename/forkname", ...
then the old-fashioned open() works with multi-fork files too.
An tools based on "open() something, then work with
the resulting file descriptor" will work _unchanged_
with such a multi-fork fs.

Tools that rely on "stat" or directory traversals might need
some updating to work perfectly with such a fs, but not something
that just opens by name.

Helge Hafting

Re: The argument for fs assistance in handling archives

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 713 bytes --]

Salut,

On Wed, Sep 08, 2004 at 11:24:40AM +0200, Helge Hafting wrote:
> If you can open a fork/substream/whatever by issuing
> open("filename/forkname", ...
> then the old-fashioned open() works with multi-fork files too.
> An tools based on "open() something, then work with
> the resulting file descriptor" will work _unchanged_
> with such a multi-fork fs.

In my version they'd run unchanged as well.

And  BTW,  I wasn't  talking  about introducing  a  new  open at  libc
level.  I was  talking about  modifying the  open system  call  in the
kernel and having libc provide compatibility for the old call.

Since  I'm not  sure  how much  breakage it  takes  to make  a file  a
directory.

			    Tonnerre


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives

[Helge Hafting]

Tonnerre wrote:

>Salut,
>
>On Wed, Sep 08, 2004 at 11:24:40AM +0200, Helge Hafting wrote:
>  
>
>>If you can open a fork/substream/whatever by issuing
>>open("filename/forkname", ...
>>then the old-fashioned open() works with multi-fork files too.
>>An tools based on "open() something, then work with
>>the resulting file descriptor" will work _unchanged_
>>with such a multi-fork fs.
>>    
>>
>
>In my version they'd run unchanged as well.
>
>And  BTW,  I wasn't  talking  about introducing  a  new  open at  libc
>level.  I was  talking about  modifying the  open system  call  in the
>kernel and having libc provide compatibility for the old call.
>
>Since  I'm not  sure  how much  breakage it  takes  to make  a file  a
>directory.
>
I believe most of the changes necessary for supporting file-as-dir
will be in the internal workings of filesystems, not so much in
the interfaces.  I don't think it is necessary to change the
open system call, because opening a file in a subdirectory
or a named substream is the same thing.

The _uses_ people have suggested for file-as-dir (complex file attributes
and different views of the data) tends to be different from what
a directory usually does, but the implementation doesn't have to be.
There's no need to add extra interfaces or restrictions.  File-as-dir that
really is a directory, so that it may contain a whole tree is more flexible.
And the underlying fs can use existing directory code instead of
something extra.

So the only change needed for the open system call is that it will
suddenly succeed in the open("filename/forkname") case that
used to complain about an invalid path because "filename"  isn't a 
directory.
"stat" may have to change though, so it can identify a file-as-dir.

Helge Hafting

Re: The argument for fs assistance in handling archives

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 449 bytes --]

Salut,

On Thu, Sep 02, 2004 at 09:12:56PM +1200, Oliver Hunt wrote:
> getNumForks(fileref){ return 1;} 

Actually it might be even cooler  to have a generic function which you
can apply to any file on a file system (or whatever) and which returns
you the  file system's capabilities  (can write, has xattrs,  has acl,
has multiple  streams, *not* has btree  lookups etc, as  that is *not*
interesting to userland anyway.)

				Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives

[Spam]

  

> Linus Torvalds wrote:

>> But _my_ point is, no user program is going to take _advantage_ of
>>
>>anything that only one filesystem on one system offers.
>>  
>>
> Apple does not have this problem....

> and yes, the apps will take advantage of it, which is different from
> depending on it.  If you use the wrong fs you will lose some of the 
> features of the app.

> For 30 years nothing much has happened in Unix filesystem semantics 
> because of sheer cowardice (excepting Clearcase, which priced itself
> into a niche market).   It is 25 years past time for someone to change
> things.  That someone will have first mover advantage, and the more 
> little semantic features possessed the more lure there will be to use it
> which will increase market share which will lure more apps into 
> depending on it and in a few years the other filesystems will 
> (deservedly) have only a small market share because the apps won't all
> work on them.

> Besides, there are enhancements which are simply compelling.  You can
> write a dramatically better performance version control system with a
> much simpler design if the FS is atomic.    Our transaction manager 
> first draft was written by a version control guy, and he would probably
> be happy to tell you how  lack of atomicity other than rename makes 
> version control software design hideous.

  Btw, version control for ordinary files would be a great feature. I
  think something like it is available through Windows 2000/3 server.
  Isn't it called "Shadow Copies". It works over network shares. :)

  It allows you to restore previous versions of the file even if you
  delete or overwrite it.

  Features like this do make a good point and helps protecting data -
  something that is important IMHO.
  
  http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp


> We have the performance lead.  By next year we will be stable enough for
> mission critical servers, and then we start the serious semantic 
> enhancements.

> If you don't embrace progress, then you doom Linux to following behind,
> because the guys at Apple are pretty aggressive now that Jobs is back,
> and they WILL change the semantics, and they will do so in compelling
> ways, and Linux will be reduced to aping them when it should be leading
> them.

> Hans

Re: The argument for fs assistance in handling archives

[Alan Cox]

On Iau, 2004-09-02 at 10:52, Spam wrote:
>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)

Netapps have done this for years with NFS, Subversion does it with DAV,
clearcase does it with NFS.

Re: The argument for fs assistance in handling archives

[V13]

On Thursday 02 September 2004 12:52, Spam wrote:
> > Linus Torvalds wrote:
> > Besides, there are enhancements which are simply compelling.  You can
> > write a dramatically better performance version control system with a
> > much simpler design if the FS is atomic.    Our transaction manager
> > first draft was written by a version control guy, and he would probably
> > be happy to tell you how  lack of atomicity other than rename makes
> > version control software design hideous.
>
>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)
>
>   It allows you to restore previous versions of the file even if you
>   delete or overwrite it.
>
>   Features like this do make a good point and helps protecting data -
>   something that is important IMHO.

I believe you mean something simillar to: 

file1.txt;1
file1.txt;2
file1.txt;3 (yeap, it's VMS) 

where you'll have to cleanup old versions when you don't need them any more... 
AFAIK that this is older than HDDs

<<V13>>

Re: The argument for fs assistance in handling archives

[John Stoffel]

V13> I believe you mean something simillar to: 

V13> file1.txt;1
V13> file1.txt;2
V13> file1.txt;3 (yeap, it's VMS) 

Or TOPS-20, a precursor to VMS in some ways.  It was a nice feature.

V13> where you'll have to cleanup old versions when you don't need
V13> them any more...  AFAIK that this is older than HDDs

It was usually an automatic cleanup past a certain point or if you
went over disk quota.  

In any case, while I do like this feature, I'm not sure how we would
cleanly implement this inside the unix namespace, or if inside a new
namespace, how that new namespace would be joined with a standard Unix
one.

John

Re: The argument for fs assistance in handling archives

[Oliver Neukum]

Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)
> 
>   It allows you to restore previous versions of the file even if you
>   delete or overwrite it.

There's no need to do that in kernel, unless you want to be able
to force it unto users.

	Regards
		Oliver

Re: The argument for fs assistance in handling archives

[Spam]

  

> Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
>>   Btw, version control for ordinary files would be a great feature. I
>>   think something like it is available through Windows 2000/3 server.
>>   Isn't it called "Shadow Copies". It works over network shares. :)
>> 
>>   It allows you to restore previous versions of the file even if you
>>   delete or overwrite it.

> There's no need to do that in kernel, unless you want to be able
> to force it unto users.

  Exactly ;)

  Difference with having it in just certain applications like Gnome-VFS
  etc is that it would work with all applications, over SMB shares
  etc. This is a great advantage and benefit for many desktop users
  but also in a corporate environment. Even though there are backups
  of users documents it is a pain to restore them every time they
  loose the original file. Besides, the backup is never as fresh
  either.
  

> 	Regards
> 		Oliver

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Oliver Neukum wrote:
| Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
|
|>  Btw, version control for ordinary files would be a great feature. I
|>  think something like it is available through Windows 2000/3 server.
|>  Isn't it called "Shadow Copies". It works over network shares. :)
|>
|>  It allows you to restore previous versions of the file even if you
|>  delete or overwrite it.
|
|
| There's no need to do that in kernel, unless you want to be able
| to force it unto users.

And on apps.  Should I teach OpenOffice.org to do version control?
Seems a lot easier to just do it in the kernel, and teach everything to
do version control in one fell swoop.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTe+NXgHNmZLgCUhAQLW+RAAnoQIQLXSd2vQudOKAaLgXXCxoa+gEWIq
zcNd+bIWPuQxwcZYlqbs/pDFiDky3XqLRaZv8PZx++rs9mMnv36eAtMX7uUp3egz
MxymeDDI796wtsDDBGcZtO+WwmqJ0fPgQMN/Q237s2yccPGJoDIVle9mzZAfNIdO
RDdLq142D+0t1iqfiVqaA+NgkArrGhcr0Hs/3TdviJX30gPL/Jm9eOvc3H3NgIQu
XToDQB1UnLoyiIwc16IY4ZxYoxf1YnFtrivZb+YgC61mIQLeqJMjixCAtBVzAfKN
2GmlsnfWcMCgXSEX8imkoNSLcmkv886+esGXZVzBbY1/Qg/f5MCXZilbb8e6+4I9
U4ReEUg5fGJW8JyKAXqKGyvbC/lim55Vgcjp/J8mrhUa3Q3cwPKoi2GKNRcbdpT/
pPB2SADA1eNrmFyGB7kLcqt0F9i4fIyTrItpHq9+m72OdFEze2OnWn+eIY8dRut7
SaGTq6ZEYtkH8oWT9xSXp1d2TUxeSVTdx8EffIIdwXAVOO0Z9xd71MJPLMp3VOMH
Qld4gjYjKFzwj3QlcWcsiVhk6Zq2fmqoOCQNfxhBzH1Le3v0F0ugammkQUXtRAub
dKEWxGfkDGOwviJXKeckDNn98x88vh7C/7gxpAOIcVSRDhEV+MDWIfMMvotE/lxe
393hENsXOgQ=
=g+3J
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Spam]

  

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1

> Oliver Neukum wrote:
> | Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
> |
|>>  Btw, version control for ordinary files would be a great feature. I
|>>  think something like it is available through Windows 2000/3 server.
|>>  Isn't it called "Shadow Copies". It works over network shares. :)
|>>
|>>  It allows you to restore previous versions of the file even if you
|>>  delete or overwrite it.
> |
> |
> | There's no need to do that in kernel, unless you want to be able
> | to force it unto users.

> And on apps.  Should I teach OpenOffice.org to do version control?
> Seems a lot easier to just do it in the kernel, and teach everything to
> do version control in one fell swoop.

  Do you mean in the kernel or as a filesystem/VFS plugin that would
  extend the functionality to include version control?

  ~S

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Spam wrote:
[...]
|   Do you mean in the kernel or as a filesystem/VFS plugin that would
|   extend the functionality to include version control?

The filesystem plugin.  But this needs some kernel support, which scares
people.  Read the archives if you don't believe me.

Besides, we'd want per-block copy-on-write support anyway to make it
sane.  That has to go in the filesystem.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTfKhXgHNmZLgCUhAQIR5w//Z6hg58uVvIJuwNCb764lFj22iHwLvg0T
JvjQeOSCz7p2/uv+dHN2MRNDeezmvGfH49Y7RE78Q9e1xrfZFGZKyybHVisfea1t
f9Bk8M6ZtykKnUdPhq5MsDH9Oo98dXMNEWhDw5WUYMzvglitMUcd4np3gGEtr539
yKHuxZhqjc/lQ9rd6PFKJYab1m9RAabQnGMHRIvx0T98iW3EI9pyOd0u1oUVCiRi
RxllyP0KCg7wxFv93b4zYHsv5BO8E7MWVI4aKuBUCqrc8ObJnDLDbO/8FfMvDt/1
Do9nlikFh4IaRvwZtmQA0nVv80uxtjnSOCs7F2r+qqNyprcHIdj2m+3x5zE8Tbfz
YCkI5BopHub94yBvhlcFGiKBtigLDsH3zHc66dVDl4SreJPKo+yirjjZ+RrU3HN1
iK+aFp5GxapcYDkMNOS0ab9TPejqLVyJ+Ci5mSNckcMARC9v+2mT7JTCBupyXZar
LSLFYef+W7ipKq5ZMurDB17I5y7VXfwOTd+oSwQdMwlsQlJs/AFOOGdJywtLGLy1
6jhejQlziPLWB9hz9wFAjWyHPrS0XsCNbtGbBESIVmqvLsZmpcAULpMn7TGc8aJg
0r79ODKlGK8lYJRIquRhZzd51oqiBfFJcX969XGPMPUBX4x3GZjaOlj79OQ7Ih3s
Qd8PSfUVzGU=
=Cw9B
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 1157 bytes --]

On Thu, 02 Sep 2004 19:43:34 CDT, David Masover said:

> And on apps.  Should I teach OpenOffice.org to do version control?
> Seems a lot easier to just do it in the kernel, and teach everything to
> do version control in one fell swoop.

Including files you didn't really want to keep version control of?

How many temp files does gcc create and unlink in the course of a kernel build?
(And remember, you can't say "don't enable that on /tmp" - gcc respects the
setting of $TMPDIR - so an 'export TMPDIR=~/tmp' confuses things quite
nicely...)

And it's hard for the kernel to know that an unlink() done by gcc should be
treated differently than the "recover the last version" you *want* it do be able
to do after you work on a source file for a long while, save it, and then
fumble-finger a 'rm * .o' - you can't even use a heuristic like "don't version
control it unless it's N seconds or more old"

(Note that the "obvious" solution of creating a chattr flag has its own
complexity issues - should versioning be turned on by default for some types
and not others, etc...)

There be dragons here - it's not as simple as "drop in a plugin and be happy".


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Valdis.Kletnieks@vt.edu wrote:
| On Thu, 02 Sep 2004 19:43:34 CDT, David Masover said:
|
|
|>And on apps.  Should I teach OpenOffice.org to do version control?
|>Seems a lot easier to just do it in the kernel, and teach everything to
|>do version control in one fell swoop.
|
|
| Including files you didn't really want to keep version control of?

No one said you have to apply it to everything.  Only that you can.
Right now vim cannot do version control, AFAIK.  But you could do
something like:
	echo 1 > foo/version_control
	vim foo

Default would be off, of course.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTfLR3gHNmZLgCUhAQKybRAAkF4sqLUCJEfpDNvKEyv8MJGcy0w+qV12
3Rf50OQeeurZMZrnli0vNIuSIWwIsGI+j7qbj1mlfi+Ps3dAjOlKXav3tGoGshLl
HH/Wl9XvGPOAWlIQOqqBGAyno8sDIvHAjh5lH3C7m+9/2Ao7s1G/bQ+hm/deYPJK
WPZxExd/1BdchrbjNjbfWHKrD6LdF/GBa/Vbj1F8m95WEvxsfbDnSTwb5HhvcUxD
TtqwkINmx7Tle2p9q7PcgR7y76dmcyoWw82ST3BfF/AbGauMQAr0h6jNjvzclfgJ
yrIOwGXdR3pn8ZK8dHVct79N0f+PYFdeAzLYxMZR2vefB2enQolbbs6Lf7X60g2g
5hrH/ezenFJIZMPhQtlJqBaFuA5GHw9B2An6VbOmDm7sSOFyHSlKoCUFlJ/I5nZd
Bcn46lhNxvTJhu3tvnvDqTB8f0/yVYt5QfoBFA1mzHKig2iNN6vr2xLGdWSLU6XD
vRjy+KyVC+PvjL13E0JrjXy9UuqY7CH2xAunSgLJ2cvfRmNRoVoV1hkp3McQcUqF
GiD3s0REcztlvsrP+DxKLRtPpnOJi5By5NmZpJkiPdVB9zpIsL/Ia7KL/A7IGGh3
IF3t/xSUmjgrek01SeadHqsIngeWQ+F8fiars7NtzNhrvRgPVfDVheey3vhQjFxM
3kDKH+I9rGk=
=3+nq
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Christoph Hellwig]

On Thu, Sep 02, 2004 at 07:43:34PM -0500, David Masover wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Oliver Neukum wrote:
> | Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
> |
> |>  Btw, version control for ordinary files would be a great feature. I
> |>  think something like it is available through Windows 2000/3 server.
> |>  Isn't it called "Shadow Copies". It works over network shares. :)
> |>
> |>  It allows you to restore previous versions of the file even if you
> |>  delete or overwrite it.
> |
> |
> | There's no need to do that in kernel, unless you want to be able
> | to force it unto users.
> 
> And on apps.  Should I teach OpenOffice.org to do version control?
> Seems a lot easier to just do it in the kernel, and teach everything to
> do version control in one fell swoop.

Just add a post-save trigger that can check it into any SCM you want.

Re: The argument for fs assistance in handling archives

[Oliver Neukum]

Am Freitag, 3. September 2004 10:13 schrieb Christoph Hellwig:
> On Thu, Sep 02, 2004 at 07:43:34PM -0500, David Masover wrote:

> > | There's no need to do that in kernel, unless you want to be able
> > | to force it unto users.
> > 
> > And on apps.  Should I teach OpenOffice.org to do version control?
> > Seems a lot easier to just do it in the kernel, and teach everything to
> > do version control in one fell swoop.
> 
> Just add a post-save trigger that can check it into any SCM you want.

That depends on whom you refer to. If you want to impose the RCS
on the users as an administrative measure and force compliance, then
you'll need to do it in kernel.
I see some nasty issues with disk quotas there.

	Regards
		Oliver

Re: The argument for fs assistance in handling archives

[Horst von Brand]

Spam <spam@tnonline.net> said:

[...]

>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)

Right. And when space runs low, the old copies get the axe. And then you
need the old version... and are royally screwed because you trusted the
system to save it for you while willy-nilly fooling around.

I never liked the MS-DOS "undelete" feature, because I saw too many people
burned. It is _less_ data security than "deleted file is gone for good", in
the end sum.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: The argument for fs assistance in handling archives

[Lee Revell]

On Thu, 2004-09-02 at 05:52, Spam wrote:

>   Btw, version control for ordinary files would be a great feature. I
>   think something like it is available through Windows 2000/3 server.
>   Isn't it called "Shadow Copies". It works over network shares. :)
> 
>   It allows you to restore previous versions of the file even if you
>   delete or overwrite it.
> 
>   Features like this do make a good point and helps protecting data -
>   something that is important IMHO.
>   
>   http://www.microsoft.com/resources/documentation/windowsserv/2003/standard/proddocs/en-us/overview_snapshot.asp
> 

Netapp's WAFL has had this for ages, they are called snapshots, and they
are one of the most useful file system features ever.  But Microsoft
certainly did not invent them.

Lee

Re: The argument for fs assistance in handling archives

[Helge Hafting]

Spam wrote:

>
>  Btw, version control for ordinary files would be a great feature. I
>  think something like it is available through Windows 2000/3 server.
>  Isn't it called "Shadow Copies". It works over network shares. :)
>
>  It allows you to restore previous versions of the file even if you
>  delete or overwrite it.
>  
>
Feel free to use cvs on your home directory. :-)

Helge Hafting

Re: The argument for fs assistance in handling archives

[John Stoffel]

Hans> For 30 years nothing much has happened in Unix filesystem
Hans> semantics because of sheer cowardice (excepting Clearcase, which
Hans> priced itself into a niche market).

Keeping a consistent namespace is cowardice?  And ClearCase is an
example of an application which sets up it's own namespace and grafts
it onto the standard Unix one.  Most standard Unix tools work just
fine inside clearcase Views, but to manage the metadata (xattrs, or
whatever you want to call them), you need to use the 'cleartool'
command.  Hmmm... sounds like 'runat' to me.

Hans> It is 25 years past time for someone to change things.  That
Hans> someone will have first mover advantage, and the more little
Hans> semantic features possessed the more lure there will be to use
Hans> it which will increase market share which will lure more apps
Hans> into depending on it and in a few years the other filesystems
Hans> will (deservedly) have only a small market share because the
Hans> apps won't all work on them.

This is all pure marketing speak and economic theory.  Show us the
*technical* advantages, not just wishful thinking.  

Hans> Besides, there are enhancements which are simply compelling.
Hans> You can write a dramatically better performance version control
Hans> system with a much simpler design if the FS is atomic. 

Define atomic please, with state diagrams and clear examples.  

Hans> We have the performance lead.  By next year we will be stable
Hans> enough for mission critical servers, and then we start the
Hans> serious semantic enhancements.

I think you've got it backwards.  Make your serious semantic
enhancements first, then make it stable, then make it fast.  Because
when you change the semantics, you break all kinds of things and then
it doesn't matter how fast you are.

Hans> If you don't embrace progress, then you doom Linux to following
Hans> behind, because the guys at Apple are pretty aggressive now that
Hans> Jobs is back, and they WILL change the semantics, and they will
Hans> do so in compelling ways, and Linux will be reduced to aping
Hans> them when it should be leading them.

Monkey see, Monkey do then.  


I'd like to point out another successful company which has extended
the standard Unix namespace and that's 'Network Appliance' with it's
.snapshot directory structure.  It's a great idea and allows my users
to restore files from snapshots without me having to think about it.  

But it still causes problems since when I use rsync to move data, I
need to put in stuff like:

  rsync -az --exclude ".snapshot" --delete --delete-excluded <src> <dest>

to make sure it doesn't descend into that directory of previous
versions and try to copy them over as well.  And of course now now
other apps can use the .snapshot name.  But what if other vendors aped
(ook ook) this decided to use their own names?  Who decides which gets
priority?  

I think you need to go back and re-read Pike's paper on namespaces
that you pointed to before and mull it over.  And look at how
simplicity is inherently powerful.  If the design is too complicated,
you're probably doing it wrong.

John

Re: The argument for fs assistance in handling archives

[Alan Cox]

On Iau, 2004-09-02 at 15:24, John Stoffel wrote:
> This is all pure marketing speak and economic theory. 

I wouldn't grace it with the label economic theory. Lets be honest about
this.. its pure -research-. Research is good but don't screw around with
Linux semantics until you've done the research, figured out how to also
make it useful to real users and proved its a good idea despite the past
evidence from things like MacOS


Alan

Re: The argument for fs assistance in handling archives

[Horst von Brand]

Hans Reiser <reiser@namesys.com> said:
> Linus Torvalds wrote:

> > But _my_ point is, no user program is going to take _advantage_ of
> >anything that only one filesystem on one system offers.

> Apple does not have this problem....

... because Apple customers are a captive lot...

> and yes, the apps will take advantage of it, which is different from 
> depending on it.  If you use the wrong fs you will lose some of the 
> features of the app.

... which is undesirable to say the very least for cross-platform Unix
(even worse, Unix + Windows) applications, so applications won't ever use
it (until Linux + super-duper filesystem is more than, say, 80% of the
market). I.e., it won't happen soon enough for this to make sense right
now. Perhaps in 13 more years...

> For 30 years nothing much has happened in Unix filesystem semantics 
> because of sheer cowardice

Careful threading there... calling respected people names without any proof
isn't exactly great PR.

>                            (excepting Clearcase, which priced itself 
> into a niche market).  It is 25 years past time for someone to change
> things.

Your opinion. Some people disagree.

>          That someone will have first mover advantage, and the more
> little semantic features possessed the more lure there will be to use it
> which will increase market share which will lure more apps into depending
> on it and in a few years the other filesystems will (deservedly) have
> only a small market share because the apps won't all work on them.

You seem to be blinded by the "success effect": You only ever see the ones
who succeed, you don't see the masses who tried to be "first movers" and
failed utterly. 

> Besides, there are enhancements which are simply compelling.  You can 
> write a dramatically better performance version control system with a 
> much simpler design if the FS is atomic.

Iff the performance of the FS is better _at doing what the system wants
than the system itself_. That the CVS is simpler/faster is no use
whatsoever if the underlying OS is more complex/slower. Besides, what is
ideal for CVS isn't necesarily good for word processing or databases, and
probably irrelevant or at least overkill for a lot of other applications,
which all end up paying the costs.

Concentrate on giving what _most_ applications want, make it possible for
applications with weird requirements to do their job.

>                                             Our transaction manager 
> first draft was written by a version control guy, and he would probably 
> be happy to tell you how  lack of atomicity other than rename makes 
> version control software design hideous.

Right. But it doesn't make the design of text editing, or word processing,
or mail handling, or... hideous, does it? Having a strong background in an
area does put blinds on one's eyes.

> We have the performance lead.

I recently saw (preliminary) numbers directly contradicting this somewhere.

>                                By next year we will be stable enough for 
> mission critical servers, and then we start the serious semantic 
> enhancements.

By which the performance lead (if any) will do down the drain. There ain't
no free lunch.

> If you don't embrace progress, then you doom Linux to following behind, 
> because the guys at Apple are pretty aggressive now that Jobs is back, 
> and they WILL change the semantics, and they will do so in compelling 
> ways, and Linux will be reduced to aping them when it should be leading 
> them.

Why on earth should Linux copy a niche system? By now Linux's marketshare
is _much_ larger than Apple's... I'd say it is the other way around: Apple
tries desperately to differentiate itself from MSFT (and traditional Unix +
Linux) in order not to dissapear (and keep their grip on their users). If
they are doing it right, only time will tell. But that is no reason to
screw up Linux' strengths.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: The argument for fs assistance in handling archives

[Tonnerre]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 878 bytes --]

Salut,

On Thu, Sep 02, 2004 at 11:32:08AM -0400, Horst von Brand wrote:
> Hans Reiser <reiser@namesys.com> said:
> > Linus Torvalds wrote:
> 
> > > But _my_ point is, no user program is going to take _advantage_ of
> > >anything that only one filesystem on one system offers.
> 
> > Apple does not have this problem....
> 
> ... because Apple customers are a captive lot...

ça suffit!  Stop  that damn evangelization. I've seen  lots of content
MacOS/X users. Personally  I use MacOS/X, NetBSD and  Linux to develop
my software, and  I quite like all the  three. I'm actively developing
Linux and MacOS/X. Why I do that? Because I'm a technician.

Linux users are no  better people whatsoever.  Restricting yourself to
one OS is  bad. I think most  of the FS developers here  will agree on
that.

				Tonnerre
		(who's a technician, not an evangelist)

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives

[Linus Torvalds]

On Thu, 2 Sep 2004, Hans Reiser wrote:
>
> Apple does not have this problem....

Deep, Hans. Deep.

You're ignoring the issue. You're flaming about not "embracing progress", 
when I was talking about how progress happens. You don't seem to 
understand _how_ people embrace progress, Hans. Railing at windmills isn't 
real smart.

		Linus

Re: The argument for fs assistance in handling archives

[Martin J. Bligh]

> For 30 years nothing much has happened in Unix filesystem semantics 
> because of sheer cowardice 

Or because it works fine, and isn't broken.

M.

Re: The argument for fs assistance in handling archives

[Christer Weinigel]

"Martin J. Bligh" <mbligh@aracnet.com> writes:

> > For 30 years nothing much has happened in Unix filesystem semantics 
> > because of sheer cowardice 
> 
> Or because it works fine, and isn't broken.

I've heard the same argument a lot of times.  People complaining that
Unix is so seventies because it sticks to the old boring philosophy of
everything is a file and that a file is a stream of bytes, nothing
more.  Modern operating systems such as VMS with basic database
handling in the OS itself, or MacOS or NT with named streams is so
much more modern.  Why don't we get with the times?

It may be because just because of the simplicity it's fairly easy to
use, harder to break och does one thing well.  If you want structured
storage, use a database, on top of the low level primitives, or use
multiple files in a directory.  Why complicate things?

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: The argument for fs assistance in handling archives

[Brian Beattie]

On Thu, 2004-09-02 at 14:38, Christer Weinigel wrote:
> "Martin J. Bligh" <mbligh@aracnet.com> writes:
> 
> > > For 30 years nothing much has happened in Unix filesystem semantics 
> > > because of sheer cowardice 
> > 
> > Or because it works fine, and isn't broken.
> 
> I've heard the same argument a lot of times.  People complaining that
> Unix is so seventies because it sticks to the old boring philosophy of
> everything is a file and that a file is a stream of bytes, nothing
> more.  Modern operating systems such as VMS with basic database
> handling in the OS itself, or MacOS or NT with named streams is so
> much more modern.  Why don't we get with the times?

Actually the handleing of special filesystem features in the OS is an
OLD idea.  VMS is actually not more modern than Unix, it is older than
Unix owing much to earlier OS's such as RSX-11.  It was a great insight
on the part of Ritche and Thompson that the kernel should present a file
as a byte stream and that the interpretation of the contents of a file
is the province of the application, not the operating system.

Now it maybe, that there are features that can only be provided by
adding support in the kernel, but we should be very careful not to be
distracted by what we see as new wizbang features.  The filesystem as a
database has been tried before, ISAM is a very old concept.

> 
> It may be because just because of the simplicity it's fairly easy to
> use, harder to break och does one thing well.  If you want structured
> storage, use a database, on top of the low level primitives, or use
> multiple files in a directory.  Why complicate things?
> 
>   /Christer
-- 
Brian Beattie   LFS12947 | "Honor isn't about making the right choices.
beattie@beattie-home.net | It's about dealing with the consequences."
www.beattie-home.net     | -- Midori Koto

Re: The argument for fs assistance in handling archives

[Steve Bergman]

On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote:
> > For 30 years nothing much has happened in Unix filesystem semantics 
> > because of sheer cowardice 
> 
> Or because it works fine, and isn't broken.

OK.  I'm not a kernel hacker.  I'm not a crack C programmer.  Nor am I a
designer of filesystems.  I'm just a guy that puts together and supports
Linux systems for my customers.

In following this thread, I may be missing huge chunks of concept.

However, a few things are becoming clear to me:

1. The file as directory thing adds complexity that the administrator
has to deal with.  Symlinks are useful, but it's still aggravating to
tar off a directory structure, take it somewhere, and then realize that
all you have is links to something not in the archive because you didn't
get your tar switches just right.  Now we're talking about adding
another set of "files which are not really files" to the semantics.
More complexity.  I'll take simplicity over some ivory tower ideal of
"unified name space" any day.

2. The use of multiple streams within files by Linux apps would make
Linux as cross-platform unfriendly as MS is trying to be.  Say these
features start getting used and you copy an OO.org document from a Linux
box to a BSD box.  It's broken.  Of course, OO.org wouldn't use the
streams in the first place because it would destroy their cross platform
portability.  So what's the point?  No one who cares about cross
platform portability can use it.  Everyone who doesn't care about cross
platform portability please raise your hand.

3. MS does require attributes and multiple streams, which makes these
features important (even essential) to Samba, and Samba alone.  Samba is
important to Linux, so this can't be ignored. (Here I am implicitly
assuming that Samba will need kernel support for this to do it right.)

So it seems to me that the only real consideration is giving Samba what
it needs without making the semantics one bit more complex than
absolutely necessary.  It might even be wise to discourage use of these
ambiguous new objects by the casual application programmer.

Then again, maybe I just have tunnel vision... 


-Steve Bergman

Re: The argument for fs assistance in handling archives

[Spam]

  

> On Thu, 2004-09-02 at 11:22 -0700, Martin J. Bligh wrote:
>> > For 30 years nothing much has happened in Unix filesystem semantics
>> > because of sheer cowardice 
>> 
>> Or because it works fine, and isn't broken.

> OK.  I'm not a kernel hacker.  I'm not a crack C programmer.  Nor am I a
> designer of filesystems.  I'm just a guy that puts together and supports
> Linux systems for my customers.

> In following this thread, I may be missing huge chunks of concept.

> However, a few things are becoming clear to me:

> 1. The file as directory thing adds complexity that the administrator
> has to deal with.  Symlinks are useful, but it's still aggravating to
> tar off a directory structure, take it somewhere, and then realize that
> all you have is links to something not in the archive because you didn't
> get your tar switches just right.  Now we're talking about adding
> another set of "files which are not really files" to the semantics.
> More complexity.  I'll take simplicity over some ivory tower ideal of
> "unified name space" any day.

  Are you afraid to learn something new? ;) Just joking. But really,
  it doesn't have to be very difficult. The extra streams etc would
  just be saved as files. If tar is patched then it would be no
  problem and no extra stuff but perhaps a switch --save-metas.

> 2. The use of multiple streams within files by Linux apps would make
> Linux as cross-platform unfriendly as MS is trying to be.  Say these
> features start getting used and you copy an OO.org document from a Linux
> box to a BSD box.  It's broken.  Of course, OO.org wouldn't use the
> streams in the first place because it would destroy their cross platform
> portability.  So what's the point?  No one who cares about cross
> platform portability can use it.  Everyone who doesn't care about cross
> platform portability please raise your hand.

  Actually MacOS and Windows support multiple streams, only Linux
  doesn't. But of course there are BSD's etc too. I'd say to leave
  them behind.

  The file streams would make my day a lot easier. The idea to
  split up contents of OO.org files into streams is bad. But that
  doesn't make the file streams bad. I see many uses that would make
  my every day life easier.

  It isn't about cross plat form compatibility, but to add features
  that are useful and meta-data and file streams are.

  Also. No one forces you to use either meta-data or streams, just as
  no one forces you to use ACLs or other things.

> 3. MS does require attributes and multiple streams, which makes these
> features important (even essential) to Samba, and Samba alone.  Samba is
> important to Linux, so this can't be ignored. (Here I am implicitly
> assuming that Samba will need kernel support for this to do it right.)

  I do not think the Samba would really require the streams support
  but it would certainly make life easier for Samba. Not to mention
  that these files would also be natively viewed on the Linux host.

> So it seems to me that the only real consideration is giving Samba what
> it needs without making the semantics one bit more complex than
> absolutely necessary.  It might even be wise to discourage use of these
> ambiguous new objects by the casual application programmer.

> Then again, maybe I just have tunnel vision... 

  I would agree with tunnel vision. The kernel should provide the
  tools and options. Users and developers can then invent new things
  to use them. :)

  ~S

> -Steve Bergman

Re: The argument for fs assistance in handling archives

[Martin J. Bligh]

>> 1. The file as directory thing adds complexity that the administrator
>> has to deal with.  Symlinks are useful, but it's still aggravating to
>> tar off a directory structure, take it somewhere, and then realize that
>> all you have is links to something not in the archive because you didn't
>> get your tar switches just right.  Now we're talking about adding
>> another set of "files which are not really files" to the semantics.
>> More complexity.  I'll take simplicity over some ivory tower ideal of
>> "unified name space" any day.
> 
>   Are you afraid to learn something new? ;) Just joking. But really,
>   it doesn't have to be very difficult. The extra streams etc would
>   just be saved as files. If tar is patched then it would be no
>   problem and no extra stuff but perhaps a switch --save-metas.

If they're saved as files, and the app has to be changed to use them
anyway, then what's the point? Just change the app to use new files
instead.

>   I would agree with tunnel vision. The kernel should provide the
>   tools and options. Users and developers can then invent new things
>   to use them. :)

Ugh. Change for changes sake is not a good thing. There's enough real
problems in the world without inventing random features. More complexity
without gain is a Bad Thing (tm).

I'm not saying streams is bad ... just that there don't seem to have
been very many convincing (to me) arguments raised for it yet. The 
versioning stuff would be nice, IMHO, because the stuff mainly using it 
wouldn't need to be modified in many instances ... "vi /etc/configfile" 
would keep old copies for you (only the recovery tool would need to
understand it). Saving icons seems easy enough to do with "foo.icon"
as another file.

M.

Re: The argument for fs assistance in handling archives

[Hacksaw]

>If tar is patched then it would be no
>problem and no extra stuff but perhaps a switch --save-metas.

Aaargh. How about a tar that recognizes when the filesystem has metas and 
saves/restores them by default, with a warning when it's losings data because 
it's restoring to a simpler filesystem?

And maybe a switch --forget-metas.

Thousands of sys-admins will thank you to not make it easier to lose data with 
the scripts they have already written.


-- 
You are in a maze of twisty passages, all alike. Again. 
http://www.hacksaw.org -- http://www.privatecircus.com -- KB1FVD

Re: The argument for fs assistance in handling archives

[Bernd Petrovitsch]

On Thu, 2004-09-02 at 22:06 +0200, Spam wrote:
[...]
>   The file streams would make my day a lot easier. The idea to
>   split up contents of OO.org files into streams is bad. But that
>   doesn't make the file streams bad. I see many uses that would make
>   my every day life easier.

When can we expect patches?

>   It isn't about cross plat form compatibility, but to add features
>   that are useful and meta-data and file streams are.

Putting everything (including web server and browser) in the kernel is
not a good (or even desireable) target - perhaps that's the reason for
Win* to be where there are.
And it *is* done in user-space (ok, there more than one apprach and
implementation but this may change. And you want to invent the next
version.).

>   Also. No one forces you to use either meta-data or streams, just as
>   no one forces you to use ACLs or other things.

Which is probably an argument against such features. How long do ACLs
exist - as well as standard and in implementation?
How many people and/or apps are using it?
No more questions.

> > 3. MS does require attributes and multiple streams, which makes these
> > features important (even essential) to Samba, and Samba alone.  Samba is
> > important to Linux, so this can't be ignored. (Here I am implicitly
> > assuming that Samba will need kernel support for this to do it right.)
> 
>   I do not think the Samba would really require the streams support
>   but it would certainly make life easier for Samba. Not to mention
>   that these files would also be natively viewed on the Linux host.

It is not in the interest of MSFT that their aplication's files are
viewable on other OSes. Expect incompatible changes to formats, streams
and metadata contents (and surely patents on them - even if the idea is
not new) if it starts to be really useful. It happended already in the
past with the SMB protocol IIRC ....

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: The argument for fs assistance in handling archives

[Spam]

  

> On Thu, 2004-09-02 at 22:06 +0200, Spam wrote:
> [...]
>>   The file streams would make my day a lot easier. The idea to
>>   split up contents of OO.org files into streams is bad. But that
>>   doesn't make the file streams bad. I see many uses that would make
>>   my every day life easier.

> When can we expect patches?

>>   It isn't about cross plat form compatibility, but to add features
>>   that are useful and meta-data and file streams are.

> Putting everything (including web server and browser) in the kernel is
> not a good (or even desireable) target - perhaps that's the reason for
> Win* to be where there are.
> And it *is* done in user-space (ok, there more than one apprach and
> implementation but this may change. And you want to invent the next
> version.).

  As I said before. It doesn't matter if it is in the kernel or user
  space. Not to me at least - as long as I could use extend the usage
  of my filesystem with meta-data, my own descriptions, perhaps
  streams, etc in a transparent way. I do not want to be confined to
  only use future Gnome apps.

>>   Also. No one forces you to use either meta-data or streams, just as
>>   no one forces you to use ACLs or other things.

> Which is probably an argument against such features. How long do ACLs
> exist - as well as standard and in implementation?
> How many people and/or apps are using it?
> No more questions.

>> > 3. MS does require attributes and multiple streams, which makes these
>> > features important (even essential) to Samba, and Samba alone.  Samba is
>> > important to Linux, so this can't be ignored. (Here I am implicitly
>> > assuming that Samba will need kernel support for this to do it right.)
>> 
>>   I do not think the Samba would really require the streams support
>>   but it would certainly make life easier for Samba. Not to mention
>>   that these files would also be natively viewed on the Linux host.

> It is not in the interest of MSFT that their aplication's files are
> viewable on other OSes. Expect incompatible changes to formats, streams
> and metadata contents (and surely patents on them - even if the idea is
> not new) if it starts to be really useful. It happended already in the
> past with the SMB protocol IIRC ....

  No perhaps. But it doesn't make a difference to the ideas of having
  plugins for reiser4 that can give such features.

  ~S

> 	Bernd

Re: The argument for fs assistance in handling archives

[Erik Hensema]

Steve Bergman (steve@rueb.com) wrote:
> However, a few things are becoming clear to me:
>
> 1. The file as directory thing adds complexity that the administrator
> has to deal with.  Symlinks are useful, but it's still aggravating to
> tar off a directory structure, take it somewhere, and then realize that
> all you have is links to something not in the archive because you didn't
> get your tar switches just right.  Now we're talking about adding
> another set of "files which are not really files" to the semantics.
> More complexity.  I'll take simplicity over some ivory tower ideal of
> "unified name space" any day.

As with any addition of complexity we have to ask ourselves: do
the advantages outweigh the disadvantages?

Adding complexity into the kernel may simplify userspace. Adding
the complexity of an IP layer into the kernel simplifies
networking applications, for example (they won't have to talk IP
to the kernel).

In this case it's hard to imagine what advantages a multistream
file will bring, simply because we aren't used to working with
them. Or maybe there aren't any advantages at all. Point is: we
don't know yet. Maybe we have to toy with the idea for some years
before we really learn why multistream files are so great.

I do realize that this isn't how Linux development is done. First
we need a problem, then somebody sits down and codes a solution.
With reiser4 we've got a solution but we haven't figured out what
the problem is yet.

> 2. The use of multiple streams within files by Linux apps would make
> Linux as cross-platform unfriendly as MS is trying to be.  Say these
> features start getting used and you copy an OO.org document from a Linux
> box to a BSD box.  It's broken.  Of course, OO.org wouldn't use the
> streams in the first place because it would destroy their cross platform
> portability.  So what's the point?  No one who cares about cross
> platform portability can use it.  Everyone who doesn't care about cross
> platform portability please raise your hand.

But do we really want to be the lowest common denominator? If you
want to lead, you can't be compatible (in some cases).
Also, if the reiser4 ideas catch on, they may spark the
development of other, similar, filesystems. Maybe we'll end up
with a ext4 in the 2.8 kernel...

> 3. MS does require attributes and multiple streams, which makes these
> features important (even essential) to Samba, and Samba alone.  Samba is
> important to Linux, so this can't be ignored. (Here I am implicitly
> assuming that Samba will need kernel support for this to do it right.)
>
> So it seems to me that the only real consideration is giving Samba what
> it needs without making the semantics one bit more complex than
> absolutely necessary.  It might even be wise to discourage use of these
> ambiguous new objects by the casual application programmer.
>
> Then again, maybe I just have tunnel vision... 

Not tunnel vision. You're just looking at what we need *now*, and
that happens to be exactly what we *have* now.

One possible use for multistream files that I see, is storage of
'not so important' data alongside of the real data of a file.
Think: the mimetype of a file; some metadata like author,
keywords, etc.

When you copy the file and lose all metadata except the file
itself, no real harm is done.

This could be used by the new mime system Gnome uses, for
example. On regular filesystems, the mimetype of a file is
determined by its extension. On reiser4(-like filesystems), the
mimetype is determined by the 'mimetype' stream.
This would clearly simplify the gnome userspace, as there is no
need anymore for a seperate extension<->mimetype conversion
anymore. It would also be compatible with other tools or desktop
environements, we only have to agree on what stream we're going
to use for what metadata.

IMHO reiser4 is a toy we should give the Linux users to play
with. Let's see what they can come up with. With the talented
hackers in the community, it should be great.

-- 
Erik Hensema <erik@hensema.net>
(attempting to write english while drinking beer)

Re: The argument for fs assistance in handling archives

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

Salut,

On Thu, Sep 02, 2004 at 08:35:58PM +0000, Erik Hensema wrote:
> In this case it's hard to imagine what advantages a multistream file
> will bring, simply  because we aren't used to  working with them. Or
> maybe there  aren't any advantages at  all. Point is:  we don't know
> yet. Maybe  we have to  toy with the  idea for some years  before we
> really learn why multistream files are so great.

Thus I recommend doing the same thing everything does to get his stuff
implemented.

How did User Mode Linux get accepted into the kernel?

In the beginning,  people couldn't imagine any advantage  of it. Then,
someone  started coding  it and  provided patches.   People  tried it,
improved it (important!  It got  to work on many platforms while being
out of the main tree!) and some day we noticed that it was really cool
for kernel hacking and virtualization. 

So when it approached a mature level where we finally saw it was cool,
it went into the mainline kernel, where it is now.

For multistream files, what is that development cycle going to be? Get
the idea, flame, flame, flame... ?

Code talks, bullshit walks.

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Steve Bergman wrote:
[...]
| In following this thread, I may be missing huge chunks of concept.

You are, but don't worry.  You aren't the only one.  (Before I get
assaulted for this, let me say that there are people on this list
smarter than me and I miss concept chunks quite frequently.)

| 1. The file as directory thing adds complexity that the administrator
| has to deal with.  Symlinks are useful, but it's still aggravating to
| tar off a directory structure, take it somewhere, and then realize that
| all you have is links to something not in the archive because you didn't
| get your tar switches just right.  Now we're talking about adding
| another set of "files which are not really files" to the semantics.
| More complexity.  I'll take simplicity over some ivory tower ideal of
| "unified name space" any day.

You want simplicity?  How does this strike you:
	scp file/serialize daemon@bsd:~/file.img

You don't have to worry about "file is not a file".  You don't have to
worry about fifos or device nodes.  You don't have to worry about files
in files.  In fact, you don't have to worry about any new kind of file
added, ever.

| 2. The use of multiple streams within files by Linux apps would make
| Linux as cross-platform unfriendly as MS is trying to be.  Say these

The use of ext3 as a filesystem isn't cross-platform.  Every disk-write
is platform-specific!  We should all be using captive-ntfs instead!

Seriously, this isn't hard -- it's called a "library".  And anything as
bloated as OO.org has got to already be modular enough to add this sort
of thing.

| features start getting used and you copy an OO.org document from a Linux
| box to a BSD box.  It's broken.  Of course, OO.org wouldn't use the
| streams in the first place because it would destroy their cross platform

If that was so, OO.org wouldn't use a GUI, because that isn't
cross-platform.  They'd use a purely commandline interface, because that
is universal.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTe8PHgHNmZLgCUhAQKiNQ//Zo6BLeGK6/GBbYzcgyp51Ra4fRf2xoVo
0aOve2vinVl+RRzFGWSa66p+/rHd339BhxTPYhlLkm9SWbU9Et80J0R/bySYtc17
RmUHhmukSEXhJx+py7Vc99jXr03zr2fNaA3UBIfpMiAlj1UB2yOls5iwJgqfPVnX
+YwYcVRfZXc3KWqH9SGcY9q+NX38wXoQh3rbeJ0OEzaIxnvPWlMvUFEN4OF92cub
on+n0h+JKptwtXOhLh9HaKAzG9fDhy4tJwmN3GIemSzKpEZ0ZGS4+Pz4dcnRXP+S
C6cMxpsmCnm5WVzmeoyBYBY28Fmnji00ShvxDGRFYOp48RPFhNaoMbVhLkQFoYfX
a5k3iedhbjaWOLmpfNGaDNdpKFtqD3LJO1xOSkXz3sndAiLW9ArKhuu2+63ASAQy
wlFknP09k13UHzD2iL5kPUPzy0A/BLqZPSpX4tXnF0xcSp4X8RoLnyu8BmrqaTjG
fi7FkEsoVEB7BWslNNtKrH596sDhWCP+BQdkAYqOBh0fzSIJ+0uyo2nJRw3d8FEi
/ynNVUqZEN8Xj08WHI+ucwUmRpTnh6spwXZC42K2tokAQa+95nSTegx3uNkypcGU
l12s+BtIHFB25BKDaBReyGoBqUj0sdFvoHVkRNc/3DNY6MfRHV3rdgjz5NieT4gH
uwMwLNstv8c=
=O3r8
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Hans Reiser]

David Masover wrote:

>
> The use of ext3 as a filesystem isn't cross-platform.  Every disk-write
> is platform-specific!  We should all be using captive-ntfs instead!

;-)

All this stuff about how no filesystem should be allowed to have 
semantic features others don't, it seems very Bolshevist to me.

Let Linux have an ecosystem with a diverse ecology of filesystems, and 
the features that work will reproduce to other filesystems.  I thought 
that was the Linus way?

If not, why did I spend 10 years laying the storage layer groundwork for 
semantic enhancements when I could have taken that job at Sun as 
filesystems architect and made a lot more money?

I want to tinker.  Let me play in my sandbox, and if you don't like what 
I do, don't imitate it.....  I think there are plenty of users who like 
reiser4 though....

Linus, trying to outguess someone who has spent 2 decades studying 
namespace design as to what will be useful to users is risky.  Look at 
reiser4's performance, see if it obsoletes V3, and if it does then let 
me play a bit.

Objecting on the grounds that it causes VFS bugs is reasonable, but I 
answered those questions and you did not respond (I can resend if 
asked).  If you really really don't like what we do to VFS, well, we can 
confine ourselves to sys_reiser4(), but that is only a last resort from 
my view.

Hans

Re: The argument for fs assistance in handling archives

[Chris Mason]

On Fri, 2004-09-03 at 02:35, Hans Reiser wrote:

> All this stuff about how no filesystem should be allowed to have 
> semantic features others don't, it seems very Bolshevist to me.
> 
> Let Linux have an ecosystem with a diverse ecology of filesystems, and 
> the features that work will reproduce to other filesystems.  I thought 
> that was the Linus way?
> 

There's no reason you can't have semantics the other filesystems don't. 
There's also no reason you shouldn't continue researching in any
direction you choose.

When you submit to the kernel (even just -mm), call the FS stable and
suggest people start broadly using it, you're telling them the parts you
have submitted are in fact fixed and reliable.

This implies you are done changing the semantics you have submitted,
which also implies you would like applications to start seriously using
it.  Those application writers will want other filesystems to be able to
support these new semantics, which means they should have a race free
framework vetted by the kernel community and available for reuse by the
other filesystems.  In other words, the semantics should be in the VFS,
without any requirement other FS's implement them, but the possibility
for every FS to implement them.

If you don't want the framework to be agreed on by the community and
available for other implementations, you either haven't stabilized the
semantics, or you don't want anyone else using them.  Either way, it
precludes inclusion in the kernel.  There are simply higher expectations
for a filesystem in these cases, and a higher burden for stable
semantics and interfaces.

This doesn't mean new semantics are impossible, it just means that
constantly changing semantics of the filesystem core are bad for users
and applications.  They belong in patches that live outside the kernel
until they are sufficiently researched to justify their inclusion.

The biggest victim of this discussion is the non-semantic part of
reiser4.  You've done a huge amount of research that is being held back
because you haven't separated the semantic research from the storage
layer areas.  If the storage layers are ready now, please please just
submit them.

-chris

Re: The argument for fs assistance in handling archives

[Horst von Brand]

Hans Reiser <reiser@namesys.com> said:
> David Masover wrote:
> > The use of ext3 as a filesystem isn't cross-platform.  Every disk-write
> > is platform-specific!  We should all be using captive-ntfs instead!

> ;-)
> 
> All this stuff about how no filesystem should be allowed to have 
> semantic features others don't, it seems very Bolshevist to me.

Haven seen that. Just requests that _experimental_ features be prototyped
in ReiserFS, and _if_ they show they are worth the hassle, _then_ design a
nice VFS interface on them.

> Let Linux have an ecosystem with a diverse ecology of filesystems, and 
> the features that work will reproduce to other filesystems.  I thought 
> that was the Linus way?

It is.

> If not, why did I spend 10 years laying the storage layer groundwork for 
> semantic enhancements when I could have taken that job at Sun as 
> filesystems architect and made a lot more money?

Dunno. That _you_ choose to do this is no reason _others_ will have to take
the results over lock, stock and barrel. 

Linux' development works because there is a lot of "wasted" effort on
alternatives that don't pan out. Many don't ever leave the drawing board,
others (like the (now defunct) Xiafs, devfs, and volume management code)
went far, even got fully implemented, only to be scrapped later. It might
just be ReiserFS' fate too..

> I want to tinker.  Let me play in my sandbox, and if you don't like what 
> I do, don't imitate it.....  I think there are plenty of users who like 
> reiser4 though....

Go tinker and play. But don't aggravate non-ReiserFS-users, please. Come up
with solutions to the issues Al Viro and Linus raised.

> Linus, trying to outguess someone who has spent 2 decades studying 
> namespace design as to what will be useful to users is risky.

I'd guess LKML adds up to a few thousand years mulling over this, so this
isn't a great point.

>                                                                Look at 
> reiser4's performance, see if it obsoletes V3, and if it does then let 
> me play a bit.

I'd say it is a free world.

> Objecting on the grounds that it causes VFS bugs is reasonable, but I 
> answered those questions and you did not respond (I can resend if 
> asked).  If you really really don't like what we do to VFS, well, we can 
> confine ourselves to sys_reiser4(), but that is only a last resort from 
> my view.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin J. Bligh wrote:
|>For 30 years nothing much has happened in Unix filesystem semantics
|>because of sheer cowardice
|
|
| Or because it works fine, and isn't broken.

Ok, maybe it wasn't cowardice.  Maybe it was laziness.

Because cowardice tells me that data corruption is bad.  Data corruption
kills kittens.  End data corruption now!

Unfortunately, the only way to allow files to survive a crash is to
implement transactions.  The simplest way is to do it in the filesystem
and to export an interface to userland.  This is what Hans is doing,
though AFAIK it's not done yet.

Laziness tells me that I should just use Windows anyway, because "it
works".  Switching to Linux requires training, porting apps requires
work, and using Linux instead of Windows is a hell of a lot more work
than just patching programs to do something like

+ start_transaction()
~  write()
~  blah()
~  write()
+ end_transaction()

How'd you like to be able to tell users "If a file gets corrupted, it's
either a bug in OpenOffice or your hardware eats data."
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTe9pngHNmZLgCUhAQLTZQ/5AZq31sPKOWB/eI4WtVhM76V2+pmQdfNk
aZkPtwhZlVrGDJscp45OVNthwMiSskD07om4obUMac79BCmA5805clZ7i0X1uzQB
SecyX/7XITWz3iwF4VIaxJgDoo7YPFfbpWS0NTmhDJPt6h5Z+goEccLdTABj5kVk
gbex4xNgXzPrnopvjnWx+K2K3ydvXiX+8bWV200F9S+j0uCObcfSZ357uQPqlReP
gKQhEs0pCcoQQua81rXJfUhP8hCbegCpUmhJNI0MwoTUvHTnBU/+99+cCsTLR4gN
1XlbS9SIaaBXG7/AV9L7iQO5GF1t8mTMSe163EOzZ7ypLDb2km5e7zZ7t2DrA6SE
Jgd0xT3cHjDgp1b3qhdeWYvt0QVfz5CGaiHukkT4pEOpsrM09BOhDhY72B74swuV
byDIJ+y4X3J08i/a0zt15hBlVz1FUq7ac7doZZDt3orj93t3B/y4Xpb7tgzVGoWP
T5AkR+uS5dGRUoMSI/btnPIK4ERlhbwFzCB5lgkas5kjp4dnawIKAk1wxPdhvEle
g3SsNqSfBsA9hs6tzVw341GaWj6AWV7oOfg3j0c9I+7nt8SX5UeuTCwBWtCuwd/H
+nnBulusiMnAm3tzQxq8aMpSVAqsZDywjgXSler1mNik4Ow5I5JdXl+Bf2ZWHF+G
VHk4JVxMt/c=
=3z3Z
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Spam]

  

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1

> Martin J. Bligh wrote:
|>>For 30 years nothing much has happened in Unix filesystem semantics
|>>because of sheer cowardice
> |
> |
> | Or because it works fine, and isn't broken.

> Ok, maybe it wasn't cowardice.  Maybe it was laziness.

> Because cowardice tells me that data corruption is bad.  Data corruption
> kills kittens.  End data corruption now!

> Unfortunately, the only way to allow files to survive a crash is to
> implement transactions.  The simplest way is to do it in the filesystem
> and to export an interface to userland.  This is what Hans is doing,
> though AFAIK it's not done yet.

  I thought reiser4 had its journaling and atomic commits. Am I
  mistaken? I run reiser4 as primary fs on my test systems and it seem
  to work as expected.

> Laziness tells me that I should just use Windows anyway, because "it
> works".  Switching to Linux requires training, porting apps requires
> work, and using Linux instead of Windows is a hell of a lot more work
> than just patching programs to do something like

> + start_transaction()
> ~  write()
> ~  blah()
> ~  write()
> + end_transaction()

> How'd you like to be able to tell users "If a file gets corrupted, it's
> either a bug in OpenOffice or your hardware eats data."
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.4 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

> iQIVAwUBQTe9pngHNmZLgCUhAQLTZQ/5AZq31sPKOWB/eI4WtVhM76V2+pmQdfNk
> aZkPtwhZlVrGDJscp45OVNthwMiSskD07om4obUMac79BCmA5805clZ7i0X1uzQB
> SecyX/7XITWz3iwF4VIaxJgDoo7YPFfbpWS0NTmhDJPt6h5Z+goEccLdTABj5kVk
> gbex4xNgXzPrnopvjnWx+K2K3ydvXiX+8bWV200F9S+j0uCObcfSZ357uQPqlReP
> gKQhEs0pCcoQQua81rXJfUhP8hCbegCpUmhJNI0MwoTUvHTnBU/+99+cCsTLR4gN
> 1XlbS9SIaaBXG7/AV9L7iQO5GF1t8mTMSe163EOzZ7ypLDb2km5e7zZ7t2DrA6SE
> Jgd0xT3cHjDgp1b3qhdeWYvt0QVfz5CGaiHukkT4pEOpsrM09BOhDhY72B74swuV
> byDIJ+y4X3J08i/a0zt15hBlVz1FUq7ac7doZZDt3orj93t3B/y4Xpb7tgzVGoWP
> T5AkR+uS5dGRUoMSI/btnPIK4ERlhbwFzCB5lgkas5kjp4dnawIKAk1wxPdhvEle
> g3SsNqSfBsA9hs6tzVw341GaWj6AWV7oOfg3j0c9I+7nt8SX5UeuTCwBWtCuwd/H
> +nnBulusiMnAm3tzQxq8aMpSVAqsZDywjgXSler1mNik4Ow5I5JdXl+Bf2ZWHF+G
> VHk4JVxMt/c=
> =3z3Z
> -----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Spam wrote:
[...]
|>   I thought reiser4 had its journaling and atomic commits. Am I
|>   mistaken? I run reiser4 as primary fs on my test systems and it seem
|>   to work as expected.

consider this:

save_file () {
	write()	/* what if the write flushes halfway through
		 * then crashes?
		 */

	blah()	/* what if "blah" crashes? */
	write()
}

Some apps need consistency across multiple files, but we don't even have
it on a single file.  You need a new interface to do that.  As you can
see, reiser4 has absolutely no way of knowing, anywhere in the above
code, when you're done writing -- and when the file is consistent.

AFAIK, all that has to be done now for this to work is for them to
finish the userland interface to the journalling and atomic commits that
already exist for kernel space.  But so far, all that is truly atomic is
metadata operations -- chmod, mv, mkdir, touch, and rm/rmdir are all
atomic, so long as you only use them on a single file/dir.  But this has
been true in reiserfs3, xfs, ext3, and others.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTfKF3gHNmZLgCUhAQKp3Q//WcijAx8VVdKjJFHENZvrA2P3/wjdCLIn
OeIGi9VxbcQhS1cBM+BU6nBN6wz8S59o9hEM/F8L5oDxto8YW6sfyceoanbtVkLo
r7k4i39A4Ao1UQFzYEzxWzgCA607WvGBboNN1J/5pYe+bWATWBw+oQgc6XikQofQ
QwFX+0KxEDUva5Xkei/wBNhvpSveFBjAm/t79nVkANOVLji2KYALtewps+7QDiGc
PNwfjSUcAbNgXrvuhwpocFgAGEhgS/Y7ANR4DqFAiUseDwY6WgkZqhTaxC8t9ZzF
/Hd82sEb/J+hGpMfrTIe+D5VYhnjX+bPPs/U0OcfoH4CfkgOFgGOeuzlWg3xeTyX
vgtgYGwuGQuWux7WNte0GOCNxfaFDE8lLGexdQYLkdVCSsLmzACA17H1bQdeswgH
UnWL3oBoBx5BcdpkW1y1ZeoJv4xInazkzeU3GtEOd50tlJmOXmdjRNfj5LTrTh+1
CUpEKVvSjQ8MY70uQW6XKEUAkCYyBmSVeMay5SSQS/1Q0ISLEG3gMqPdsL2BOBO/
LkkQKjLe3LQS913wUtibL1CCfJCS/BrDswNCMsboivcg1SnwY+xJb9ffFMW0blBL
kQ0B1AGZCjWHDfxXKIlC31PJYKzgE5vaKZ6+INj8mG8t5PzqgikBDQStvsXqczJq
VgvvOVCjvho=
=I+Xw
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Hans Reiser]

David Masover wrote:

> Spam wrote:
> [...]
> |>   I thought reiser4 had its journaling and atomic commits. Am I
> |>   mistaken? I run reiser4 as primary fs on my test systems and it seem
> |>   to work as expected.
>
> consider this:
>
> save_file () {
>     write()    /* what if the write flushes halfway through
>          * then crashes?

reiser4 does protect from this.  reiserfs v3 does not.

>          */
>
>     blah()    /* what if "blah" crashes? */
>     write()
> }
>
> Some apps need consistency across multiple files, but we don't even have
> it on a single file.  You need a new interface to do that.  As you can
> see, reiser4 has absolutely no way of knowing, anywhere in the above
> code, when you're done writing -- and when the file is consistent.
>
> AFAIK, all that has to be done now for this to work is for them to
> finish the userland interface to the journalling and atomic commits that
> already exist for kernel space.  But so far, all that is truly atomic is
> metadata operations -- chmod, mv, mkdir, touch, and rm/rmdir are all
> atomic, so long as you only use them on a single file/dir.  But this has
> been true in reiserfs3, xfs, ext3, and others.

Re: The argument for fs assistance in handling archives

[Timothy Miller]

Hey, you know how device nodes have a bit set, indicating that they're 
device nodes and not regular files?  Can a set of such properties be 
defined for reiser4 metadata properties?

Like a "metadata" bit so you can distinguish (if you wish) between 
regular files and metadata objects, and in addition an "archivable 
metadata" bit which indicates that the given piece of metadata is not 
automatically generated and should be archived during backup (some 
manually-generated metadata which does not need to be backed up will not 
have this bit set -- perhaps add another flag indicating that it's not 
automatic but unnecessary to archive).

Re: The argument for fs assistance in handling archives

[Helge Hafting]

Timothy Miller wrote:

>
> Hey, you know how device nodes have a bit set, indicating that they're 
> device nodes and not regular files?  Can a set of such properties be 
> defined for reiser4 metadata properties?
>
> Like a "metadata" bit so you can distinguish (if you wish) between 
> regular files and metadata objects, and in addition an "archivable 
> metadata" bit which indicates that the given piece of metadata is not 
> automatically generated and should be archived during backup (some 
> manually-generated metadata which does not need to be backed up will 
> not have this bit set -- perhaps add another flag indicating that it's 
> not automatic but unnecessary to archive). 


Interesting idea, particularly for autogenerated metadata.  Some 
metadata, like
thumbnails and icons might as well be ordinary files though.  (Like some
icon and want to use it as background bitmap?  Just copy it, because
it is a plain file.  Or want to use it as clipart - fine, the word processor
supports bitmap files. . .) 

This is a case where something is "metadata"  only because it is used as 
such,
not because it has to be.  File-as-dir is an interesting concept for 
such cases,
because it is convenient to group the "metadata" with the file, have it
move with the file and so on.  But still have this metadata accessible as
a file because it isn't all that special.

Helge Hafting

Re: The argument for fs assistance in handling archives

[Hans Reiser]

Linus Torvalds wrote:

>
>But _my_ point is, no user program is going to take _advantage_ of
>anything that only one filesystem on one system offers.
>
>So there's no point.
>
>I
>
Filesystem diversity will only be crippling for Linux if you choose for 
it to be so.

Re: The argument for fs assistance in handling archives

[James Bruce]

Linus Torvalds wrote:

>...
>But _my_ point is, no user program is going to take _advantage_ of
>anything that only one filesystem on one system offers.
>
>So there's no point.
>
>It's much saner (from _any_ app standpoint) to roll their own database in 
>user space - that way it just works.
>
>In other words, nobody is really ever going to take advantage of this. 
>This is _not_ how technical advanncement happens. The way you get people 
>to take advantage of something is to have a nice gradual ramp-up, not a 
>sudden new feature that they can't realistically use.
>...
>

Sure, but there are plenty of existing interfaces that you could 
emulate.  One could make a small library to use a transactional 
filesystem to implement the Berkely DB interface (libdb) for example.  
So on filesystems without such support your app could use the regular 
userspace database, but on a transactional filesystem it'd just use 
regular files, which would simplify database management and likely 
increase performance over the userspace-only version (libdb is pretty 
slow).  In terms of functionality it'd just be a drop-in replacement, 
just like math libraries that use MMX/SSE when available.

    - Jim Bruce

Re: The argument for fs assistance in handling archives

[Linus Torvalds]

On Thu, 2 Sep 2004, James Bruce wrote:
> Linus Torvalds wrote:
>
> >In other words, nobody is really ever going to take advantage of this. 
> >This is _not_ how technical advanncement happens. The way you get people 
> >to take advantage of something is to have a nice gradual ramp-up, not a 
> >sudden new feature that they can't realistically use.
> 
> Sure, but there are plenty of existing interfaces that you could 
> emulate. 

Absolutely. We should look at them, and whether they solve any issues. 

I'm not saying that Hans would have to make up a new interface. Quite the
reverse. One interface I suggested to use for attributes (and everybody
hates ;) was the existing one from Solaris. Similarly, there are probably 
perfectly fine interfaces for the issue of transactions..

		Linus

Re: The argument for fs assistance in handling archives

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 222 bytes --]

Salut,

On Thu, Sep 02, 2004 at 08:50:10AM -0400, James Bruce wrote:
> filesystem to implement the Berkely DB interface (libdb) for example.  

Problem is that BerkeleyDB changes its API all the time.

			Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Linus Torvalds wrote:
[...]
| It's much saner (from _any_ app standpoint) to roll their own database in
| user space - that way it just works.

Then why do we have apps which use Windows' Internet Options where it
can, and otherwise ask the user for proxy settings?

|
| In other words, nobody is really ever going to take advantage of this.
| This is _not_ how technical advanncement happens. The way you get people
| to take advantage of something is to have a nice gradual ramp-up, not a
| sudden new feature that they can't realistically use.
|
| So before you do transactions in the filesystem, you have to be able
to do
| them in user space - and then make the filesystem version be _compatible_
| with that user space library. That way you can get people to use the
| library ("hey, it works anywhere") and then you can get them to use your
| filesystem ("hey, if you use our super-duper filesystem, you can do what
| you are already doing five times faster").

Why not:
	do transactions in the filesystem
	write a library to talk to that filesystem
	have the library do emulation on other systems

in that order.

| See? You're starting at the wrong end. Give me a portable interface to
use
| _first_. Then do transactions in the filesystem.

If you're going to have to do both the portable interface and the fs
support, does it really matter which order you do them in?

Again, this is curiosity, not sarcasm.  Here my reasons for wanting the
fs support first:

You don't always know exactly how the filesystem transactions will work.
~ You don't know until it's done whether you'll deviate from your design,
cut certain features, replace them with others...

And if you write the portable library first, and it ends up supporting
features a, b, and c, while the kernel supports features b, c, and d,
which obsoletes a, you have to add some features and remove some
features from the portable library to make it sane.

It'd be like writing OpenGL entirely in software, before hardware
accelerators work, and at the last minute have to change the library to
use triangles instead of splines.  What's more, every single app now has
to be rewritten or use the library's emulation -- that's assuming many
apps would consider using a 3D library that doesn't have hardware accel
support yet.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTfEsHgHNmZLgCUhAQK1ORAAiSQ6UHSYc/PPMMOv2VjTkSuK2MgPylKP
d2jEitIuvWx01xAa++CLcUS6ncaiLuFYDA7TTHjhKYKAj+zuNN9OTI9EMGc6D+Ua
lFcbcRFapqbT9HPrUnGPM8VQIR8UrBt54hRAGm0/jOCc8DKLa3d6xnRfqmKQhnkv
w3ut4FiNo0GCd19hF82NxEA1YtwbOIgOIUxVlbADGI9QT/d+I3ZMA8QUNCrgrudn
e9g7o0yTa2c16t/rrqkiYCqSHIhCgY8ZH8VQzG4uYSjU+qYS+QbpJakD82EyuF/k
4T4jyJOqqSv6tLwE1j2y0gteXDYOdmfda/TpDeBSaRvGf+SmNxfSDFL448B2QJgA
e3V0HZHVVums1Vl6kX4pBaWM01V3n9BGeLaF2S9XSP7WCSWgW7t4ONRSLE3Bf2mm
1e5cMJ63mrtImb8Jyd8GhyCQ8/tkGnEYOIAHL3laGOjuc2ARBeyqmWrGKJsQ8iwu
i9NsyukQC3NP2FY1nBSmKs9J2bWtixqk5OXpzupkqBkbpeKL1H/b9Mxpnn5+Aomi
PjI/5i4QcaHMNBggxCG3B5zGJIWmvMkmZmHrHxvbLeENGvS5fiZ0Kscq7PL0FnE0
/2etDgjg65I66SqguXMzvtZdQ0hYgdwvoR83nZymHMSLYrFJ8FOxuB7kYQBeKZnu
QpUz0UDTUZA=
=jVvF
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 448 bytes --]

On Thu, 02 Sep 2004 20:11:13 CDT, David Masover said:

> It'd be like writing OpenGL entirely in software, before hardware
> accelerators work, and at the last minute have to change the library to
> use triangles instead of splines.

I expect that SGI did a software-only version of IrisGL first, so they could
figure out what the hardware accelerators needed to support.  And even then,
the API for IrisGL got modified when it became OpenGL.....


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: The argument for fs assistance in handling archives

[Hans Reiser]

The reason for supporting transactions in the kernel is that there is a 
whole slew of security bugs that have their origins in fs semantics 
being non-transactional.  rename does not cut it as a transactions API.  
rename results in horribly performing apps that engage in contortions 
that cost a lot of programmer time to code them to stuff a transaction 
into a file that can be renamed.  Most of the time programmers who 
should use a transaction/rename are just too busy to do it, and they 
take the risk that the user will lose data and be less secure rather 
than write the code to do it right.  When they do do it, because it is 
complex code, they often screw it up.

That said, there is a whole slew of reasons for not supporting 
transactions in the kernel:

* the history of the literature stars transactional filesystems that 
paid too high a cost in performance

* isolation is very tricky to do without killing performance

* transactions which are kept open too long are a problem

* avoiding problems with locking shared data structures long enough to 
cause painful exclusion of things that should be able to run in parallel 
just fine

Reiser4, by being very modest in its ambitions, manages to not pay a 
performance cost.   We say, isolation and rollback are for user space to 
manage until we can find a clever implementation, if such an 
implementation can even be found.  Even if such an implementation can be 
found, there are plenty of cases where special case knowledge of what 
the app is doing can improve performance a lot, and thus for that app it 
will belong in user space.

All reiser4 does, is allow you to guarantee (using sys_reiser4) that a 
limited size set of fs operations will be fused into one atom together, 
and therefor will either all survive a crash or none will survive.  This 
does however meet a lot of apps needs.

Hans

David Masover wrote:

> Linus Torvalds wrote:
> [...]
> | It's much saner (from _any_ app standpoint) to roll their own 
> database in
> | user space - that way it just works.
>
> Then why do we have apps which use Windows' Internet Options where it
> can, and otherwise ask the user for proxy settings?
>
> |
> | In other words, nobody is really ever going to take advantage of this.
> | This is _not_ how technical advanncement happens. The way you get people
> | to take advantage of something is to have a nice gradual ramp-up, not a
> | sudden new feature that they can't realistically use.
> |
> | So before you do transactions in the filesystem, you have to be able
> to do
> | them in user space - and then make the filesystem version be 
> _compatible_
> | with that user space library. That way you can get people to use the
> | library ("hey, it works anywhere") and then you can get them to use your
> | filesystem ("hey, if you use our super-duper filesystem, you can do what
> | you are already doing five times faster").
>
> Why not:
>     do transactions in the filesystem
>     write a library to talk to that filesystem
>     have the library do emulation on other systems
>
> in that order.
>
> | See? You're starting at the wrong end. Give me a portable interface to
> use
> | _first_. Then do transactions in the filesystem.
>
> If you're going to have to do both the portable interface and the fs
> support, does it really matter which order you do them in?
>
> Again, this is curiosity, not sarcasm.  Here my reasons for wanting the
> fs support first:
>
> You don't always know exactly how the filesystem transactions will work.
> ~ You don't know until it's done whether you'll deviate from your design,
> cut certain features, replace them with others...
>
> And if you write the portable library first, and it ends up supporting
> features a, b, and c, while the kernel supports features b, c, and d,
> which obsoletes a, you have to add some features and remove some
> features from the portable library to make it sane.
>
> It'd be like writing OpenGL entirely in software, before hardware
> accelerators work, and at the last minute have to change the library to
> use triangles instead of splines.  What's more, every single app now has
> to be rewritten or use the library's emulation -- that's assuming many
> apps would consider using a 3D library that doesn't have hardware accel
> support yet.

Re: The argument for fs assistance in handling archives

[Robin Rosenberg]

On Thursday 02 September 2004 09.15, Hans Reiser wrote:
> Linus Torvalds wrote:
> >. Doing transactions on one file is
> >only the beginning - you'll find people who want transactions across file
> >boundaries etc.
>
> Yup, that's what reiser4 is aiming at and where things get exciting for
> version control systems and the like.  We are just missing the api code
> for it at the moment, all the infrastructure is there, and the api is in
> progress in sys_reiser4.

Hope that is the same as or implied XA-support. Having to syncrhonized file 
updates with database transactions is not an uncommon problem. 

-- robin

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Alan Cox]

On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
> and quite frankly, I think you can do the above pretty much totally in
> user space with a small library and a daemon (in fact, ignoring security
> issues you probably don't even need the daemon). And if you can prototype
> it like that, and people actually find it useful, I suspect kernel support
> for better performance might be possible.

Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
it since the late 1990's.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Jamie Lokier]

Alan Cox wrote:
> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
> > and quite frankly, I think you can do the above pretty much totally in
> > user space with a small library and a daemon (in fact, ignoring security
> > issues you probably don't even need the daemon). And if you can prototype
> > it like that, and people actually find it useful, I suspect kernel support
> > for better performance might be possible.
> 
> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
> it since the late 1990's.

Firstly, if I have to do it from a Gnome program, about the only
program where looking in a tar file is visibly useful is Nautilus.
Ironically, clicking on a tar file in Nautilus doesn't work, despite
having a dependency on gnome-vfs2. :/

Secondly, no, Gnome and MC don't support entering a container file,
letting you make changes in it, and remembering those changes to
_lazily_ regenerate the container file when you need it linearized,
possibly months later or never, by some unrelated program.

Thirdly, you must be referring to the Gnome versions of Bash, Make,
GCC, coreutils and Perl which I haven't found.  Perhaps we have a
different idea of what "supports this" means :)

uservfs, which is based on gnome-vfs and getting a bit rusty due to
disuse, does try to solve the last problem.  Unfortunately it needs
further work to have a nicer interface, and the second problem is
still not solved by gnome-vfs.

-- Jamie

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Dave Kleikamp]

On Thu, 2004-09-02 at 11:11, Jamie Lokier wrote:

> Firstly, if I have to do it from a Gnome program, about the only
> program where looking in a tar file is visibly useful is Nautilus.
> Ironically, clicking on a tar file in Nautilus doesn't work, despite
> having a dependency on gnome-vfs2. :/

This should be fixed in Nautilus, not the kernel.

> Secondly, no, Gnome and MC don't support entering a container file,
> letting you make changes in it, and remembering those changes to
> _lazily_ regenerate the container file when you need it linearized,
> possibly months later or never, by some unrelated program.

Why do this in a tar file?  tar = "tape archive".  It isn't designed to
be a file system.  Sure, it's nice to have tools that make it easier to
access files in a tar file, but to this isn't a job for the kernel.

> Thirdly, you must be referring to the Gnome versions of Bash, Make,
> GCC, coreutils and Perl which I haven't found.  Perhaps we have a
> different idea of what "supports this" means :)

Please don't tell me that we have expectations to run make from within a
tar file.  This is getting silly.  tar does a pretty good job of
extracting files into real directories, and putting them back into an
archive.  I don't see a need to teach the kernel how to deal with
compound files when user space can do it very easily.

Shaggy

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dave Kleikamp wrote:
[...]
| Please don't tell me that we have expectations to run make from within a
| tar file.  This is getting silly.  tar does a pretty good job of
| extracting files into real directories, and putting them back into an
| archive.  I don't see a need to teach the kernel how to deal with
| compound files when user space can do it very easily.

Suppose I've got a tar file with an index attached.  Suppose it's
something like /usr/src/linux.  Am I expected to extract all code for
all architectures, with all drivers, all docs, etc?  Now, yes -- or I
have to figure out exactly which ones I need before I extract them
manually, one by one.

But with tar support for make (and so on), files can be extracted on
demand.  It's possible to do this in userspace, with named pipes, but
that's much slower and insanely clumsy.

This has further implications -- imagine a desktop, binary distro
shipped with all files except the very most basic stuff as package
archives.  They can all be extracted, on demand -- the first time I run
OpenOffice.org, it's installed.  If there needs to be post-installation,
that's handled by the .deb plugin (or whatever).

I don't know offhand how big OOo is.  I think it's something like this:
~ The installer is at most half (and maybe only a third) of the full
installation. That's a HUGE optimization!
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTe5/HgHNmZLgCUhAQI5vQ/+NyU/tbW1Dyaf/OlDUEScH8jHghdcMPQQ
qcyBbzid9hMT0pm4fRX4CQJ/vm+VLhfvYzEmgRUCyNY3JybCKeS/EynRt/ybdblu
aB+hO8meFitBmAa7kYrj1UhWvoSvDSZgAwC9k50DYPuQO1kVZFjFYcPee1P54iwJ
UMn9RE01aeufCt1+jWFxhsEZKfNWvXDCaQtqa483A2AWWzklwF25ZW2kSfp6G+i0
g1jND8pPDkQcP8ujGTuDxEI8LsN62glNzVZ8MhPa65lZI1vO5Ll2dDL2QKgNwziK
MqtMMJD1d3HWa7QBHwMegJ0teR/hiqJ62SgQr3QpW4Xy9Ss0VUVH1HNuhxwPB2rl
YYomqw2yO/GGSDs5XuXm/cRM5E9d+nvu1V8bsrSa5LK/64Vlp6huLkLNvOZ3y6vK
38ELPBxbmIA3iWTgaYDPANX/vrpnA0K8JQU9M4LMveaHhxfEcDbH+iZHtpjsYqF3
allfHH2SEZRFlXGxKBNZsXTcrudAHjoyEOQ+UiI9QLCM83G4bFGr1WEGOEHmD0ry
hBETe8GkwuQK1CfxFm5obgFUmE4TwVRIWVD71EvoJFuBS+dlezO6GOZ5mDf91tSe
goPS2f/9XKwyYfOnEnnfXT17k5SYjTB9m0upi6q7dJpxvg1535E6N1nCqdLZzTcJ
mVkdhfFsUqI=
=YjJH
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Dave Kleikamp]

On Thu, 2004-09-02 at 19:25, David Masover wrote:
> Dave Kleikamp wrote:
> [...]
> | Please don't tell me that we have expectations to run make from within a
> | tar file.  This is getting silly.  tar does a pretty good job of
> | extracting files into real directories, and putting them back into an
> | archive.  I don't see a need to teach the kernel how to deal with
> | compound files when user space can do it very easily.
> 
> Suppose I've got a tar file with an index attached.  Suppose it's
> something like /usr/src/linux.  Am I expected to extract all code for
> all architectures, with all drivers, all docs, etc?  Now, yes -- or I
> have to figure out exactly which ones I need before I extract them
> manually, one by one.

I don't think it's unreasonable to expect someone to either extract the
whole tar file, or identify what files they want from it.  If you think
there is too much in the tar file, roll your own with only the files you
need.

> But with tar support for make (and so on), files can be extracted on
> demand.  It's possible to do this in userspace, with named pipes, but
> that's much slower and insanely clumsy.

This doesn't justify bloating the kernel.  untar the darn thing and user
space does fine.

> This has further implications -- imagine a desktop, binary distro
> shipped with all files except the very most basic stuff as package
> archives.  They can all be extracted, on demand -- the first time I run
> OpenOffice.org, it's installed.  If there needs to be post-installation,
> that's handled by the .deb plugin (or whatever).

Are you saying install it on demand the first time it's run?  This
doesn't take any new kernel function.

Or are you saying that the files are never installed on the filesystem,
but always accessed from the package archives?  In this case, why not
ship each package as a compressed iso, and have the system mount the iso
to run the app.  I really don't see the point though, in that disk space
is very cheap these days.

> I don't know offhand how big OOo is.  I think it's something like this:
> ~ The installer is at most half (and maybe only a third) of the full
> installation. That's a HUGE optimization!

Optimization?  How much performance are you willing to give up to save a
little bit of disk space?

-- 
David Kleikamp
IBM Linux Technology Center

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dave Kleikamp wrote:
| On Thu, 2004-09-02 at 19:25, David Masover wrote:
|
|>Dave Kleikamp wrote:
|>[...]
|>| Please don't tell me that we have expectations to run make from within a
|>| tar file.  This is getting silly.  tar does a pretty good job of
|>| extracting files into real directories, and putting them back into an
|>| archive.  I don't see a need to teach the kernel how to deal with
|>| compound files when user space can do it very easily.
|>
|>Suppose I've got a tar file with an index attached.  Suppose it's
|>something like /usr/src/linux.  Am I expected to extract all code for
|>all architectures, with all drivers, all docs, etc?  Now, yes -- or I
|>have to figure out exactly which ones I need before I extract them
|>manually, one by one.
|
|
| I don't think it's unreasonable to expect someone to either extract the
| whole tar file, or identify what files they want from it.  If you think
| there is too much in the tar file, roll your own with only the files you
| need.

That is not a solution, that is exactly the opposite of a solution.
"Roll my own" implies that I must download the tar file, extract the
whole thing, grab what I need, and then tar it up again.  Then I extract
it again.  Doh!

More seriously, suppose it's a format like zip, where I don't have to
decompress the whole file to get a listing.  And suppose I want to
compile some little example from it, but the file itself is huge -- half
a gig, say, and I only need ten megabytes.

But the only way I'm going to find out _which_ ten megabytes I need is
to extract the Makefile, read it, then go find all the other little
Makefiles, not to mention the configure script, the header files, and so
on...

But suppose that make can _transparently_ extract _only_ the files I
need for this?

|>But with tar support for make (and so on), files can be extracted on
|>demand.  It's possible to do this in userspace, with named pipes, but
|>that's much slower and insanely clumsy.
|
|
| This doesn't justify bloating the kernel.  untar the darn thing and user
| space does fine.

Does it really bloat the kernel?  My kernel doesn't feel bloated, and
it's got reiser4 and much of what's needed for this.

Remember, saying "I can access foo.zip/contents" doesn't mean "zip is in
the kernel".

|>This has further implications -- imagine a desktop, binary distro
|>shipped with all files except the very most basic stuff as package
|>archives.  They can all be extracted, on demand -- the first time I run
|>OpenOffice.org, it's installed.  If there needs to be post-installation,
|>that's handled by the .deb plugin (or whatever).
|
|
| Are you saying install it on demand the first time it's run?  This
| doesn't take any new kernel function.

It does for it to appear installed.  And to only install the pieces that
are needed at that time.

| Or are you saying that the files are never installed on the filesystem,
| but always accessed from the package archives?  In this case, why not

No, only that if I only wanted OpenOffice writer, why should I install
all of OpenOffice?  Yes, it could be broken into smaller packages.  What
I'm talking about is exactly the pieces of an installation that you need
- -- insanely fine granularity -- done automatically, on demand.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTkLiHgHNmZLgCUhAQK7+A//TATQvQ3U61VA/mdVqylnrkWC7tNOewwq
MqF+KKU3Cc/n54mOlGTpph2qpPJTzv1y4KlVgcDM/d0bn1cDPx41n//xEe9QXlqu
vOywb8g11HSlAhKmbl4APwCHFHr1HibHgYqM7PmeVSD+Xfy5gJvIW5Oc44f16+q9
agEXWAk0EgM0WCAKEQFxN56i8e7qHq28PPGzcpGcn08xmBmD9Ik71jjpLY88csYy
kjH32ExEy+uABq+Tglfr0EBZR4RDuqkxsei7cL3Rn58O8twJtn8UP3VcukTLciZw
jb7If3ekuO7BXYJbwB/foFEESFql68jNKH7c7+Bzeb5pREloreVine/2rRM1iekD
FUeTv78kn+6G/INl9XwUB2ER0KZOy8n2wZut35T5w94GtZgmdHpm+3mCOscS6BdG
JNx/HRGJJXfm0P/7tKbgZ/3wjQlFzbC5HcByn9Ocfm8qrNsoLxwtF/8aId/9ctD9
lEmDMUHYuVC51/m5ka+i/XUQeuzgbtY5QKoNsxWXYZfeBNQfMqfMOsVWP1wFMlpB
mPmf6w+4idp1aIYwgvPyQee1BZiXmkbncglcnY+J4y46AZ+tDEO5eqjJrEU9kA7v
NNVfCehCZ2IIo1TMHcQizsHQ53NEmZ5gwCYXvymGS1+6fyE0SqcP9EolVuUfnRN9
yNdulMJ0gDg=
=TeOM
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 458 bytes --]

Salut,

On Thu, Sep 02, 2004 at 07:25:32PM -0500, David Masover wrote:
> This has further implications -- imagine a desktop, binary distro
> shipped with all files except the very most basic stuff as package
> archives.  They can all be extracted, on demand -- the first time I run
> OpenOffice.org, it's installed.  If there needs to be post-installation,
> that's handled by the .deb plugin (or whatever).

zsh using apt extensions does that.

			Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Pavel Machek]

Hi!

> > Thirdly, you must be referring to the Gnome versions of Bash, Make,
> > GCC, coreutils and Perl which I haven't found.  Perhaps we have a
> > different idea of what "supports this" means :)
> 
> Please don't tell me that we have expectations to run make from within a
> tar file.  This is getting silly.  tar does a pretty good job of
> extracting files into real directories, and putting them back into an
> archive.  I don't see a need to teach the kernel how to deal with
> compound files when user space can do it very easily.

Actually its not easy. User has to manually extract it and manually
delete it when he's done. Not nice.
								Pavel
-- 
When do you have heart between your knees?

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Paul Jakma]

On Thu, 2 Sep 2004, Jamie Lokier wrote:

> Firstly, if I have to do it from a Gnome program, about the only 
> program where looking in a tar file is visibly useful is Nautilus. 
> Ironically, clicking on a tar file in Nautilus doesn't work, 
> despite having a dependency on gnome-vfs2. :/

Do you have file-roller installed?

I can open tar/zip/rar/etc.. files from anywhere in gnome2, eg in 
Galeon I can click on a tar.gz URL (http or whatever) and have it 
open it in file-roller, from where i can browse the files to my 
hearts content.

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
Why isn't there a special name for the tops of your feet?
 		-- Lily Tomlin

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  

> On Thu, 2 Sep 2004, Jamie Lokier wrote:

>> Firstly, if I have to do it from a Gnome program, about the only 
>> program where looking in a tar file is visibly useful is Nautilus. 
>> Ironically, clicking on a tar file in Nautilus doesn't work, 
>> despite having a dependency on gnome-vfs2. :/

> Do you have file-roller installed?

> I can open tar/zip/rar/etc.. files from anywhere in gnome2, eg in 
> Galeon I can click on a tar.gz URL (http or whatever) and have it 
> open it in file-roller, from where i can browse the files to my 
> hearts content.

  But can you actually do things with these files? Can you run
  applications or edit files directly, or is there need for temporary
  unzip first?
  
  ~S
  
> regards,

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Alan Cox]

On Gwe, 2004-09-03 at 00:49, Spam wrote:
>   But can you actually do things with these files? Can you run
>   applications or edit files directly, or is there need for temporary
>   unzip first?

You always need that for zip files. Firstly because executables are
paged so you need an accessible random access copy of the bits. Secondly
because data may be paged, and also for seek performance.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  

> On Gwe, 2004-09-03 at 00:49, Spam wrote:
>>   But can you actually do things with these files? Can you run
>>   applications or edit files directly, or is there need for temporary
>>   unzip first?

> You always need that for zip files. Firstly because executables are
> paged so you need an accessible random access copy of the bits. Secondly
> because data may be paged, and also for seek performance.

  Yes, some archive types can't be partially unzipped either. But my
  point is that it wouldn't be transparent to the application/user in
  the same way.

  ~S

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Paul Jakma]

On Fri, 3 Sep 2004, Spam wrote:

>  Yes, some archive types can't be partially unzipped either. But my
>  point is that it wouldn't be transparent to the application/user in
>  the same way.

It doesnt matter whether it is transparent to the application. It can 
be the application which implements the required level of 
transparency.

User doesnt care what provides the transparency or how it's 
implemented.

>  ~S

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
A committee is a group that keeps the minutes and loses hours.
 		-- Milton Berle

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  

> On Fri, 3 Sep 2004, Spam wrote:

>>  Yes, some archive types can't be partially unzipped either. But my
>>  point is that it wouldn't be transparent to the application/user in
>>  the same way.

> It doesnt matter whether it is transparent to the application. It can
> be the application which implements the required level of 
> transparency.

> User doesnt care what provides the transparency or how it's 
> implemented.

  Indeed. I hope I didn't say otherwise :). Just that I think it will
  be very difficult to have this transparency in all apps. Just
  thinking of "nano file.jpg/description.txt" or "ls
  file.tar/untar/*.doc". Sure in some environments like Gnome it could
  work, but it still doesn't for the rest of the flora of Linux
  programs.



  ~S

> regards,

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Dave Kleikamp]

On Thu, 2004-09-02 at 19:39, Spam wrote:
>   
> 
> > On Fri, 3 Sep 2004, Spam wrote:
> 
> >>  Yes, some archive types can't be partially unzipped either. But my
> >>  point is that it wouldn't be transparent to the application/user in
> >>  the same way.
> 
> > It doesnt matter whether it is transparent to the application. It can
> > be the application which implements the required level of 
> > transparency.
> 
> > User doesnt care what provides the transparency or how it's 
> > implemented.
> 
>   Indeed. I hope I didn't say otherwise :). Just that I think it will
>   be very difficult to have this transparency in all apps.

You're missing the point.  We don't need transparency in all apps.  You
can write an application to be as transparent as you want, but you don't
need every app to to understand every file format.

>  Just
>   thinking of "nano file.jpg/description.txt" or "ls
>   file.tar/untar/*.doc".

I don't do much image editting, but I'm sure there are applications that
let you edit the description in a text file.  You can even create a
script that extracts it, runs nano, and puts it back into the jpeg.

This works for me:
tar -tf file.tar | grep '\.doc'

There are userland tools that deal with hundreds of file formats.  Use
the tool you need, rather than try to have the kernel do everything.

>  Sure in some environments like Gnome it could
>   work, but it still doesn't for the rest of the flora of Linux
>   programs.

Just choose the right program.  tar groks tar files, not ls.

-- 
David Kleikamp
IBM Linux Technology Center

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  

> On Thu, 2004-09-02 at 19:39, Spam wrote:
>>   
>> 
>> > On Fri, 3 Sep 2004, Spam wrote:
>> 
>> >>  Yes, some archive types can't be partially unzipped either. But my
>> >>  point is that it wouldn't be transparent to the application/user in
>> >>  the same way.
>> 
>> > It doesnt matter whether it is transparent to the application. It can
>> > be the application which implements the required level of 
>> > transparency.
>> 
>> > User doesnt care what provides the transparency or how it's 
>> > implemented.
>> 
>>   Indeed. I hope I didn't say otherwise :). Just that I think it will
>>   be very difficult to have this transparency in all apps.

> You're missing the point.  We don't need transparency in all apps.  You
> can write an application to be as transparent as you want, but you don't
> need every app to to understand every file format.

  No, but not every user "can write an application" either, or even
  have the skills to apply patches. What I was talking about wasn't
  just tar, which itself isn't the best example anyway, but the idea
  that users can load plugins that will extend the functionality of
  their filesystems. That idea seem to be to be _much_ better than
  trying to teach every user how to write applications or patch
  existing ones.

>>  Just
>>   thinking of "nano file.jpg/description.txt" or "ls
>>   file.tar/untar/*.doc".

> I don't do much image editting, but I'm sure there are applications that
> let you edit the description in a text file.  You can even create a
> script that extracts it, runs nano, and puts it back into the jpeg.

> This works for me:
> tar -tf file.tar | grep '\.doc'

> There are userland tools that deal with hundreds of file formats. Use
> the tool you need, rather than try to have the kernel do everything.

  No, but if I wanted to have an encryption plugin active for some of
  my files or directories then why should I not be able to? I still
  want to edit, view and save my encrypted files.

  Again, this was just an example of what could be done with plugins.
  It is not said that every conceivable plugin will be written, nor
  loaded per default. Even though plugins cannot today be dynamically
  used, they will be eventually. Reiser4 is still very young.

  Please separate your thoughts for specific plugins from those of the
  idea to have plugins at all.

  ~S

>>  Sure in some environments like Gnome it could
>>   work, but it still doesn't for the rest of the flora of Linux
>>   programs.

> Just choose the right program.  tar groks tar files, not ls.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Dave Kleikamp]

On Fri, 2004-09-03 at 08:16, Spam wrote:
> > You're missing the point.  We don't need transparency in all apps.  You
> > can write an application to be as transparent as you want, but you don't
> > need every app to to understand every file format.
> 
>   No, but not every user "can write an application" either, or even
>   have the skills to apply patches. What I was talking about wasn't
>   just tar, which itself isn't the best example anyway,

That was one of the examples you gave, that and .jpg.  I believe they
are both ridiculous.

>  but the idea
>   that users can load plugins that will extend the functionality of
>   their filesystems. That idea seem to be to be _much_ better than
>   trying to teach every user how to write applications or patch
>   existing ones.

If I understand Hans' plugins, they are not user-loadable, but rather a
statically built part of the kernel.

> 
>   No, but if I wanted to have an encryption plugin active for some of
>   my files or directories then why should I not be able to? I still
>   want to edit, view and save my encrypted files.

I would not argue against an encryption plugin.

>   Again, this was just an example of what could be done with plugins.
>   It is not said that every conceivable plugin will be written, nor
>   loaded per default.

This I agree with.

>  Even though plugins cannot today be dynamically
>   used, they will be eventually. Reiser4 is still very young.

As kernel modules, this would make sense.  I don't see just any
unprivileged user being able to add code into the file system, though.

>   Please separate your thoughts for specific plugins from those of the
>   idea to have plugins at all.

I'm not against reiser4 plugins.  I don't think file system code should
care about the type of data in a file, and do any interpretation based
on it.

Shaggy
-- 
David Kleikamp
IBM Linux Technology Center

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Horst von Brand]

Spam <spam@tnonline.net> said:
> Dave Kleikamp <shaggy@austin.ibm.com> said:
> >Spam <spam@tnonline.net> said:

[...]

> > You're missing the point.  We don't need transparency in all apps.  You
> > can write an application to be as transparent as you want, but you don't
> > need every app to to understand every file format.

>   No, but not every user "can write an application" either, or even
>   have the skills to apply patches. What I was talking about wasn't
>   just tar, which itself isn't the best example anyway, but the idea
>   that users can load plugins that will extend the functionality of
>   their filesystems. That idea seem to be to be _much_ better than
>   trying to teach every user how to write applications or patch
>   existing ones.

Why compare "write application or apply patches" to "load plugin"? It
would be locical to compare running applications with loading plugins (and
even so, loading plugins is presumably root-only).

[...]

> > There are userland tools that deal with hundreds of file formats. Use
> > the tool you need, rather than try to have the kernel do everything.

>   No, but if I wanted to have an encryption plugin active for some of
>   my files or directories then why should I not be able to? I still
>   want to edit, view and save my encrypted files.

Use an editor that knows about encrypted files. Decrypt/edit/encrypt if no
other option (I'm sure emacs can be coerced to do that transparently ;-). I
for one would be _way_ more confortable with my users doing that than them
loading random modules into the kernel. Besides, if one of my users doesn't
trust the system encryption programs, and prefers hers, she can be happy.

>   Again, this was just an example of what could be done with plugins.
>   It is not said that every conceivable plugin will be written, nor
>   loaded per default. Even though plugins cannot today be dynamically
>   used, they will be eventually. Reiser4 is still very young.

Modules of the kernel were supposed to have all those magic properties too,
until there were nasty races... and it was _seriously_ considered to take
them out. They stayed because they are root-only business, and (un)loading
is rare. FS plugins are kernel modules, AFAIU, and are subject to the same
problems.

>   Please separate your thoughts for specific plugins from those of the
>   idea to have plugins at all.

If you can't find concrete uses for specific plugins, they are the
proverbial solution searching for a problem.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
[...]
| Use an editor that knows about encrypted files. Decrypt/edit/encrypt if no

You use emacs.  I use vim.  My brother uses gedit.  My parents use
abiword.  Perhaps I should patch them all?  If that was so easy, why is
there cryptoloop/dm-crypt?

| is rare. FS plugins are kernel modules, AFAIU, and are subject to the same
| problems.

Actually, FS plugins currently cannot be modules.  They are currently
called "plugins" because they share some concepts with browser plugins,
and it sounds great in marketing.

| If you can't find concrete uses for specific plugins, they are the
| proverbial solution searching for a problem.

Fine, let them be.  They are just very well structured code.  If you use
reiser4 and don't look in the source or the "metas" dir, they are
completely invisible to you.  I used the betas for months, and "metas"
never burned me.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTlUt3gHNmZLgCUhAQJVGhAAki5RLZckm5jiZnw7MXQLqURBvm+cz7wU
eFPfGyOE0SpFKJLFntWr0zrAgyK5ClgwqB7wsJEldWUfGBPdYpH5lroOQEHVEGBs
4X+ze/xyUOL6z3S07a85jNibYamDeoCDc5P0Vc6GWrdpsU7FQGXrHykNyglDxFJ1
MiYEQkB8NYDzQukl+7HPR3qPhQpAl5hx3XtmOcC5w0/88ATMqXg81DoVzPAPlsrL
IPu4ai7KjXRaY1sKo8SU4orj7iQHmmkiFJJg+QwVU9sO2GMBGpXZRSr3KcUL3ux5
nr+++ceVXyLADZaJRYp5LoTxL0KPJUKhaa9ABLmN2zQ5hT/v6AlQmKKD3s6ca02a
A8MQxy69hG50RVSeJm9yjRYQQBvATEXslCQXPXSAlLJGrPZ1FZgQdYyo2wNboD23
ep+JP2qTPdyTFFl2TOtoeR7fIsjg6DF5Bq0uh0maqC0UXXIo1GRO/OQGsNMfCN88
pevDg0GvE+bdeL8CEZYfDzu4aaUs+ltzZSPEKlXHCGFORL9iSuYhqdUCRPbcKYOy
uyhE7fgZoPYsOZLfChmXllEF69Cs5Vm5R0ymIgHqprfAjfqYf4ypbU0fukFDQ5dS
GGFTfxketTHdhYr7ATAfZg08ZMP819UnvcwISflLlDBwvpL4BDAWOrnDFnbHSugk
lPAxyPnfECM=
=h24Y
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Hans Reiser]

David Masover wrote:

>   I used the betas for months, and "metas"
> never burned me.

metas is much less likely than what clearcase uses to be hit 
accidentally (After all these years I forget what exactly clearcase 
special cases, maybe it was "@@").  When people using clearcase suffer a 
namespace collision, life goes on, no big deal, they structure a 
filename slightly differently and so what?  I mean, just how much do we 
suffer from not being able to use '/' in filenames?  Every once in a 
while it is annoying, but not that much....

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Dave Kleikamp <shaggy@austin.ibm.com> said:
>> >Spam <spam@tnonline.net> said:

> [...]

>> > You're missing the point.  We don't need transparency in all apps.  You
>> > can write an application to be as transparent as you want, but you don't
>> > need every app to to understand every file format.

>>   No, but not every user "can write an application" either, or even
>>   have the skills to apply patches. What I was talking about wasn't
>>   just tar, which itself isn't the best example anyway, but the idea
>>   that users can load plugins that will extend the functionality of
>>   their filesystems. That idea seem to be to be _much_ better than
>>   trying to teach every user how to write applications or patch
>>   existing ones.

> Why compare "write application or apply patches" to "load plugin"? It
> would be locical to compare running applications with loading plugins (and
> even so, loading plugins is presumably root-only).

  Doesn't matter if it is root-only. Ask the admin about the specific
  plugin the user wants. For most other users they can su to root and
  do this on their own computers. Not everyone is using Linux is a
  corporate world.

> [...]

>> > There are userland tools that deal with hundreds of file formats. Use
>> > the tool you need, rather than try to have the kernel do everything.

>>   No, but if I wanted to have an encryption plugin active for some of
>>   my files or directories then why should I not be able to? I still
>>   want to edit, view and save my encrypted files.

> Use an editor that knows about encrypted files. Decrypt/edit/encrypt if no
> other option (I'm sure emacs can be coerced to do that transparently ;-). I
> for one would be _way_ more confortable with my users doing that than them
> loading random modules into the kernel. Besides, if one of my users doesn't
> trust the system encryption programs, and prefers hers, she can be happy.

  That just doesn't do it. I doubt there will be an option to save
  encrypted Word and Excel files and be able to open them in Abiword,
  StarOffice or OpenOffice unless the decryption/encryption is done on
  a lower level.

  Also, eventually there may be a userland interface for loading
  certain modules without root access.?

>>   Again, this was just an example of what could be done with plugins.
>>   It is not said that every conceivable plugin will be written, nor
>>   loaded per default. Even though plugins cannot today be dynamically
>>   used, they will be eventually. Reiser4 is still very young.

> Modules of the kernel were supposed to have all those magic properties too,
> until there were nasty races... and it was _seriously_ considered to take
> them out. They stayed because they are root-only business, and (un)loading
> is rare. FS plugins are kernel modules, AFAIU, and are subject to the same
> problems.

>>   Please separate your thoughts for specific plugins from those of the
>>   idea to have plugins at all.

> If you can't find concrete uses for specific plugins, they are the
> proverbial solution searching for a problem.

  To me there are fine and good uses for plugins. Everyone seem to
  think different though.

  Sure many things can perhaps be done by linking and piping lots of
  programs, just to access some data. But that certainly is far from
  user friendly.

  ~S

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dave Kleikamp wrote:
| On Thu, 2004-09-02 at 19:39, Spam wrote:
|
|>
|>
|>
|>>On Fri, 3 Sep 2004, Spam wrote:
|>
|>>> Yes, some archive types can't be partially unzipped either. But my
|>>> point is that it wouldn't be transparent to the application/user in
|>>> the same way.
|>
|>>It doesnt matter whether it is transparent to the application. It can
|>>be the application which implements the required level of
|>>transparency.
|>
|>>User doesnt care what provides the transparency or how it's
|>>implemented.
|>
|>  Indeed. I hope I didn't say otherwise :). Just that I think it will
|>  be very difficult to have this transparency in all apps.
|
|
| You're missing the point.  We don't need transparency in all apps.  You
| can write an application to be as transparent as you want, but you don't
| need every app to to understand every file format.

You don't need every app to understand every filesystem, either.  Come
to think of it, why would I want anything other than CD burning software
and file browsers to understand a CD drive?

Let's all just wait -- quietly -- for the powers that be to work out
whether it's feasable and sane to support it in all apps.  Because
wouldn't that be better?

|> Just
|>  thinking of "nano file.jpg/description.txt" or "ls
|>  file.tar/untar/*.doc".
|
|
| I don't do much image editting, but I'm sure there are applications that
| let you edit the description in a text file.  You can even create a
| script that extracts it, runs nano, and puts it back into the jpeg.

That is a PITA.  Because you'll need to make more scripts.  And still more.

Say you want to grep for a jpeg with a particular description.  Say you
want to copy the description from one jpeg into another.

Maybe you could make a general-purpose command, like "jpeg_run", which
runs a command on the jpeg description, but it'd still be hackish, slow,
and more to type and remember.  Consider that you might have no idea
what "jpeg_run" is called, but you can always do "ls file.jpeg/metas" to
find out how to edit it.
|
| This works for me:
| tar -tf file.tar | grep '\.doc'

And then you need to run "tar -tf" a minute later, this time looking for
*.xls.  Maybe file.tar is actually a several gigabyte file.tar.bz2.

|> Sure in some environments like Gnome it could
|>  work, but it still doesn't for the rest of the flora of Linux
|>  programs.
|
|
| Just choose the right program.  tar groks tar files, not ls.

tar groks tar.  bzip2 groks bz2.  gzip groks gz.  "mount -o loop" groks
images.  zip groks zip.  rar groks rar.  openoffice groks .sxw, unless
staroffice does.  xmms can read id3 info from mp3s.  gcc compiles C.

If the file is an object, it's easier.  You can ask the file what it is,
and what you can do with it.  And you can tell it to do certain standard
things.  You can do all of this while reading almost no documentation on
what the file is.  At the end of the day, it won't matter to you whether
a file is zip, rar, or tar -- it's an archive, and you can extract it by
copying files out of its /contents directory.

Your way (the traditional way) means you have to learn to use which
program is the right program and how to use that program, and you'll
have to remember it constantly.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTkIq3gHNmZLgCUhAQJaNRAAnp/o0Wr+SJK5tkgYFCuc+CKIP+eALJo/
9wHnqK0nLdnji+vG0Czd9TUj1vWtoMrUichAwFoguMHHg3VeZGu61YwZoZ4idLNM
QbZ+CuQdUygNmyT0byGMFemP+cSbyvff1PRMy2BlSHKW3gUhvnQyggLGVKxpMRWf
VNnEHvkvJeA9PpEm6QGi1VRNp5bc0+Ocl4kO4CJk5ZYZ9D+BV6NwN/MZwqwlsu+Y
RZsYYEa6mLiCnU4rEo0tEAvvwMdC0e/9s9TQMcmJbT6JnybkWIRFMrTS4pmSabKg
uYGG9p1WrX8/V8WRNnaodlvx35gRPQj5S5SWDBoSWr999nmq31Y5RZ9QwbVj0d+U
yB4yNu0NpvFBJfwg8nVIKUa+bhPLCkdY5w+GnlEYGweSN20FYOaLiqiPtXBqJm8a
PP+8OCL35zy+1X7t/tq+JG/K91fYbECPR/qrAyHDXzNSuxdqidvBjfsEPBHOGwu6
RpsPoKyEvkzdXgauotWbVzWLt1ijGmYd/8Uk19OnmiloggViQhUWAJTIVGcvMNt8
Tnk8ESQMzGHbbVOIu3gB6DZD6P0IEEN4L6+gOZWoYqe2nvaYKvIL1My/i4BMU5Ml
cxDkzzeq58iVK07Has8lwgFp0U9iD1LrbIT5p0ZJn/52EXqlX4dJCYSSNaHzUqE8
hcLG4aDELEI=
=Wcyv
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Stephan von Krawczynski]

On Fri, 03 Sep 2004 19:13:31 -0500
David Masover <ninja@slaphack.com> wrote:

> | Just choose the right program.  tar groks tar files, not ls.
> 
> tar groks tar.  bzip2 groks bz2.  gzip groks gz.  "mount -o loop" groks
> images.  zip groks zip.  rar groks rar.  openoffice groks .sxw, unless
> staroffice does.  xmms can read id3 info from mp3s.  gcc compiles C.
> 
> If the file is an object, it's easier.  You can ask the file what it is,
> and what you can do with it.  And you can tell it to do certain standard
> things.  You can do all of this while reading almost no documentation on
> what the file is.  At the end of the day, it won't matter to you whether
> a file is zip, rar, or tar -- it's an archive, and you can extract it by
> copying files out of its /contents directory.
> 
> Your way (the traditional way) means you have to learn to use which
> program is the right program and how to use that program, and you'll
> have to remember it constantly.

Just a short input from someone listening quite a while to this thread:
I think your approach to the problem leads actually nowhere. The reason for
this is implicit in your own explanation. _Currently_ you talk of archives, but
as soon as you got that, what's next?
Your idea needs abstraction, then you see the problem more clearly:
An archive is only some sort of file type, just as (you already named) mp3 or
.sxw is another type. If you really want to do something _generally_ useful you
should think of a method to parse and use all kinds of filetypes and create an
interface for that.
And one thing is clear: as there are numerous different types you cannot pull
all that code inside the kernel. Obviously there should be a way some
application can install a hook, a helper, a plugin or whatever to provide
extended functionality on its special filetypes. If you don't want to use tar,
you don't need the plugin either.
If you want tar, you should (as a user) be able to install the "fs-plugin"
(just a name, do not shoot me for it) together with tar as an application. You
get the idea? Obviously this must be possible during runtime, everything else
is senseless.
So, please, if you go on investigating this whole stuff do it in a more general
way, because "tar inside kernel" is not really what your idea is all about, no?
And one last obvious thing: if one doesn't want this kind of file-handling, he
should not need it. It has to be an add-on functionality, not a replacement.

Regards,
Stephan

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Stephan von Krawczynski wrote:
| On Fri, 03 Sep 2004 19:13:31 -0500
| David Masover <ninja@slaphack.com> wrote:
|
|
|>| Just choose the right program.  tar groks tar files, not ls.
|>
|>tar groks tar.  bzip2 groks bz2.  gzip groks gz.  "mount -o loop" groks
|>images.  zip groks zip.  rar groks rar.  openoffice groks .sxw, unless
|>staroffice does.  xmms can read id3 info from mp3s.  gcc compiles C.
|>
|>If the file is an object, it's easier.  You can ask the file what it is,
|>and what you can do with it.  And you can tell it to do certain standard
|>things.  You can do all of this while reading almost no documentation on
|>what the file is.  At the end of the day, it won't matter to you whether
|>a file is zip, rar, or tar -- it's an archive, and you can extract it by
|>copying files out of its /contents directory.
|>
|>Your way (the traditional way) means you have to learn to use which
|>program is the right program and how to use that program, and you'll
|>have to remember it constantly.
|
|
| Just a short input from someone listening quite a while to this thread:
| I think your approach to the problem leads actually nowhere. The
reason for
| this is implicit in your own explanation. _Currently_ you talk of
archives, but
| as soon as you got that, what's next?

Anything sane.  Currently I talk of archives, because I need examples,
and archives have been a good example for awhile of something that can
gain things from plugins which they cannot get anywhere else, which have
implications immediately after the archive plugin is wrote, without
requiring any application support.

| Your idea needs abstraction, then you see the problem more clearly:
| An archive is only some sort of file type, just as (you already named)
mp3 or
| .sxw is another type. If you really want to do something _generally_
useful you
| should think of a method to parse and use all kinds of filetypes and
create an
| interface for that.

Fine.  It'd certainly be nice to use vim to edit an id3 tag, or a text
portion of a .sxw document (openoffice is HUGE compared to vim).

| And one thing is clear: as there are numerous different types you
cannot pull
| all that code inside the kernel. Obviously there should be a way some

I never said it should all go in the kernel.  I repeatedly said that
most of it should not -- all that goes in the kernel is the most generic
interface that is sane.  Obviously an archive plugin is different than a
compression plugin, but once you've got the archive plugin, the rest
(individual format support) can go in userland.

| application can install a hook, a helper, a plugin or whatever to provide
| extended functionality on its special filetypes. If you don't want to
use tar,
| you don't need the plugin either.

So you disable it at compile time.  I don't want to use XFS, and I
certainly don't want to use ATI's video drivers.  The latter I choose
not to download, and the former I choose not to activate.

| If you want tar, you should (as a user) be able to install the "fs-plugin"

If by "as a user" you mean "not root", I have to disagree.  I think it'd
have security implications.  It'd be great if I'm wrong.

| (just a name, do not shoot me for it) together with tar as an
application. You
| get the idea? Obviously this must be possible during runtime,
everything else
| is senseless.

I would love for this same attitude of "runtime or bust" to be more
common.  I would love for there to be a generic way to write a userland
fs plugin, just as there's a generic way to write a userland filesystem
(lufs).  Just understand that some plugins will be in userland, and some
will be in the kernel.

And, in fact, the part of the archive plugin that knows about
tar/zip/rar/whatever goes in userspace.  But the part that does caching
probably goes in kernel space, even if it becomes a more generic
"caching plugin".

| So, please, if you go on investigating this whole stuff do it in a
more general
| way, because "tar inside kernel" is not really what your idea is all
about, no?

Absolutely not.

| And one last obvious thing: if one doesn't want this kind of
file-handling, he
| should not need it. It has to be an add-on functionality, not a
replacement.

For awhile, yes.  But I can imagine that sometime in the future, an app
would need this kind of functionality.  And why not?  Is it so much
worse for an app to insist on "mp3 metadata plugin" than to insist on
"libid3"?

In any case, look at the current implementation.  I can do
	echo 750 > file/metas/rwx
but I can also use chmod, which is currently much more efficient.  I
think Hans wants to eventually create an efficient interface to access
lots of small files at once (or maybe this is done?) -- and his dream is
to see things like chmod replaced with a call to this interface.  But at
best, chmod will become a library call sometime years from now.  It will
never go away entirely.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQToI+3gHNmZLgCUhAQKEWg/9F0FZGxB2yXoo9l5UKI7Fvh19RlnpJvZE
D3DajNqxXZ5smYIba7QxATEwMLWNMwOdiJTDIcQ7jylbD3hTkp0uFQLXcS5FeWt5
yxA0oDYEHsW14aQov4pHGplHVvCHSkm6vJn/zU91QdJAZVYnH4na5NFuWsAIoUWi
5vigYTefavabFv0/RnyPCN6zFfn3hrjOnKqSELUXXleG/fV+XJVw3Nyg4DSe1hK+
xmXHFWkGC832tH052EH/CFlE17/K05QdpfBKYNzCFA5dvTtCs11OQZSggtGVipmn
cSycyDHtJraMdF78Qa8LZzjGiZbT/3befM6wrnekcwhjxCqok8CSbOvbVh2ENtiu
klNtHeQEEtm+4MhbEEkdiiuWaU7EHIFJFGYC6LJE0B4DAKDdi4TM9QbqxLW8HghQ
WUJfMh8H2FtqotjMsdDW0OCmSuzh0B4ZZ9EXvSMIttXBJ7T2+q442hm4PAFQWXSO
FPCAiUgGZ+ZCbyZI1/pjM+CGblMTCfCBOm8UI3MSjTTI7cVzWFrQHJSAUhCPIsxA
9RZlEvmg+ElbnGrJkpDggpToDFN5uHXj41r4CZBnXgjfUdA0jmHnWpzO29Smfu7i
cJPtN+LzR1Se+fz4IvYamWd1p/Mi3vQACPt4r3IGVKa27tYXzDw+CNa+CliHnczt
lAhISMnhoVo=
=7rV6
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Paul Jakma]

On Fri, 3 Sep 2004, Spam wrote:

>  Indeed. I hope I didn't say otherwise :).

Sure.

>  Just that I think it  will
>  be very difficult to have this transparency in all apps. Just
>  thinking of "nano file.jpg/description.txt" or "ls
>  file.tar/untar/*.doc". Sure in some environments like Gnome it could
>  work, but it still doesn't for the rest of the flora of Linux
>  programs.

"will it be transparent for all apps?", whether that's worth doing 
depends on the technical implications. Thankfully we have Al and 
Linus to make the judgement call on that ;)

Personally, I think that if GNOME can provide transparency for GNOME 
users, I think that's probably enough - unless there are literally no 
issues in adding some kind of VFS support.

The nano / ls /tar user is likely a very different user to the GNOME 
user. That user is also likely to appreciate the problems with 
backups and such more.

Anyway, userspace transparency is sufficient for most classes of 
users. Only reason to provide some kernel support is if it makes 
sense ("but not all apps can use GNOME transparency" not being one of 
those reasons).

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
Beat your son every day; you may not know why, but he will.

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Jakma wrote:
| On Fri, 3 Sep 2004, Spam wrote:
|
|>  Indeed. I hope I didn't say otherwise :).
|
|
| Sure.
|
|>  Just that I think it  will
|>  be very difficult to have this transparency in all apps. Just
|>  thinking of "nano file.jpg/description.txt" or "ls
|>  file.tar/untar/*.doc". Sure in some environments like Gnome it could
|>  work, but it still doesn't for the rest of the flora of Linux
|>  programs.
|
|
| "will it be transparent for all apps?", whether that's worth doing
| depends on the technical implications. Thankfully we have Al and Linus
| to make the judgement call on that ;)

So far, the technical implications are mainly "does it create a serious
issue in the distant future" or "does some exotic new feature that
doesn't exist yet cause problems" and not "it's currently broken".

Right now, I can edit a file's permissions, transparently, from any app,
and I haven't noticed any stability issues at all.

| Personally, I think that if GNOME can provide transparency for GNOME
| users, I think that's probably enough - unless there are literally no
| issues in adding some kind of VFS support.

Only it can't.  Especially if it's a typical GNOME user, who gets used
to having their files encrypted in foo.tar.pgp, say.  Works in abiword,
works in gedit, breaks in OpenOffice.  Not good.

| The nano / ls /tar user is likely a very different user to the GNOME
| user. That user is also likely to appreciate the problems with backups
| and such more.

I'm a vim/ls/tar user.  And I believe the problems with backups to be
very minor.  There are other issues that I don't understand as
thoroughly, so I'll leave them to Al and Linus.

| Anyway, userspace transparency is sufficient for most classes of users.
| Only reason to provide some kernel support is if it makes sense ("but
| not all apps can use GNOME transparency" not being one of those reasons).

Can you justify why "not all apps can use GNOME transparency" is not a
valid reason?  Would you still think so if GNOME transparency was the
only way to read isofs?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTkCL3gHNmZLgCUhAQJVSxAAiFd+e/Q0VjAvPaQL1IDwA56SP7LF2M1v
jnk5wehjvicPEfSn2HpWik7Pr55K2/7dUn6mEJO4N43x/zmfyiA2wXLibfWJtD3X
VgtFMgbOeKu+WobfdnvlBPAKYoZ+MYsC2jyDYwOHguJbR9epmFtHv0mMhmD60s+B
3JDLaYXwPjmjs7zezy2B/Xc3mgm060VgDZRVOnrYZLzHAi3dLvZLzGjORf5kBZtd
NtsiPUZw3oIqORBHqfWDWUUi4irfzklZkiuYp44RNFZs0z0xvPrUJ1klXiJpbQ9t
HTkz9qRe0sxjRYfLcJzWxfahaNN+2SXyR5FFqkpzfUfyNuoOUMZDtin+ZM83qyVc
J/zRCZvWQOaZUtgas9KCCYnbwgCFLcDEE0xLLDRMGAMKeHOwCImU55ChTC/0HTas
Q7Bd0df456dAv+ktbdMRaznWeNOpW5rJWXtxNeWnncTYKG0ogXct9mD0+BEIssaK
qaEYUousp6ZRvOSlCCNCF5hOvfEuZLuAZe2Q8zKH39B4Hg7BMnmIOw6VPLmlLKds
pGNuIKfKjyzkxcdHwqGp36e27wizyKtCbZ58Zd0x1wnf47gkm8kgkY38tb0ct7Xf
Ch+s3+zLsOtVnnipnlGhIPEtxYv78DiqhGUc+pdZisi5VlfA4PTXZAAjoICeXA/H
EqfVU6yYsu8=
=ggxZ
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 148 bytes --]

Salut,

On Fri, Sep 03, 2004 at 02:39:06AM +0200, Spam wrote:
> "nano file.jpg/description.txt"

Is that supposed to do steganography?

			Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  

> Salut,

> On Fri, Sep 03, 2004 at 02:39:06AM +0200, Spam wrote:
>> "nano file.jpg/description.txt"

> Is that supposed to do steganography?

  No, not at all. I already use similar things in Windows this is why
  I suggested it here. Having a separate file for descriptions are not
  as convenient. Especially if you have a few hundred JPEGs and I want
  to quickly view some properties of the one you have just looked at.

  ~S

> 			Tonnerre

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Thu, 2 Sep 2004, Alan Cox wrote:
>
> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
> > and quite frankly, I think you can do the above pretty much totally in
> > user space with a small library and a daemon (in fact, ignoring security
> > issues you probably don't even need the daemon). And if you can prototype
> > it like that, and people actually find it useful, I suspect kernel support
> > for better performance might be possible.
> 
> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
> it since the late 1990's.

And nobody has asked for kernel support that I know of.

So either "it just works" in user space, or people haven't figured out the 
kernel could help them. Or decided it's not worth it, exactly because 
they'd still have to support systems/filesystems that can't be converted.

		Linus

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Thu, Sep 02, 2004 at 10:46:32AM -0700, Linus Torvalds wrote:
> > On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
> > > and quite frankly, I think you can do the above pretty much totally in
> > > user space with a small library and a daemon (in fact, ignoring security
> > > issues you probably don't even need the daemon). And if you can prototype
> > > it like that, and people actually find it useful, I suspect kernel support
> > > for better performance might be possible.
> > 
> > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
> > it since the late 1990's.
> 
> And nobody has asked for kernel support that I know of.
> 
> So either "it just works" in user space, or people haven't figured out the 
> kernel could help them. Or decided it's not worth it, exactly because 
> they'd still have to support systems/filesystems that can't be converted.

http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto
a kernel filesystem.  The code is horrible, but it shows that it can
be done.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Thu, 2 Sep 2004, Christoph Hellwig wrote:
> 
> http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto
> a kernel filesystem.  The code is horrible, but it shows that it can
> be done.

I do like the setup where the extended features are done as a "view" on 
top of some other filesystem, so that you can choose to _either_ access 
the raw (and supposedly stable, simply by virtue of simplicity) or the 
"fancy" interface. Without having to reformat the disk to a filesystem you 
don't trust, or you have other reasons you can't use (disk sharing with 
other systems, whatever).

It doesn't have to be "user", btw, in the sense that a lot of the normal 
code could be in kernel mode. Same way as Tux handling all the regular 
static requests entirely in kernel mode, but having the ability for 
calling down to apache..

		Linus

Re: The argument for fs assistance in handling archives

[Linh Dang]

Linus Torvalds <torvalds@osdl.org> wrote:

>
>
> On Thu, 2 Sep 2004, Christoph Hellwig wrote:
>>
>> http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs
>> onto a kernel filesystem.  The code is horrible, but it shows that
>> it can be done.
>
> I do like the setup where the extended features are done as a "view"
> on top of some other filesystem, so that you can choose to _either_
> access the raw (and supposedly stable, simply by virtue of
> simplicity) or the "fancy" interface. Without having to reformat the
> disk to a filesystem you don't trust, or you have other reasons you
> can't use (disk sharing with other systems, whatever).

It'd be something similar to what clearcase does (not that I like
clearcase, I hate it with a passion for other reasons!)

On such a system, one would have multiple virtual views mounted (by
root) under:

        /view/tar, /view/dpkg, /view/rpm, etc.

for every regular file /home/joe/blah.tar

the path /view/tar/home/joe/blah.tar/ is a directory where member of
the archives directly accessible.

old tools continue work as is. new tools can take a look on virtual
views for virtual access. 

Not sure how such a system would work with the dentry cache.

-- 
Linh Dang

Re: The argument for fs assistance in handling archives

[Giuseppe Bilotta]

Linh Dang wrote:
> On such a system, one would have multiple virtual views mounted (by
> root) under:
> 
>         /view/tar, /view/dpkg, /view/rpm, etc.
> 
> for every regular file /home/joe/blah.tar
> 
> the path /view/tar/home/joe/blah.tar/ is a directory where member of
> the archives directly accessible.
> 
> old tools continue work as is. new tools can take a look on virtual
> views for virtual access. 
> 
> Not sure how such a system would work with the dentry cache.

How does it cope with 'view withint view'? Say you have a 
.zipped ISO. Can you do

cd /view/iso/view/zip/home/joe/mycd.zip/mycd.iso/

?

-- 
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Alan Cox]

On Iau, 2004-09-02 at 18:46, Linus Torvalds wrote:
> > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
> > it since the late 1990's.
> 
> And nobody has asked for kernel support that I know of.

I asked our desktop people. They want something like inotify because
dontify doesn't cut it. They have zero interest in the multiple streams
and hiding icons in streams type stuff.

Alan

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Thu, 2 Sep 2004, Alan Cox wrote:
> 
> I asked our desktop people. They want something like inotify because
> dontify doesn't cut it.

Well, dnotify() really _is_ inotify(), since it does actually work on 
inodes, not dentries.

I think what they are really complaining about is that dnotify() only 
notifies the _directory_ when a file is changed, and they'd like it to 
notify the file itself too. Which is a one-liner, really.

Does the following make sense? (Totally untested, use-at-your-own-risk, 
I've-never-actually-used-dnotify-in-user-space, whatever).

		Linus

===== fs/dnotify.c 1.17 vs edited =====
--- 1.17/fs/dnotify.c	2004-08-09 18:45:22 -07:00
+++ edited/fs/dnotify.c	2004-09-02 13:21:26 -07:00
@@ -160,6 +160,8 @@
 	if (!dir_notify_enable)
 		return;
 
+	__inode_dir_notify(dentry->d_inode, event);
+
 	spin_lock(&dentry->d_lock);
 	parent = dentry->d_parent;
 	if (parent->d_inode->i_dnotify_mask & event) {

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Stephen Rothwell]

[-- Attachment #1: Type: text/plain, Size: 1236 bytes --]

On Thu, 2 Sep 2004 13:22:41 -0700 (PDT) Linus Torvalds <torvalds@osdl.org> wrote:
>
> Well, dnotify() really _is_ inotify(), since it does actually work on 
> inodes, not dentries.

The "d" stands for directory not dentry :-)

> I think what they are really complaining about is that dnotify() only 
> notifies the _directory_ when a file is changed, and they'd like it to 
> notify the file itself too. Which is a one-liner, really.

I don't think so, since this notify will only happen if the process has
registered for the notification and there is no way to register unless the
file is a directory ...

> Does the following make sense? (Totally untested, use-at-your-own-risk, 
> I've-never-actually-used-dnotify-in-user-space, whatever).

I had intended to extend dnotify to do file notifies, but I think the
real killer is needing the keep the file open that you want to be
notified about when you want to be notified about lots of files ...

I think that is what inotify was trying to fix (but I haven't had a chance
to look at it recently).  It reminds me of omirr that we had many years
ago - I wonder what happened to it?

-- 
Cheers,
Stephen Rothwell                    sfr@canb.auug.org.au
http://www.canb.auug.org.au/~sfr/

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Bill Huey]

On Thu, Sep 02, 2004 at 07:46:05PM +0100, Alan Cox wrote:
> On Iau, 2004-09-02 at 18:46, Linus Torvalds wrote:
> > > Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
> > > it since the late 1990's.
> > 
> > And nobody has asked for kernel support that I know of.
> 
> I asked our desktop people. They want something like inotify because
> dontify doesn't cut it. They have zero interest in the multiple streams
> and hiding icons in streams type stuff.

It also depends on who you ask. I can't take a lot of the mainstream
X folks serious since they are still using integer math as parameters
to half broken drawing primitives and barely discovered things like OpenGL.
Their attitude doesn't treat these things as first class citizens in
what ever software system they create. They also haven't create a modern
and highly dynamic structured document system that's in wide use yet,
so this problem space hasn't really been pushed as hard as other much
more dynamic systems. And the advent of XML (basically a primitive and
flat model of what Hans is doing) for .NET style systems are going to
push these systems into those areas in new and unique ways. (Actually
retro Smalltalk-ish)

It seems that many of the original ideas about "why" GUI systems exist
have been lost to older commericial interests (Microsoft Win32) and that
has wiped out the fundamental classic computer science backing this from
history. This simple "MP3 metadata" stuff is a very superficial example
of how something like this is used.

The problems are fundamentally about data representation in a manner so
simple that its "expressive power" (Hans here) can extend itself to even
the dorkiest of shell scripts. To have that power immediately available
as network/local objects and to have their relationships clearly defined
is a very powerful manner to build software systems. 

Unix folks tend to forget that since they either have never done this
kind of programming or never understood why this existed in the first
place. It's about a top-down methodology effecting the entire design of
the software system, not just purity Unix. If it can be integrate
smoothly into the system, then it should IMO.

The folks against this system forget about how important the context of
all this is work is set in... The mindset is fundamentally different and
I'm quite sick of hearing "It's not Unix" over and over again. And
notion of Linux being marginalize to a minority OS over this stuff is
just plain crazy.
 
bill

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Alan Cox]

On Iau, 2004-09-02 at 22:56, Bill Huey wrote:
> It also depends on who you ask. I can't take a lot of the mainstream
> X folks serious since they are still using integer math as parameters

The X folks know what they are doing. Modern X has a complete
compositing model. Modern X has a superb font handling system. Nobody
broke anything along the way. The new API's can be mixed with the old,
there are good fallbacks for old servers.

In fact they are so good at it that most people don't notice beyond the
fact their UI looks better than before.

That is how you do change *right*

> more dynamic systems. And the advent of XML (basically a primitive and
> flat model of what Hans is doing) for .NET style systems are going to

I see you don't really get XML either. XML is just an encoding. Its
larger and prettier than ASN.1 and easier to hack about with perl. You
can do the same thing with lisp lists for that matter.

> have been lost to older commericial interests (Microsoft Win32) and that
> has wiped out the fundamental classic computer science backing this from
> history. This simple "MP3 metadata" stuff is a very superficial example
> of how something like this is used.

The trouble with computer science is that most of it sucks in the real
world. We don't write our OS's in Standard ML, we don't implement some
of the provably secure capability computing models. At the end of the
day they are neat, elegant and useless to real people.

> Unix folks tend to forget that since they either have never done this
> kind of programming or never understood why this existed in the first
> place. It's about a top-down methodology effecting the entire design of
> the software system, not just purity Unix. If it can be integrate
> smoothly into the system, then it should IMO.

The Unix world succeeded because Unix (at least in v7 days) was the
other way around to every other grungy OS on the planet. It had only
thing things it needed. I've used some of the grungy crawly horrors that
were its rivals and there is a reason they don't exist any more.

I would sum up the essence of the unix kernel side as
	- Does only what it must do
	- "Makes the usual easy makes the unusual possible"
	- Has an API that is small enough for developers to learn
	  easily (an API so good every other OS promptly ripped it off)
	  People forget the worlds of SYS$QIO, RMS, FCB's and the like

Its worked remarkably well for a very very long time, and most of the
nasties have come from people trying to break that model or not
understanding it.

Alan

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Bill Huey]

On Thu, Sep 02, 2004 at 11:48:10PM +0100, Alan Cox wrote:
> On Iau, 2004-09-02 at 22:56, Bill Huey wrote:
> > It also depends on who you ask. I can't take a lot of the mainstream
> > X folks serious since they are still using integer math as parameters
> 
> The X folks know what they are doing. Modern X has a complete
> compositing model. Modern X has a superb font handling system. Nobody
> broke anything along the way. The new API's can be mixed with the old,
> there are good fallbacks for old servers.

Keith Packard knows what he's doing. The rest that don't follow his lead
on these issues are useless or detracting and redirecting resources away
from critical problems that have been solved under other, more mature,
operating systems, specifically Apple's OS X.

Notice, that you didn't deflect the claim that folks are still using
integer math to draw things. This is critical...

> In fact they are so good at it that most people don't notice beyond the
> fact their UI looks better than before.
> 
> That is how you do change *right*

It's one way, but it's not top-level redesign heavily enough that you still
have a substandard rendering system. Notice that were wasn't a mention of
device independent rendering, WYSIWYG (what you get is what you see) in
their rasterization model, nor was there any mention of how a scalable
unified rasterization model used in an application. OpenStep ruled during
it's time and rendered and generate better PostScript than Macintosh
applications because of this unified rasterization model. Split development
and the lack of "connecting the dots" across systems is one of the reasons
why X has such poor app development. It takes way more time to do things
in this environment than in other system.

What folks X land are missing here is a notion of design verses piecewise
hacks to half emulate a critical function. The fault in this matter is
really spread out to multipule camps not doing their homework regarding
these matters.

Also, there's no mention of making OpenGL a first class citizen in this
system so that all compositing is hardware accelerate. What OS has this ?
You guess it Apple's OS X has this with full transparency and bitmap
backstore for region invalidating and redrawing. X currently has none of
this stuff. You could say that this is planned in the future, but there
has been no push for doing this in this aggressive a manner which is a
leadership failure of that community as a whole.

Also, the notion of color space, not just RGB, but CMYK is a first class
citizen in Display Postscript. X is also missing this. These aren't
things that you can gradually add in. It's about looking at the system
from the top-down and solving these problems using other systems that.

X is at best piecewise and substandard to just about any modern rendering
system that I've used.

> > more dynamic systems. And the advent of XML (basically a primitive and
> > flat model of what Hans is doing) for .NET style systems are going to
> 
> I see you don't really get XML either. XML is just an encoding. Its
> larger and prettier than ASN.1 and easier to hack about with perl. You
> can do the same thing with lisp lists for that matter.

I get it fine. What you don't get is something called Model View Controller
in that .NET applications have the ability to take that serialized structure,
apart of their entire object store/serialization system and transform
it into many views. The same widget data can be used not only to build HTML
for a web page, but also a standard component driven GUI system a native
windowing kit. This exploits polymorphism and how OOP should have been
done in the first place. This use of it is very important and obviously
useful in this case.

In MVC, all data has relationships and XML s the kind of DB format that these
systems use for object serialization. XML is the storage layer in these systems.
It's used for every object, remote or local.

This is why something like Reiser FS 4 with it's persistent object store
capability may have a critical influence on how these systems are built.
These systems are returning nothing but structured storage in the form
of XML analougs.

Your Perl/Python, what ever example here doesn't cut it and explicate
how XML based system are being generically used in this sense. It's too
technically shallow for this context.

> > have been lost to older commericial interests (Microsoft Win32) and that
> > has wiped out the fundamental classic computer science backing this from
> > history. This simple "MP3 metadata" stuff is a very superficial example
> > of how something like this is used.
> 
> The trouble with computer science is that most of it sucks in the real
> world. We don't write our OS's in Standard ML, we don't implement some
> of the provably secure capability computing models. At the end of the
> day they are neat, elegant and useless to real people.

Java and .NET are just those things that you have been talking about.
As these systems become more and more prevalent (they will) they're
influence is just the beginning. These system have pretty much happened
overnight and can't be ignored by kernel folks.

> > Unix folks tend to forget that since they either have never done this
> > kind of programming or never understood why this existed in the first
> > place. It's about a top-down methodology effecting the entire design of
> > the software system, not just purity Unix. If it can be integrate
> > smoothly into the system, then it should IMO.
> 
> The Unix world succeeded because Unix (at least in v7 days) was the
> other way around to every other grungy OS on the planet. It had only
> thing things it needed. I've used some of the grungy crawly horrors that
> were its rivals and there is a reason they don't exist any more.

Look, I agree with you for the most part about Unix, but these issues
have to be reexamined again after some period of time has elapsed.

I can only guess that some of these system are horribly implemented
minimally, but I can't comment about their original intentions since
I've personally never used them.
 
> I would sum up the essence of the unix kernel side as
> 	- Does only what it must do
> 	- "Makes the usual easy makes the unusual possible"
> 	- Has an API that is small enough for developers to learn
> 	  easily (an API so good every other OS promptly ripped it off)
> 	  People forget the worlds of SYS$QIO, RMS, FCB's and the like
> 
> Its worked remarkably well for a very very long time, and most of the
> nasties have come from people trying to break that model or not
> understanding it.

This is a difficult topic, but I don't how folks can dimiss something
like Reiser FS semantics and the importance of systems like this for
future software design across all application boundaries.

There has to be a way of getting this stuff in and keeping the traditional
semantics in place.

I was thinking about have a dual mounted file system, one with traditional
Unix semantics with read-only permissions, but then having it also being
mounted in a special namespace directory structure (something like /proc)
that would reveal other streams for "stream aware" applications with
read/write permissions.

It might solve both problems in this case. Don't know.

I've been on vacation for the last couple of days at Burning Man, which
is why this response has been delayed. :)

bill

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  



> On Thu, 2 Sep 2004, Alan Cox wrote:
>>
>> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
>> > and quite frankly, I think you can do the above pretty much totally in
>> > user space with a small library and a daemon (in fact, ignoring security
>> > issues you probably don't even need the daemon). And if you can prototype
>> > it like that, and people actually find it useful, I suspect kernel support
>> > for better performance might be possible.
>> 
>> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
>> it since the late 1990's.

> And nobody has asked for kernel support that I know of.

  Actually. Doesn't matter if it is in kernel or not for the users as
  long as it works.

  The problem is that I do not see either Gnome or KDE to ever get
  along to form one standard that everyone will use. Their libraries
  are huge and memory hogging which so many Linux users just do not
  like. What if a user doesn't want KDE or Gnome? Would all files
  created with either be broken?

  I doubt that something like file streams and meta-data can
  successfully be implemented purely in user-space and get the same
  support (ie be used by many programs) if this change doesn't come
  from the kernel. I just do not see it happen.

> So either "it just works" in user space, or people haven't figured out the
> kernel could help them. Or decided it's not worth it, exactly because
> they'd still have to support systems/filesystems that can't be converted.

> 		Linus

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Spam wrote:
[...]
|   I doubt that something like file streams and meta-data can
|   successfully be implemented purely in user-space and get the same
|   support (ie be used by many programs) if this change doesn't come
|   from the kernel. I just do not see it happen.

The issue is not "many programs".  The issue is "all programs".

Even if the political issues were solved -- even if Linus said to us all
"Thou shalt use this library or suffer my wrath!" -- it'd have to be
everywhere.  Bash.  Perl.  Make.  Gcc.  Vim.

And btw, if it was political, you'd get no sympathy here.  ("Oh no,
everyone's using their own game engine!  We need to put the doom3 engine
in the kernel, now!")  No, _people_ solve political problems.  Kernels
don't.

Kernel support automatically adds support for a lot of features without
patching a thing.

There should be an interface in the filesystem.  And for certain things,
uservfs will be incredibly slower than reiser4 as that interface.
(Remember Linus' point about TUX and Apache.)

I'll say this again:  most of it -- all but the bare interface stuff --
should be in userspace.  In fact, let's all add this to our signature so
no one brings it up again. ("Oh no, they want to put tar support in the
kernel!" no, we don't.)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTe4BXgHNmZLgCUhAQKH1g//WXaL5xdLpX37TdgFhXidiRJGe/ojehj3
CZ7kseI9GkOBSxHt/yb5/xC6r+XT7JLlvJZybT7HLIRIxGp+WQHHBOD/xezWC+eX
OyRaOlkZ7o9HRQKhKNRIAwI4jgftpLhFUhePgibubS4UdxtzN2FWuULfKvMKIGHn
L4Zv4Dpje5ld7l7ce8jhfcURJ7AgAPwja3Tc7C38pmG+dSo2mj0I+YlCUED7mx3R
ZSv6WtdAUCZjnKv9hSQVruk3fjYZc4dLEGzGH1ZJsD1ZkH5wNmWds5gHGEvQrc4Z
9reNanTxy+0ECxndk2H/ukw5Wv011rJWubLy/CnaPakPrSvrsmmoEs8ZcVZavlg9
ABJX/NtyBVl/y8+6Eh6/BdAhQr30U+c/UZLNbOflmcPGPiJCiXfBuaX1OF+qffQ1
QQvAGPgO2R9egHJWqFhBaLHtBAmiXSRWUU4+4nPBYZ/X5dCmGuV46knQGHdqoAQc
l/qILh+spY09q9g118QbdnXBseiuVh/a+vf2GrbxbEMuWQu1kAI0DJbN0KKgUdtE
ZkmIqXYULO6QZsYk3L41ZyKyE7oFMUqbT0uxSQZUCjOcnuBpMn/PzwM0yMJeDUKx
295Yzq5lkqkGmHJHi7XGOOI5XVIPb++DWXXv9E6Bfgoj4TSZscfwM69PQRoSq0hu
iL9VOiyLRBU=
=zrvZ
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 1399 bytes --]

Salut,

On Thu, Sep 02, 2004 at 09:50:31PM +0200, Spam wrote:
> Their  libraries are  huge and  memory hogging  which so  many Linux
> users just do not like.

This  is rather  a  fud argument:  both  the gnome  VFS  code and  the
KIOserver/KIOslave  code aren't really  large. You  don't want  to use
them for a busybox/tinylogin system, however.

> What if  a user doesn't want  KDE or Gnome? Would  all files created
> with either be broken?

The files  still work well, just  that you can't access  them over the
old fancy URL schemes.

> I  doubt  that  something   like  file  streams  and  meta-data  can
> successfully be  implemented purely in  user-space and get  the same
> support (ie  be used by many  programs) if this  change doesn't come
> from the kernel. I just do not see it happen.

Actually,  practical  discordianism.  If  you develop  a  common  API,
there'll always be people disagreeing.

GTK+ with all  its features is just cool. Desktop  warping is a really
nice  thing. But  there are  people out  there who  don't want  to use
it. They use QT, or even plain old Athena Widgets. So what? Will we be
implementing the X toolkits into the kernel?

In case of marketing it's up to the distributions to provide something
concise  so  everyone  can  use  their  programs  through  a  coherent
namespace. (I.e. port all the apps they ship to gnome-vfs or kio).

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  

> Salut,

> On Thu, Sep 02, 2004 at 09:50:31PM +0200, Spam wrote:
>> Their  libraries are  huge and  memory hogging  which so  many Linux
>> users just do not like.

> This  is rather  a  fud argument:  both  the gnome  VFS  code and the
> KIOserver/KIOslave  code aren't really  large. You  don't want  to use
> them for a busybox/tinylogin system, however.

  Then it is good. Just I see no programs other than Gnome or KDE apps
  that are using them.

>> What if  a user doesn't want  KDE or Gnome? Would  all files created
>> with either be broken?

> The files  still work well, just  that you can't access  them over the
> old fancy URL schemes.

  No only, but if you cp then with a non KDE/Gnome app then you will
  loose the meta-data and extra info too. That was my point.

>> I  doubt  that  something   like  file  streams  and  meta-data can
>> successfully be  implemented purely in  user-space and get  the same
>> support (ie  be used by many  programs) if this  change doesn't come
>> from the kernel. I just do not see it happen.

> Actually,  practical  discordianism.  If  you develop  a  common  API,
> there'll always be people disagreeing.

> GTK+ with all  its features is just cool. Desktop  warping is a really
> nice  thing. But  there are  people out  there who  don't want  to use
> it. They use QT, or even plain old Athena Widgets. So what? Will we be
> implementing the X toolkits into the kernel?

  This is certainly not what I said or wanted.

> In case of marketing it's up to the distributions to provide something
> concise  so  everyone  can  use  their  programs  through  a  coherent
> namespace. (I.e. port all the apps they ship to gnome-vfs or kio).

  Do you really believe this will happen? Good if it did. I do not
  believe it. And I certainly do not see the thousands of man hours
  needed to actually provide all the patches as a benefit.

  ~S

> 			    Tonnerre

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 1257 bytes --]

Salut,

On Mon, Sep 06, 2004 at 10:05:34AM +0200, Spam wrote:
> Then it is good. Just I see no programs other than Gnome or KDE apps
> that are using them.

Because KDE people  hate Gnome people and vice  versa, and because the
rest of the world just neglects the two races for political reasons.

Maybe   the  Freedesktop  project   should  provide   some  convenient
specification/code to  do it.  Like they  do for HAL  and DBUS (Please
note that this is something  interesting because it does clever things
on hardware without requiring to patch the kernel.)

> > In  case of  marketing it's  up  to the  distributions to  provide
> > something  concise so everyone  can use  their programs  through a
> > coherent namespace. (I.e. port all the apps they ship to gnome-vfs
> > or kio).
> 
> Do you really believe this will happen?

If the distributors  really want to be able to gain  money, and if the
Free Unix community wants to  gain a significant market share, this is
supposed to  happen. It's  the question of  whether we can  ignore our
childish concept  wars, or if we're  always going to stay  at that low
level we're at now.

Actually, this can't be fixed by putting everything into the kernel.

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives

[Dr. Giovanni A. Orlando]

Tonnerre wrote:

>Salut,
>
>On Mon, Sep 06, 2004 at 10:05:34AM +0200, Spam wrote:
>  
>
>>Then it is good. Just I see no programs other than Gnome or KDE apps
>>that are using them.
>>    
>>
>
>Because KDE people  hate Gnome people and vice  versa, and because the
>rest of the world just neglects the two races for political reasons.
>  
>
Hi,

    This is completely wrong. Neither KDE people hate GNOME people
    nor viceversa.

    Time ago, I repeat in a conference the GNOME people need to jump
    on the KDE wagon, so the train will move to a single direction, and
    people does not approve this comment.

    More than on the license, QPL, there are a developer problem.

    Develop in C (Gtk, GNOME) is a lot more easy than develop in C++ 
(Qt, KDE)
   
    But, for the GUI, the C++ approach is a lot superior ... a lot.

    The problem is this. They start to develop on Gtk to create GNOME, in C
    for a license problem.

    Now, they said why I need to change?

    But they need to change, because it is supeiror. Everyone may of 
course wants to do what prefer.

>Maybe   the  Freedesktop  project   should  provide   some  convenient
>specification/code to  do it.  Like they  do for HAL  and DBUS (Please
>note that this is something  interesting because it does clever things
>on hardware without requiring to patch the kernel.)
>
>  
>
I don't agree FreeDesktop.org, because it is handled by a RedHat employee
and the code is made on Gtk2, generally.

... They are not neutral.

Thanks,
Giovanni

>>>In  case of  marketing it's  up  to the  distributions to  provide
>>>something  concise so everyone  can use  their programs  through a
>>>coherent namespace. (I.e. port all the apps they ship to gnome-vfs
>>>or kio).
>>>      
>>>
>>Do you really believe this will happen?
>>    
>>
>
>If the distributors  really want to be able to gain  money, and if the
>Free Unix community wants to  gain a significant market share, this is
>supposed to  happen. It's  the question of  whether we can  ignore our
>childish concept  wars, or if we're  always going to stay  at that low
>level we're at now.
>
>Actually, this can't be fixed by putting everything into the kernel.
>
>			    Tonnerre
>  
>


-- 

-- 

--
Check FT Websites ... 
http://www.futuretg.com  - ftp://ftp.futuretg.com
http://www.FTLinuxCourse.com
	http://www.FTLinuxCourse.com/Certification
http://www.rpmparadaise.org
http://GNULinuxUtilities.com
http://www.YourPersonalOperatingSystem.com

--

Re: The argument for fs assistance in handling archives

[Grzegorz Jaśkiewicz]

On Mon, 06 Sep 2004 11:15:59 +0200, Dr. Giovanni A. Orlando
<gorlando@futuretg.com> wrote:
> Tonnerre wrote:
> >Because KDE people  hate Gnome people and vice  versa, and because the
> >rest of the world just neglects the two races for political reasons.
>     This is completely wrong. Neither KDE people hate GNOME people
>     nor viceversa.
> 
>     Time ago, I repeat in a conference the GNOME people need to jump
>     on the KDE wagon, so the train will move to a single direction, and
>     people does not approve this comment.
> 
>     More than on the license, QPL, there are a developer problem.

KDE is covered with LGPL/GPL licence, so is QT, so I don't see a problem here.

 
>     Develop in C (Gtk, GNOME) is a lot more easy than develop in C++
> (Qt, KDE)
Well, some ppl knowing well C++ will say something completly different.
Since I've started QT/KDE development, my life as developer has changed.
I am developing everything in QT, as far as desktop stuff goes. And
it's much more clear and easier.
You just need to lear C++ well ;)
 
>     But, for the GUI, the C++ approach is a lot superior ... a lot.
> 
>     The problem is this. They start to develop on Gtk to create GNOME, in C
>     for a license problem.
> I don't agree FreeDesktop.org, because it is handled by a RedHat employee
> and the code is made on Gtk2, generally.
> 
> ... They are not neutral.
Agreed.

-- 
GJ

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Thu, Sep 02, 2004 at 10:46:05AM +0100, Alan Cox wrote:
> On Mer, 2004-09-01 at 21:50, Linus Torvalds wrote:
> > and quite frankly, I think you can do the above pretty much totally in
> > user space with a small library and a daemon (in fact, ignoring security
> > issues you probably don't even need the daemon). And if you can prototype
> > it like that, and people actually find it useful, I suspect kernel support
> > for better performance might be possible.
> 
> Gnome already supports this in the gnome-vfs2 layer. "MC" has supported
> it since the late 1990's.

Can it do this:

	cd FC2-i386-disc1.iso
	ls

or this:

	cd /dev/sda1
	ls
	cd /dev/floppy
	ls
	cd /dev/cdrom
	ls

?

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Dave Kleikamp]

On Thu, 2004-09-02 at 15:38, Frank van Maarseveen wrote:
> Can it do this:
> 
> 	cd FC2-i386-disc1.iso
> 	ls
> 
> or this:
> 
> 	cd /dev/sda1
> 	ls
> 	cd /dev/floppy
> 	ls
> 	cd /dev/cdrom
> 	ls
> 
> ?

We have the mount command for that.  :^)
-- 
David Kleikamp
IBM Linux Technology Center

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Thu, Sep 02, 2004 at 04:36:34PM -0500, Dave Kleikamp wrote:
> On Thu, 2004-09-02 at 15:38, Frank van Maarseveen wrote:
> > Can it do this:
> > 
> > 	cd FC2-i386-disc1.iso
> > 	ls
> > 
> > or this:
> > 
> > 	cd /dev/sda1
> > 	ls
> > 	cd /dev/floppy
> > 	ls
> > 	cd /dev/cdrom
> > 	ls
> > 
> > ?
> 
> We have the mount command for that.  :^)

mount is nice for root, clumsy for user. And a rather complicated
way of accessing data the kernel has knowledge about in the first
place. For filesystem images, cd'ing into the file is the most
obvious concept for file-as-a-dir IMHO.

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[viro]

On Thu, Sep 02, 2004 at 11:48:06PM +0200, Frank van Maarseveen wrote:
> mount is nice for root, clumsy for user. And a rather complicated
> way of accessing data the kernel has knowledge about in the first
> place. For filesystem images, cd'ing into the file is the most
> obvious concept for file-as-a-dir IMHO.

The hell it is.

a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
b) kernel has no way to guess which options to use
c) fs _type_ is a fundamental part of mount - device(s) (if any) involved
are arguments to be interpreted by that particular fs driver.
d) permissions required for that lovely operation (and questions like
whether we force nosuid/noexec, etc.) are nightmare to define.

Frankly, the longer that thread grows, the more obvious it becomes that
file-as-a-dir is a solution in search of problem.  Desperate search, at
that.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Thu, Sep 02, 2004 at 11:00:27PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> The hell it is.
> 
> a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.

excuse me, but how does the kernel mount the root fs?

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Fri, Sep 03, 2004 at 12:02:42AM +0200, Frank van Maarseveen wrote:
> On Thu, Sep 02, 2004 at 11:00:27PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> > 
> > The hell it is.
> > 
> > a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
> 
> excuse me, but how does the kernel mount the root fs?

trial and error.  That's why you see all thos ext3 mounted as ext2
problems, or $RANDOMFS as fat.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Fri, Sep 03, 2004 at 12:03:54AM +0200, Christoph Hellwig wrote:
> On Fri, Sep 03, 2004 at 12:02:42AM +0200, Frank van Maarseveen wrote:
> > On Thu, Sep 02, 2004 at 11:00:27PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> > > 
> > > The hell it is.
> > > 
> > > a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
> > 
> > excuse me, but how does the kernel mount the root fs?
> 
> trial and error.  That's why you see all thos ext3 mounted as ext2
> problems, or $RANDOMFS as fat.

Andb btw, for an lkml discussion RTFS wouldn't hurt.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[viro]

On Fri, Sep 03, 2004 at 12:02:42AM +0200, Frank van Maarseveen wrote:
> On Thu, Sep 02, 2004 at 11:00:27PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> > 
> > The hell it is.
> > 
> > a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
> 
> excuse me, but how does the kernel mount the root fs?

By trying all fs types it has registered in a more or less random (OK, defined
by order of fs type registration, which is kinda-sorta deterministic at
boot time) order.  With no flags, unless you pass them explicitly in kernel
command line.  Fs types list can also be set explicitly in the command line.

Next question?

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Thu, Sep 02, 2004 at 11:06:40PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> On Fri, Sep 03, 2004 at 12:02:42AM +0200, Frank van Maarseveen wrote:
> > > a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
> > 
> > excuse me, but how does the kernel mount the root fs?
> 
> By trying all fs types it has registered in a more or less random (OK, defined
> by order of fs type registration, which is kinda-sorta deterministic at
> boot time) order.  With no flags, unless you pass them explicitly in kernel
> command line.  Fs types list can also be set explicitly in the command line.

Of course I know that: the point is, the kernel _has_ knowlegde contrary
to what you blatantly said.

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[viro]

On Fri, Sep 03, 2004 at 12:11:33AM +0200, Frank van Maarseveen wrote:
> On Thu, Sep 02, 2004 at 11:06:40PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> > On Fri, Sep 03, 2004 at 12:02:42AM +0200, Frank van Maarseveen wrote:
> > > > a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
> > > 
> > > excuse me, but how does the kernel mount the root fs?
> > 
> > By trying all fs types it has registered in a more or less random (OK, defined
> > by order of fs type registration, which is kinda-sorta deterministic at
> > boot time) order.  With no flags, unless you pass them explicitly in kernel
> > command line.  Fs types list can also be set explicitly in the command line.
> 
> Of course I know that: the point is, the kernel _has_ knowlegde contrary
> to what you blatantly said.

What knowledge does the kernel have about fs type that could deal with the
contents of given device?  Details, please.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Thu, Sep 02, 2004 at 11:17:22PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> What knowledge does the kernel have about fs type that could deal with the
> contents of given device?  Details, please.

Try a "make tags;grep SUPER_MAGIC tags".
Or is it there for a different purpose?

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[viro]

On Fri, Sep 03, 2004 at 12:26:50AM +0200, Frank van Maarseveen wrote:
> On Thu, Sep 02, 2004 at 11:17:22PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> > 
> > What knowledge does the kernel have about fs type that could deal with the
> > contents of given device?  Details, please.
> 
> Try a "make tags;grep SUPER_MAGIC tags".
> Or is it there for a different purpose?

RTFS and you'll see.  Individual fs generally knows how to check if it
would be immediately unhappy with given image (not all types do, BTW).
Exact form of checks depends on fs type; for crying out loud, there's
not even a promise that they are mutually exclusive!

Read the fucking source.  Read through the code that "chooses" fs type
of root fs.  Look at it.  Then use whatever you have between your ears.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Thu, Sep 02, 2004 at 11:33:24PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> RTFS and you'll see.  Individual fs generally knows how to check if it
> would be immediately unhappy with given image (not all types do, BTW).
> Exact form of checks depends on fs type; for crying out loud, there's
> not even a promise that they are mutually exclusive!

so?

A user can stick an USB memory card with _any_ malformed fs data and
make troubles via the automounter or user mounts. Yes, mount might do
some more checks but it sure won't do an fsck.

The user gets what he deserves when sticking crap in an USB port.

And that doesn't mean that the kernel should accept any fs image
when a user tries to cd into the file.

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[viro]

On Fri, Sep 03, 2004 at 12:56:34AM +0200, Frank van Maarseveen wrote:
> And that doesn't mean that the kernel should accept any fs image
> when a user tries to cd into the file.

No, it shouldn't.  It should say "it's not a directory, bugger off".
Which it does.

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Helge Hafting]

Frank van Maarseveen wrote:

>On Thu, Sep 02, 2004 at 11:33:24PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
>  
>
>>RTFS and you'll see.  Individual fs generally knows how to check if it
>>would be immediately unhappy with given image (not all types do, BTW).
>>Exact form of checks depends on fs type; for crying out loud, there's
>>not even a promise that they are mutually exclusive!
>>    
>>
>
>so?
>
>A user can stick an USB memory card with _any_ malformed fs data and
>make troubles via the automounter or user mounts. Yes, mount might do
>some more checks but it sure won't do an fsck.
>
>The user gets what he deserves when sticking crap in an USB port.
>
>And that doesn't mean that the kernel should accept any fs image
>when a user tries to cd into the file.
>  
>
You don't need kernel support for cd'ing into fs images.
You need a shell (or GUI app) that:
1. notices that user tries to CD into a file, not a directory
2. Attempts fs type detection and do a loop mount.
3. Give error message if it wasn't a supported fs image.

Helge Hafting

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Fri, Sep 03, 2004 at 10:43:44AM +0200, Helge Hafting wrote:
> >
> You don't need kernel support for cd'ing into fs images.
> You need a shell (or GUI app) that:
> 1. notices that user tries to CD into a file, not a directory
> 2. Attempts fs type detection and do a loop mount.
> 3. Give error message if it wasn't a supported fs image.

Ok, and right now I'm in vim typing this message and want
to ":new /tmp/backup.iso/etc/fstab"

modifying 1000+ applications is not an option IMO. Putting it in
a preloaded library might be doable except maybe for permission
problems but this is incredibly clumsy and lacks the possibility
to implement sane caching behavior because it is process bound.

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Fri, Sep 03, 2004 at 10:50:18AM +0200, I wrote:
> and lacks the possibility
> to implement sane caching behavior because it is process bound.

sorry, this part is crap.

-- 
Frank

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Helge Hafting wrote:
[...]
| You don't need kernel support for cd'ing into fs images.
| You need a shell (or GUI app) that:
| 1. notices that user tries to CD into a file, not a directory
| 2. Attempts fs type detection and do a loop mount.
| 3. Give error message if it wasn't a supported fs image.

You can argue this about FS plugins, too.  You don't need kernel support
for cryptocompress, for example.  Just have a fifo farm.  Every file
that has compression turned on is actually a named pipe, the original
file is hidden (starts with a dot), and you have a script running to
each fifo which runs zcat.

The kernel support (read: interface) is just cleaner in some ways.

In the loop mount example, it isn't universal -- the user can't CD into
a file if they use the wrong shell, and two different shells might make
two different mounts.  How many mounts before we run out of loop devices?

In the cryptocompress example, there's no caching or compress-on-flush
optimization, and it isn't entirely transparent to anything.  It doesn't
look like a file, it looks like a fifo.  ls will make it look yellow.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTkEWngHNmZLgCUhAQKmMhAAkECgB9hUCxM2vNn0NgzQmkuz028QsAEN
6B1ej6tCbVyT8whsVD2YA7rQH5UavUeUKjuFNJNwelXeEYqG8tbv4vcE1Z65U5tR
GUJkDxWwgjfx4JtE137VmWXcMWW/6FeIBbHuKFohDeBz2nBYmc3w/b4NjgT//JOG
0iUQGRDQcGxDuCYqMqCO8+m/LOSL9wbYDG+bXa4NxVAcghmxAhAJFiDpRDQHcYxX
CeUI7ZkFnNX/4A52YTAFyHQmeoQRX1SLxJsn0GH9mcf2Gsxs8AtoTQgtedDsz9I2
Ct6Dil6NfI94VfOx0EWa1y4I0p7UnFjAsye9zeElhumhocFvEcd5Lzn6SnCY/rQh
o4VukNKUtDE92noL62BZGgCc9ek3/YZivMMFfmQOrjKjfQxEMwSvyjKWIj9XPATT
FWkN7mZn7JSxluB1n1BtHTYvdieD65RGWF1aW1atJZ1r0z0xzy95LF/kqm9Rkigr
zv6+1j9ND79pBikYuq4ol916R9bW93TkriSYbENBP5dtMx4nXUYI6kRU53Y1UxZH
+mstouIpGr9MB7kHLLDYFFFV8upRJi2EQHA2sAtHOaMmjZUoJrbsIdA9n1r1LsKt
6Wjq8MofdyvZWXbzh9aGD4QkovNf6tw4UHAGzVf9Uh3n6mwh7yw4qyDC5h8dqMEe
P4wWkRawtHc=
=GC9p
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Helge Hafting]

David Masover wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Helge Hafting wrote:
> [...]
> | You don't need kernel support for cd'ing into fs images.
> | You need a shell (or GUI app) that:
> | 1. notices that user tries to CD into a file, not a directory
> | 2. Attempts fs type detection and do a loop mount.
> | 3. Give error message if it wasn't a supported fs image.
>
> You can argue this about FS plugins, too.  You don't need kernel support
> for cryptocompress, for example.  Just have a fifo farm.  Every file
> that has compression turned on is actually a named pipe, the original
> file is hidden (starts with a dot), and you have a script running to
> each fifo which runs zcat.

Eww.  A shell/filemanager supporting fs images is one running program.
The fifo farm is one per file.  Ouch.

>
> The kernel support (read: interface) is just cleaner in some ways.
>
> In the loop mount example, it isn't universal -- the user can't CD into
> a file if they use the wrong shell, and two different shells might make
> two different mounts.  How many mounts before we run out of loop devices?

This limitation apply to the kernel interface too.  But you can 
configure for an
arbitrary number of loop mounts anyway. The user will have to mount
using the correct tool - after that the door is opened for anything.

>
> In the cryptocompress example, there's no caching or compress-on-flush
> optimization, and it isn't entirely transparent to anything.  It doesn't
> look like a file, it looks like a fifo.  ls will make it look yellow. 


Even the kernel supported "cd into fs images" has problems.  How should
the kernel know which files you want to support with mounting, and which
ones are the wast majority of plain files?   A file manager doesn't
have to do this fully automatic, it can have a button/hotkey for "loop 
mount this".
So the user will have to tell it what to do, but in a much easier way
than typing "mount -o loop . . . "  The file manager can also ask for
encryption keys - something the kernel cannot do because it doesn't
know what user interface(s) is in use by the process that attempts the "cd".
There may be one interface (stdin, X) there may be both, i.e. something
started from an xterm can use both, there may be none, (the 
web/ftp/file-server
got a request for something inside a encrypted fs image.)

And even if the kernel can figure out the correct user interface, do we
want it to contain an X client for asking for keys? :-/

Helge Hafting

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 787 bytes --]

Salut,

On Fri, Sep 03, 2004 at 12:26:50AM +0200, Frank van Maarseveen wrote:
> Try a "make tags;grep SUPER_MAGIC tags".
> Or is it there for a different purpose?

Problem is:

There are  cool superblock magics  for reiserfs. And for  ext[23]. And
even for good old Minix. Cool.

However,  there  are   also  ugly  file  systems,  such   as  fat  for
example. Fat has been defined as  "something that can be read by a fat
driver", which  can be pretty  much anything. You can't  really detect
whether it is fat. Even Microsoft only guess whether some FS is fat or
not.

And don't  say detect it  via a partition  table.  There are a  lot of
cases where you get absolutely no  or wrong information out of it, and
there are also lots of cases where you plainly have none.

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Mon, Sep 06, 2004 at 10:04:24AM +0200, Tonnerre wrote:
> 
> Problem is:
> 
> There are  cool superblock magics  for reiserfs. And for  ext[23]. And
> even for good old Minix. Cool.
> 
> However,  there  are   also  ugly  file  systems,  such   as  fat  for
> example. Fat has been defined as  "something that can be read by a fat

This problem is not new. Kernel probes for filesystems in a particular
order for mounting the root fs. And mount understands the fstype=auto in
/etc/fstab. There is no perfect solution but it's sure possible to come
up with something acceptable and workable with not too much effort, for
a configuratble subset of file-systems. This is not that much different
from an automounter/usermount mounting a USB storage device or cdrom:
ext3, ext2, udf, iso9660, vfat, read-only or not, just to name a few
things. This should work anyway.

-- 
Frank

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

viro@parcelfarce.linux.theplanet.co.uk wrote:
| On Thu, Sep 02, 2004 at 11:48:06PM +0200, Frank van Maarseveen wrote:
|
|>mount is nice for root, clumsy for user. And a rather complicated
|>way of accessing data the kernel has knowledge about in the first
|>place. For filesystem images, cd'ing into the file is the most
|>obvious concept for file-as-a-dir IMHO.
|
|
| The hell it is.
|
| a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.

reiser4 kernel will contain knowledge of fs type contained in a file.

"file/..metas/type" might contain a mime type.  Mime type might have to
be guessed, but at least if it's made by a local "mkisofs" then we're fine.

Indeed, that's not the only interface that's been discussed.
"file/..metas/is_isofs" might be consulted.


| b) kernel has no way to guess which options to use
| c) fs _type_ is a fundamental part of mount - device(s) (if any) involved
| are arguments to be interpreted by that particular fs driver.

Unless there's some severe security issues with "mount this iso as fat
and you get root access", this should work fine.

I see no reason why there can't be a global setting of the mount
commandline to use.

And it doesn't all have to be in the kernel.  Only it'd be nice to have
some of it there because the kernel knows how to deal with an isofs,
even if it won't know what it looks like.

| d) permissions required for that lovely operation (and questions like
| whether we force nosuid/noexec, etc.) are nightmare to define.

They are quite simple, actually.  Just set them globally -- some admins
would force nosetuid/noexec, some wouldn't.  And the operation happens
transparently -- you need no "permissions" other than to read the
directory which would contain the mount.

| Frankly, the longer that thread grows, the more obvious it becomes that
| file-as-a-dir is a solution in search of problem.  Desperate search, at
| that.

Actually, the longer this thread grows, the more obvious it is how when
there's a hot issue, everyone has an opinion, even if the same opinion
has been expressed ten or twenty times already.

File-as-a-dir has numerous advantages, but enough have been discussed.
Short list is image mounts, tarballs, streams, metas, and namespace
unification.  Longer list and explanations can be found if you RTFA.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTe19XgHNmZLgCUhAQJdow/+Knsw1GgpauqDUcg8sKtxzgXZ18OxMQ3Q
By8sRrSTuKAzI5A3BtYIzndsj1veP+7wndG7nYPz8NS1fU2+xWSIhoGq/YMaQsu4
70uMLu448PFXZua4hZMk2w4mkULXbGyYHJ1Bf+2Z7QkQ/8W08hozC8QQynxMXIkX
SrcWCS5hK8Nh7Ol691sDpPqexH7F1GwUyoslNGj63U5r6ViLAawt2ZKDYdT7ZPo8
0a/pWUHoHMPbv/KwqZZxRr1/qncA9QYQo6JqQBPPCr+tWNJs/ei3nAKGi58iOt1M
DK1TEKd2lpbmwiK5pWDwGz+nwWmaFTAyfTEEEcP4gZedSJtRXaxyNh0jRl1iLATB
SCO5Eb4jkQs8hdjHqQcQ1q7XKFX9eSXWeDdrGrtWaYC/QYOHxT+ci3lnKBKCG99Y
YTqg3sNEZlV1N0jIcNvFSDEYbbX12v1Y6xbwvUx48+sMyUj3suT76niTRbwEydfO
MA9y+wE2k4wF+h+sJCbTjimCNFvvuFTTJuBCbQTpfY4eOYBAFalxnWmrpTEfVzka
4iAqAYygWObGDkFFy/rp1HEVZPIKM0NwGLOsRwJsgyUMOsccBrEc0bg8sgMECVfs
5qNIb27tLokh8NBR6RodAv2NZYKC+foM0T+PC5bZMFD/Q7f6yDklqK4C4RCIYaXj
xO9z1C6FPcM=
=/gmK
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Linus Torvalds]

On Thu, 2 Sep 2004, David Masover wrote:
> 
> reiser4 kernel will contain knowledge of fs type contained in a file.

That's a disaster, btw.

There is no one "fs type" of a file. Files have at _least_ one type
(bytestream), but most have more. Which is why automatically doing the
right thing (in the sense you seem to want) in kernel space is simply not
possible.

			Linus

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Linus Torvalds wrote:
|
| On Thu, 2 Sep 2004, David Masover wrote:
|
|>reiser4 kernel will contain knowledge of fs type contained in a file.
|
|
| That's a disaster, btw.
|
| There is no one "fs type" of a file. Files have at _least_ one type
| (bytestream), but most have more. Which is why automatically doing the
| right thing (in the sense you seem to want) in kernel space is simply not
| possible.

Oops, my bad.  The interface is kernel, the file type database is user.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTfH7XgHNmZLgCUhAQL0aw//SG/RDCy30xcVh36FVVAOG7GjP6UkHBeo
14/O3hSELxwm9r8z0YnGnbpoJj9atbOCu6VJhiumtjeyikdH5OaOrcMo7DQYp47V
holyEKvjd+YUkCBJSjFBmG279Ac7wuk2orFlG76gCkFjNk11W/FQsjEM3gi8yKa0
ZODFmh0XJa2p0mKRnApwynO1ma7HgyqNRKrjmxMUK2VHpOsbqjVe1+Gc2lk0E9gv
xc2PIdCgi3yDEpDpxSY8LEBXad7GTwa2VEID5G7C6Z0wOH38YyMEpoHQz03se9zO
aCnVg8LPdFkZveWXPiiyJrwDERCyKt/yrRxrVznWyNVaWmhoxXmw7TmQexvGiDD2
E6+XT8uWfdeMZRPhv284qcBegIDw5c1wHgz+GRso61T6x04hfJyY/onbF5lHz661
lfys4JRT0zmYbrG1b3GVgmfKDv7t3V6DkY+Pi1E5raeZr2F0idiZo+7uKh2NZAML
PLhb8LLr/lbHOSRG3Rq0YY/l+Q6wZXy770qF/51jp/c3UXXJyVusK+bCLrVZ3VPa
0Uf3KkuFLcoD4jDTNjt79QVJXIvEHmNOxp/g80nJUFM0WDj7u4a7pFpLCHZzFeBl
5uoS9c7dYjuV+/EkQ2S8YwryeyCHQNt381icb8HtbdlQOCVuK0v9OspA3IZmphcB
oD1RBCkVTYQ=
=l8Vz
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Hans Reiser]

Linus Torvalds wrote:

>On Thu, 2 Sep 2004, David Masover wrote:
>  
>
>>reiser4 kernel will contain knowledge of fs type contained in a file.
>>    
>>
Whoa!  No it won't.  It will allow you to create a metafile named type 
if you choose.  Or maybe a metadirectory named types, I don't really 
care much about this stuff yet, we need to do other things to the 
semantics more urgently than this, Typing requires a lot of cooperation 
from user space apps.  I don't really expect to have that much social 
pull on the issue, but if someone chooses to design their app to use a 
types metadirectory, I don't mind accomodating on it.  I apologize if  
from my saying that I gave the impression that I think files should be 
strongly typed.

I am quite well aware of the disadvantages of OS/400 hindering usability 
with type information, as well the advantages of having types available 
for those that want to look at them.....

>
>That's a disaster, btw.
>  
>
In every implementation so far it has been a net disaster, because OS 
designers who type things have been willing to make users pay attention 
to type when they don't want to.  Sometimes trying to make too much out 
of something makes it a mistake when being a bit more laid back would 
make it ok.  Me, I am so laid back, I prefer to work on other features 
for a few years first.....;-)

>There is no one "fs type" of a file. Files have at _least_ one type
>(bytestream), but most have more. Which is why automatically doing the
>right thing (in the sense you seem to want) in kernel space is simply not
>possible.
>
>			Linus
>
>
>  
>

Re: The argument for fs assistance in handling archives

[viro]

On Thu, Sep 02, 2004 at 07:08:21PM -0500, David Masover wrote:
> | a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
> 
> reiser4 kernel will contain knowledge of fs type contained in a file.

Right.  And when I decide to talk to delusional nutcases I will go to
talk.origin and find a creationist to chat with.

Time to extend the killfile...

*plonk*

Re: The argument for fs assistance in handling archives

[Spam]

  

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1

> viro@parcelfarce.linux.theplanet.co.uk wrote:
> | On Thu, Sep 02, 2004 at 11:48:06PM +0200, Frank van Maarseveen wrote:
> |
|>>mount is nice for root, clumsy for user. And a rather complicated
|>>way of accessing data the kernel has knowledge about in the first
|>>place. For filesystem images, cd'ing into the file is the most
|>>obvious concept for file-as-a-dir IMHO.
> |
> |
> | The hell it is.
> |
> | a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.

> reiser4 kernel will contain knowledge of fs type contained in a file.

> "file/..metas/type" might contain a mime type.  Mime type might have to
> be guessed, but at least if it's made by a local "mkisofs" then we're fine.

> Indeed, that's not the only interface that's been discussed.
> "file/..metas/is_isofs" might be consulted.

 What you are talking about isn't the kernel or such, but plugins that
 could extend the filesystem. Plugins could store information about
 contents, encodings, formatting, filesystems, etc, as meta-info. If
 you have a plugin that would allow you to traverse files as disk
 images then it could read those meta-data. But before those plugins
 exist then there is no such standard for info stored as meta-data and
 the kernel wouldn't know anything about this to begin with.


> | b) kernel has no way to guess which options to use
> | c) fs _type_ is a fundamental part of mount - device(s) (if any) involved
> | are arguments to be interpreted by that particular fs driver.

> Unless there's some severe security issues with "mount this iso as fat
> and you get root access", this should work fine.

> I see no reason why there can't be a global setting of the mount
> commandline to use.

> And it doesn't all have to be in the kernel.  Only it'd be nice to have
> some of it there because the kernel knows how to deal with an isofs,
> even if it won't know what it looks like.

> | d) permissions required for that lovely operation (and questions like
> | whether we force nosuid/noexec, etc.) are nightmare to define.

> They are quite simple, actually.  Just set them globally -- some admins
> would force nosetuid/noexec, some wouldn't.  And the operation happens
> transparently -- you need no "permissions" other than to read the
> directory which would contain the mount.

> | Frankly, the longer that thread grows, the more obvious it becomes that
> | file-as-a-dir is a solution in search of problem.  Desperate search, at
> | that.

> Actually, the longer this thread grows, the more obvious it is how when
> there's a hot issue, everyone has an opinion, even if the same opinion
> has been expressed ten or twenty times already.

> File-as-a-dir has numerous advantages, but enough have been discussed.
> Short list is image mounts, tarballs, streams, metas, and namespace
> unification.  Longer list and explanations can be found if you RTFA.
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.4 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

> iQIVAwUBQTe19XgHNmZLgCUhAQJdow/+Knsw1GgpauqDUcg8sKtxzgXZ18OxMQ3Q
> By8sRrSTuKAzI5A3BtYIzndsj1veP+7wndG7nYPz8NS1fU2+xWSIhoGq/YMaQsu4
> 70uMLu448PFXZua4hZMk2w4mkULXbGyYHJ1Bf+2Z7QkQ/8W08hozC8QQynxMXIkX
> SrcWCS5hK8Nh7Ol691sDpPqexH7F1GwUyoslNGj63U5r6ViLAawt2ZKDYdT7ZPo8
> 0a/pWUHoHMPbv/KwqZZxRr1/qncA9QYQo6JqQBPPCr+tWNJs/ei3nAKGi58iOt1M
> DK1TEKd2lpbmwiK5pWDwGz+nwWmaFTAyfTEEEcP4gZedSJtRXaxyNh0jRl1iLATB
> SCO5Eb4jkQs8hdjHqQcQ1q7XKFX9eSXWeDdrGrtWaYC/QYOHxT+ci3lnKBKCG99Y
> YTqg3sNEZlV1N0jIcNvFSDEYbbX12v1Y6xbwvUx48+sMyUj3suT76niTRbwEydfO
> MA9y+wE2k4wF+h+sJCbTjimCNFvvuFTTJuBCbQTpfY4eOYBAFalxnWmrpTEfVzka
> 4iAqAYygWObGDkFFy/rp1HEVZPIKM0NwGLOsRwJsgyUMOsccBrEc0bg8sgMECVfs
> 5qNIb27tLokh8NBR6RodAv2NZYKC+foM0T+PC5bZMFD/Q7f6yDklqK4C4RCIYaXj
> xO9z1C6FPcM=
> =/gmK
> -----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Spam wrote:
[...]
| Indeed, that's not the only interface that's been discussed.
| "file/..metas/is_isofs" might be consulted.
|
|
|>  What you are talking about isn't the kernel or such, but plugins that

Plugins are kernel-space.  As Linus points out, dealing with mime-types
in the kernel is uncool.

|>  could extend the filesystem. Plugins could store information about
|>  contents, encodings, formatting, filesystems, etc, as meta-info. If
|>  you have a plugin that would allow you to traverse files as disk
|>  images then it could read those meta-data. But before those plugins
|>  exist then there is no such standard for info stored as meta-data and
|>  the kernel wouldn't know anything about this to begin with.

So implement a plugin which knows how to talk to a userland program
which knows about metadata.  The plugin controls access to file-type.

Maybe there ought to be a general-purpose userland plugin interface?  So
that the only things left in the kernel are things that have to be there
for speed and/or sanity reasons?  (Things like cryptocompress and
standard file/directory plugins.)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTfIs3gHNmZLgCUhAQJCqA//cPGI/TPfgtwovk5a8HdcY9TbjJELGtIb
CHcNa2LKGUB8uVvoL9M0Pe9ei4iVK2QOL1QjUAbIE0Dx7t18KR/5qEIfCHTSw0sJ
8u/r3aaJhFjFwMcVLQOZWJYuTeodgMwkV96GgViGoHoiqDiV7BzZgjd43qwiH8rW
FUTKlPn2VmijyTTbf5VfX4hvmsU/He+5W0t08/xe3vpCa+ihNFLJQAwGioo/wzFq
aHl9jBT1esFLxONd7OxQpgVl/2uHx+rSAY6F5RyBqL/Tpm1ZKlrMdAzmdDWcAJA9
KnOLN8ltcPmjP0eCzgCO/iq8yczcwcagfmbD+WcYOmQbTXMTjxktrZqRuLrUHU+7
tl8JISKch5epHfOnQ/RTMEgotlcQ0SCoE7K5lIUuyheMYRWoVDJSvy3okET6ZxQL
NP3PVHguQbu1Bo2X8LNsrnU0KFT7XrejpXzKalPWVbQxayEEWUwLXBdNIpgBoYwX
FZXQKEpS4MmB/8kEC9xuQ077PfAclcjcHSj4B7phbSVMioFy4/HojOLOLA8gjHyr
7p+wimfDpsuioDLlccwc1b+fLB2eJM5G+zfzh//uy7LdfFxuI+074FJYxebRI2eL
N/rpjT+S0S7L0/k71Rwp/5y4YgDgdQqU0wyGaBLRjP0qD7k2S41g+yrua6qArhzP
baR9dCu1zAQ=
=Nrye
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Chris Dukes]

On Thu, Sep 02, 2004 at 08:28:20PM -0500, David Masover wrote:
> 
> So implement a plugin which knows how to talk to a userland program
> which knows about metadata.  The plugin controls access to file-type.
> 
> Maybe there ought to be a general-purpose userland plugin interface?  So
> that the only things left in the kernel are things that have to be there
> for speed and/or sanity reasons?  (Things like cryptocompress and
> standard file/directory plugins.)

Ahem,
Wasn't this the goal of GNU HURD?

I really think you should ask them why they haven't delivered
something useful, then come back to this thread.

Thanks.
-- 
Chris Dukes
Warning: Do not use the reflow toaster oven to prepare foods after
it has been used for solder paste reflow. 
http://www.stencilsunlimited.com/stencil_article_page5.htm

Re: The argument for fs assistance in handling archives

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris Dukes wrote:
| On Thu, Sep 02, 2004 at 08:28:20PM -0500, David Masover wrote:
|
|>So implement a plugin which knows how to talk to a userland program
|>which knows about metadata.  The plugin controls access to file-type.
|>
|>Maybe there ought to be a general-purpose userland plugin interface?  So
|>that the only things left in the kernel are things that have to be there
|>for speed and/or sanity reasons?  (Things like cryptocompress and
|>standard file/directory plugins.)
|
|
| Ahem,
| Wasn't this the goal of GNU HURD?

The goal of GNU HURD was to take everything out of the kernel and make
it entirely daemons.  That's a far cry from keeping a file-type database
(historically the realm of file managers) out of the kernel.

Most people hate putting things in the kernel that don't belong there.
Most of what's in the kernel is for speed and/or sanity -- filesystems
are there for speed, device drivers (and scheduling, and vm, and so on)
are there for sanity.

Correct me if I'm wrong on this.

| I really think you should ask them why they haven't delivered
| something useful, then come back to this thread.

Honestly?  I think it's mostly got nothing to do with architecture.  I
think it's mostly got to do with politics.  Most people would rather
hack on Linux, which is already done, than try to develop HURD, which is
something new.  Most people also enjoy working with Linus (or prefer
Linus to the FSF).

I do not like how Linux is monolithic.  I do not like having to reboot
to upgrade the kernel, and I do not like having to run closed software
(the nvidia drivers) in the kernel (as in, full privelages, can crash
entire system, yadda yadda).  But Linux is the best we have.

The HURD people have delivered at least something.  I think there's even
a Debian/HURD distro.  Whether it's useful probably has to do with
whether it's stable/fast, which isn't likely.  You hear of Linux news
every day, you hear of HURD maybe once in a lifetime -- "Hey, HURD exists!"

But you are right -- I need to take a break from this thread.  It's
becoming addictive, and it's hard keeping up with so many people who are
so much smarter than I am.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTf0/3gHNmZLgCUhAQKGrw//XMBsWBo7uqlocjdvSati4GkxsB+f2GPW
hI3JboYhWVMB7Ya+mS9VbrruxVn7uCCUwLwNDDnMncHTf09Em+WhzwJCQXZ14Y2r
AxZ8iaVR9Qv8hIHGw59uJYWvtTSHgSmWiFFVa4UB4D3NneSFr2vgXXDbU8LjhHQB
46qs8IM6s+RCb0L608uGml6AmyrxFvDgWz9p+++tWicLjQPVhBnLQQnqYy92we7x
qUwxCyGQywmdcSsJlkcjKzLQ/dCDwQRq7LXmSCx3qbSRr27D6zpE6l9He62ZuoBR
I9NmO12S6VgYifO5fnq6pxSu2+GuJVW469tm7EiXmKZ/rjfMJKp0cqg+mtGTU91m
gk7cZTBbe4id9+IQPJPJJ08IMH5XTI4DI6dXCkKIB0TZ8teFnU2+DsUdZ50ZXwiL
etFUjortzjPjEqZoKfvK+4n7HOMu1w1og/7aE6P5UNTbj5ViwLQB9KjwLBDyBMJr
dTmcLRyxzHtDZJMQuLwvU/SsdF86CdlR4spkEI6CitRMiiWvYghOeOkasOO8qxVj
ckzZzUCRHNF/PTPSAEiIaw2HyjVzf0Nru96Y80/hik3hRYO0XuTP/3K/twHoebkz
Fo1NLtGU3g631pDYcpqnkkAcQN+3fiyZ3VEiNoYYLhZpkQiEH8goRv5lof9XJPAC
krhwM8lNHoM=
=JJEe
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Horst von Brand]

David Masover <ninja@slaphack.com> said:
> Chris Dukes wrote:
> | On Thu, Sep 02, 2004 at 08:28:20PM -0500, David Masover wrote:
> |
> |>So implement a plugin which knows how to talk to a userland program
> |>which knows about metadata.  The plugin controls access to file-type.
> |>
> |>Maybe there ought to be a general-purpose userland plugin interface?  So
> |>that the only things left in the kernel are things that have to be there
> |>for speed and/or sanity reasons?  (Things like cryptocompress and
> |>standard file/directory plugins.)
> |
> |
> | Ahem,
> | Wasn't this the goal of GNU HURD?

> The goal of GNU HURD was to take everything out of the kernel and make
> it entirely daemons.  That's a far cry from keeping a file-type database
> (historically the realm of file managers) out of the kernel.

Yep. Traditional microkernel.

[...]

> | I really think you should ask them why they haven't delivered
> | something useful, then come back to this thread.

> Honestly?  I think it's mostly got nothing to do with architecture.  I
> think it's mostly got to do with politics.  Most people would rather
> hack on Linux, which is already done, than try to develop HURD, which is
> something new.  Most people also enjoy working with Linus (or prefer
> Linus to the FSF).

Partly right. Hurd is _much_ older than Linux, and didn't start from
scratch (Mach was around, and worked). Yet I remember hearing "any day now"
for Hurd from 86 or 87... and as of now, AFAIU even the people hacking on
Hurd prefer Linux (because Linux does work, Hurd doesn't).

> I do not like how Linux is monolithic.  I do not like having to reboot
> to upgrade the kernel,

Tough luck. An upgrade of the core will _always_ mean rebooting. Plus you
can replace modules in Linux without a reboot.

>                        and I do not like having to run closed software
> (the nvidia drivers) in the kernel (as in, full privelages, can crash
> entire system, yadda yadda).

Me neither. But that is not exactly Linux' fault, binary drivers are quite
strongly discouraged.

>                               But Linux is the best we have.

By far.

> The HURD people have delivered at least something.  I think there's even
> a Debian/HURD distro.  Whether it's useful probably has to do with
> whether it's stable/fast, which isn't likely.  You hear of Linux news
> every day, you hear of HURD maybe once in a lifetime -- "Hey, HURD
> exists!"

It is still vaporware, some 20 years later.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: The argument for fs assistance in handling archives

[Hans Reiser]

I don't think streams are a good thing, I think that all of the 
different pieces of additional functionality necessary to emulate them 
with files and directories are a good thing.

Keeping streams out of linux was one of the (less important) ideas 
behind reiser4.  Streams are a rigid hack.  The toolkit that can emulate 
them, is useful, and the perceived importance of that emulation will 
fade as people start to use the toolkit for things that are much more 
fun than streams.

I agree with most of the rest of what you say though David.

Hans


David Masover wrote:

> File-as-a-dir has numerous advantages, but enough have been discussed.
> Short list is image mounts, tarballs, streams, metas, and namespace
> unification.  Longer list and explanations can be found if you RTFA.

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Please read the FAQ at  http://www.tux.org/lkml/

Re: The argument for fs assistance in handling archives

[John Stoffel]

>>>>> "David" == David Masover <ninja@slaphack.com> writes:

David> File-as-a-dir has numerous advantages, but enough have been
David> discussed.  Short list is image mounts, tarballs, streams,
David> metas, and namespace unification.  Longer list and explanations
David> can be found if you RTFA.

And it has numerous dis-advantages as well, since it doesn't have a
good set of semantics and syntax defined yet, nor does it explain
except by vigorous handwaving the performance and security impacts it
can have.

My personal feeling is that the mount(8) command should be the tool
used to extract and expose the internal namespace of files like this
and to then graft it onto the standard Unix namespace with gross Unix
semantics, but it's own wacky internal semantics.  This way, standard
tools don't care, but special tools which know how to handle it can do
what they want.


> mount -t tarfs /some/place/on/disk/foo.tar.gz /mnt/tar
> cp /var/tmp/img.gif .
> umount /mnt/tar

Oops!  Someone did a rm /some/place/on/disk/foo.tar.gz between steps
one and two.  Now what happens?  Please define those semantics...

John

Re: The argument for fs assistance in handling archives

[Spam]

  

>>>>>> "David" == David Masover <ninja@slaphack.com> writes:

David>> File-as-a-dir has numerous advantages, but enough have been
David>> discussed.  Short list is image mounts, tarballs, streams,
David>> metas, and namespace unification.  Longer list and explanations
David>> can be found if you RTFA.

> And it has numerous dis-advantages as well, since it doesn't have a
> good set of semantics and syntax defined yet, nor does it explain
> except by vigorous handwaving the performance and security impacts it
> can have.

> My personal feeling is that the mount(8) command should be the tool
> used to extract and expose the internal namespace of files like this
> and to then graft it onto the standard Unix namespace with gross Unix
> semantics, but it's own wacky internal semantics.  This way, standard
> tools don't care, but special tools which know how to handle it can do
> what they want.


>> mount -t tarfs /some/place/on/disk/foo.tar.gz /mnt/tar
>> cp /var/tmp/img.gif .
>> umount /mnt/tar

> Oops!  Someone did a rm /some/place/on/disk/foo.tar.gz between steps
> one and two.  Now what happens?  Please define those semantics...

  Uhm, can you delete a file (loop) that is mounted?

> John

Re: The argument for fs assistance in handling archives

[Michelle Konzack]

[-- Attachment #1: Type: text/plain, Size: 761 bytes --]

Am 2004-09-03 16:01:17, schrieb Spam:
> >>>>>> "David" == David Masover <ninja@slaphack.com> writes:

> >> mount -t tarfs /some/place/on/disk/foo.tar.gz /mnt/tar
> >> cp /var/tmp/img.gif .
> >> umount /mnt/tar
> 
> > Oops!  Someone did a rm /some/place/on/disk/foo.tar.gz between steps
> > one and two.  Now what happens?  Please define those semantics...
> 
>   Uhm, can you delete a file (loop) that is mounted?

Not as $USER but root... :-/

> > John

Greetings
Michelle

-- 
Linux-User #280138 with the Linux Counter, http://counter.li.org/ 
Michelle Konzack   Apt. 917                  ICQ #328449886
                   50, rue de Soultz         MSM LinuxMichi
0033/3/88452356    67100 Strasbourg/France   IRC #Debian (irc.icq.com)

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 582 bytes --]

Salut,

On Fri, Sep 03, 2004 at 04:01:17PM +0200, Spam wrote:
> >> mount -t tarfs /some/place/on/disk/foo.tar.gz /mnt/tar
> >> cp /var/tmp/img.gif .
> >> umount /mnt/tar
> 
> > Oops!  Someone did a rm /some/place/on/disk/foo.tar.gz between steps
> > one and two.  Now what happens?  Please define those semantics...
> 
>   Uhm, can you delete a file (loop) that is mounted?

"Text file busy"

But for files this isn't true.

Currently, we  have another method assuring data  coherency: we remove
the inode only when the last reference goes away.

				Tonnerre


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives

[Clemens Schwaighofer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank van Maarseveen wrote:

|>
|>We have the mount command for that.  :^)
|
|
| mount is nice for root, clumsy for user. And a rather complicated
| way of accessing data the kernel has knowledge about in the first
| place. For filesystem images, cd'ing into the file is the most
| obvious concept for file-as-a-dir IMHO.

thats why we have automount.

lg, clemens
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBPF3djBz/yQjBxz8RAmVhAKCODKNKOV9V0I59SUfQ1pp+qk88xwCg6+4p
Q+JcbO+W0+6cUfSEf3z/Iwk=
=IwZQ
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Spam]

  

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1

> Frank van Maarseveen wrote:

|>>
|>>We have the mount command for that.  :^)
> |
> |
> | mount is nice for root, clumsy for user. And a rather complicated
> | way of accessing data the kernel has knowledge about in the first
> | place. For filesystem images, cd'ing into the file is the most
> | obvious concept for file-as-a-dir IMHO.

> thats why we have automount.

  Which still needs to be setup in fstab, right?

> lg, clemens
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.2.5 (GNU/Linux)
> Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

> iD8DBQFBPF3djBz/yQjBxz8RAmVhAKCODKNKOV9V0I59SUfQ1pp+qk88xwCg6+4p
> Q+JcbO+W0+6cUfSEf3z/Iwk=
> =IwZQ
> -----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Clemens Schwaighofer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Spam wrote:

| thats why we have automount.
|
|
|>   Which still needs to be setup in fstab, right?

no, in /etc/automount* actually. I have no fstab entry here and still I
can mount various samba shares, cdroms, etc ...

lg, clemens

even my usbhd is automount :)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBPF8tjBz/yQjBxz8RApB5AKDOjvCM2foy0lTXAy3IOVXXxdBcUACgxUyn
1WlAMXdnleNZOTOwvVRQCz0=
=LX+8
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Frank van Maarseveen]

On Mon, Sep 06, 2004 at 09:59:25PM +0900, Clemens Schwaighofer wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Spam wrote:
> 
> | thats why we have automount.
> |
> |
> |>   Which still needs to be setup in fstab, right?
> 
> no, in /etc/automount* actually. I have no fstab entry here and still I
> can mount various samba shares, cdroms, etc ...

but can you access images that way?

-- 
Frank

Re: The argument for fs assistance in handling archives

[Spam]

  

> On Mon, Sep 06, 2004 at 09:59:25PM +0900, Clemens Schwaighofer wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>> 
>> Spam wrote:
>> 
>> | thats why we have automount.
>> |
>> |
>> |>   Which still needs to be setup in fstab, right?
>> 
>> no, in /etc/automount* actually. I have no fstab entry here and still I
>> can mount various samba shares, cdroms, etc ...

> but can you access images that way?

  Probably, but only predefined ones.. IE, no random automount?

  ~S

Re: The argument for fs assistance in handling archives

[Clemens Schwaighofer]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Frank van Maarseveen wrote:
| On Mon, Sep 06, 2004 at 09:59:25PM +0900, Clemens Schwaighofer wrote:
|
|>-----BEGIN PGP SIGNED MESSAGE-----
|>Hash: SHA1
|>
|>Spam wrote:
|>
|>| thats why we have automount.
|>|
|>|
|>|>   Which still needs to be setup in fstab, right?
|>
|>no, in /etc/automount* actually. I have no fstab entry here and still I
|>can mount various samba shares, cdroms, etc ...
|
|
| but can you access images that way?
|

in theory yes, but only predefined ones. But I don't see automount for
this. more for cdroms and other removable medias.

I haven't had the need to mount and iso for quite some time, so I don't
know if I need root access for that, but I think yes like for all mount
things.

lg, clemens
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFBPGNgjBz/yQjBxz8RAsm7AJsEwa18ymMBWKf1rJ2PAwPOfCMIJgCglaNS
d8OqXmD0jsYezcE8u7BGrkM=
=ctEE
-----END PGP SIGNATURE-----

Re: The argument for fs assistance in handling archives

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 227 bytes --]

Salut,

On Mon, Sep 06, 2004 at 09:53:49PM +0900, Clemens Schwaighofer wrote:
> thats why we have automount.

Plus the HAL automounting code which dynamically generates mountpoints
for HAL detected devices...

			    Tonnerre


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 360 bytes --]

On Thu, 02 Sep 2004 22:38:54 +0200, Frank van Maarseveen said:

> Can it do this:
> 
> 	cd FC2-i386-disc1.iso
> 	ls

That one's at least theoretically doable, assuming that it really *IS* the
Fedora Core disk and an ISO9660 format...

> 	cd /dev/cdrom
> 	ls

And the CD in the drive at the moment is AC/DC "Back in Black".  What
should this produce as output?

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Thu, Sep 02, 2004 at 07:19:47PM -0400, Valdis.Kletnieks@vt.edu wrote:
> 
> > 	cd FC2-i386-disc1.iso
> > 	ls
> 
> That one's at least theoretically doable, assuming that it really *IS* the
> Fedora Core disk and an ISO9660 format...

Impressive.

But when it comes to file-systems like ext[23] I think an in-kernel
solution might be preferable to get the exact semantics wrt locking,
atomicity, synchronization, coherency, whatever a kernel is good at
for filesystems.

> > 	cd /dev/cdrom
> > 	ls
> 
> And the CD in the drive at the moment is AC/DC "Back in Black".  What
> should this produce as output?

bash: cd: /dev/cdrom: Not a directory

When it doesn't match a (sub)set of known file system types I think.
Because that is the area the kernel has knowledge about. Nothing
else, no tarballs for example unless of course the kernel has "tarfs" :-)

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  

> On Thu, 02 Sep 2004 22:38:54 +0200, Frank van Maarseveen said:

>> Can it do this:
>> 
>> 	cd FC2-i386-disc1.iso
>> 	ls

> That one's at least theoretically doable, assuming that it really *IS* the
> Fedora Core disk and an ISO9660 format...

>> 	cd /dev/cdrom
>> 	ls

> And the CD in the drive at the moment is AC/DC "Back in Black".  What
> should this produce as output?

  Yes why not? If there was any filesystem drivers for the AudioCD
  format then it could.

  I had such a driver for Windows 9x which would display several
  folders and files for inserted AudioCD's:

  D: (cdrom)
    Stereo
      22050
        Track01.wav
        Track02.wav
        ...
      44100
        Track01.wav
        ...
    Mono
      22050
        Track01.wav
        ...
      44100
        Track01.wav
        ...

  Normal AudioCD players would also work even though this driver was
  installed. These files were also visible for legacy applications in
  the command prompt (inside Windows).

  I do not see why this would not be possible in Linux. Of course, it
  would perhaps require a filesystem driver/module to be present when
  you mount.

  If you just want to do a cd file.iso then it may be a totally
  different thing. Either you would have a automount feature or a
  filesystem/vfs plugin that could load secondary modules to support
  this kind of thing.

  ~S

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Fri, Sep 03, 2004 at 01:43:02AM +0200, Spam wrote:
> 
>   Yes why not? If there was any filesystem drivers for the AudioCD
>   format then it could.
> 
>   I had such a driver for Windows 9x which would display several
>   folders and files for inserted AudioCD's:
> 
>   D: (cdrom)
>     Stereo
>       22050
>         Track01.wav
>         Track02.wav
>         ...
>       44100
>         Track01.wav
>         ...
...
> 
>   If you just want to do a cd file.iso then it may be a totally
>   different thing. Either you would have a automount feature or a
>   filesystem/vfs plugin that could load secondary modules to support
>   this kind of thing.

Why is this so different, compared to your example? They are both
filesystem drivers.

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 708 bytes --]

Salut,

On Fri, Sep 03, 2004 at 01:43:02AM +0200, Spam wrote:
>   Yes why not? If there was any filesystem drivers for the AudioCD
>   format then it could.
> 
>   I had such a driver for Windows 9x which would display several
>   folders and files for inserted AudioCD's:
> 
>   D: (cdrom)
>     Stereo
>       22050
>         Track01.wav
>         Track02.wav
>         ...
>       44100
>         Track01.wav
>         ...
>     Mono
>       22050
>         Track01.wav
>         ...
>       44100
>         Track01.wav
>         ...

So you'd like the kernel to know  about raw CD PCM and RIFF PCM format
and conversion? Great.. That's really Solarisish!

			    Tonnerre


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Spam]

  

> Salut,

> On Fri, Sep 03, 2004 at 01:43:02AM +0200, Spam wrote:
>>   Yes why not? If there was any filesystem drivers for the AudioCD
>>   format then it could.
>> 
>>   I had such a driver for Windows 9x which would display several
>>   folders and files for inserted AudioCD's:
>> 
>>   D: (cdrom)
>>     Stereo
>>       22050
>>         Track01.wav
>>         Track02.wav
>>         ...
>>       44100
>>         Track01.wav
>>         ...
>>     Mono
>>       22050
>>         Track01.wav
>>         ...
>>       44100
>>         Track01.wav
>>         ...

> So you'd like the kernel to know  about raw CD PCM and RIFF PCM format
> and conversion? Great.. That's really Solarisish!

  If it could be done in userland, and still be accessible for most
  applications then that is great.

  If the user things that such a plugin is of value for him then I do
  not see why he should not be able to use it? Again these are example
  of stuff that could be done if there was a plugin/extensible
  interface.

  You make it sound like everything on the planet should be predefined
  and included in the kernel. This is not what I was saying or wanted.

  From what I understood from Hans, there will be a way to load
  plugins without having to recompile reiser4 module (or the kernel).
  This is what I'd like to see.

  Some things/plugins may even be partially user-space.

  ~S

  

> 			    Tonnerre

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Jan Harkes]

On Thu, Sep 02, 2004 at 07:19:47PM -0400, Valdis.Kletnieks@vt.edu wrote:
> On Thu, 02 Sep 2004 22:38:54 +0200, Frank van Maarseveen said:
> > 	cd /dev/cdrom
> > 	ls
> 
> And the CD in the drive at the moment is AC/DC "Back in Black".  What
> should this produce as output?

Hehe, cdfs already figured that one out. Ofcourse you show the
individual tracks as .wav files.

Jan

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 857 bytes --]

On Fri, 03 Sep 2004 03:13:35 EDT, Jan Harkes said:
> On Thu, Sep 02, 2004 at 07:19:47PM -0400, Valdis.Kletnieks@vt.edu wrote:
> > On Thu, 02 Sep 2004 22:38:54 +0200, Frank van Maarseveen said:
> > > 	cd /dev/cdrom
> > > 	ls
> > 
> > And the CD in the drive at the moment is AC/DC "Back in Black".  What
> > should this produce as output?
> 
> Hehe, cdfs already figured that one out. Ofcourse you show the
> individual tracks as .wav files.

That's sidestepping the *real* issue - which is that you need reasonable semantics
even if you don't have cdfs or whatever special gee-wizz-bang driver handy.

Consider an embedded system - it may have iso9660 support, and boot off a CD
like Knoppix, but not have cdfs.  What do you do then?

(For bonus points, figure out the security issues involved in dealing with an intentionall
corrupted image on a CD......)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 784 bytes --]

Salut,

On Thu, Sep 02, 2004 at 10:38:54PM +0200, Frank van Maarseveen wrote:
> Can it do this:
> 
> 	cd FC2-i386-disc1.iso
> 	ls
> 
> or this:
> 
> 	cd /dev/sda1
> 	ls
> 	cd /dev/floppy
> 	ls
> 	cd /dev/cdrom
> 	ls
> 
> ?

Actually                  I                  see                  some
small-security-hole-you-can-drive-a-big-yellow-truck-with-flashes-on-through
problem.

$ cat fs_header owner_root flags_with_suid evil_program > evil.iso
$ ls -l evil.iso/evil_program
-rwsr-xr-x    1 root     root           24 2003-11-09 16:41 evil.iso/evil_program
$ evil.iso/evil_program
Mwahaha, I got root!
Copy-right violating IP... done.
Hijacking administrators wife... done.
Overwriting OS with Parachute 2010... done.

etc. pp.

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Mon, Sep 06, 2004 at 09:56:03AM +0200, Tonnerre wrote:
> 
> $ cat fs_header owner_root flags_with_suid evil_program > evil.iso
> $ ls -l evil.iso/evil_program

It should of course be equivalent to a user mount: nodev nosuid etc.

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Herbert Poetzl]

On Mon, Sep 06, 2004 at 10:08:45AM +0200, Frank van Maarseveen wrote:
> On Mon, Sep 06, 2004 at 09:56:03AM +0200, Tonnerre wrote:
> > 
> > $ cat fs_header owner_root flags_with_suid evil_program > evil.iso
> > $ ls -l evil.iso/evil_program
> 
> It should of course be equivalent to a user mount: nodev nosuid etc.

hmm, sounds reasonable, but what if root accesses it?
(or somebody with the 'right' capability)

 - it might be strange if even root is not able to
   open device nodes or execute files from an archive

 - it might lead to interesting situations if the
   archive is opened by root, but accessed by an user
   (thinking of caches and such)
  
best,
Herbert

> -- 
> Frank
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Frank van Maarseveen]

On Mon, Sep 06, 2004 at 02:43:57PM +0200, Herbert Poetzl wrote:
> hmm, sounds reasonable, but what if root accesses it?
> (or somebody with the 'right' capability)
> 
>  - it might be strange if even root is not able to
>    open device nodes or execute files from an archive

Yes, but if the file is owned by or writable for non-root then
you've got a security problem. So, unless owned by root and not
writable (readable, executable?) for anyone else "nodev" and
'nosuid" are mandatory.

> 
>  - it might lead to interesting situations if the
>    archive is opened by root, but accessed by an user
>    (thinking of caches and such)

See the above.
Alternatively, each process could have its own vfsmount (please don't
shoot me for suggesting this ;-)

-- 
Frank

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Pavel Machek]

Hi!

> I bet you could write a small library to test this out for a few types.  
> See if it's useful to you. And only if it's useful (and would make a huge
> performance difference) would it be worth putting in the kernel.

It seemed really usefull in uservfs incarnation. Unfortunately the
daemon was not multihreaded at that time, so it was not really usefull
on multiuser systems :-(

> Implementation of the _user_space_ library would be something like this:
> 
> 	#define MAXNAME 1024
> 	int open_cached_view(int base_fd, char *type, char *subname)

Well, you'd need more than simple open. For caching tar (etc), you'd
need stat_cached_view and opendir_cached_view and ...

And this really works, only that its called mc_open(), mc_stat() etc.

Gnome actually uses newer incarnation of mc_open etc, but they had to
introduce rather ugly interface to make it asynchronous.
								Pavel
-- 
When do you have heart between your knees?

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 1631 bytes --]

Salut,

On Wed, Sep 01, 2004 at 01:50:56PM -0700, Linus Torvalds wrote:
> 	#define MAXNAME 1024
> 	int open_cached_view(int base_fd, char *type, char *subname)
> 	{
> 	        struct stat st;
> 	        char filename[PATH_MAX];
> 	        char name[MAXNAME];
> 	        int len, cachefd;
> 	
> 	        if (fstat(base_fd, &st) < 0)
> 	                return -1;
> 	        sprintf(name, "/proc/self/fd/%d", base_fd);
> 	        len = readlink(name, filename, sizeof(filename)-1);
> 	        if (len < 0)
> 	                return -1;
> 	        filename[len] = 0;
> 	
> 	        /* FIXME! Replace '/' with '#' in "type" and "subname" */
> 	        len = snprintf(name, sizeof(name),
> 	                        "%04llx/%04llx/%s/%s/%s",
> 	                        (unsigned long long) st.st_dev,
> 	                        (unsigned long long) st.st_ino,
> 	                        type ? : "default",
> 	                        subname,
> 	                        filename);
> 	        errno = ENAMETOOLONG;
> 	        if (len >= sizeof(name))
> 	                return -1;
> 	        cachefd = open(name, O_RDONLY);
> 	        if (cachefd >= 0) {
> 	                /* Check mtime here - maybe we could have kernel support */
> 	                return cachefd;
> 	        }
> 	        if (errno != ENOENT)
> 	                return -1;
> 	        /*
> 	        .. try to generate cache file here ..
> 	         */

Around of  what I've  had in  mind. Only that  one might  use libmagic
instead of the type argument. The  rest can be done by a corresponding
MIME plugin.

				Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Horst von Brand]

Jamie Lokier <jamie@shareable.org> said:
> I'm going to explain why filesystem support for .tar.gz or other
> "document container"-like formats is useful.

Nobody disagrees there (I think), the disagreement is on whether the
usefullness is worth the hassle.

>                                               This does _not_ mean tar
> in the kernel (I know someone who can't read will think that if I
> don't say it isn't); it does mean hooks in the kernel for maintaining
> coherency between different views and filesystem support for cacheing.

"Coherency" and "different views" implies atomic transactions, and being
able to tell that an access to the file requieres updating random junk
about it. It requires being able to guess if it is worth updating now
(given that the file might be modified a dozen times more before the junk
being checked).

> The vision I'm going for here is:
> 
>   1. OpenOffice and similar programs store compound documents in
>      standard file formats, such as .tar.gz, compressed XML and such.

And they are doing fine AFAICS. Besides, they won't exactly jump on the
possibility of leaving behind all other OSes on which they run to become a
Linux-only format.

>      Fs support can reduce CPU time handling these sorts of files, as
>      I explain below, while still working with the standard file formats.

I don't buy this one. A tar.gz must be uncompressed and unpacked, and
whatever you could save is surely dwarfed by those costs.

>      With appropriate userspace support, programs can be written which
>      have access to the capabilities on all platforms, but reduced CPU
>      time occurs only on platforms with the fs support.

Userspace support isn't there on any of the platforms right now, if ever it
will be a strange-Linux-installation thing for quite some time to come. Not
exactly attractive for application writers.

>   2. Real-time indexing and local search engine tools.

Sure! Gimme the CPU power and disk throughput for that, pretty please. [No,
I won't tell I have better use for those right now...]

>                                                         This isn't
>      just things like local Google; it's also your MP3 player scanning
>      for titles & artists, your email program scanning for subject
>      lines to display the summary fast, your blog server caching built
>      pages, your programming environment scanning for tags, and your
>      file transfer program scanning for shared deltas to reduce bandwidth.

With no description on how this is supposed to work, this is pure science
fiction/wet dreams.

>      I won't explain how these work as it would make this mail too
>      long.  It should be clear to anyone who thinks about it why the
>      coherency mechanism is essential for real-time, and a consistent
>      interface to container internals helps with performance.

Coherency is essential, but it isn't free. Far from it. The easiest way of
getting coherency is having _one_ authoritative source. That way you don't
need coherency, and don't pay for it. Anything in this class must by force
be just hints, to be recomputed at a moment's notice. I.e., let the
application who might use it check and recompute as needed.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Jamie Lokier]

Horst von Brand wrote:
> >                                               This does _not_ mean tar
> > in the kernel (I know someone who can't read will think that if I
> > don't say it isn't); it does mean hooks in the kernel for maintaining
> > coherency between different views and filesystem support for cacheing.
> 
> "Coherency" and "different views" implies atomic transactions,

Wrong.  They imply ordering.

> and being able to tell that an access to the file requieres updating
> random junk about it.

Correct.

> It requires being able to guess if it is worth updating now
> (given that the file might be modified a dozen times more before the junk
> being checked).

Wrong.  Lazy and deterministic are compatible.

> 
> > The vision I'm going for here is:
> > 
> >   1. OpenOffice and similar programs store compound documents in
> >      standard file formats, such as .tar.gz, compressed XML and such.
> 
> And they are doing fine AFAICS. Besides, they won't exactly jump on the
> possibility of leaving behind all other OSes on which they run to become a
> Linux-only format.

You replied before reading the rest of the mail again, didn't you?
The whole purpose of the idea is to work with formats which are _not_
Linux-only.

> >      Fs support can reduce CPU time handling these sorts of files, as
> >      I explain below, while still working with the standard file formats.
> 
> I don't buy this one. A tar.gz must be uncompressed and unpacked, and
> whatever you could save is surely dwarfed by those costs.

Didn't they explain what "lazy" means in your computer science class?
Google for "lazy evaluation".

In this context it means _unnecessary_ uncomressions and compressions,
i.e. repeated ones, or ones where you never really needed the
compressed form, are eliminated.  Necessary ones are of course done.

Those are things which are currently wasting your CPU because apps
cannot eliminate them: to do so implies synchronisation between
different apps running at different times.  Surprise surprise, that's
something filesystems and kernels are good for.

> >      With appropriate userspace support, programs can be written which
> >      have access to the capabilities on all platforms, but reduced CPU
> >      time occurs only on platforms with the fs support.
> 
> Userspace support isn't there on any of the platforms right now, if ever it
> will be a strange-Linux-installation thing for quite some time to come. Not
> exactly attractive for application writers.

Again, you have a strange vision of what userspace support means.  I
say it means a portable and simple library for accessing components
inside a compressed container file in a standard format.

Something I believe OpenOffice et al. already has - so to say it
doesn't exist is both incorrect and missing the point.  The point is
to change that library which already exists, so it uses the filesystem
facility when available.  And to make the library better in standalone
form, so that other tools which manipulate container-like formats are
inclined to work with it as plugins, instead of creating their own
interface as cli tools.

_At no point is portability an issue_ - that library is already
portable to everything - and it's certainly not a strange-Linux thing
nor meant to become one.


> >   2. Real-time indexing and local search engine tools.
> 
> Sure! Gimme the CPU power and disk throughput for that, pretty please. [No,
> I won't tell I have better use for those right now...]

You still haven't grasped the idea of an algorithm which is more
complex but reduces CPU time and disk throughput, have you?

*Today*, when I open Rhythmbox I have to wait 5 minutes while it wastes
my CPU power and disk throughput scanning all the files in my Music
directory.

The entire point of these indexing schemes is so that programs like
Rhythmbox will display the same data without having to scan the disk
for 5 minutes every time they start up.  That's a near infinite
_saving_ of CPU power and time.

If you don't see that I'm just going to have to suggest you google for
"cache" and learn a bit about them.

> >      This isn't
> >      just things like local Google; it's also your MP3 player scanning
> >      for titles & artists, your email program scanning for subject
> >      lines to display the summary fast, your blog server caching built
> >      pages, your programming environment scanning for tags, and your
> >      file transfer program scanning for shared deltas to reduce bandwidth.
> 
> With no description on how this is supposed to work, this is pure science
> fiction/wet dreams.

Sigh.  If you must continue to hurl your blunt instruments around,
here is a description.  I didn't want to write this becuase it is off
topic, and you insist on not understanding the basic of algorithm
complexity and CPU usage, so you'll probably not understand this
either, but I'll give it a go.

    1. Local Google (by which I mean a search engine on your local machine),
       Real-time (by which I mean the results are always up to date):

       Every file modified since last search is known to the query engine.
       This is a reality: BeOS does it; WinFS is expected to do it.

       We know that it's possible to update free text indexes with
       small amounts of known changes quickly, at the time of a query.

       Thus we have real-time local free text search engine, and other
       features like searching inside files and for file names.  The
       point is the real-time nature of it: the results you get
       correspond to exactly the contents of the filesystem at the time of
       the query (writes which occur _during_ a query are obviously
       not coherent with this, but writes which complete before the
       query, even immediately before, appear in the results).

       Note that file write timing need not be affected - all the work
       can happen during queries (although it is best to have a
       nightly re-index or whatever so that the delay at query times
       is kept moderate).

    2. MP3 player scanning artists & titles:

       Easy.  MP3 player does what it does, shows the extracted ID
       tags from all .mp3 files in your home directory.  They do this
       already!  The difference is that with fs coherency hooks, they
       can _store_ that ID information, retrieve it later without
       having to scan all the .mp3s again (see Rhythmbox earlier), and
       keep their lists updated on the screen as soon as any .mp3 is
       changed or even any new ones created anywhere.

       Technically it's a simpler subset of real-time queries in 1.

    3. Email program scanning for subject lines fast:

       See Evolution; the only difference is stat() on a thousand
       files won't be required, it'll be exposed through a standard
       query instead of an Evolution-only cache method so other mail
       programs may use it as well as shell commands, and you can
       pretend you have mbox instead of maildir (mbox is a container
       format...).

    4. Blog server caching built pages:

       It's a reality already, obviously.  The difference is it'll
       make sense to built the pages through an indexed query on files
       instead of a database, e.g. one file per article, and
       (independent of the first point) the process of building can
       keep track of all the prerequisite files and scripts and
       templates used to produce the page, and actually expect to
       know, _with coherent accuracy_, if any of those prerequisites
       is different the next time the page is needed.

       In other words, complex script-generated web pages, cached
       output, and with no overhead when serving each page to check
       prerequisites (i.e. no 100 stat() calls), because we know the
       filesystem would have told us if any prerequisite had changed
       prior to the moment we begin serving the cached output.

       You can do something close to this already with dnotify (if you
       ignore that it doesn't tell you about new hard links, which is
       a dnotify bug), although doing the query part is unrealistic
       with dnotify and stat() prerequisites unless the directory
       names are structured thoughtfully.

    5. Programming environment scanning for tags:

       By now this should be obvious.  No need to run "exuberant
       ctags" or whatever class-hierarchy-extraction and
       documentation-extraction program every so often after making
       changes, and yet your IDE's clickable list of tags and notes
       stays up to date with file modifications in real time.  I think
       some IDEs do this already with moderate size trees, using
       stat(), but it's not realistic when you have tens of thousands
       of source files.

    6. File transfer program scanning for shared deltas.

       This is nothing more than searching all files for common
       subsequences which match strong hashes which are expected to be
       common during file transfer operations.  E.g. the GPL header at
       the start of many source files would have one such hash.  A
       hash of every whole file is another one, along with the name of
       the file (it is a key which indicates a "likely to match the
       corresponding hash" condition), as is a hash of every aligned
       64k subsequence, or whatever is appropriate to reduce disk I/O.

       Having an index of these can speed up some file transfer
       protocols, in the manner of rsync but comparing among a large
       group of files instead of just two.  The point is that kind of
       protocol can be more efficient than rsync (sometimes
       dramatically so), but it's only a net gain to use that kind of
       algorithm if you have a handy _and reliable_ index of likely
       common subsequences and whole-file hashes, otherwise it uses
       far too much disk I/O to check the group.  The index needs
       filesystem coherency support, otherwise it is not reliable
       enough for this bandwidth optimisation.

> >      I won't explain how these work as it would make this mail too
> >      long.  It should be clear to anyone who thinks about it why the
> >      coherency mechanism is essential for real-time, and a consistent
> >      interface to container internals helps with performance.
> 
> Coherency is essential, but it isn't free. Far from it. The easiest way of
> getting coherency is having _one_ authoritative source. That way you don't
> need coherency, and don't pay for it. Anything in this class must by force
> be just hints, to be recomputed at a moment's notice. I.e., let the
> application who might use it check and recompute as needed.

(a) Without some kind of minimal hook, there is no way for the application
to check without re-reading all the files it has used for a
computation, starting at the moment where it's about to potentially
recompute the result.

Sometimes you can get away with stat() on the files (e.g. Evolution
and fontconfig do that), but often you can't because that's not
trustworthy enough (security checks, protocol optimisations,
transparent "as if I didn't cache anything" semantics), and sometimes
stat() calls are far too slow anyway (anything that involves a lot of files).

(The trustworthiness problem is likely to be solved, for some
applications, by an xattr or such which is guaranteed to be deleted or
changed when a file is modified.  That is really a very minimal hook,
but it is one).

(b) Even with all the misgivings of stat(), you can't realistically
update real time displays, e.g. the lists of artists & albums on the
screen in your MP3 player whenever you modify anything in your music
collection using another program.  At best, you have to expect the
display to not notice the change for a while (minutes in the mp3
case), or until you click an "Explicit Refresh" button.  Neither
should be necessary.


It's true that the simplest coherency is one authoratitive source
which you have to check all the time (although even that doesn't work
for some things).  But checking all the time does rule out, due to its
high algorithmic complexity, a lot of very interesting applications
and even some interesting simple unixish tools which you might like,
such as real-time "locate".

-- Jamie

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Christer Weinigel]

Jamie Lokier <jamie@shareable.org> writes:

>     1. Local Google (by which I mean a search engine on your local machine),
>        Real-time (by which I mean the results are always up to date):
> 
>        Every file modified since last search is known to the query engine.
>        This is a reality: BeOS does it; WinFS is expected to do it.
> 
>        Thus we have real-time local free text search engine, and other
>        features like searching inside files and for file names.  The
>        point is the real-time nature of it: the results you get
>        correspond to exactly the contents of the filesystem at the time of
>        the query (writes which occur _during_ a query are obviously
>        not coherent with this, but writes which complete before the
>        query, even immediately before, appear in the results).

Can be done with dnotify/inotify and a cache daemon keeping track of
mtime.  Yes, this will need a kernel change to make sure mtime always
changed when the file changes, but it does not require anything else.

>     2. MP3 player scanning artists & titles:

Same.

>     3. Email program scanning for subject lines fast:

Same here.

>     4. Blog server caching built pages:
>     5. Programming environment scanning for tags:
>     6. File transfer program scanning for shared deltas.

And so on.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Jamie Lokier]

Christer Weinigel wrote:
> Can be done with dnotify/inotify and a cache daemon keeping track of
> mtime.  Yes, this will need a kernel change to make sure mtime always
> changed when the file changes, but it does not require anything else.

    - Can the daemon keep track of _every_ file on my disk like this?
      That's more than a million files, and about 10^5 directories.
      dnotify would require the daemon to open all the directories.
      I'm not sure what inotify offers.

    - What happens at reboot - I guess the daemon has to call stat()
      on every file to verify its indexes? Have you any idea how long
      it takes to call stat() on every file in my home directory?

    - The ordering problem: I write to a file, then the program
      returns.  System is very busy compiling.  2 minutes later, I
      execute a search query.  The file I wrote two minute ago doesn't
      appear in the search results.  What's wrong?

      Due to scheduling, the daemon hasn't caught up yet.  Ok, we can
      accept that's just hard life.  Sometimes it takes a while for
      something I write to appear in search results.

      But!  That means I can't use these optimised queries as drop-in
      replacements for calling grep and find, or for making Make-like
      programs run faster (by eliminating parsing and stat() calls).
      That's a shame, it would have been nice to have a mechanism that
      could transparently optimise prorgrams that do calculations....

Do you see what I'm getting at?  There's building some nice GUI
and search engine like functionality, where changes made by one
program _eventually_ show up in another (i.e. not synchronously).

That's easy.

And then there's optimising things like grep, find, perl, gcc,
make, httpd, rsync, in a way that's semantically transparent, but
executes faster _as if_ they had recalculated everything they
need to every time.  That's harder.

> >     3. Email program scanning for subject lines fast:
> 
> Same here.
> 
> >     4. Blog server caching built pages:
> >     5. Programming environment scanning for tags:
> >     6. File transfer program scanning for shared deltas.
> 
> And so on.

No, not 3, 4 or 6.  For correct behaviour those require synchronous
query results.  Think about 6, where one important cached query is
"what is the MD5 sum of this file", and another critical one, which
can only work through indexing, is "give me the name of any file whose
MD5 sum matches $A_SPECIFIC_MD5".  Trusting the async results for
those kind of queries from your daemon would occasionally result in
data loss due to race conditions.  So you wouldn't trust the async
results, and you fail to get those CPU-saving and bandwidth-saving
optimisations.

-- Jamie

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Alan Cox]

On Iau, 2004-09-02 at 22:47, Jamie Lokier wrote:
>     - Can the daemon keep track of _every_ file on my disk like this?
>       That's more than a million files, and about 10^5 directories.
>       dnotify would require the daemon to open all the directories.
>       I'm not sure what inotify offers.

This is currently a real issue for both desktop search and for virus
scanners. They want a "what changed and where" system wide (or at least
per namespace/mount).

Re: Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Jon Smirl]

On Thu, 02 Sep 2004 23:27:06 +0100, Alan Cox <alan@lxorguk.ukuu.org.uk> wrote:
> On Iau, 2004-09-02 at 22:47, Jamie Lokier wrote:
> >     - Can the daemon keep track of _every_ file on my disk like this?
> >       That's more than a million files, and about 10^5 directories.
> >       dnotify would require the daemon to open all the directories.
> >       I'm not sure what inotify offers.
> 
> This is currently a real issue for both desktop search and for virus
> scanners. They want a "what changed and where" system wide (or at least
> per namespace/mount).

In the database work everything is a transaction and the transactions
are logged. Reiser4 is fully atomic and logged. So to get the "what
changed and where" you just process the transaction logs from the
point of your last marked checkpoint. Hot backup db servers work this
way too by listening to the transaction log stream. You don't need a
daemon in this model.

Jon Smirl
jonsmirl@gmail.com

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Christer Weinigel]

Jamie Lokier <jamie@shareable.org> writes:

> Christer Weinigel wrote:
> > Can be done with dnotify/inotify and a cache daemon keeping track of
> > mtime.  Yes, this will need a kernel change to make sure mtime always
> > changed when the file changes, but it does not require anything else.
> 
>     - Can the daemon keep track of _every_ file on my disk like this?
>       That's more than a million files, and about 10^5 directories.
>       dnotify would require the daemon to open all the directories.
>       I'm not sure what inotify offers.

I don't think dnotify/inotify handles subdirectories well yet, so I
suppose that they would have to be extended.

>     - What happens at reboot - I guess the daemon has to call stat()
>       on every file to verify its indexes? Have you any idea how long
>       it takes to call stat() on every file in my home directory?

The daemon saves state before it shuts down and reloads the state
after a reboot.  You have to make sure that it is started first and
stopped last during the boot process.  How would a kernel plugin
handle things that happen before or after the plugin module has been
loaded?  It's the same problem.

>     - The ordering problem: I write to a file, then the program
>       returns.  System is very busy compiling.  2 minutes later, I
>       execute a search query.  The file I wrote two minute ago doesn't
>       appear in the search results.  What's wrong?
>
>       Due to scheduling, the daemon hasn't caught up yet.  Ok, we can
>       accept that's just hard life.  Sometimes it takes a while for
>       something I write to appear in search results.
> 
>       But!  That means I can't use these optimised queries as drop-in
>       replacements for calling grep and find, or for making Make-like
>       programs run faster (by eliminating parsing and stat() calls).
>       That's a shame, it would have been nice to have a mechanism that
>       could transparently optimise prorgrams that do calculations....

Sure you can.  First of all, you can just wait for the daemon to
finish indexing any files that it has been notified about changes in.
This is no different from you having to wait for the kernel to finish
indexing the files.  Or are you suggesting that the kernel should stop
all other processes until the indexing is done?

> Do you see what I'm getting at?  There's building some nice GUI
> and search engine like functionality, where changes made by one
> program _eventually_ show up in another (i.e. not synchronously).
> 
> That's easy.
> 
> And then there's optimising things like grep, find, perl, gcc,
> make, httpd, rsync, in a way that's semantically transparent, but
> executes faster _as if_ they had recalculated everything they
> need to every time.  That's harder.

If you have a good notify, it's not harder.  

> No, not 3, 4 or 6.  For correct behaviour those require synchronous
> query results.  Think about 6, where one important cached query is
> "what is the MD5 sum of this file", and another critical one, which
> can only work through indexing, is "give me the name of any file whose
> MD5 sum matches $A_SPECIFIC_MD5".  Trusting the async results for
> those kind of queries from your daemon would occasionally result in
> data loss due to race conditions.  So you wouldn't trust the async
> results, and you fail to get those CPU-saving and bandwidth-saving
> optimisations.

So how do you calculate the MD5 sum of a file that is in the process
of being modified?  It's not possible to do that unless you block all
other access to that file and recalculate the MD5 sum after each
write.  With a notifier that tells the daemon that it has stale data
and needs to reprocess the file, it's no different.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

[Chris Dukes]

On Mon, Sep 06, 2004 at 05:55:21PM +0200, Christer Weinigel wrote:

Could ya'll take this to slashdot for a bit just to improve the level
of commentary there?
http://slashdot.org/article.pl?sid=04/09/06/1235236

-- 
Chris Dukes
Warning: Do not use the reflow toaster oven to prepare foods after
it has been used for solder paste reflow. 
http://www.stencilsunlimited.com/stencil_article_page5.htm

Re: silent semantic changes with reiser4

[Timothy Miller]

Jamie Lokier wrote:
[snip
> When a simple "cd" into .tar.gz or .iso is implemented properly, it
> will have _no_ performance penalty after you have first looked in the
> file, so long as it remains in the on-disk cache.  And, the filesystem
> will manage that cache intelligently.
> 
> Imagine: for looking at source files and such, you probably won't
> bother untarring in future, and you won't bother keeping untarred
> source trees in your home directory for easy access to things you look
> at often.  Why waste the space?  You could install whole applications
> as a .tar and run them from within it, with no performance penalty.
> 
> Similarly, the filesystem will be able to archive directories
> automatically that haven't been touched in a long time, with no
> visible change except increased storage space.  "grep" will be a bit
> slower, but you'll have a useful search tool by then (using coherent
> indexes) which will be more useful than grep, and much faster.
> 

[Again, pardon me for being 5000 messages behind.]

Anyhow, I recall reading an article about 'unified name spaces', and I'm 
not referring to what's on namesys.com.  It mentioned how putting device 
nodes into the file system is a very powerful innovation of UNIX. 
Having files and devices in the same name space simplifies tools and 
makes the environment much more powerful, because you can connect data 
sources and targets together more arbitarily.

If I recall correctly, the article mentioned something about Plan9 
taking things to a greater extreme than UNIX.

Going along with what you said, Jamie, if "containers" like tar files 
could be accessed like directories without being first extracted, it 
would increase the power and flexibility of the whole system.  Windows 
XP does something like this with the way it presents ZIP files as 
directories, and although I'm sure the Windows way isn't nearly as 
efficient and universal as how Linux would do it, I find the feature to 
be INCREDIBLY useful.

Of course, we don't necessarily want codecs for every archive format 
ever invented to be embedded in the kernel.  Instead, we need to do 
something like how you can mount an ISO on a loopback device, only more 
transparently and more flexibly, and with the less performance-critical 
codecs being implemented as daemons in userspace.

Also, there would be limitations.  For instance, many such 
pseudo-directories would be read-only, and some would be writable only 
when on some file systems (like writing to a compressed archive might be 
much easier to implement on Reiser4 than something else, or perhaps the 
ext3 version would have to do a lot more copying, while the Reiser 
version could take advantage of Reiser-specific features like inserting 
in the middle of a file).

Re: silent semantic changes with reiser4

[Hans Reiser]

A friend asked me a question, and because he is very bright it reminded 
me that I have not done a good job of reviewing the history of the 
design's evolution.

He asked me, why not just access a filename's size as filename/size?

So, the original idea was to access metafiles as just files within a 
directory, and it actually remains that way. However, I first decided to 
make the standard unix metafiles less intrusive on the namespace. So 
that led to calling it filename/..size and filename/..owner, etc. In 
this scheme, the use of '..' was just a style convention for metafiles, 
and not a requirement in anyway.

This was actually pretty decent as a design, but then a user on the 
mailing list suggested replacing the '..' prefix with a subdirectory 
prefix. I forget who suggested this and what the prefix was exactly. So 
we then created a '..metas' subdirectory, and this had the advantage 
that one could ls it to see all the builtins supported by a given 
plugin. This is not an important advantage, and I encourage others to 
critique it.

So, instead of filename/size we have filename/..metas/size. The only 
thing gained by this is that 'size' has a rarer name of '..metas/size'. 
The use of the '..metas/' prefix is purely a non-mandated style 
convention. File plugins that dislike it are free to violate the 
convention. There is no deep semantic to it, just a cowardly aversion to 
intruding on current namespace usage with something as common as 'size'. 
It has the significant disadvantage of being a longer name than 'size' 
or '..size'. I could be talked out of it.

Hans

Re: silent semantic changes with reiser4

[Peter Foldiak]

On Fri, 2004-09-10 at 06:22, Hans Reiser wrote:
> He asked me, why not just access a filename's size as filename/size?

I now understand that you need a way to distinguish between something
like

shoe/size

and

shoe/.../size   (or shoe/..size)

The first one is the size of the shoe, the second is the automatically
generated size of the file (object). You would get into trouble if you
would not allow the user to use shoe/size for shoe size. Peter

Re: silent semantic changes with reiser4

[Hans Reiser]

Peter Foldiak wrote:

>On Fri, 2004-09-10 at 06:22, Hans Reiser wrote:
>  
>
>>He asked me, why not just access a filename's size as filename/size?
>>    
>>
>
>I now understand that you need a way to distinguish between something
>like
>
>shoe/size
>
>and
>
>shoe/.../size   (or shoe/..size)
>
>The first one is the size of the shoe, the second is the automatically
>generated size of the file (object). You would get into trouble if you
>would not allow the user to use shoe/size for shoe size. Peter
>
>
>
>
>  
>
exactly.

Of course, problem/shoe/size could refer to shoe size in centimeters of 
a problem shoe or the size of the problem relating to a shoe in units of 
reporters providing press coverage of it or....

So there are lots of opportunities for ambiguity in semantics....

Still, widely used builtins seem like they should be moderately evasive 
of commonly used names.

Re: silent semantic changes with reiser4

[Timothy Miller]

Hans Reiser wrote:
> Peter Foldiak wrote:
> 
>> On Fri, 2004-09-10 at 06:22, Hans Reiser wrote:
>>  
>>
>>> He asked me, why not just access a filename's size as filename/size?
>>>   
>>
>>
>> I now understand that you need a way to distinguish between something
>> like
>>
>> shoe/size
>>
>> and
>>
>> shoe/.../size   (or shoe/..size)
>>
>> The first one is the size of the shoe, the second is the automatically
>> generated size of the file (object). You would get into trouble if you
>> would not allow the user to use shoe/size for shoe size. Peter
>>
>>
>>
>>
>>  
>>
> exactly.
> 
> Of course, problem/shoe/size could refer to shoe size in centimeters of 
> a problem shoe or the size of the problem relating to a shoe in units of 
> reporters providing press coverage of it or....
> 
> So there are lots of opportunities for ambiguity in semantics....
> 
> Still, widely used builtins seem like they should be moderately evasive 
> of commonly used names.

You know, if tools all need to be rewritten anyway to deal with the file 
metadata "directory", then why not change the symbol that delimits the 
metadata key?

Everyone likes ':', so we'd have "problem/shoe:size".  (Don't bother to 
complain about files which have : in them, because I already know it 
sucks, but it's an example.)

See, unless you can come up with a way to seamlessly make old tools work 
with the new semantics, then there's no reason not to make more than one 
change to tools at at the same time.

Also, if you take the ':' example literally, then the file system would 
need to be able to figure out that a file whose literal name is 
"C:\MYPR0NDIR\BODY_PARTS.JPG" isn't referring to metadata for a file 
named "C".  :)

Re: silent semantic changes with reiser4

[Wayne Scott]

From: Timothy Miller <miller@techsource.com>
> Everyone likes ':', so we'd have "problem/shoe:size".  (Don't bother to 
> complain about files which have : in them, because I already know it 
> sucks, but it's an example.)

[[ I just joined this discussion, so pardon if this is already known.]]

One advantage of ':' is that portable programs already have to avoid
it because of NTFS alternate data streams:
  http://www.diamondcs.com.au/index.php?page=archive&id=ntfs-streams

For example on an XP box with NTFS:

$ mkdir j
$ cd j
$ echo hi > foo:bar
$ ls -l
total 0
-rw-r--r--    1 wscott   Administ        0 Sep 10 10:45 foo
$ cat foo
$ cat foo:bar
hi
$ rm foo
$ cat foo:bar
cat: foo:bar: No such file or directory

-Wayne

Re: silent semantic changes with reiser4

[Horst von Brand]

Timothy Miller <miller@techsource.com> said:
> > Everyone likes ':', so we'd have "problem/shoe:size".  (Don't bother to 
> > complain about files which have : in them, because I already know it 
> > sucks, but it's an example.)
> 
> [[ I just joined this discussion, so pardon if this is already known.]]

> One advantage of ':' is that portable programs already have to avoid
> it because of NTFS alternate data streams:

Portable programs working on POSIX don't give a damn that C:, CON:, etc are
reserved in MS-DOS.

Windows warts shouldn't infect Linux.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Uwe Ohse]

Wayne Scott wrote:

>One advantage of ':' is that portable programs already have to avoid
>it because of NTFS alternate data streams:

this is not going to work:

lynx www.site.tld:8080
ls /usr/share/man/man3/*::*            ## perl stuff
http://cr.yp.to/proto/maildir.html

Regards, Uwe

Re: silent semantic changes with reiser4

[Hans Reiser]

Timothy Miller wrote:

>
>
> Hans Reiser wrote:
>
>> Peter Foldiak wrote:
>>
>>> On Fri, 2004-09-10 at 06:22, Hans Reiser wrote:
>>>  
>>>
>>>> He asked me, why not just access a filename's size as filename/size?
>>>>   
>>>
>>>
>>>
>>> I now understand that you need a way to distinguish between something
>>> like
>>>
>>> shoe/size
>>>
>>> and
>>>
>>> shoe/.../size   (or shoe/..size)
>>>
>>> The first one is the size of the shoe, the second is the automatically
>>> generated size of the file (object). You would get into trouble if you
>>> would not allow the user to use shoe/size for shoe size. Peter
>>>
>>>
>>>
>>>
>>>  
>>>
>> exactly.
>>
>> Of course, problem/shoe/size could refer to shoe size in centimeters 
>> of a problem shoe or the size of the problem relating to a shoe in 
>> units of reporters providing press coverage of it or....
>>
>> So there are lots of opportunities for ambiguity in semantics....
>>
>> Still, widely used builtins seem like they should be moderately 
>> evasive of commonly used names.
>
>
> You know, if tools all need to be rewritten anyway to deal with the 
> file metadata "directory", then why not change the symbol that 
> delimits the metadata key?

because it is useful that it is only a style convention.  Changing the 
symbol makes it mandatory to distinguish metafiles from files.

>
> Everyone likes ':', so we'd have "problem/shoe:size".  (Don't bother 
> to complain about files which have : in them, because I already know 
> it sucks, but it's an example.)
>
> See, unless you can come up with a way to seamlessly make old tools 
> work with the new semantics, then there's no reason not to make more 
> than one change to tools at at the same time.
>
> Also, if you take the ':' example literally, then the file system 
> would need to be able to figure out that a file whose literal name is 
> "C:\MYPR0NDIR\BODY_PARTS.JPG" isn't referring to metadata for a file 
> named "C".  :)
>
>
>

Re: silent semantic changes with reiser4

[Horst von Brand]

Hans Reiser <reiser@namesys.com> said:
> Timothy Miller wrote:

[...]

> > You know, if tools all need to be rewritten anyway to deal with the 
> > file metadata "directory", then why not change the symbol that 
> > delimits the metadata key?

> because it is useful that it is only a style convention.  Changing the 
> symbol makes it mandatory to distinguish metafiles from files.

If they are really different than directories, it should be clear what is
what, IMVHO. If they aren't different, they have no place here. Trying to
break hoary Unix tradition (files and directories are separate) while
pretending nothing has changed (see, a file can be handled just like a
directory) just doesn't cut it.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Timothy Miller]

Horst von Brand wrote:
> Hans Reiser <reiser@namesys.com> said:
> 
>>Timothy Miller wrote:
> 
> 
> [...]
> 
> 
>>>You know, if tools all need to be rewritten anyway to deal with the 
>>>file metadata "directory", then why not change the symbol that 
>>>delimits the metadata key?
> 
> 
>>because it is useful that it is only a style convention.  Changing the 
>>symbol makes it mandatory to distinguish metafiles from files.
> 
> 
> If they are really different than directories, it should be clear what is
> what, IMVHO. If they aren't different, they have no place here. Trying to
> break hoary Unix tradition (files and directories are separate) while
> pretending nothing has changed (see, a file can be handled just like a
> directory) just doesn't cut it.


I think that sums up what I was trying to say better than what I said 
did.  :)

Re: silent semantic changes with reiser4

[Horst von Brand]

Hans Reiser <reiser@namesys.com> said:
> A friend asked me a question, and because he is very bright it reminded 
> me that I have not done a good job of reviewing the history of the 
> design's evolution.

The design's evolution, and your friend's question, are irrelevant. Might
be a nice story, but not relevant to discuss the _current_ design, its
shortcommings, and possible solutions.

> He asked me, why not just access a filename's size as filename/size?

Urgh.

> So, the original idea was to access metafiles as just files within a 
> directory, and it actually remains that way. However, I first decided to 
> make the standard unix metafiles less intrusive on the namespace. So 
> that led to calling it filename/..size and filename/..owner, etc. In 
> this scheme, the use of '..' was just a style convention for metafiles, 
> and not a requirement in anyway.

Oh, so metafiles could have any names at all. Awful.

> This was actually pretty decent as a design,

Could have been worse, true. Not very much, anyway.

>                                              but then a user on the 
> mailing list suggested replacing the '..' prefix with a subdirectory 
> prefix. I forget who suggested this and what the prefix was exactly. So 
> we then created a '..metas' subdirectory,

Bad design. As Ted explained, _any_ "normal" name in there will lead to
legacy applications do the wrong thing unaware of the special status of
this stuff, and this is sure to break things and/or to lead to nasty
security problems.

>                                           and this had the advantage 
> that one could ls it to see all the builtins supported by a given 
> plugin.

What kind of builtins? What is a "plugin"?

>         This is not an important advantage, and I encourage others to 
> critique it.

If it is not important, let's postpone any discussion on it until (if ever)
the important problems are solved.

> So, instead of filename/size we have filename/..metas/size. The only 
> thing gained by this is that 'size' has a rarer name of '..metas/size'. 

English, please.

> The use of the '..metas/' prefix is purely a non-mandated style 
> convention.

So it isn't mandated, and I could decide placing my own stuff under other
names, willy-nilly? See the comment above on "normal names" being
dangerous; if it hasn't a builtin way of ensuring legacy (or even new
applications) don't trip over this stuff, it is a moby nightmare security
and correctness wise.

>             File plugins that dislike it are free to violate the 
> convention. There is no deep semantic to it, just a cowardly aversion to 
> intruding on current namespace usage

"Common" and "current usage" is important to the average user, "normal" is
crucial for correctness and security,

>                                      with something as common as 'size'. 
> It has the significant disadvantage of being a longer name than 'size' 
> or '..size'. I could be talked out of it.

So far, I've seen no real reason to introduce this. If it is scrapped,
there will be no need to talk anybody out of silly ideas.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Joel Becker]

On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote:
> Yes, this was part of the plan, tar file-directory plugins would be cute.

	Question:  Is "cat /foo/bar/baz.tar.gz/metas" the attribute
directory or a directory in the tarball named "metas"?

Joel

-- 

"I'm so tired of being tired,
 Sure as night will follow day.
 Most things I worry about
 Never happen anyway."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: joel.becker@oracle.com
Phone: (650) 506-8127

Re: silent semantic changes with reiser4

[Jamie Lokier]

Joel Becker wrote:
> On Thu, Aug 26, 2004 at 01:49:15AM -0700, Hans Reiser wrote:
> > Yes, this was part of the plan, tar file-directory plugins would be cute.
> 
> 	Question:  Is "cat /foo/bar/baz.tar.gz/metas" the attribute
> directory or a directory in the tarball named "metas"?

This needs to be designed.

Perhaps /foo/bar/baz.tar.gz/tar/metas is the directory in the tarball
named "metas".

Or perhaps /foo/bar/baz.tar.gz/x/metas is: it's independent of archive
format, and I personally tend to extract things into a directory
called "x". [*]

Or perhaps /foo/bar/baz.tar.gz/metas is, and the attribute directory
is /foo/bar/baz.tar.gz/../metas, to be perverse ;)

I prefer the second one, ("x/metas"), but not with any conviction.

-- Jamie


[*] Actually I prefer:

      /foo/bar/baz.tar.gz/content/metas
      /foo/bar/baz-0.01.tar.gz/content/baz-0.01/metas

           Archives always in "content".  One layer of decompression
           always tried for .tar files and other uncompressed archive
           formats.

      /foo/bar/baz.tar.gz/x -> content/
      /foo/bar/baz-0.01.tar.gz/x -> content/baz-0.01/

           If the root of the archive contains a single directory, "x"
           is a symlink to it.  Otherwise "x" is a symlink to the root
           directory of the archive.  This is comfortable with the
           common practice by which archives are distributed, without
           making a mess when someone forgets to put everything in a
           top-level directory.

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 11:35:32AM -0700, Joel Becker wrote:
>	Question:  Is "cat /foo/bar/baz.tar.gz/metas" the attribute
>directory or a directory in the tarball named "metas"?

This has been fought over on the reiserfs-list ad nauseaum, but
it's a valid point.

That's why I tend to rename metas/ into ..metas/ to avoid
name clashes, even if I've never had a directory named metas/
apart from what Reiser4 ships.

It is then debatable if it should be renamed before it's too
late, have it renamable in the kernel configs (and the name
exported via /sys or something) or just leave it be.

-- 
mjt

Re: silent semantic changes with reiser4

[Matthew Wilcox]

On Fri, Aug 27, 2004 at 12:19:54PM +0300, Markus   Törnqvist wrote:
> That's why I tend to rename metas/ into ..metas/ to avoid
> name clashes, even if I've never had a directory named metas/
> apart from what Reiser4 ships.

http://packages.debian.org/cgi-bin/search_contents.pl?word=metas&searchmode=searchfilesanddirs&case=insensitive&version=unstable&arch=i386

OK, those are capital METAS rather than junior metas, but it does show
this is not a unique word to reiser4.

-- 
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain

Re: silent semantic changes with reiser4

[Chris Wedgwood]

On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote:

> One of the big potential uses for file-as-directory is to go inside
> archive files, ELF files, .iso files and so on in a convenient way.

Arguably this belongs in userspace --- and people have put it there.


  --cw

Re: silent semantic changes with reiser4

[Jamie Lokier]

Chris Wedgwood wrote:
> On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote:
> 
> > One of the big potential uses for file-as-directory is to go inside
> > archive files, ELF files, .iso files and so on in a convenient way.
> 
> Arguably this belongs in userspace --- and people have put it there.

I agree that these belong in userspace, and that there's plenty* of
userspace code doing a similar thing already.  I don't think there's
any argument over it.

However, as far as I know it's not accessible in a file-as-directory
form as yet.  In my opinion that is the most natural form and it would
be very intuitive to use.  I hope we can pick a useful semantics for
them, and also provide filesystem-independent plugins with GNU
Hurd-like per-user extensibility.

-- Jamie

* plenty == too much.
  Gnome, KDE, Emacs and Bash all see different virtual filesystems.
  (All but Bash implement their own virtual filesystem extensions).
  That makes them much less useful than they could be.

Re: silent semantic changes with reiser4

[Spam]

> Chris Wedgwood wrote:
>> On Thu, Aug 26, 2004 at 02:00:49AM +0100, Jamie Lokier wrote:
>> 
>> > One of the big potential uses for file-as-directory is to go inside
>> > archive files, ELF files, .iso files and so on in a convenient way.
>> 
>> Arguably this belongs in userspace --- and people have put it there.

> I agree that these belong in userspace, and that there's plenty* of
> userspace code doing a similar thing already.  I don't think there's
> any argument over it.

> However, as far as I know it's not accessible in a file-as-directory
> form as yet.  In my opinion that is the most natural form and it would
> be very intuitive to use.  I hope we can pick a useful semantics for
> them, and also provide filesystem-independent plugins with GNU
> Hurd-like per-user extensibility.

> -- Jamie

> * plenty == too much.
>   Gnome, KDE, Emacs and Bash all see different virtual filesystems.
>   (All but Bash implement their own virtual filesystem extensions).
>   That makes them much less useful than they could be.

  Exactly,  and  I  doubt  they  have the ability to join together and
  create  a  common  uniform standard out of it either. They have just
  far to different views on Linux and how they want things to be done.

Re: silent semantic changes with reiser4

[Horst von Brand]

Spam <spam@tnonline.net> said:
> > Chris Wedgwood wrote:

[...]

> >   Gnome, KDE, Emacs and Bash all see different virtual filesystems.
> >   (All but Bash implement their own virtual filesystem extensions).
> >   That makes them much less useful than they could be.

>   Exactly,  and  I  doubt  they  have the ability to join together and
>   create  a  common  uniform standard out of it either. They have just
>   far to different views on Linux and how they want things to be done.

And the solution is... inventing yet another, incompatible way of doing
things that nobody outside its followers will ever use. Inside the kernel.
Should have thought of that before.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Will Dyson]

Jamie Lokier wrote:

> However, as far as I know it's not accessible in a file-as-directory
> form as yet.  In my opinion that is the most natural form and it would
> be very intuitive to use.  I hope we can pick a useful semantics for
> them, and also provide filesystem-independent plugins with GNU
> Hurd-like per-user extensibility.
> 
> -- Jamie
> 
> * plenty == too much.
>   Gnome, KDE, Emacs and Bash all see different virtual filesystems.
>   (All but Bash implement their own virtual filesystem extensions).
>   That makes them much less useful than they could be.

It has always bugged me that Gnome and KDE implement their own VFS layers.

It seems to me that having a standard userspace filesystem layer like 
FUSE (http://sourceforge.net/projects/avf) would provide a lot of the 
benefit that HURD users (all 5 of them) see from filesystem 
"translators". Now, safely allowing unprivileged users to run arbitrary 
userspace filesystems would be a real trick. But, if it were possible, 
it could be combined with files-as-directories in some rather neat ways.

-- 
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 11:50, Will Dyson wrote:
> Jamie Lokier wrote:
> 
> > However, as far as I know it's not accessible in a file-as-directory
> > form as yet.  In my opinion that is the most natural form and it would
> > be very intuitive to use.  I hope we can pick a useful semantics for
> > them, and also provide filesystem-independent plugins with GNU
> > Hurd-like per-user extensibility.
> > 
> > -- Jamie
> > 
> > * plenty == too much.
> >   Gnome, KDE, Emacs and Bash all see different virtual filesystems.
> >   (All but Bash implement their own virtual filesystem extensions).
> >   That makes them much less useful than they could be.
> 
> It has always bugged me that Gnome and KDE implement their own VFS layers.
> 

Same here.  This always seemed like something the kernel should be able
to handle.  It seems to me that if reiser4 had been available at the
time the Gnome and KDE developers would not have needed to do this.

Lee

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell:

> > It has always bugged me that Gnome and KDE implement their own VFS layers. 
> 
> Same here.  This always seemed like something the kernel should be able
> to handle.  It seems to me that if reiser4 had been available at the
> time the Gnome and KDE developers would not have needed to do this.

Well, the kernel doesn't have a filesystem that speaks http, scp and
those things. GnomeVFS is URL-based. It has some pseudo-protocols that
extract a pseudo directory-tree for all installed applications + the
changes the used made, created on the fly from a set of XML files that
are read-only and system-wide and the user-overridden changes. I don't
know if all of these things would really make sense inside the kernel.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 16:50, Christophe Saout wrote:
> Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell:
> 
> > > It has always bugged me that Gnome and KDE implement their own VFS layers. 
> > 
> > Same here.  This always seemed like something the kernel should be able
> > to handle.  It seems to me that if reiser4 had been available at the
> > time the Gnome and KDE developers would not have needed to do this.
> 
> Well, the kernel doesn't have a filesystem that speaks http, scp and
> those things. GnomeVFS is URL-based. It has some pseudo-protocols that
> extract a pseudo directory-tree for all installed applications + the
> changes the used made, created on the fly from a set of XML files that
> are read-only and system-wide and the user-overridden changes. I don't
> know if all of these things would really make sense inside the kernel.
> 

True.  FWIW, I never use most of those features.  It's just too damn
slow.  Windows seems to implement all of the useful features of
GnomeVFS, and they are 10x faster.

Lee

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote:
>On Thu, 2004-08-26 at 16:50, Christophe Saout wrote:
>> are read-only and system-wide and the user-overridden changes. I don't
>> know if all of these things would really make sense inside the kernel.
>True.  FWIW, I never use most of those features.  It's just too damn
>slow.  Windows seems to implement all of the useful features of
>GnomeVFS, and they are 10x faster.

Are they in the kernel in Windows?

-- 
mjt

Re: silent semantic changes with reiser4

[Spam]

Hello Markus,

Friday, August 27, 2004, 11:21:31 AM, you wrote:

> On Thu, Aug 26, 2004 at 04:57:26PM -0400, Lee Revell wrote:
>>On Thu, 2004-08-26 at 16:50, Christophe Saout wrote:
>>> are read-only and system-wide and the user-overridden changes. I don't
>>> know if all of these things would really make sense inside the kernel.
>>True.  FWIW, I never use most of those features.  It's just too damn
>>slow.  Windows seems to implement all of the useful features of
>>GnomeVFS, and they are 10x faster.

> Are they in the kernel in Windows?

  Some are, some aren't :)

  Many  of the features are between the FS and the application layers.
  Therefore  they will work with all programs that use the Windows API
  to  access  files  etc.  I  do not think that many programs actually
  access   disks  directly  at the FS or below level. Exceptions would
  perhaps be file-recovery tools etc.

  Others are simply plugins to Windows Explorer.

  I  don't  know  if  all,  any,  or none, are installed at the kernel
  level. I doubt they are.

  But  the  problem  with  Linux  seem to be that programs actually do
  access  the  FS  directly.  Therefore  everything  will break if you
  change the FS structures to much.

  Like I said, this is IMHO something that Linux should somehow solve.
  Applications  should  really  not  access  the filesystems directly.
  Perhaps Grub and stuff like it should, but not many other.

  ~S

Re: silent semantic changes with reiser4

[Jamie Lokier]

Lee Revell wrote:
> Same here.  This always seemed like something the kernel should be able
> to handle.  It seems to me that if reiser4 had been available at the
> time the Gnome and KDE developers would not have needed to do this.

reiser4 has prompted the discussion again, but it doesn't provide the
virtual filesystem capabilities that we're talking about, which Gnome
and KDE implement in userspace.  The FUSE project is much more
relevant for that.

We had file-as-directory discussions years ago, and 5 years ago I was
experimenting with doing virtual filesystems with that capability over NFS.

podfuk offered something close to Gnome and KDE but in kernel space:
virtual filesytems accessible through the kernel, but it didn't get
too popular.  It was based on Midnight Commander's VFS (which has
evolved into Gnome's VFS), using the CODA filesystem interface to
provide the kernel hooks.  It had some problems, and was a bit fiddly
to use.  We still don't have something that's particularly nice to use
for ordinary users.

-- Jamie

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > > One of the big potential uses for file-as-directory is to go inside
> > > archive files, ELF files, .iso files and so on in a convenient way.
> > 
> > Arguably this belongs in userspace --- and people have put it there.
> 
> I agree that these belong in userspace, and that there's plenty* of
> userspace code doing a similar thing already.  I don't think there's
> any argument over it.
> 
> However, as far as I know it's not accessible in a file-as-directory
> form as yet.  In my opinion that is the most natural form and it would
> be very intuitive to use.  

It does not work. .deb file is ar archive. It is also debian
package. How do you know if I want to see it as a package or as a archive?
How do you identify file types, anyway?

Same happens for .tar.gz.

uservfs does

cd foo.deb#uar
vs.
cd foo.deb#udeb

and

cd foo.tgz#utar
vs.
cat foo.tgz#ugz


-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pavel Machek wrote:
[...]
| uservfs does
|
| cd foo.deb#uar
cd foo.deb/ar
| vs.
| cd foo.deb#udeb
cd foo.deb/deb

and why would you want that, instead of just:
cd foo.deb	# for the ar
dpkg -i foo.deb	# for the deb

|
| and
|
| cd foo.tgz#utar
cd foo.tgz
| vs.
| cat foo.tgz#ugz
cat foo.tgz

Just want to extract the tar file?  Maybe something like
cat foo.tgz/gunzip
In which case (of course) foo.tgz/gunzip has exactly the same directory
contents as foo.tgz

Looks different, that's all.

In fact, for just about any syntax anyone could suggest, I can't really
see why you can't just replace all weird symbols with a slash and a
symbol.  Instead of
	foo.tgz#utar
you have
	foo.tgz/#/utar
Only difference is, some things which used to require special tools can
now be serviced by less than what's in busybox.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTPNpngHNmZLgCUhAQJ0PA//RmPgHL4MgiQn7Li5b52DI/0p+Zp9JUs7
OaA+k4wHeXEIdBProtBQ+3541noEg086bVriYRnajsA9xgd/P9ncE7ci/d9Kagdl
8KmH4eWnCjNkL4x8GnDeR+EM5EfcgoSxR7ezO1KO9uMMcPFnCvmeB1yB5U+Fr9Q1
HVPHfGsh+ZWU2CFJHoJx27Q07UK+bFvzGr4JkhLbRMbPeEROrkHJHp3wj/N2yXXb
5icLgFa0g/b6wM8SuBAaO5xVM8jkmECI/P5Jo3n/d/CTUumb/BoKMIgbMurR61CG
hrZ40mizpCMFAnLrgm9FIvjKtZUsigG+oM/2xTtku2Z2nSM16/ChKtJOAj6MoMGT
xvWzLdTY6kEL30VDaHtcVvIyUFtUe5oYxEasnrcKseV7hJtccgjTYKB21PnWyZFz
/KIdh8sBg6i1nuYOtHncL6agD1M2bg1vBYwbMvIRfa1YbAFUbW1A6hDTftxJIhLn
Beso6d68SyfNMdQ8yWloR7sIufmbDut2fx1SHS3wPt2Z5W1e0XBexcpSqdD8KPns
lvmC0ggPDuQzXjPCIsS17Uk5vrCtVEdrsCCaj62a8UE2d1mCqu4CyLwMYCFv3Sve
SwZ36iOv4fzdvgjPOUg27tMwbpa/WiDAcZJ/31BfZfTDl4E9LbkQjWYT/YgsJuaQ
wp3Hi/Cgh/4=
=8YVp
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> | uservfs does
> |
> | cd foo.deb#uar
> cd foo.deb/ar
> | vs.
> | cd foo.deb#udeb
> cd foo.deb/deb
> 
> and why would you want that, instead of just:
> cd foo.deb	# for the ar
> dpkg -i foo.deb	# for the deb

Because I want to see contents of that .deb package, nicely parsed?

> Just want to extract the tar file?  Maybe something like
> cat foo.tgz/gunzip
> In which case (of course) foo.tgz/gunzip has exactly the same directory
> contents as foo.tgz

Yes, that would work.

> In fact, for just about any syntax anyone could suggest, I can't really
> see why you can't just replace all weird symbols with a slash and a
> symbol.  Instead of
> 	foo.tgz#utar
> you have
> 	foo.tgz/#/utar
> Only difference is, some things which used to require special tools can
> now be serviced by less than what's in busybox.

That would work, too. I do not get your comment about busybox.
								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: silent semantic changes with reiser4

[Horst von Brand]

Pavel Machek <pavel@ucw.cz> said:
> David Masover <ninja@slaphack.com> said:

[...]

> > Just want to extract the tar file?  Maybe something like
> > cat foo.tgz/gunzip
> > In which case (of course) foo.tgz/gunzip has exactly the same directory
> > contents as foo.tgz
> 
> Yes, that would work.
> 
> > In fact, for just about any syntax anyone could suggest, I can't really
> > see why you can't just replace all weird symbols with a slash and a
> > symbol.  Instead of
> > 	foo.tgz#utar
> > you have
> > 	foo.tgz/#/utar
> > Only difference is, some things which used to require special tools can
> > now be serviced by less than what's in busybox.
> 
> That would work, too. I do not get your comment about busybox.

You do need extra tools anyway, placing them in the kernel is cheating (and
absolutely pointless, IMHO).
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Horst von Brand wrote:
> 
> You do need extra tools anyway, placing them in the kernel is cheating (and
> absolutely pointless, IMHO).

I agree.

There's no point to having the kernel export information that is already 
inherent in the main stream.

I've seen all these examples of exposing MP3 ID information as a "side 
stream", and that's TOTALLY POINTLESS! The information is already there, 
it's in a standard format, and exporting it as a stream buys you 
absolutely nothing.

Where named attributes make sense is when they are _independent_ data.  
Data that is only tangentially related to the main stream itself, and
which the main stream _cannot_ encompass because of some real technical
issue.

In a graphical environment, the "icon" stream is a good example of this.  
It literally has _nothing_ to do with the data in the main stream. The
only linkage is a totally non-technical one, where the user wanted to
associate a secondary stream with the main stream _without_ altering the
main one. THAT is where named streams make sense.

But if you want to look at one particular file inside a tar-file, do so in 
user space. There are zero advantages to exposing it as a side-stream, and 
there are absolutely _tons_ of disadvantages.

In short, named streams only make sense if:
 - they are tied to the file some way that is _independent_ of the file 
   contents (since if it's dependent on the file contents, you're just a 
   ton better off regenerating it with a caching server)
 - there are serious reasons to keep the lookup synchronized (since if 
   there isn't such a reason, you're just better off with a separate 
   shadow tree in user space)

And realize that the "separate shadow tree" actually works very well.  
That's how version control systems like CVS have always worked. It's
certainly how you can make icon information work too. If you use a tool
for accessing the data, the tool can maintain coherency and you'll never
care about the side stream.

Which means that normally we really don't _want_ named streams. In 99% of
all cases we can use equally good - and _much_ simpler - tool-based
solutions.

Which means that the only _real_ technical issue for supporting named
streams really ends up being things like samba, which want named streams
just because the work they do fundamentally is about them, for externally
dictated reasons. Doing named streams for any other reason is likely just
being stupid.

Once you do decide that you have to do named streams, you might then 
decide to use them for convenient things like icons. But it should very 
much be a secondary issue at that point.

		Linus

Re: silent semantic changes with reiser4

[Spam]

> On Tue, 31 Aug 2004, Horst von Brand wrote:
>> 
>> You do need extra tools anyway, placing them in the kernel is cheating (and
>> absolutely pointless, IMHO).

> I agree.

> There's no point to having the kernel export information that is already
> inherent in the main stream.

  There are two things you are saying in the same sentence here. One I
  do  not  agree  with.  There is point to export information from the
  main  stream  to  secondary file streams. Just it doesn't have to be
  done by the kernel.

  Just  do  not limit things to a few defined types of things that can
  be  stored in a stream. IMHO a file stream should be able to contain
  just  about  anything  the user wishes. That said, the kernel should
  not have to know about the contents.

> I've seen all these examples of exposing MP3 ID information as a "side
> stream", and that's TOTALLY POINTLESS! The information is already there,
> it's in a standard format, and exporting it as a stream buys you 
> absolutely nothing.

  True.  But this is the case for this specific example. I have stated
  previously  about what I think would be legitimate uses as examples.
  Of  course  these  apply  to  me  and  what  I work with. One of the
  examples   were  to  add  streams  that  contained  descriptions  or
  keywords.

> Where named attributes make sense is when they are _independent_ data.
> Data that is only tangentially related to the main stream itself, and
> which the main stream _cannot_ encompass because of some real technical
> issue.

> In a graphical environment, the "icon" stream is a good example of this.
> It literally has _nothing_ to do with the data in the main stream. The
> only linkage is a totally non-technical one, where the user wanted to
> associate a secondary stream with the main stream _without_ altering the
> main one. THAT is where named streams make sense.

> But if you want to look at one particular file inside a tar-file, do so in
> user space. There are zero advantages to exposing it as a side-stream, and
> there are absolutely _tons_ of disadvantages.

  The  tar archive support seem (is) to be useless because tar is also
  so supported and implemented in every type of application out there.
  It is also easy to script and otherwise work with.

  But  the  idea to be able to extend the filesystem to show/represent
  data  as  file streams and meta directores is very good.

  Someone might come up with an idea to allow browsing and changing of
  data  inside  XML files. Of course this may already be done with sed
  and other tools.

> In short, named streams only make sense if:
>  - they are tied to the file some way that is _independent_ of the file
>    contents (since if it's dependent on the file contents, you're just a
>    ton better off regenerating it with a caching server)
>  - there are serious reasons to keep the lookup synchronized (since if
>    there isn't such a reason, you're just better off with a separate
>    shadow tree in user space)

> And realize that the "separate shadow tree" actually works very well.
> That's how version control systems like CVS have always worked. It's
> certainly how you can make icon information work too. If you use a tool
> for accessing the data, the tool can maintain coherency and you'll never
> care about the side stream.

  Just  that  with  shadow  trees  (I assume they are stored simply in
  another  folder  beside  the  file)  you  won't be able to "cp filea
  /mnt/backup" and keep the extra stuff.

  Sure  icons  could  also easily be stores as file.icon, just like on
  the Amiga :). It just isn't as nice.

> Which means that normally we really don't _want_ named streams. In 99% of
> all cases we can use equally good - and _much_ simpler - tool-based
> solutions.

> Which means that the only _real_ technical issue for supporting named
> streams really ends up being things like samba, which want named streams
> just because the work they do fundamentally is about them, for externally
> dictated reasons. Doing named streams for any other reason is likely just
> being stupid.

  I  doubt  it would be stupid. Perhaps today just not needed for many
  things.

> Once you do decide that you have to do named streams, you might then
> decide to use them for convenient things like icons. But it should very
> much be a secondary issue at that point.

  I think we focus to much on the actual contents of these streams. Do
  we  really  have  to  know  what will in the future be stored inside
  these streams?

  For  example.  Applications may use file streams for locking instead
  of having a .lock file. The future isn't known, and we shouldn't try
  to  limit  what  we can put in the streams. You said 99% of the uses
  won't  need  them.  True.  The  same is with xattr, ACLs and so many
  other things.

  Even  if 0.1% of the users today would use it I think we should have
  the support for named streams.

  ~S

>     Linus

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > You do need extra tools anyway, placing them in the kernel is cheating (and
> > absolutely pointless, IMHO).
> 
> I agree.
> 
> There's no point to having the kernel export information that is already 
> inherent in the main stream.
> 
> I've seen all these examples of exposing MP3 ID information as a "side 
> stream", and that's TOTALLY POINTLESS! The information is already there, 
> it's in a standard format, and exporting it as a stream buys you 
> absolutely nothing.

It buys me caching. I do quite often

bzcat patch.2.6.8.bz2 | less (read the patch)
(sometimes repeat that few times because I hit ^c when I should not
have etc).
cd ...clean; bzcat patch.2.6.8.bz2 | patch -Esp1
cd ...linux; bzcat patch.2.6.8.bz2 | patch -Esp1

Now... that's total waste of cpu. bzip2 decompression takes quite some
time. 

I could do

bzcat patch.2.6.8.bz2 > /tmp/delete.me.when.you.are.done

...but I'd probably forget to delete that one and anyway, it requires
me to think about it. Nicest way would be

cat patch.2.6.8.bz2/ubz | less
cd ...clean; cat patch.2.6.8.bz2/ubz | patch -Esp1
cd ...linux; cat patch.2.6.8.bz2/ubz | patch -Esp1

with kernel intelligently caching uncompressed data. I believe this
can not be done completely in userspace.
								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Pavel Machek wrote:
> 
> It buys me caching.

I'll actually buy into that. If only because I consider caching to be one 
of the more important things that the kernel does (caches are a _classic_ 
case of "shared data that needs synchronization").

However, that said, user space can trivially cache things in the 
filesystem, so while this may be a convenient feature, I think you should 
look at perhaps doing it in the _shell_ instead..

		Linus

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > It buys me caching.
> 
> I'll actually buy into that. If only because I consider caching to be one 
> of the more important things that the kernel does (caches are a _classic_ 
> case of "shared data that needs synchronization").
> 
> However, that said, user space can trivially cache things in the 
> filesystem, so while this may be a convenient feature, I think you should 
> look at perhaps doing it in the _shell_ instead..

That cache should disappear as soon as I need disk
space. I.e. userspace should never see -ENOSPC because of this kind of
caching. This need some kernel support. Ouch and cached file should
atomically go away as soon as main file changes, otherwise I do not
see how multiple processes could cooperate on caching...

chattr +kill-this-file-when-low-on-disk-space patch.bz2...ubz 

would solve first problem. Not sure how to do the second one.

								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Pavel Machek wrote:
> > 
> > However, that said, user space can trivially cache things in the 
> > filesystem, so while this may be a convenient feature, I think you should 
> > look at perhaps doing it in the _shell_ instead..
> 
> That cache should disappear as soon as I need disk
> space. I.e. userspace should never see -ENOSPC because of this kind of
> caching. This need some kernel support. Ouch and cached file should
> atomically go away as soon as main file changes, otherwise I do not
> see how multiple processes could cooperate on caching...

Well, what other projects have done is to just reserve a certain amount of 
disk for caching. See "ccache", which solves both of the above problems 
(it doesn't shrink the cache on ENOSPC, but reserving diskspace is 
accepted practice for things like this..)

		Linus

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > > However, that said, user space can trivially cache things in the 
> > > filesystem, so while this may be a convenient feature, I think you should 
> > > look at perhaps doing it in the _shell_ instead..
> > 
> > That cache should disappear as soon as I need disk
> > space. I.e. userspace should never see -ENOSPC because of this kind of
> > caching. This need some kernel support. Ouch and cached file should
> > atomically go away as soon as main file changes, otherwise I do not
> > see how multiple processes could cooperate on caching...
> 
> Well, what other projects have done is to just reserve a certain amount of 
> disk for caching. See "ccache", which solves both of the above problems 
> (it doesn't shrink the cache on ENOSPC, but reserving diskspace is 
> accepted practice for things like this..)

Okay, that does work, it just is not really nice. Just as reserving
fixed ammount of space for disk cache is bad, reserving fixed ammount
of space for ccache (and similar) is bad. When there are few of such
caches, balancing between them starts to matter...
								Pavel
-- 
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!

Re: silent semantic changes with reiser4

[Christer Weinigel]

Pavel Machek <pavel@ucw.cz> writes:

> Okay, that does work, it just is not really nice. Just as reserving
> fixed ammount of space for disk cache is bad, reserving fixed ammount
> of space for ccache (and similar) is bad. When there are few of such
> caches, balancing between them starts to matter...

So try to convince them to use the same cache daemon or the same
shared cache manager library then.  It isn't all that different from
implementing a kernel interface that everyone is supposed to use.

A cache manager daemon could sit and watch the free space on the disk
every other second and start deleting the cached files (according to
some LCU heuristics or whatever) whenever free space is getting low.

To see if the original file is newer than the cached file, good old
mtime can be used.  

I can't remember who right now, but someone mentioned a few problems
with timestamps:

1. The resolutions is too low in current kernels, but that just means
   that someone(tm) needs to implement nanosecond timestamps in the
   kernel and for the filesystems.

2. Even nanosecond timestams may become to coarse in the future.  One
   way of getting around this is to keep track of the latest timestamp
   written and make sure the next timestamp is one higher than the
   latest one.  This way timestamps are always increasing, and I doubt
   that we can have modifications each nanosecond for a long time with
   the hardware available for the next ten years.

3. If time goes backwards beacuse somebody changes the clock, the
   caching (or makefile rules) can get confued.  This is a hard one,
   but I actually we can use the doctors response "don't do that
   then".  With NTP we can guarantee a monotonically increasing clock.

I belive the kernel could give some assistance to make it easier to
see if a file has been modified, I remember that a few suggestions
were thrown around the last time Samba and dcache aliases were
discussed on l-k.  I definitely belive that kind of infrastructure
belongs in the kernel.  But the cache manager itself, no.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christer Weinigel wrote:
| Pavel Machek <pavel@ucw.cz> writes:
|
|
|>Okay, that does work, it just is not really nice. Just as reserving
|>fixed ammount of space for disk cache is bad, reserving fixed ammount
|>of space for ccache (and similar) is bad. When there are few of such
|>caches, balancing between them starts to matter...
|
|
| So try to convince them to use the same cache daemon or the same
| shared cache manager library then.  It isn't all that different from
| implementing a kernel interface that everyone is supposed to use.

You're right, it's not.  Is there some huge performance loss by having a
kernel interface that then turns around and calls a userland daemon?  If
yes, I can understand the argument.  If no, then the kernel interface
should ultimately be able to replace gnome-vfs -- and it should rely
heavily on userland utilities.

Caching is very generic.  It's so generic that there should be exactly
one cache manager, the way there is exactly one VM manager for Linux.
If there's a choice, choose it at boot or compile-time.

Abiword uses gzipped files.  So does gnumeric, gnucash, cube (a cute
little game engine), and many others.  There's no way every single one
will suddenly use a new library/daemon.  But with plugins and
intelligent caching, any kind of compression could be supported without
the app needing to support it natively -- either it goes in a compressed
folder or the user browses to it and changes an attribute.


| A cache manager daemon could sit and watch the free space on the disk
| every other second and start deleting the cached files (according to
| some LCU heuristics or whatever) whenever free space is getting low.

Agreed.  The daemon should go in userspace.  But the interface should be
in the kernel.  Just as no one suggested putting tar in the kernel, I
don't think anyone would really insist that the daemon needs to go in
the kernel.

That is, unless critical system files start actually being cached
streams of the actual package from the distro...  But this way leads to
madness.  Any serious issues here are better solved by an initrd.

[...]
| I belive the kernel could give some assistance to make it easier to
| see if a file has been modified, I remember that a few suggestions
| were thrown around the last time Samba and dcache aliases were
| discussed on l-k.  I definitely belive that kind of infrastructure
| belongs in the kernel.  But the cache manager itself, no.

Isn't there already dnotify?

Dnotify is wrong here, because I don't think it would be sane in a
sufficiently large setup.  But wouldn't it be nice if the kernel not
only told you whether a file has been modified, but told you immediately
after it was modified?

Yet another reason why there has to be a common cache and significant
kernel management -- if I've got a drive of less than, say, 5 gigs, I am
simply not willing to reserve any large amount of it for cache.  Take
the example of a source tarball being automatically extracted and the
contents cached.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUpPHgHNmZLgCUhAQLhIg/9G87W4OCt8hOvTK+XniPgWGn+YC7t4fKy
mi4QowX1eLmnFEovprjSUzPgKRMeDSChZ8M1pvXXWJQ/iULE44WH8vNiH8H974zw
h5O8dNI/AczuzaCalQ+BYXEGM8pasaCGCI/sLrr+WBy97NOpM2VUUKY+qKxSFWFt
x7u7/XCGGWSHVG3t3KzpkqD2K8u3IGSrV/vHIR2PLatEXu3G8K8sUAhfXMgFsTFp
gQAuQN/YzQKGUphnMzFWLDMYyEYjAA4VACZFqFz9QJIYeWzP+YmGdLfZ5mzN91JF
g5ACGHUxbYdjZsjTh6RbalVr1HpAAy8mvGNydbwUvuOxaXBvH44vUfRLnvIUoZs9
BB0Tq4xPqAwAtP0Z7b5Qem1ef3gqGXo1hfsEJWJZPv1lssYzn1+Jpn99p6N6VkXI
fbAHnypbHeYbf30lC5lvID0yngb+s5EUaxzxOYqFA8QpEJD/bl3S9Qx6bpom/9AL
U8kdK8QjNbhHSMrOpl8mg77VRLXdINoV7R5iSXXUMdIcA1j+4R1AIPh62JAMwQDF
/Vzek8KNyxvJDgF/y78LiGETZcG/l2W4szEsJFFF4iIKLri5+OFGiLFz8jVNViKu
Lr7Q7u7TgWSSykdTrHFcR/WRupAHeL5RB5DA9HBZErqgAgJWz/Rjv743G8PYFKs8
oPhRyePnFkE=
=cQLE
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> I belive the kernel could give some assistance to make it easier to
> see if a file has been modified, I remember that a few suggestions
> were thrown around the last time Samba and dcache aliases were
> discussed on l-k.  I definitely belive that kind of infrastructure
> belongs in the kernel.  But the cache manager itself, no.

Well, callback "filesystem full" would be nice, too; but I guess that
is "support user cache", too...
								Pavel
-- 
When do you have heart between your knees?

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 416 bytes --]

Salut,

On Wed, Sep 01, 2004 at 01:19:00AM +0200, Christer Weinigel wrote:
> To see if the original file is newer than the cached file, good old
> mtime can be used.  

I'd not use a daemon or any  mtime for it. I'd just put it into xattrs
and let  it up  to the applications  to poke  the values in  case they
changed. And copy them on cp, since it would have to be changed either
way.

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christer Weinigel]

Linus Torvalds <torvalds@osdl.org> writes:

> In a graphical environment, the "icon" stream is a good example of this.  
> It literally has _nothing_ to do with the data in the main stream. The
> only linkage is a totally non-technical one, where the user wanted to
> associate a secondary stream with the main stream _without_ altering the
> main one. THAT is where named streams make sense.

I think that the "icon" argument for named streams is a silly
argument, since different users may want to have different icons for
the same file.  Say that I want /usr/bin/emacs to have the enterprise
icon and someone else wants the gnu head icon.  And besides, root owns
the file anyways, so neither of us mortal users should be able to add
a stream to it.

Another reason for named streams that usually crops up is the ability
set a "preferred application" for a certain file, so that when I
double click on a document I want to open it with antiword instead of
openoffice.  But the same contra-argument applies here, different
users have different preferences.

I can see the argument for having the equivalent of Content-type or
Content-transfer-encoding as a named stream though.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Spam]

  

> Linus Torvalds <torvalds@osdl.org> writes:

>> In a graphical environment, the "icon" stream is a good example of this.
>> It literally has _nothing_ to do with the data in the main stream. The
>> only linkage is a totally non-technical one, where the user wanted to
>> associate a secondary stream with the main stream _without_ altering the
>> main one. THAT is where named streams make sense.

> I think that the "icon" argument for named streams is a silly
> argument, since different users may want to have different icons for
> the same file.  Say that I want /usr/bin/emacs to have the enterprise
> icon and someone else wants the gnu head icon.  And besides, root owns
> the file anyways, so neither of us mortal users should be able to add
> a stream to it.

  Yet again are we thinking in blocking ways. Firstly this was an
  example. Usually, though, most users accept the default icon for a
  file. If they do not they can still change the icon for the link
  they make on their start-menu/home folder/etc.

> Another reason for named streams that usually crops up is the ability
> set a "preferred application" for a certain file, so that when I
> double click on a document I want to open it with antiword instead of
> openoffice.  But the same contra-argument applies here, different
> users have different preferences.

  I can make the same argument as for the icons.

> I can see the argument for having the equivalent of Content-type or
> Content-transfer-encoding as a named stream though.

  That would be a nice thing.

>   /Christer

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 407 bytes --]

Salut,

On Wed, Sep 01, 2004 at 01:02:24AM +0200, Christer Weinigel wrote:
> I can see the argument for having the equivalent of Content-type or
> Content-transfer-encoding as a named stream though.

Why having them as  named streams if we can get them  as xattrs for no
additional pain? (Since fileutils would  have to be changed anyway, we
can even make cp copy and emacs update xattrs.)

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Grzegorz Jaśkiewicz]

On Sun, 5 Sep 2004 13:17:43 +0200, Tonnerre <tonnerre@thundrix.ch> wrote:
> Salut,
> 
> On Wed, Sep 01, 2004 at 01:02:24AM +0200, Christer Weinigel wrote:
> > I can see the argument for having the equivalent of Content-type or
> > Content-transfer-encoding as a named stream though.
> 
> Why having them as  named streams if we can get them  as xattrs for no
> additional pain? (Since fileutils would  have to be changed anyway, we
> can even make cp copy and emacs update xattrs.)

Because some of as requested to be able to open tar,iso, and few other
formats this way. So I can simply access it with cat (as a dir/file).



-- 
GJ

Re: silent semantic changes with reiser4

[Spam]

  

> Salut,

> On Wed, Sep 01, 2004 at 01:02:24AM +0200, Christer Weinigel wrote:
>> I can see the argument for having the equivalent of Content-type or
>> Content-transfer-encoding as a named stream though.

> Why having them as  named streams if we can get them  as xattrs for no
> additional pain? (Since fileutils would  have to be changed anyway, we
> can even make cp copy and emacs update xattrs.)

  What if I do not use emacs, but vim, mcedit, gedit, or some other
  editor? It doesn't seem logical to have to patch every application
  that uses files.

  ~S

> 			    Tonnerre

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 483 bytes --]

Salut,

On Sun, Sep 05, 2004 at 01:57:49PM +0200, Spam wrote:
>   What if I do not use emacs, but vim, mcedit, gedit, or some other
>   editor? It doesn't seem logical to have to patch every application
>   that uses files.

We would have to do that in  either case, so let's patch them to do it
in a nonintrusive way. And as to reading and writing inside tar files,
write and/or  use a really nice  userspace library to do  it. (As does
MacOS/X, as does KDE, etc.)

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Spam]

  

> Salut,

> On Sun, Sep 05, 2004 at 01:57:49PM +0200, Spam wrote:
>>   What if I do not use emacs, but vim, mcedit, gedit, or some other
>>   editor? It doesn't seem logical to have to patch every application
>>   that uses files.

> We would have to do that in  either case, so let's patch them to do it
> in a nonintrusive way. And as to reading and writing inside tar files,
> write and/or  use a really nice  userspace library to do  it. (As does
> MacOS/X, as does KDE, etc.)

  No, not with the current setup with file-as-dir. It also works in
  Gnome/Natilus. Just tested:

  mkdir test
  chmod +x test
  cd test
  dir metas
  echo 0777>rwx

  In Nautilus I just typed in in the address bar /test/metas/
  I could open meta-data and change with GEdit too.

  I do not see that I have had to patch any programs to get this to
  work.

  ~S

> 			    Tonnerre

Re: silent semantic changes with reiser4

[Spam]

  

> Salut,

> On Sun, Sep 05, 2004 at 01:57:49PM +0200, Spam wrote:
>>   What if I do not use emacs, but vim, mcedit, gedit, or some other
>>   editor? It doesn't seem logical to have to patch every application
>>   that uses files.

> We would have to do that in  either case, so let's patch them to do it
> in a nonintrusive way. And as to reading and writing inside tar files,
> write and/or  use a really nice  userspace library to do  it. (As does
> MacOS/X, as does KDE, etc.)

  The problem with the userspace library is standardization. What
  would be needed is a userspace library that has a extensible plugin
  interface that is standardized. Otherwise we would need lots of
  different libraries, and I seriously doubt that 1) this will happen
  and 2) we get all Linux programs to be patched to use it.

  ~S

> 			    Tonnerre

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> >>   What if I do not use emacs, but vim, mcedit, gedit, or some other
> >>   editor? It doesn't seem logical to have to patch every application
> >>   that uses files.
> 
> > We would have to do that in  either case, so let's patch them to do it
> > in a nonintrusive way. And as to reading and writing inside tar files,
> > write and/or  use a really nice  userspace library to do  it. (As does
> > MacOS/X, as does KDE, etc.)
> 
>   The problem with the userspace library is standardization. What
>   would be needed is a userspace library that has a extensible plugin
>   interface that is standardized. Otherwise we would need lots of
>   different libraries, and I seriously doubt that 1) this will happen
>   and 2) we get all Linux programs to be patched to use it.

libvfs from midnight commander (and anything build on the top of it)
already is very extensible.
								Pavel

Re: silent semantic changes with reiser4

[Spam]

  

> Hi!

>> >>   What if I do not use emacs, but vim, mcedit, gedit, or some other
>> >>   editor? It doesn't seem logical to have to patch every application
>> >>   that uses files.
>> 
>> > We would have to do that in  either case, so let's patch them to do it
>> > in a nonintrusive way. And as to reading and writing inside tar files,
>> > write and/or  use a really nice  userspace library to do  it. (As does
>> > MacOS/X, as does KDE, etc.)
>> 
>>   The problem with the userspace library is standardization. What
>>   would be needed is a userspace library that has a extensible plugin
>>   interface that is standardized. Otherwise we would need lots of
>>   different libraries, and I seriously doubt that 1) this will happen
>>   and 2) we get all Linux programs to be patched to use it.

> libvfs from midnight commander (and anything build on the top of it)
> already is very extensible.

  This wasn't my point. My point was about all applications using it.
  Most aren't.

> 								Pavel

Re: silent semantic changes with reiser4

[Christer Weinigel]

Spam <spam@tnonline.net> writes:

> >>   The problem with the userspace library is standardization. What
> >>   would be needed is a userspace library that has a extensible plugin
> >>   interface that is standardized. Otherwise we would need lots of
> >>   different libraries, and I seriously doubt that 1) this will happen
> >>   and 2) we get all Linux programs to be patched to use it.
> 
> > libvfs from midnight commander (and anything build on the top of it)
> > already is very extensible.
> 
>   This wasn't my point. My point was about all applications using it.
>   Most aren't.

So why do you think that just because you put an interface into the
everyone will automatically start using it and use it without any
conflicts (i.e. is foo/icon a windows icon, a gif, png, or jpeg file,
what size is it, 16x16, 48x48...).  

When you add a new shared namespace, userspace must be taught about it
anyways.  So start with a userspace library, convince people to use
it, and when you know what people want, _then_ put it into the kernel
to increase performance or security.  Don't start with "wouldn't it be
nice if" without any thought behind it, it will just mean that we get
another useless and misdesigned interface in the kernel that we have
to live with for years.

Do you know how long it took to get rid of /dev/cua0 and the locking
mess associated with it?  (Deity!  I just checked on my FC2 system,
it's still there.  I thought it would be dead by now.)

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> writes:

>> >>   The problem with the userspace library is standardization. What
>> >>   would be needed is a userspace library that has a extensible plugin
>> >>   interface that is standardized. Otherwise we would need lots of
>> >>   different libraries, and I seriously doubt that 1) this will happen
>> >>   and 2) we get all Linux programs to be patched to use it.
>> 
>> > libvfs from midnight commander (and anything build on the top of it)
>> > already is very extensible.
>> 
>>   This wasn't my point. My point was about all applications using it.
>>   Most aren't.

> So why do you think that just because you put an interface into the
> everyone will automatically start using it and use it without any
> conflicts (i.e. is foo/icon a windows icon, a gif, png, or jpeg file,
> what size is it, 16x16, 48x48...).  

> When you add a new shared namespace, userspace must be taught about it
> anyways.  So start with a userspace library, convince people to use
> it, and when you know what people want, _then_ put it into the kernel
> to increase performance or security.

  Several aspects of this;

  1) Programs will need to actually be coded against this shared
  library which probably will be more advanced that just usning normal
  file access code.
  
  2) Then we redo it all again to fit in the kernel, as a kernel
  plugin or user-level plugin.

> Don't start with "wouldn't it be
> nice if" without any thought behind it, it will just mean that we get
> another useless and misdesigned interface in the kernel that we have
> to live with for years.

  I am not saying that there should not be any thought behind
  implementing something. But from many hear I just hear a big no,
  without any thought about it either.

  Named streams and meta files are just one example of a general
  access to extra data that belongs to files. It seem to me as a much
  simpler way for program to use those than to link in several various
  libraries to do the same thing. 

  Plugins were just another thing that could extend the functionality
  of these streams and meta data. Reiser4 has a plugin architecture,
  although not yet any run-time support for loading them. Is this so
  bad that we have to prevent it?

  No one has to use them. maybe at some point it would be possible to
  specify which classes of plugins that could be loaded by a user and
  his files.

  What are the complaints about. The technical challenge to make
  things like this secure, or the concept to have this functionality
  at all. I do not think that because something is hard to do, it
  should be dismissed.

  ~S

  
> Do you know how long it took to get rid of /dev/cua0 and the locking
> mess associated with it?  (Deity!  I just checked on my FC2 system,
> it's still there.  I thought it would be dead by now.)

>   /Christer

Re: silent semantic changes with reiser4

[Christer Weinigel]

Spam <spam@tnonline.net> writes:

> > When you add a new shared namespace, userspace must be taught about it
> > anyways.  So start with a userspace library, convince people to use
> > it, and when you know what people want, _then_ put it into the kernel
> > to increase performance or security.
> 
>   Several aspects of this;
> 
>   1) Programs will need to actually be coded against this shared
>   library which probably will be more advanced that just usning normal
>   file access code.

Not at all.  As many others have already shown, trying to use files as
directories won't work since then you can't have named streams or
attributes for real directories.  What does foo/icon refer to when foo
is a directory?  There must be some way of differentiating between
them, which in turn means that we need a new way of referring to the
stream.  So the simple, hackish, solution really won't fly.

Besides, it will probably break a lot of security critical assumptions
in todays software.  What if you have a filter in your web server that
excludes executable files or files that end with .php, can you imagine
the mess if you could just open "foo.php/main-stream" instead?

>   2) Then we redo it all again to fit in the kernel, as a kernel
>   plugin or user-level plugin.

No, we use the same user space library, but let the user space library
use helper functions in the kernel.

We could for example add a function that opens a named stream based on
a filename which uses a .xattr directory in the same directory as the
file, or in database in our home directory.  If we add a solaris-like
openat(3) syscall we modify the library to take advantage of it.

>   Plugins were just another thing that could extend the functionality
>   of these streams and meta data. Reiser4 has a plugin architecture,
>   although not yet any run-time support for loading them. Is this so
>   bad that we have to prevent it?

Take an example: "expose a tar-file as streams below the file" which
as been suggested here is IMNSHO plain silly.  I'm not saying anything
about mounting a tar file via userfs somewhere else in the file
system, that is quite ok, but trying to mount it on top of the same
file which suddenly and automagically turns into a sort-of-directory
breaks too many thing.  Let your file manager do the choice instead,
based on the users preferences.  For example, with a tar.gz-file, I'd
like to have a choice of "open file as a seekable file" which would
do:

    mount -t userfs -o driver=gunzip foo.tar.gz /tmp/xyzzy

Then I can work with /tmp/xyzzy as a normal file (maybe even with
write access if the userfs driver can handle this).  Another choice in
the same file manager would be "open file as a directory" which would
mount the same file in _another place_ as a directory, and I can even
have different views of the same file mounted at the same time.  With
the named streams junk that have been suggested here, having two
different views would be impossible.

Sure, we could say that we add another level of indirection to the
named streams, so that we specify the driver as the first component of
te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to
the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show
the tar file as a directory, but really, this isn't any more useful
than doing a userfs mount.  The userfs mount does not break existing
semantics (anymore than mount -o loop does today), and it will work
with the existing infrastructure in the linux kernel.  The only
advantage of files-as-directories with magic plugins in the kernel is
that one can look at it and say "look, how neat, the filenames look
almost the same".

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> Sure, we could say that we add another level of indirection to the
> named streams, so that we specify the driver as the first component of
> te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to
> the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show
> the tar file as a directory, but really, this isn't any more useful
> than doing a userfs mount.  The userfs mount does not break existing
> semantics (anymore than mount -o loop does today), and it will work
> with the existing infrastructure in the linux kernel.  The only
> advantage of files-as-directories with magic plugins in the kernel is
> that one can look at it and say "look, how neat, the filenames look
> almost the same".

Who is going to umount it when application crashes, etc? Plus mount
required root priviledges last time I checked.

								Pavel

Re: silent semantic changes with reiser4

[Christer Weinigel]

Pavel Machek <pavel@ucw.cz> writes:

> Who is going to umount it when application crashes, etc? 

Who removes temporary files left behind when an application crashes?
I'd guess a daemon such as autofs could do a very good job here.

>Plus mount required root priviledges last time I checked.

    bash$ ls -l /bin/mount 
    -rwsr-xr-x  1 root root 78504 May  4 23:34 /bin/mount

with proper policies in userspace it allows users to perform mounts.  

I'm not suggesting that the kernel should be unchanged, but really,
some of the proposals here, to put a hell of a lot of complexity into
the kernel it just wet dreams with not much thought of how it affects
the kernel.  What happened to the philosophy of putting complexity and
policy in userspace?  Look at khttpd and tux, they were hacks in the
kernel to try things out.  But what ended up in the kernel is generic
infrastructure that is useful for a lot more applications than just a
web server.  That is the right way to do things.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> >Plus mount required root priviledges last time I checked.
> 
>     bash$ ls -l /bin/mount 
>     -rwsr-xr-x  1 root root 78504 May  4 23:34 /bin/mount
> 
> with proper policies in userspace it allows users to perform mounts.  

You want to do exec() each time you enter a archive? I do not think
that is good idea.

> I'm not suggesting that the kernel should be unchanged, but really,
> some of the proposals here, to put a hell of a lot of complexity into
> the kernel it just wet dreams with not much thought of how it affects
> the kernel.  What happened to the philosophy of putting complexity and
> policy in userspace?  Look at khttpd and tux, they were hacks in the
> kernel to try things out.  But what ended up in the kernel is generic
> infrastructure that is useful for a lot more applications than just a
> web server.  That is the right way to do things.

Well, generic infrastructure for uservfs would be enough for me... And
it is few lines in kernel.
								Pavel

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 617 bytes --]

Salut,

On Mon, Sep 06, 2004 at 05:54:56PM +0200, Pavel Machek wrote:
> Who is going to umount it when application crashes, etc?

This  has   been  discussed  along   with  the  HAL  people   a  while
ago. Actually, file systems can  introduce a refcount, where we need a
decrement function  which automatically unmounts the  filesystem if we
decrement the use count to zero. Kind of an automatic umount.

How  do  you tell  which  file  systems  shall be  autoumounted?   HAL
suggestion: introduce a no-op mount  option. For some classes of fs'es
we might as well make that obligatory, such as tar.

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Alan Cox]

On Mer, 2004-09-08 at 07:11, Tonnerre wrote:
> This  has   been  discussed  along   with  the  HAL  people   a  while
> ago. Actually, file systems can  introduce a refcount, where we need a
> decrement function  which automatically unmounts the  filesystem if we
> decrement the use count to zero. Kind of an automatic umount.

We've supported file system garbage collection when they become unused
since umm about 2.4.10 I think.

Re: silent semantic changes with reiser4

[Spam]

>>   Plugins were just another thing that could extend the functionality
>>   of these streams and meta data. Reiser4 has a plugin architecture,
>>   although not yet any run-time support for loading them. Is this so
>>   bad that we have to prevent it?

> Take an example: "expose a tar-file as streams below the file" which
> as been suggested here is IMNSHO plain silly.  I'm not saying anything
> about mounting a tar file via userfs somewhere else in the file
> system, that is quite ok, but trying to mount it on top of the same
> file which suddenly and automagically turns into a sort-of-directory
> breaks too many thing.  Let your file manager do the choice instead,
> based on the users preferences.  For example, with a tar.gz-file, I'd
> like to have a choice of "open file as a seekable file" which would
> do:

>     mount -t userfs -o driver=gunzip foo.tar.gz /tmp/xyzzy

 Where is the difference? Both would be handled by a specific driver
 or module and export the same semantics (files+dirs) with permissions
 etc to the user. With your idea you still would need the userfs
 kernel module, and with "magic plugins", as you said, you will need a
 vfs/reiser4 kernel plugin.

> Then I can work with /tmp/xyzzy as a normal file (maybe even with
> write access if the userfs driver can handle this).  Another choice in
> the same file manager would be "open file as a directory" which would
> mount the same file in _another place_ as a directory, and I can even
> have different views of the same file mounted at the same time.  With
> the named streams junk that have been suggested here, having two
> different views would be impossible.

> Sure, we could say that we add another level of indirection to the
> named streams, so that we specify the driver as the first component of
> te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to
> the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show
> the tar file as a directory, but really, this isn't any more useful
> than doing a userfs mount.  The userfs mount does not break existing
> semantics (anymore than mount -o loop does today), and it will work
> with the existing infrastructure in the linux kernel.  The only
> advantage of files-as-directories with magic plugins in the kernel is
> that one can look at it and say "look, how neat, the filenames look
> almost the same".

  No there are _usability_ differences. I cirtanly do not want to
  mount lots of files with mount -t userfs, just to see extra
  meta-data that I want to quickly be able to use. And it also
  wouldn't work generically (searchable) with tools like find, grep,
  etc either.

  ~S

>   /Christer

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christer Weinigel wrote:
| Spam <spam@tnonline.net> writes:
|
|
|>>When you add a new shared namespace, userspace must be taught about it
|>>anyways.  So start with a userspace library, convince people to use
|>>it, and when you know what people want, _then_ put it into the kernel
|>>to increase performance or security.
|>
|>  Several aspects of this;
|>
|>  1) Programs will need to actually be coded against this shared
|>  library which probably will be more advanced that just usning normal
|>  file access code.
|
|
| Not at all.  As many others have already shown, trying to use files as
| directories won't work since then you can't have named streams or
| attributes for real directories.  What does foo/icon refer to when foo

Oh yes, you can.  That's what "metas" is for.  I wanted to call it ...
| is a directory?  There must be some way of differentiating between

When foo is a directory, foo/icon is a normal file.  But foo/metas/icon
(or foo/.../icon if you like) is the metadata.

This already works.  I can use it right now for file permissions:

echo 755 > /bin/bash/metas/rwx
echo 755 > ./metas/rwx

| Besides, it will probably break a lot of security critical assumptions
| in todays software.  What if you have a filter in your web server that
| excludes executable files or files that end with .php, can you imagine
| the mess if you could just open "foo.php/main-stream" instead?

Why would you have a foo.php/main-stream?

And what would be so hard about allowing read access to foo, but not
access to foo's metas?  This is essentially the same as having mode 6 on
a directory.

[...]
|>  Plugins were just another thing that could extend the functionality
|>  of these streams and meta data. Reiser4 has a plugin architecture,
|>  although not yet any run-time support for loading them. Is this so
|>  bad that we have to prevent it?
|
|
| Take an example: "expose a tar-file as streams below the file" which
| as been suggested here is IMNSHO plain silly.  I'm not saying anything
| about mounting a tar file via userfs somewhere else in the file
| system, that is quite ok, but trying to mount it on top of the same
| file which suddenly and automagically turns into a sort-of-directory
| breaks too many thing.  Let your file manager do the choice instead,

What things?  It's just dangerously different thinking.  I don't see it
breaking anything on my system right now.  If anything, the nvidia
binary drivers are a bigger issue...

| based on the users preferences.  For example, with a tar.gz-file, I'd
| like to have a choice of "open file as a seekable file" which would
| do:
|
|     mount -t userfs -o driver=gunzip foo.tar.gz /tmp/xyzzy
|
| Then I can work with /tmp/xyzzy as a normal file (maybe even with
| write access if the userfs driver can handle this).  Another choice in
| the same file manager would be "open file as a directory" which would
| mount the same file in _another place_ as a directory, and I can even
| have different views of the same file mounted at the same time.  With
| the named streams junk that have been suggested here, having two
| different views would be impossible.
|
| Sure, we could say that we add another level of indirection to the
| named streams, so that we specify the driver as the first component of
| te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to
| the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show
| the tar file as a directory, but really, this isn't any more useful
| than doing a userfs mount.  The userfs mount does not break existing

Yes, it is more useful than doing a userfs mount.  The userfs mount has
to be done manually or by a file manager.  I do not use a file manager.
~ I am not alone here.  Unless you're going to have the entire FS mounted
userfs and have that be the file manager, which wouldn't be nearly as
fast as plugins.

I've heard autofs mentioned -- which only works on specific files which
it's been preconfigured for.  Not good here.

| semantics (anymore than mount -o loop does today), and it will work
| with the existing infrastructure in the linux kernel.  The only

That's true -- it's easier to implement if you ignore the number of apps
which need patching.

There's also the fact that most people here won't be patching apps,
they'll just be breathing a sigh of relief that they didn't have to
patch the kernel.

Just because someone else has to do it doesn't make it any less work.

| advantage of files-as-directories with magic plugins in the kernel is
| that one can look at it and say "look, how neat, the filenames look
| almost the same".

And this is a usability issue.  Not all newbies use Nautilus.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTy3MngHNmZLgCUhAQJ/lQ//UmQOJLZAmsFFD8TJehD8WitMrImrcrBC
SJ/mCkIaPq5JNo9R/ovd1UoGlf47JUvaaT5G1d/yOMA+/0kPNqjJL49Km9A3Ze+C
00Uq12yP8yp6NjBzq9HKAjvyHVghl1A5ZllIdiwuc8Fywbl6UKUHlN1D4PpUp1uA
znKPjmGpBOsTzgNWJYbAfjns60C7DHZoD/S+ldt3af2PsJKOpx0v4fusV8Y2+XRG
kWybZLl1YPwPLfJLCaa3KpbY1KkSuk6TSeut90+zRV6s1WJ8LE98NpDBGUDt/+cm
AXcp/zyrUmbpvQtZBaUFYwZ8imizspiSHYnBwdhVm9tfERFOFI9diBidvanHk/s0
D+FSJhnQSeo/hMnbHlfv4E6p4k1jDvJk06XJ/U45sEym4hCp9roDPB0tUic7bOnE
mxNxzogp92KHfsVPuqkidq255ztAjRbzdnt8zsBOPImqTgfsO3xVfT3svIixNxWv
+qaAlrR7dAri9bi3wNLCjzRhp3id0fspetLJrrCHI/8hMlu7a6nsIHLocuvG069W
OzdXtTPRYsV8YlElsdL5d/3grZSL3CWNpQxoRb9gzj0hKHHPjrXw9ACcQthiknNc
DkjjcLxV2gPqhKq3WgMTKXHhLDWEfJpFVQ8PccJHsKaaUgGtDUg0ETBT54h8Mo8E
I0bkcYOLtHk=
=o3vQ
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Horst von Brand]

Spam <spam@tnonline.net> said:

[...]

>   The problem with the userspace library is standardization. What
>   would be needed is a userspace library that has a extensible plugin
>   interface that is standardized. Otherwise we would need lots of
>   different libraries, and I seriously doubt that 1) this will happen
>   and 2) we get all Linux programs to be patched to use it.

What is the difference with a kernel implementation? Not by being in-kernel
will it make all the incompatible ways of doing this magically vanish, and
give outstanding performance. Plus handling and maintaining the in-kernel
stuff is _much_ harder than userspace libraries.

I'd go the other way around: Get userspace to agree on a common framework,
make it work in userspace; if (extensive, hopefully) experience shows that
a pure userspace solution has issues that can't be solved except by kernel
assistance, so be it.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
| Spam <spam@tnonline.net> said:
|
| [...]
|
|
|>  The problem with the userspace library is standardization. What
|>  would be needed is a userspace library that has a extensible plugin
|>  interface that is standardized. Otherwise we would need lots of
|>  different libraries, and I seriously doubt that 1) this will happen
|>  and 2) we get all Linux programs to be patched to use it.
|
|
| What is the difference with a kernel implementation? Not by being
in-kernel
| will it make all the incompatible ways of doing this magically vanish, and
| give outstanding performance. Plus handling and maintaining the in-kernel
| stuff is _much_ harder than userspace libraries.

First of all, only the interface has to be in the kernel.  I haven't
heard anyone suggest otherwise.

Second, there are quite a few things which I might want to do, which can
be done with this interface and without patching programs, but would
require massive patches to userspace.  There have been numerous examples.

There are some things which can't be solved without patching.  Version
control is one such thing.  But then there can be more generic patches
- -- as soon as the transaction API is done, you only have to patch apps
to use that, and have a version control reiser4 plugin.

| I'd go the other way around: Get userspace to agree on a common framework,
| make it work in userspace; if (extensive, hopefully) experience shows that
| a pure userspace solution has issues that can't be solved except by kernel
| assistance, so be it.

We already have such a framework -- it's called "VFS".
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTyyGXgHNmZLgCUhAQLUeQ/8D3tWL9l/zQeGylpVbOe6SkcSPOOmlIFR
tcjh0y1Y4ET17ATFKbKbzQYYDgd49AqU/gZnro27jYun3Yi6U0fWGGQFfi1A9O5E
gSCmsjSWjfDfx4gu3EU1x0Bhkd6Mo8GCrC7L5gj5C+L4c5ZAnffeGloF8nM4hCex
Wsb0PgOSxXuoQcd2EELVEXYdq0RCnxrmuszl1B2SE6w1ImONWMoXJ9fDGDf0aIUu
rwrrZnlH4zp0bQ0dXDGXqUYYT5h5DAhbh96IWLrPbWMB0vLBqIP+95P2/vTHb7EL
RwVKBV1UuuZ2ANPbImoIuxHWF+PCx/HwFs/mUolw0D2Yn3u2HgmPVFemyPnlCfeX
yGPhJgnieRuGntgUZcfbqk7ZO3y0y5eRDq6N4eMHMlWYV9LC5kyP7OxcQ8SAF8P6
Sk4iylYN1AMiy5Bp9odScauST0NT9CLmi1Ps0DYwgVN1H+ldS1l+4ITokb1Ex1+4
ZLq1HhPNaYYWoA4VPuwxl0XrB4wGrMbOt1w4+TNM3AG9MvzqTGgSrh2rXfXkPHGZ
7LNHuinRyJt3dcF0vPS4WHG6FtVsO8XVPaY55tYQIYZEBtZl3mattBb9gM3WDJmw
M/pxbAQTDZHloR9/7TGEF8gD3AjPBexTfvojqVHK2VvRu4/2Ku17wvK82v68LuyU
bFxxxgj9IgY=
=BxAT
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Christer Weinigel]

David Masover <ninja@slaphack.com> writes:

> Second, there are quite a few things which I might want to do, which can
> be done with this interface and without patching programs, but would
> require massive patches to userspace.  There have been numerous
> examples.

Please give one example?  There is a lot of handwaving here on all
sides.

> | I'd go the other way around: Get userspace to agree on a common framework,
> | make it work in userspace; if (extensive, hopefully) experience shows that
> | a pure userspace solution has issues that can't be solved except by kernel
> | assistance, so be it.
> 
> We already have such a framework -- it's called "VFS".

That you'd like to break the semantics of.  Great.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Horst von Brand]

David Masover <ninja@slaphack.com> said:
> Horst von Brand wrote:
> | Spam <spam@tnonline.net> said:
> |
> | [...]
> |
> |
> |>  The problem with the userspace library is standardization. What
> |>  would be needed is a userspace library that has a extensible plugin
> |>  interface that is standardized. Otherwise we would need lots of
> |>  different libraries, and I seriously doubt that 1) this will happen
> |>  and 2) we get all Linux programs to be patched to use it.
> |
> |
> | What is the difference with a kernel implementation? Not by being
> | in-kernel will it make all the incompatible ways of doing this
> | magically vanish, and give outstanding performance. Plus handling
> | and maintaining the in-kernel stuff is _much_ harder than userspace
> | libraries.

> First of all, only the interface has to be in the kernel.  I haven't
> heard anyone suggest otherwise.

Right. But it is _another_ interface in the kernel, plus special userland
code supporting it.

> Second, there are quite a few things which I might want to do, which can
> be done with this interface and without patching programs,

Such as?

>                                                            but would
> require massive patches to userspace.  There have been numerous examples.

Haven't seen any that made sense to me, sorry.

> There are some things which can't be solved without patching.

Maybe. Question is, is it worth it (kernel modifications + userland
support, or just userland support, or leave it alone). Sure, it might make
your particular application easier to write (at a cost for _all_ filesystem
hackers!), perhaps even a bit faster; but is _your_ particular convenience
worth the cost for _everybody_?

>                                                                Version
> control is one such thing.

bk, cvs, svn, rcs, ... are working just fine here, thank you so much. Used
to work on SunOS and Solaris, even SCO Unix (I used at least rcs and cvs
there). No Reiser4 in sight.

>                             But then there can be more generic patches
> - -- as soon as the transaction API is done, you only have to patch apps
> to use that, and have a version control reiser4 plugin.

Again, _what_ version control exactly? Will the above packages be able to
make use of it (remember they all are cross-platform (at least cross-Unix),
and so quite unlikely to make use of a Reiser4 on Linux whackiness...)?

> | I'd go the other way around: Get userspace to agree on a common framework,
> | make it work in userspace; if (extensive, hopefully) experience shows that
> | a pure userspace solution has issues that can't be solved except by kernel
> | assistance, so be it.

> We already have such a framework -- it's called "VFS".

Right. It offers what applications need to build their own stuff. It is
minimalistic (well, sort of) and  time-proven.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
[...]
|>Second, there are quite a few things which I might want to do, which can
|>be done with this interface and without patching programs,
|
|
| Such as?

They've been mentioned.
|
|
|>                                                           but would
|>require massive patches to userspace.  There have been numerous examples.
|
|
| Haven't seen any that made sense to me, sorry.

Sorry if they don't make sense to you, but I don't feel like discussing
them now.  Either you get it or you don't, either you agree or you
don't.  Read the archives.
|
|
|>There are some things which can't be solved without patching.
|
|
| Maybe. Question is, is it worth it (kernel modifications + userland
| support, or just userland support, or leave it alone). Sure, it might make
| your particular application easier to write (at a cost for _all_
filesystem
| hackers!), perhaps even a bit faster; but is _your_ particular convenience
| worth the cost for _everybody_?

There are far more userland developers than filesystem hackers.  If
you're a filesystem hacker, it's easier to understand how much work it
is to do something in the filesystem/kernel, and harder to understand
how it works in userland, or for the actual user.
|
|
|>                                                               Version
|>control is one such thing.
|
|
| bk, cvs, svn, rcs, ... are working just fine here, thank you so much. Used
| to work on SunOS and Solaris, even SCO Unix (I used at least rcs and cvs
| there). No Reiser4 in sight.

Transparently?

It _works_ to do
	zcat file.gz > /tmp/file
	vim /tmp/file
	gzip -c /tmp/file > file.gz
	rm /tmp/file
You can even do that as a script, call it zvim.  You could even do it as
a generic script, where "vim" is replaced with "$1".  But is it as
elegent as transparently compressed files?

I'm not sure about the version control thing -- I don't think people
have hit every conceptual issue about it yet.  But the point is:

Moving complexity from kernel space to user space (or the other way)
doesn't make it any less complex.


|>                            But then there can be more generic patches
|>- -- as soon as the transaction API is done, you only have to patch apps
|>to use that, and have a version control reiser4 plugin.
|
|
| Again, _what_ version control exactly? Will the above packages be able to
| make use of it (remember they all are cross-platform (at least
cross-Unix),
| and so quite unlikely to make use of a Reiser4 on Linux whackiness...)?

Probably.  It wasn't specified what version control would be used --
that's currently just as abstract as what compression algorithm to use
to transparently compress files.

Maybe something new.  Version control just doesn't look that complex
once you've got the interface and the data storage done.  I do
incremental backups using rsync and hardlinks -- why would version
control be so much more complex than that?

Said backup system uses hardlinks, btw, which not all systems support.
And rsync is fairly broken on some platforms.

|>| I'd go the other way around: Get userspace to agree on a common
framework,
|>| make it work in userspace; if (extensive, hopefully) experience
shows that
|>| a pure userspace solution has issues that can't be solved except by
kernel
|>| assistance, so be it.
|
|
|>We already have such a framework -- it's called "VFS".
|
|
| Right. It offers what applications need to build their own stuff. It is
| minimalistic (well, sort of) and  time-proven.

x86 assemby is minimalistic and time-proven.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQT1MGHgHNmZLgCUhAQLxoQ/+Mglz6tr0A4wQmAnP1y9q7WVqQQ8s3Cwg
b3VyOHaIIZSw3lod+bWDgE9LJJsQOxO4SThlWLyMEHipFJT0iHVIYjLSrC5e4DOb
Wl5E51yz3ZVx11mRTM7uEw+Ez6wOcaWIFLTvarDxzmDaxS6oh2ZpN2Ibnf8h/1lT
0YJnz6C964TkaA8jYIsQljIWkMMk7EGzP6UlrQOBbo4xMJNT+wIZMJWX5JAsULc2
EGzTO8dgZB0MbJ5STufS4h5tudny/lz4TO9iTv9CRGhusf2am7k7PVFPG4RcjXKU
iMgLTHKsXbDjncj39JZqwQtBZP8nuf72pMzbEe5iiYYIHGWu4Mm/JSxUmuJZ/YXX
yZIS6DKwSrKzDf/p1chvLVCaScxsaIWuetoe4ODFUoWMRUfCdxK+r7+6j9tKn+hn
LK5iYVAs3/xpU64jpBKvrKlRlISm8++GvGD+tdnZKAnetmaS0QHb2DrbwSbONvC1
4RvABUYC2IoVoDuAsueRDQxqTJjGPWn7DBvVUI5SCQZVlJPMavHmrv9qRVtNA4Y4
LBzfYK6aCprbmmX2Axs8FS4ptNGdwGpUhwuVKfSzlOgUS5gIaWlMKOGxWOTgDx7U
Q53mVhdZIinHH+/h4xBpcP3Q1fk8nTQ1gqfYVTseNlrMZvgFAax7E2VINiJWohis
KH6z9bFgkZA=
=aNM7
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Hans Reiser]

I just want to remind that no one has been able to offer a good way of 
handling attributes/streams/metafiles other than reiser4 in which 
CTRL-XCTRL-Ffilename within emacs to edit the stream/attribute/metafile 
will work without modifying the emacs source.  David Masover is right 
that there are far more application writers than kernel hackers, and we 
should make the kernel more complicated if it makes a few thousand apps 
simpler.

Re: silent semantic changes with reiser4

[William Lee Irwin III]

On Mon, Sep 06, 2004 at 11:02:01PM -0700, Hans Reiser wrote:
> I just want to remind that no one has been able to offer a good way of 
> handling attributes/streams/metafiles other than reiser4 in which 
> CTRL-XCTRL-Ffilename within emacs to edit the stream/attribute/metafile 
> will work without modifying the emacs source.  David Masover is right 
> that there are far more application writers than kernel hackers, and we 
> should make the kernel more complicated if it makes a few thousand apps 
> simpler.

This thread is getting a bit soft on technical details and/or value.

Hans, please post the results of fsstress (the bits from ext3 cvs) on
reiser4 on SMP machines of non-i386 architectures (e.g. ppc64, sparc64,
ia64) as well as the results of transferring reiser4 filesystems
between machines whose PAGE_SIZE, wordsize and endianness vary.


-- wli

Re: silent semantic changes with reiser4

[William Lee Irwin III]

On Mon, Sep 06, 2004 at 11:28:06PM -0700, William Lee Irwin III wrote:
> This thread is getting a bit soft on technical details and/or value.
> Hans, please post the results of fsstress (the bits from ext3 cvs) on
> reiser4 on SMP machines of non-i386 architectures (e.g. ppc64, sparc64,
> ia64) as well as the results of transferring reiser4 filesystems
> between machines whose PAGE_SIZE, wordsize and endianness vary.

Okay, sounds like time to run these tests and post the results myself.

Hans, these things should be part of your standard QA. It would likely
make a better impression if there were some record of these kinds of
things having been successfully tested prior to your releases. For
future reference, Andrew, Christoph, myself, and others can provide
more detailed references to suites of stress tests and various kinds of
tests filesystems should pass before being considered stable, and we
(it's a relatively safe presumption that I speak for all of us when I
say this) would appreciate this kind of testing in the future.


-- wli

Re: silent semantic changes with reiser4

[William Lee Irwin III]

[-- Attachment #1: Type: text/plain, Size: 5599 bytes --]

On Tue, Sep 07, 2004 at 03:38:01PM -0700, William Lee Irwin III wrote:
> Okay, sounds like time to run these tests and post the results myself.
> Hans, these things should be part of your standard QA. It would likely
> make a better impression if there were some record of these kinds of
> things having been successfully tested prior to your releases. For
> future reference, Andrew, Christoph, myself, and others can provide
> more detailed references to suites of stress tests and various kinds of
> tests filesystems should pass before being considered stable, and we
> (it's a relatively safe presumption that I speak for all of us when I
> say this) would appreciate this kind of testing in the future.

Step 1, the tools are very broken. This level of nonfunctionality
of the reiser4 toolchain precludes any kind of exposure to the kind of
testing I've asked about. I would very strongly prefer not to have to
become a reiser4 implementor and furthermore fix numerous bugs just to
have the smallest bit of assurance that this thing won't generate bug
reports en masse once merged.

The following is from an UltraSPARC system (64-bit wordsize, big-endian,
8KB pagesize, 32-bit userspace, including 32-bit compiled reiser4 tools)
running 2.6.9-rc1-mm3:


# /usr/local/sbin/mkfs.reiser4 -f -b 4096 /dev/loop0
/usr/local/sioctl32(mkfs.reiser4:1949): Unknown cmd fd(3) cmd(40081272){00} arg(efffc810) on /dev/loop0
4boicnt/lm3k2f(sm.krfesi.sreeri4s e1r.40:.109
 9C)o:p yUrnikgnhotw n( Cc)m d2 0f0d1(,3 )2 0c0m2d,( 420000831,2 7220)0{40 0b}y raHragn(se fRfefics8e1r0,)  loinc e/ndseivn/gl ogoopv0e
 ned by
reiser4progs/COPYING.

Block size 4096 will be used.
Linux 2.6.9-rc1-mm3 is detected.
Reiser4 is going to be created on /dev/loop0.
(Yes/No): Yes
 ioctl32(mkfs.reiser4:1949): Unknown cmd fd(3) cmd(40081272){00} arg(efffc738) on /dev/loop0
i o c t l 3 2 ( m k f s . r e i s e r 4 : 1 9 4 9 ) :   U n k n o w n   c m d   sC r/edaetvi/nlgo orpe0i0 8 1 2 7 2 ) { 0 0 }   a r g ( e f f f c 7 3 8 )
 er4 on /dev/loop0 ... Bus error
#

With the below patch, I get:
# /usr/local/sbin/mkfs.reiser4 -f -b 4096 /dev/loop0                            /usr/local/sbin/mkfs.reiser4 1.0.0
Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by
reiser4progs/COPYING.

Block size 4096 will be used.
Linux 2.6.9-rc1-mm3 is detected.
Reiser4 is going to be created on /dev/loop0.
(Yes/No): Yes
Creating reiser4 on /dev/loop0 ... Bus error
#

The S_ISBLK() check is so that other assumptions about the file being
a block device elsewhere won't be tripped up; it's not strictly
necessary, only retained so as not to expose unaudited code to a new
situation for which it's not prepared.

strace(1) before and after the patch is included here. The backtrace
from the core dump is:

#0  0x7005e464 in cde40_insert_units () from /usr/local/lib/libreiser4-1.0.so.0
(gdb) bt
#0  0x7005e464 in cde40_insert_units () from /usr/local/lib/libreiser4-1.0.so.0
#1  0x7004fa44 in node40_modify () from /usr/local/lib/libreiser4-1.0.so.0
#2  0x7003c608 in cb_node_insert () from /usr/local/lib/libreiser4-1.0.so.0
#3  0x7003c58c in reiser4_node_modify ()
   from /usr/local/lib/libreiser4-1.0.so.0
#4  0x7003ee7c in reiser4_tree_modify ()
   from /usr/local/lib/libreiser4-1.0.so.0
#5  0x7006bc10 in obj40_insert () from /usr/local/lib/libreiser4-1.0.so.0
#6  0x7006d9e8 in dir40_create () from /usr/local/lib/libreiser4-1.0.so.0
#7  0x70041610 in reiser4_object_create ()
   from /usr/local/lib/libreiser4-1.0.so.0
#8  0x00012160 in main ()


-- wli

Index: libaal-1.0.0/src/file.c
===================================================================
--- libaal-1.0.0.orig/src/file.c	2004-01-08 06:49:40.000000000 -0800
+++ libaal-1.0.0/src/file.c	2004-09-07 19:36:49.593844072 -0700
@@ -193,59 +193,31 @@
 	return !aal_strncmp(file1, file2, aal_strlen(file1));
 }
 
-#if defined(__linux__) && defined(_IOR) && !defined(BLKGETSIZE64)
-#   define BLKGETSIZE64 _IOR(0x12, 114, uint64_t)
-#endif
-
 /* Handler for "len" operation for use with file device. See bellow for
    understanding where it is used. */
 static count_t file_len(
 	aal_device_t *device)	    /* file device, lenght will be obtained from */
 {
-	uint64_t size;
-	off_t max_off = 0;
+	off_t off, size;
+	struct stat stat_buf;
+	int fd;
 
-	if (!device) 
+	if (!device)
 		return INVAL_BLK;
-    
-#ifdef BLKGETSIZE64
-	if ((int)ioctl(*((int *)device->entity), BLKGETSIZE64, &size) >= (int)0) {
-		uint32_t block_count;
-		
-		size = (size / 4096) * 4096 / device->blksize;
-		block_count = size;
-		
-		if ((uint64_t)block_count != size) {
-			aal_fatal("The partition size is too big.");
-			return INVAL_BLK;
-		}
-		
-		return (count_t)block_count;
-	}
-    
-#endif
-
-#ifdef BLKGETSIZE    
-	{
-		unsigned long l_size;
-		
-		if (ioctl(*((int *)device->entity), BLKGETSIZE, &l_size) >= 0) {
-			size = l_size;
-			return (count_t)((size * 512 / 4096) * 4096 / 
-				device->blksize);
-		}
-	}
-    
-#endif
-    
-	if ((max_off = lseek(*((int *)device->entity), 
-			     0, SEEK_END)) == (off_t)-1) 
-	{
-		file_error(device);
+	fd = *(int *)device->entity;
+	if (fstat(fd, &stat_buf))
+		return INVAL_BLK;
+	if (!S_ISBLK(stat_buf.st_mode))
+		return INVAL_BLK;
+	off = lseek(fd, 0, SEEK_CUR);
+	size = lseek(fd, 0, SEEK_END);
+	if (lseek(fd, off, SEEK_SET) == (off_t)-1)
+		errno = 0;
+	if (size == (off_t)-1) {
+		errno = 0;
 		return INVAL_BLK;
 	}
-    
-	return (count_t)(max_off / device->blksize);
+	return (size & ~4096ULL)/device->blksize;
 }
 
 /* Initializing the file device operations. They are used when any operation of

[-- Attachment #2: mkfs.reiser4.log.1.gz --]
[-- Type: application/octet-stream, Size: 2695 bytes --]

[-- Attachment #3: mkfs.reiser4.log.2.gz --]
[-- Type: application/octet-stream, Size: 2679 bytes --]

Re: silent semantic changes with reiser4

[William Lee Irwin III]

On Tue, Sep 07, 2004 at 07:43:19PM -0700, William Lee Irwin III wrote:
> -	return (count_t)(max_off / device->blksize);
> +	return (size & ~4096ULL)/device->blksize;

Correcting this to return (size & ~4095ULL)/device->blksize does not
fix the coredumps.


-- wli

Re: silent semantic changes with reiser4

[Christer Weinigel]

David Masover <ninja@slaphack.com> writes:

> |>Second, there are quite a few things which I might want to do, which can
> |>be done with this interface and without patching programs,
> | Such as?
> They've been mentioned.

> | Haven't seen any that made sense to me, sorry.
> Sorry if they don't make sense to you, but I don't feel like discussing
> them now.  Either you get it or you don't, either you agree or you
> don't.  Read the archives.

Great argument.  Not.  There has been so much shit thrown around here
so that it's impossible to keep track of all examples.

Could you please try summarize a few of the arguments that you find
especially compelling?  This thread has gotten very confused since
there are a bunch of different subjects all being intermixed here.

What are we discussing?

1. Do we want support for named streams?

   I belive the answer is yes, since both NTFS and HFS (that's the
   MacOS filesystem, isn't it?) supports streams we want Linux to
   support this if possible.

   Anyone disagreeing?

2. How do we want to expose named streams?

   One suggestion is file-as-directory in some form.

   Another suggestion made is to expose named streams somewhere under
   /proc/self/fd.

   Yet another suggestion is to use the openat(3) API from solaris.

   Some filesystems exposes extra data in a special directory in the
   same directory as the file, such as netapps .snapshot directories
   or the extra directories that netatalk expects.  This has the
   advantage that it even works on non-named stream capable
   filesystems, but it has a lot of problems too.

   Linux already has limited support for names streams via the xattr
   interface, but it's not a good interface for people wanting to
   have large files as named streams.

4. What belongs in the generic VFS, what belongs in Reiser4?

   Some things reiser4 do, such as files-as-directories need changes
   to the VFS because it breaks assumptions that the VFS makes
   (i.e. a deadlock or an oops when doing a hard link out of one).

   Some other things reiser4 can do would be better if they were in
   the VFS since other filesystems might want to support the same
   functionality. 

   Or Linux may not support some of the things reiserfs at all.

5. What belongs in the kernel, what belongs in userspace?

   This is mostly what I have been trying to argue about.

So, to try to summarize my opinion, regarding file-as-directory, I
belive it's fatally flawed because it breaks a lot of assumptions that
existing code make.  One example of an application that will break is
a web server that tries to filter out accesses to "bad" files,
files-as-directories suddenly means that part of those files will be
accessible (and there are a _lot_ of CERT reports on just this kind of
problems with Windows web servers due to access to named streams not
being restricted or ways to access files with non-canonical names that
also managed to bypass access restrictions).

Files-as-directories also does not give us named streams on
directories.  The suggestion to have dir/metas access the named
streams means that if someone already has a file named metas in a
directory that file will be lost.  (Does anyone remember the
discussions about the linux kernel having a directory named "core" and
the problems this caused for some people?)

All this suggests to me that named streams must live in another
namespace than the normal one.  To me, openat(3) seems like a good
choice for an API and it has the advantage that someone else, Solaris,
already has implemented it.

Additionally, files-as-directores does not solve the problem of 
"cp a b" losing named streams.  There is curently no copyfile syscall
in the Linux kernel, "cp a b" essentially does "cat a >b".  So unless
cp is modified we don't gain anything.  If cp is modified to know
about named streams, it really does not matter if named streams are
accessed as file-as-directories, via openat(3) or via a shared library
with some other interface.

Regarding the kernel or userspace discussion.  In my opinion anything
that can be done in user space should be done in userspace.  If the
performance sucks, or it has security problems, or needs caching that
cant be solved in userspace it can be moved to the kernel, but in that
case the smallest and cleanest API possible should be implemented.

If, for historical reasons, an API must be in the kernel, there is not
much we can do about it either.  It'll have to stay there, but we can
avoid making the same mistakes again.

So, for all the examples of the kernel having plugins that
automatically lets an application see a tar-file as a directory, I
really, really don't belive this belongs in the kernel.  First of all,
this is the file-as-directory discussion again, I belive it is a
mistake to expose the contents as a directory on top of the file
because it breaks a lot of assumptions that unix programs make.

It's much better to expose the contents at another place in the
filesystem by doing a temporary mount of the file with the proper
filesystem.  As Pavel Machek pointed out, this has the problem of who
cleans up the mount if the application crashes.  One way to handle
this could be something like this:

    mount -t tarfs -o loop bar.tar /tmp/bar-fabb50509
    chdir /tmp/bar-fabb50509
    umount -f /tmp/bar-fabb50509

This will require the ability to unmount busy filesystems (but I
belive Alexander Viro already has implemented the infrastructure
needed for this).

Or for files that we don't have a real filesystem driver (or on other
systems where userspace mounts are not allowed), we could just unpack
the contents into /tmp.  For cleanup we could let whatever cleans up
/tmp anyways handle it, or have a cache daemon that keeps track of
untarred directories and removes them after a while.

Another way is to completely forget about presenting the contents of a
tar file as a real files, and just use a shared library to get at the
contents (now we just have to convince everyone to use the shared
library).  This would also be portable to other systems.

If we do this right, it could all be hidden in a shared library, and
if the system below it supports more advanced features, it can use it.

Regarding the "I want a realtime index of all files".  I belive that a
notifier that can tell me when a file has been changed and a userspace
daemon ought to handle most of the cases that have been mentioned.
The suggested problems of not getting an up to date query response can
be handled by just asking the daemon "are you done with indexing yet".
The design of such a daemon and the support it needs from the kernel
can definitely be discussed.  But to put the indexer itself in the
kernel sounds like a bad idea.  Even adding an API to query the
indexer into the kernel sounds pointless, why do that instead of just
opening a Unix socket to the indexer and asking it directly?

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Gunnar Ritter]

Christer Weinigel <christer@weinigel.se> wrote:

> Additionally, files-as-directores does not solve the problem of 
> "cp a b" losing named streams.  There is curently no copyfile syscall
> in the Linux kernel, "cp a b" essentially does "cat a >b".  So unless
> cp is modified we don't gain anything.  If cp is modified to know
> about named streams, it really does not matter if named streams are
> accessed as file-as-directories, via openat(3) or via a shared library
> with some other interface.

You cannot just 'modify cp'. cp is a programming interface standardized
in POSIX.1. You can of course add non-standard extensions to some cp
implementations, but it seems hardly evitable then that you either have
to use cp in a non-standard manner regularly with Linux or risk to lose
data.

This is even more severe with tar/pax. Just patching GNU tar for file
streams, as it was suggested earlier in this discussion, is still far
away from a real solution because it neither solves the issues with
the POSIX.1 pax standard nor those with other implementations of it.

Given these facts, it does not seem so clear to me that adding named
streams for Windows and Mac OS interoperability would be a win to Linux
in the end. The loss of interoperability to Unix/POSIX/today's Linux
might have much worse effects.

The current xattr extension is much less of a problem because it only
holds metadata, which is mostly not applicable to other environments
anyway.

	Gunnar

-- 
http://omnibus.ruf.uni-freiburg.de/~gritter

Re: silent semantic changes with reiser4

[Hans Reiser]

Gunnar Ritter wrote:

>
>
>You cannot just 'modify cp'. 
>
People who think that POSIX is the objective rather than the least 
common denominator of OS design have had their head screwed on backwards 
to better look at where their competitors used to be.

However, I agree that streams suck.  That is why reiser4 just has files 
and directories and not streams.  Our files and directories just happen 
to be able to do all that streams can do.

Hans

Re: silent semantic changes with reiser4

[Gunnar Ritter]

Hans Reiser <reiser@namesys.com> wrote:

> Gunnar Ritter wrote:
> >You cannot just 'modify cp'. 
> >
> People who think that POSIX is the objective rather than the least 
> common denominator of OS design

I am not principally adversed against extensions to POSIX. My mailx
implementation 'nail' has e.g. perhaps more extensions than there are
commands and options in the POSIX standard for it.

POSIX is also not against extensions. In fact, POSIX development
generally works as follows: One vendor creates something as an extension,
other vendors follow to implement it, and later on it is discussed if it
is desirable to integrate the feature into the standard itself. It is
absolutely possible that Sun's openat() might be in POSIX.1-2010 one day,
for example. Useful extensions are thus welcome to POSIX.

This does not mean, however, that one should not clearly distinct between
standard and extensions, and that extensions should be created at will
without carefully weighting pros and cons.

I did not say: POSIX forbids to handle streams or directory/file mixes.
This would not even have been true. However, POSIX restricts the choice
of possible interfaces for them. One of those restrictions is that 'cp'
may not copy streams if used in strict accordance with POSIX. As you
acknowledged in your reply, POSIX is the least common denominator. Thus
'cp' implementations should not be modified in a way that violates it.

This means, in effect, that a strictly conforming POSIX application (i.e.
something like a shell script that uses no POSIX extensions or methods
which are not clearly defined in the standard) will very likely be unable
to copy streams, unless some other, conforming, method is found. Which is
a problem one should know about when discussing this.

> have had their head screwed on backwards 
> to better look at where their competitors used to be.

But there are not only forwards and backwards directions. Sideways might
lead to nowhere.

	Gunnar

-- 
http://omnibus.ruf.uni-freiburg.de/~gritter

Re: silent semantic changes with reiser4

[Herbert Poetzl]

On Tue, Sep 07, 2004 at 01:55:32PM +0200, Christer Weinigel wrote:
> David Masover <ninja@slaphack.com> writes:
> 
> > |>Second, there are quite a few things which I might want to do, which can
> > |>be done with this interface and without patching programs,
> > | Such as?
> > They've been mentioned.
> 
> > | Haven't seen any that made sense to me, sorry.
> > Sorry if they don't make sense to you, but I don't feel like discussing
> > them now.  Either you get it or you don't, either you agree or you
> > don't.  Read the archives.
> 
> Great argument.  Not.  There has been so much shit thrown around here
> so that it's impossible to keep track of all examples.
> 
> Could you please try summarize a few of the arguments that you find
> especially compelling?  This thread has gotten very confused since
> there are a bunch of different subjects all being intermixed here.
> 
> What are we discussing?
> 
> 1. Do we want support for named streams?
> 
>    I belive the answer is yes, since both NTFS and HFS (that's the
>    MacOS filesystem, isn't it?) supports streams we want Linux to
>    support this if possible.

well, yes HFS has this, is it advantageous, no
it's kind of heritage ...

>    Anyone disagreeing?

yes, MacOS X allows to use UFS instead of HFS+
which doesn't support the fancy/confusing streams

I, for my part, do not like the idea of multiple
streams for one file, IMHO all features can be
provided by using directories instead, which does
not break any userspace tools _and_ sounds natural
to me ...

best,
Herbert

> 2. How do we want to expose named streams?
> 
>    One suggestion is file-as-directory in some form.
> 
>    Another suggestion made is to expose named streams somewhere under
>    /proc/self/fd.
> 
>    Yet another suggestion is to use the openat(3) API from solaris.
> 
>    Some filesystems exposes extra data in a special directory in the
>    same directory as the file, such as netapps .snapshot directories
>    or the extra directories that netatalk expects.  This has the
>    advantage that it even works on non-named stream capable
>    filesystems, but it has a lot of problems too.
> 
>    Linux already has limited support for names streams via the xattr
>    interface, but it's not a good interface for people wanting to
>    have large files as named streams.
> 
> 4. What belongs in the generic VFS, what belongs in Reiser4?
> 
>    Some things reiser4 do, such as files-as-directories need changes
>    to the VFS because it breaks assumptions that the VFS makes
>    (i.e. a deadlock or an oops when doing a hard link out of one).
> 
>    Some other things reiser4 can do would be better if they were in
>    the VFS since other filesystems might want to support the same
>    functionality. 
> 
>    Or Linux may not support some of the things reiserfs at all.
> 
> 5. What belongs in the kernel, what belongs in userspace?
> 
>    This is mostly what I have been trying to argue about.
> 
> So, to try to summarize my opinion, regarding file-as-directory, I
> belive it's fatally flawed because it breaks a lot of assumptions that
> existing code make.  One example of an application that will break is
> a web server that tries to filter out accesses to "bad" files,
> files-as-directories suddenly means that part of those files will be
> accessible (and there are a _lot_ of CERT reports on just this kind of
> problems with Windows web servers due to access to named streams not
> being restricted or ways to access files with non-canonical names that
> also managed to bypass access restrictions).
> 
> Files-as-directories also does not give us named streams on
> directories.  The suggestion to have dir/metas access the named
> streams means that if someone already has a file named metas in a
> directory that file will be lost.  (Does anyone remember the
> discussions about the linux kernel having a directory named "core" and
> the problems this caused for some people?)
> 
> All this suggests to me that named streams must live in another
> namespace than the normal one.  To me, openat(3) seems like a good
> choice for an API and it has the advantage that someone else, Solaris,
> already has implemented it.
> 
> Additionally, files-as-directores does not solve the problem of 
> "cp a b" losing named streams.  There is curently no copyfile syscall
> in the Linux kernel, "cp a b" essentially does "cat a >b".  So unless
> cp is modified we don't gain anything.  If cp is modified to know
> about named streams, it really does not matter if named streams are
> accessed as file-as-directories, via openat(3) or via a shared library
> with some other interface.
> 
> Regarding the kernel or userspace discussion.  In my opinion anything
> that can be done in user space should be done in userspace.  If the
> performance sucks, or it has security problems, or needs caching that
> cant be solved in userspace it can be moved to the kernel, but in that
> case the smallest and cleanest API possible should be implemented.
> 
> If, for historical reasons, an API must be in the kernel, there is not
> much we can do about it either.  It'll have to stay there, but we can
> avoid making the same mistakes again.
> 
> So, for all the examples of the kernel having plugins that
> automatically lets an application see a tar-file as a directory, I
> really, really don't belive this belongs in the kernel.  First of all,
> this is the file-as-directory discussion again, I belive it is a
> mistake to expose the contents as a directory on top of the file
> because it breaks a lot of assumptions that unix programs make.
> 
> It's much better to expose the contents at another place in the
> filesystem by doing a temporary mount of the file with the proper
> filesystem.  As Pavel Machek pointed out, this has the problem of who
> cleans up the mount if the application crashes.  One way to handle
> this could be something like this:
> 
>     mount -t tarfs -o loop bar.tar /tmp/bar-fabb50509
>     chdir /tmp/bar-fabb50509
>     umount -f /tmp/bar-fabb50509
> 
> This will require the ability to unmount busy filesystems (but I
> belive Alexander Viro already has implemented the infrastructure
> needed for this).
> 
> Or for files that we don't have a real filesystem driver (or on other
> systems where userspace mounts are not allowed), we could just unpack
> the contents into /tmp.  For cleanup we could let whatever cleans up
> /tmp anyways handle it, or have a cache daemon that keeps track of
> untarred directories and removes them after a while.
> 
> Another way is to completely forget about presenting the contents of a
> tar file as a real files, and just use a shared library to get at the
> contents (now we just have to convince everyone to use the shared
> library).  This would also be portable to other systems.
> 
> If we do this right, it could all be hidden in a shared library, and
> if the system below it supports more advanced features, it can use it.
> 
> Regarding the "I want a realtime index of all files".  I belive that a
> notifier that can tell me when a file has been changed and a userspace
> daemon ought to handle most of the cases that have been mentioned.
> The suggested problems of not getting an up to date query response can
> be handled by just asking the daemon "are you done with indexing yet".
> The design of such a daemon and the support it needs from the kernel
> can definitely be discussed.  But to put the indexer itself in the
> kernel sounds like a bad idea.  Even adding an API to query the
> indexer into the kernel sounds pointless, why do that instead of just
> opening a Unix socket to the indexer and asking it directly?
> 
>   /Christer
> 
> -- 
> "Just how much can I get away with and still go to heaven?"
> 
> Freelance consultant specializing in device driver programming for Linux 
> Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: silent semantic changes with reiser4

[Oliver Neukum]

Am Dienstag, 7. September 2004 14:30 schrieb Herbert Poetzl:
> > 1. Do we want support for named streams?
> > 
> >    I belive the answer is yes, since both NTFS and HFS (that's the
> >    MacOS filesystem, isn't it?) supports streams we want Linux to
> >    support this if possible.
> 
> well, yes HFS has this, is it advantageous, no
> it's kind of heritage ...

HFS doesn't support _named_ streams. There are just two forks
per file.

	Regards
		Oliver

Re: silent semantic changes with reiser4

[Spam]

> David Masover <ninja@slaphack.com> writes:

>> |>Second, there are quite a few things which I might want to do, which can
>> |>be done with this interface and without patching programs,
>> | Such as?
>> They've been mentioned.

>> | Haven't seen any that made sense to me, sorry.
>> Sorry if they don't make sense to you, but I don't feel like discussing
>> them now.  Either you get it or you don't, either you agree or you
>> don't.  Read the archives.

> Great argument.  Not.  There has been so much shit thrown around here
> so that it's impossible to keep track of all examples.

> Could you please try summarize a few of the arguments that you find
> especially compelling?  This thread has gotten very confused since
> there are a bunch of different subjects all being intermixed here.

  Indeed. We are discussion changes to the heart of Linux. It is bound
  to get a little heated :/

> What are we discussing?

> 1. Do we want support for named streams?

>    I belive the answer is yes, since both NTFS and HFS (that's the
>    MacOS filesystem, isn't it?) supports streams we want Linux to
>    support this if possible.

>    Anyone disagreeing?

     No :)
     
> 2. How do we want to expose named streams?

>    One suggestion is file-as-directory in some form.

>    Another suggestion made is to expose named streams somewhere under
>    /proc/self/fd.

>    Yet another suggestion is to use the openat(3) API from solaris.

>    Some filesystems exposes extra data in a special directory in the
>    same directory as the file, such as netapps .snapshot directories
>    or the extra directories that netatalk expects.  This has the
>    advantage that it even works on non-named stream capable
>    filesystems, but it has a lot of problems too.

>    Linux already has limited support for names streams via the xattr
>    interface, but it's not a good interface for people wanting to
>    have large files as named streams.

> 4. What belongs in the generic VFS, what belongs in Reiser4?

>    Some things reiser4 do, such as files-as-directories need changes
>    to the VFS because it breaks assumptions that the VFS makes
>    (i.e. a deadlock or an oops when doing a hard link out of one).

>    Some other things reiser4 can do would be better if they were in
>    the VFS since other filesystems might want to support the same
>    functionality. 

>    Or Linux may not support some of the things reiserfs at all.

> 5. What belongs in the kernel, what belongs in userspace?

>    This is mostly what I have been trying to argue about.

> So, to try to summarize my opinion, regarding file-as-directory, I
> belive it's fatally flawed because it breaks a lot of assumptions that
> existing code make.  One example of an application that will break is
> a web server that tries to filter out accesses to "bad" files,
> files-as-directories suddenly means that part of those files will be
> accessible (and there are a _lot_ of CERT reports on just this kind of
> problems with Windows web servers due to access to named streams not
> being restricted or ways to access files with non-canonical names that
> also managed to bypass access restrictions).

  But restrictions could be controlled. But exactly how depends on
  what solution for named streams is chosen. For example, there might
  be separate permissions for named streams other than the main
  stream.

> Files-as-directories also does not give us named streams on
> directories.  The suggestion to have dir/metas access the named
> streams means that if someone already has a file named metas in a
> directory that file will be lost.  (Does anyone remember the
> discussions about the linux kernel having a directory named "core" and
> the problems this caused for some people?)

  Yes, reserved keywords needs to be chosen very carefully if they are
  used. Other examples here has been ..metas or ...

> All this suggests to me that named streams must live in another
> namespace than the normal one.  To me, openat(3) seems like a good
> choice for an API and it has the advantage that someone else, Solaris,
> already has implemented it.

> Additionally, files-as-directores does not solve the problem of 
> "cp a b" losing named streams.  There is curently no copyfile syscall
> in the Linux kernel, "cp a b" essentially does "cat a >b".  So unless
> cp is modified we don't gain anything.  If cp is modified to know
> about named streams, it really does not matter if named streams are
> accessed as file-as-directories, via openat(3) or via a shared library
> with some other interface.

  One suggestion is missed. It is to provide system calls for copy.
  That would also solve the problem. Named streams and metas would
  then be handled correctly. It also allows further changes to
  filesystems without having to patch applications yet again.

  A copy system call would also be large beneficial for networked
  filesystems (NFS, Samba, etc) as data wouldn't have to be
  transferred over the network and back.

> Regarding the kernel or userspace discussion.  In my opinion anything
> that can be done in user space should be done in userspace.  If the
> performance sucks, or it has security problems, or needs caching that
> cant be solved in userspace it can be moved to the kernel, but in that
> case the smallest and cleanest API possible should be implemented.

  Yes. I agree. I do not think most people arguing for plugins or
  usage for named streams actually wanted to put everything in the
  kernel, but rather to have a kernel interface to which user-level
  modules, plugins and applications could work.

> If, for historical reasons, an API must be in the kernel, there is not
> much we can do about it either.  It'll have to stay there, but we can
> avoid making the same mistakes again.

> So, for all the examples of the kernel having plugins that
> automatically lets an application see a tar-file as a directory, I
> really, really don't belive this belongs in the kernel.  First of all,
> this is the file-as-directory discussion again, I belive it is a
> mistake to expose the contents as a directory on top of the file
> because it breaks a lot of assumptions that unix programs make.

> It's much better to expose the contents at another place in the
> filesystem by doing a temporary mount of the file with the proper
> filesystem.  As Pavel Machek pointed out, this has the problem of who
> cleans up the mount if the application crashes.  One way to handle
> this could be something like this:

>     mount -t tarfs -o loop bar.tar /tmp/bar-fabb50509
>     chdir /tmp/bar-fabb50509
>     umount -f /tmp/bar-fabb50509

> This will require the ability to unmount busy filesystems (but I
> belive Alexander Viro already has implemented the infrastructure
> needed for this).

> Or for files that we don't have a real filesystem driver (or on other
> systems where userspace mounts are not allowed), we could just unpack
> the contents into /tmp.  For cleanup we could let whatever cleans up
> /tmp anyways handle it, or have a cache daemon that keeps track of
> untarred directories and removes them after a while.

  But now, as you said, we are not talking about named streams, but
  about a specic type of plugin and its implications.

  Can we make a plugin infrastructure that will let user-space plugins
  to be loaded for certain directories or files? If we can, then it
  would present a much cleaner and easier way for the user to access
  data he wants. In this particular example it was a tar file.

> Another way is to completely forget about presenting the contents of a
> tar file as a real files, and just use a shared library to get at the
> contents (now we just have to convince everyone to use the shared
> library).  This would also be portable to other systems.

  Yes. If it was possible. I do honestly think it will be easier and
  less man hour work to create a run-time loadable user-space plugin
  interface than to convince thousands of application developers to
  use this shared library.
  
> If we do this right, it could all be hidden in a shared library, and
> if the system below it supports more advanced features, it can use it.

> Regarding the "I want a realtime index of all files".  I belive that a
> notifier that can tell me when a file has been changed and a userspace
> daemon ought to handle most of the cases that have been mentioned.

  We have FAM (File Alteration Monitor) that notifies applications of
  changes to files they are interested in.

  http://oss.sgi.com/projects/fam/

> The suggested problems of not getting an up to date query response can
> be handled by just asking the daemon "are you done with indexing yet".
> The design of such a daemon and the support it needs from the kernel
> can definitely be discussed.  But to put the indexer itself in the
> kernel sounds like a bad idea.  Even adding an API to query the
> indexer into the kernel sounds pointless, why do that instead of just
> opening a Unix socket to the indexer and asking it directly?

  I think that we do not need a indexer with Reiser4 anyway. It is
  already a database that could be queried directly. There shouldn't
  be any need to build a database on top of Reiser4 (like the
  updatedb) that holds the same information already existing in
  Reiser4.

  Now, it is just a matter of having an interface to query the Reiser4
  db. One of the suggested ways were through meta-data. For example:

  ls -l ..metas/keyword/.pdf

  ~S
  
>   /Christer

Re: silent semantic changes with reiser4

[Christer Weinigel]

Spam <spam@tnonline.net> writes:

> > Additionally, files-as-directores does not solve the problem of 
> > "cp a b" losing named streams.  There is curently no copyfile syscall
> > in the Linux kernel, "cp a b" essentially does "cat a >b".  So unless
> > cp is modified we don't gain anything.  If cp is modified to know
> > about named streams, it really does not matter if named streams are
> > accessed as file-as-directories, via openat(3) or via a shared library
> > with some other interface.
> 
>   One suggestion is missed. It is to provide system calls for copy.
>   That would also solve the problem. Named streams and metas would
>   then be handled correctly. It also allows further changes to
>   filesystems without having to patch applications yet again.

But this still solves only part of the problem.  A backup application
won't have any use for a copyfile syscall, it will need to be taught
about streams.

>   A copy system call would also be large beneficial for networked
>   filesystems (NFS, Samba, etc) as data wouldn't have to be
>   transferred over the network and back.

Definitely.  

>   Can we make a plugin infrastructure that will let user-space plugins
>   to be loaded for certain directories or files? If we can, then it
>   would present a much cleaner and easier way for the user to access
>   data he wants. In this particular example it was a tar file.

In that case I'd argue that:

    mount -t userfs -o driver=tarfs foo /tmp/foo

is a rather good kernel interface for plugins.  userfs (or something
based on userfs) is the plugin API and tarfs is a plugin. :-)

To make this efficient, well have to allow non-root users to perform
the mount syscall (with the limitation that they can only mount on top
of directories they own and that the mounts have the nosuid and nodev
flags set).

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> writes:

>> > Additionally, files-as-directores does not solve the problem of 
>> > "cp a b" losing named streams.  There is curently no copyfile syscall
>> > in the Linux kernel, "cp a b" essentially does "cat a >b".  So unless
>> > cp is modified we don't gain anything.  If cp is modified to know
>> > about named streams, it really does not matter if named streams are
>> > accessed as file-as-directories, via openat(3) or via a shared library
>> > with some other interface.
>> 
>>   One suggestion is missed. It is to provide system calls for copy.
>>   That would also solve the problem. Named streams and metas would
>>   then be handled correctly. It also allows further changes to
>>   filesystems without having to patch applications yet again.

> But this still solves only part of the problem.  A backup application
> won't have any use for a copyfile syscall, it will need to be taught
> about streams.

  Yes, but backup programs always needed to be taught about new
  features. Be it new type of files, attributes or meta-data. I think
  that teaching backup applications is far better than teaching every
  application.

>>   A copy system call would also be large beneficial for networked
>>   filesystems (NFS, Samba, etc) as data wouldn't have to be
>>   transferred over the network and back.

> Definitely.  

>>   Can we make a plugin infrastructure that will let user-space plugins
>>   to be loaded for certain directories or files? If we can, then it
>>   would present a much cleaner and easier way for the user to access
>>   data he wants. In this particular example it was a tar file.

> In that case I'd argue that:

>     mount -t userfs -o driver=tarfs foo /tmp/foo

> is a rather good kernel interface for plugins.  userfs (or something
> based on userfs) is the plugin API and tarfs is a plugin. :-)

> To make this efficient, well have to allow non-root users to perform
> the mount syscall (with the limitation that they can only mount on top
> of directories they own and that the mounts have the nosuid and nodev
> flags set).

  Yes, this seem to be one solution. It isn't very dynamic in usage
  though. You can't use this directly from applications wihout
  manually doing the mount.

  This is only a solution to browsing contents of files. It doesn't
  provide a solution for using meta-data streams or other things like
  this.

  ~S

>   /Christer

Re: silent semantic changes with reiser4

[Horst von Brand]

Spam <spam@tnonline.net> said:
> Christer Weinigel <christer@weinigel.se> said:

[...]

> > But this still solves only part of the problem.  A backup application
> > won't have any use for a copyfile syscall, it will need to be taught
> > about streams.

>   Yes, but backup programs always needed to be taught about new
>   features. Be it new type of files, attributes or meta-data. I think
>   that teaching backup applications is far better than teaching every
>   application.

Strange... tar(1) is quite capable of backing up .mp3 files, which weren't
around when tar was born...
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Hans Reiser]

Christer Weinigel wrote:

>Spam <spam@tnonline.net> writes:
>
>  
>
>>>Additionally, files-as-directores does not solve the problem of 
>>>"cp a b" losing named streams.
>>>
reiser4 does not support streams, it supports files that can do what 
streams do. cp -r does not currently lose files.

Re: silent semantic changes with reiser4

[Christer Weinigel]

Hans Reiser <reiser@namesys.com> writes:

> Christer Weinigel wrote:
> 
> >Spam <spam@tnonline.net> writes:
> >
> >
> >>> Additionally, files-as-directores does not solve the problem of
> >>> "cp a b" losing named streams.
> >>>
> reiser4 does not support streams, it supports files that can do what
> streams do. cp -r does not currently lose files.

I'm not talking about reiser4 only.  I'm interested in semantics that
can be sanely apply to reiser4, NTFS, HFS, or any other file system
that supports XATTRS.

I relize that resier4 is your baby and you're only interested in it,
but I'm much more interested in linux in general.

  /Christer


-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Horst von Brand]

Spam <spam@tnonline.net> said:
> Christer Weinigel <christer@weinigel.se> said:

[...]

> > Could you please try summarize a few of the arguments that you find
> > especially compelling?  This thread has gotten very confused since
> > there are a bunch of different subjects all being intermixed here.

>   Indeed. We are discussion changes to the heart of Linux. It is bound
>   to get a little heated :/

True. But without any specific applications, just "this would be nice to
have", the discussion can't go forward.

> > What are we discussing?

> > 1. Do we want support for named streams?
> 
> >    I belive the answer is yes, since both NTFS and HFS (that's the
> >    MacOS filesystem, isn't it?) supports streams we want Linux to
> >    support this if possible.
> 
> >    Anyone disagreeing?
> 
>      No :)

There are many people around here who disagree (that is precisely the heart
of the discussion). I for one don't think Linux has to get $RANDOM_FEATURE
just because $SOME_OTHER_OS has got it. Either the feature stands on its
own _in the context of POSIX/Unix/Linux_ (possibly as an extension or
modification of said standards) or it isn't worth it.

> > 2. How do we want to expose named streams?
> 
> >    One suggestion is file-as-directory in some form.

Which is broken, as it forbids hard links to files.

It also has quite interesting issues with moving streams around among
containers, or renaming them. How would you handle permissions on streams
inside containers? Links to streams? What happens if the principal stream
is deleted? Renamed? Moved into another container?

Now you have 3 principal types of objects: Directories, containers (files
with streams), and files (no streams). What is the essential difference
between containers and directories? Same for containers vs plain files? If
none, containers are just complicating stuff, and should not exist at all.

If you add streams to directories, you get 4 types. Again, tell what the
exact differences are, and what each one can do none of the others can.

In the preceding two exercises, it is not enough to show something that
would be dificult or messy to do with what we have now, it must also be
some operation(s) that are natural and common enough to warrant the extra
complexity (in kernel, in applications, in wetware). Consider support in
form of existing or to-be-written tools for the task. What of all this can
the GUI hide from (or fake for) the unsuspecting end-user?

Also, there have been suggestions of cached data in streams (like thumbnail
version of graphic file, contents listing of archive, ...), where losing
the stream is of no real consecuence (except performance). There have also
been requests for metadata that can't be recovered from the file (author,
comments on the file, older versions of the file (i.e., some form of file
version control), permissions). And finally there have been requests for
structured contents manipulable in other forms (f.ex. .tar.bz2 file in
unpacked form, where you can directly read or even modify the contents).
Some suggestions go for a limited number of small(ish), even standardized,
extra streams with no further structure, others ask for full directories
under streams (which could even be much larger than the principal stream,
like the tar.bz2 case). And there is the possibility of recursive
application: If you want to be consecuent, the files under a .tar.bz2
should again be able to be .tar.bz2'es with their own streams, or files
with streams in them.

What permissions should apply to streams? The ones of the file (i.e., I can
see the icon stream of a file I can't read), or extra permissions (i.e.,
icon is writable to me (so I can adjust it to taste) but I can't write to
the file; now I can screw up the icon for everybody...). Streams on a
per-user basis (i.e., icon stream for each user)?

We need to sort out exactly how far it makes sense to go, by showing
concrete, down to earth uses for whatever substructure we want. Then show
the effect can't be easily gotten through tools for power users or faking
it for unsuspecting users via GUI, and that overall the complexity and
performance cost is less than the win. Note that the success of the Unix
way is in large part due to its use of few, simple concepts that can be
combined endlessly; and tools following the same strategy. Adding extra
concepts that current tools can't naturally handle has to be considered
with extreme care.

[-- much snippage --]

> > The suggested problems of not getting an up to date query response can
> > be handled by just asking the daemon "are you done with indexing yet".
> > The design of such a daemon and the support it needs from the kernel
> > can definitely be discussed.  But to put the indexer itself in the
> > kernel sounds like a bad idea.  Even adding an API to query the
> > indexer into the kernel sounds pointless, why do that instead of just
> > opening a Unix socket to the indexer and asking it directly?

>   I think that we do not need a indexer with Reiser4 anyway. It is
>   already a database that could be queried directly. There shouldn't
>   be any need to build a database on top of Reiser4 (like the
>   updatedb) that holds the same information already existing in
>   Reiser4.

Indexing/general queries of the filesystem are _hard_ (you need a complete
database behind it for this, plus handling transactions for full
functionality). Applications who _really_ need this use DBMSes, and are
prepared to pay the ensuing cost. They are far in between, in any case.

Besides, this is completely orthogonal to the streams discussion.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Hans Reiser]

Horst von Brand wrote:

>Spam <spam@tnonline.net> said:
>  
>
>>Christer Weinigel <christer@weinigel.se> said:
>>    
>>
>
>[...]
>
>  
>
>>>Could you please try summarize a few of the arguments that you find
>>>especially compelling?  This thread has gotten very confused since
>>>there are a bunch of different subjects all being intermixed here.
>>>      
>>>
>
>  
>
>>  Indeed. We are discussion changes to the heart of Linux. It is bound
>>  to get a little heated :/
>>    
>>
>
>True. But without any specific applications, just "this would be nice to
>have", the discussion can't go forward.
>
>  
>
>>>What are we discussing?
>>>      
>>>
>
>  
>
>>>1. Do we want support for named streams?
>>>      
>>>
>>>   I belive the answer is yes, since both NTFS and HFS (that's the
>>>   MacOS filesystem, isn't it?) supports streams we want Linux to
>>>   support this if possible.
>>>      
>>>
>>>   Anyone disagreeing?
>>>      
>>>
>>     No :)
>>    
>>
>
>There are many people around here who disagree (that is precisely the heart
>of the discussion). I for one don't think Linux has to get $RANDOM_FEATURE
>just because $SOME_OTHER_OS has got it. Either the feature stands on its
>own _in the context of POSIX/Unix/Linux_ (possibly as an extension or
>modification of said standards) or it isn't worth it.
>
>  
>
>>>2. How do we want to expose named streams?
>>>      
>>>
>>>   One suggestion is file-as-directory in some form.
>>>      
>>>
>
>Which is broken, as it forbids hard links to files.
>  
>
No, it forbids hard links to the directory aspect of the file-directory 
duality.

>
>
>Now you have 3 principal types of objects: Directories, containers (files
>with streams), and files (no streams).
>
No, the reiser4 design supports only files and directories, but makes 
them able to do what people use streams for.

The reiser4 design is based on a hatred of streams, and a desire to show 
that adding more features to files and directories makes streams 
unnecessary.

Hans

Re: silent semantic changes with reiser4

[Horst von Brand]

Hans Reiser <reiser@namesys.com> said:
> Horst von Brand wrote:
> >Spam <spam@tnonline.net> said:
> >>Christer Weinigel <christer@weinigel.se> said:

[...]

> >>>2. How do we want to expose named streams?

> >>>   One suggestion is file-as-directory in some form.

> >Which is broken, as it forbids hard links to files.

> No, it forbids hard links to the directory aspect of the file-directory 
> duality.

How do you distinguish a "hard link to the directory personality" from
"hard link to the file personality"? And how do you make sure that only one
of them can be followed if you go to the directory, while allowing several
to the exact same file?!

> >Now you have 3 principal types of objects: Directories, containers (files
> >with streams), and files (no streams).

> No, the reiser4 design supports only files and directories, but makes 
> them able to do what people use streams for.

I.e., you have files/directories with/without "use as streams" stuff. 4
types (or uses) of things.

> The reiser4 design is based on a hatred of streams, and a desire to show 
> that adding more features to files and directories makes streams 
> unnecessary.

Trade one ugly wart for others...
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Hans Reiser]

Horst von Brand wrote:

>Hans Reiser <reiser@namesys.com> said:
>  
>
>>Horst von Brand wrote:
>>    
>>
>>>Spam <spam@tnonline.net> said:
>>>      
>>>
>>>>Christer Weinigel <christer@weinigel.se> said:
>>>>        
>>>>
>
>[...]
>
>  
>
>>>>>2. How do we want to expose named streams?
>>>>>          
>>>>>
>
>  
>
>>>>>  One suggestion is file-as-directory in some form.
>>>>>          
>>>>>
>
>  
>
>>>Which is broken, as it forbids hard links to files.
>>>      
>>>
>
>  
>
>>No, it forbids hard links to the directory aspect of the file-directory 
>>duality.
>>    
>>
>
>How do you distinguish a "hard link to the directory personality" from
>"hard link to the file personality"?
>
Put in (undoubtedly overly) simple terms, if you can do it to a file you 
can do it to the file personality, but if you currently can only do it 
to a directory and get an error from attempting it to a file then in the 
new scheme doing it to the hard link only gives the same error.

Or, we can ask Alexander to help us use his deadlock detection algorithm 
and try to do things right....

Re: silent semantic changes with reiser4

[Horst von Brand]

Hans Reiser <reiser@namesys.com> said:
> Horst von Brand wrote:
> >Hans Reiser <reiser@namesys.com> said:
> >>Horst von Brand wrote:
> >>>Spam <spam@tnonline.net> said:
> >>>>Christer Weinigel <christer@weinigel.se> said:

[...]

> >>>>>2. How do we want to expose named streams?
> >>>>>  One suggestion is file-as-directory in some form.

> >>>Which is broken, as it forbids hard links to files.

> >>No, it forbids hard links to the directory aspect of the file-directory 
> >>duality.

> >How do you distinguish a "hard link to the directory personality" from
> >"hard link to the file personality"?

> Put in (undoubtedly overly) simple terms, if you can do it to a file you 
> can do it to the file personality, but if you currently can only do it 
> to a directory and get an error from attempting it to a file then in the 
> new scheme doing it to the hard link only gives the same error.

Let me sort this out: If it can't be done POSIXly to a directory, it can't
be done in Reiser4 to a file (which really is a directory too). So there
can be exactly _one_ hard link to a file. Way borken.

> Or, we can ask Alexander to help us use his deadlock detection algorithm 
> and try to do things right....

Good luck with that one. I'd suspect if it can be made to work, it will
have _huge_ overhead, so much that it is useless. I'd love to be proven
wrong, but I won't hold my breath.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Spam]

  

> Hans Reiser <reiser@namesys.com> said:
>> Horst von Brand wrote:
>> >Hans Reiser <reiser@namesys.com> said:
>> >>Horst von Brand wrote:
>> >>>Spam <spam@tnonline.net> said:
>> >>>>Christer Weinigel <christer@weinigel.se> said:

> [...]

>> >>>>>2. How do we want to expose named streams?
>> >>>>>  One suggestion is file-as-directory in some form.

>> >>>Which is broken, as it forbids hard links to files.

>> >>No, it forbids hard links to the directory aspect of the file-directory
>> >>duality.

>> >How do you distinguish a "hard link to the directory personality" from
>> >"hard link to the file personality"?

>> Put in (undoubtedly overly) simple terms, if you can do it to a file you
>> can do it to the file personality, but if you currently can only do it
>> to a directory and get an error from attempting it to a file then in the
>> new scheme doing it to the hard link only gives the same error.

> Let me sort this out: If it can't be done POSIXly to a directory, it can't
> be done in Reiser4 to a file (which really is a directory too). So there
> can be exactly _one_ hard link to a file. Way borken.

  But you can make a hard link to a file in reiser4, and you can
  access the metadata in both. I did this test:
  echo "moooo" > test
  ln test moo
  chmod +x moo test
  echo "0700" > test/metas/rwx
  dir moo test
      -rwx------  2 root root 10 Sep  7 23:17 moo*
      -rwx------  2 root root 10 Sep  7 23:17 test*
      
  echo "0777" > test/metas/rwx
  dir moo test
      -rwxrwxrwx  2 root root 10 Sep  7 23:17 moo*
      -rwxrwxrwx  2 root root 10 Sep  7 23:17 test*
      
  rm moo test
  mkdir test
  ln test moo
      ln: `test': hard link not allowed for directory

  If this is the intended behaviour I do not know, but it shows that
  hard links works as normal.

  You can also do linking to meta-data:

  echo moo > moo
  ln -s moo/metas/rwx test
  dir
      -rwxrwxrwx   1 root    root      0 Sep  7 23:24 moo*
      lrwxrwxrwx   1 root    root     13 Sep  7 23:25 test -> moo/metas/rwx
  echo 0700 > test
  dir moo
      -rwx------   1 root    root      0 Sep  7 23:24 moo*

      
>> Or, we can ask Alexander to help us use his deadlock detection algorithm
>> and try to do things right....

> Good luck with that one. I'd suspect if it can be made to work, it will
> have _huge_ overhead, so much that it is useless. I'd love to be proven
> wrong, but I won't hold my breath.

Re: silent semantic changes with reiser4

[Horst von Brand]

Spam <spam@tnonline.net> said:
> Horst von Brand <vonbrand@inf.utfsm.cl> said:
> > Hans Reiser <reiser@namesys.com> said:
> >> Horst von Brand wrote:
> >> >Hans Reiser <reiser@namesys.com> said:
> >> >>Horst von Brand wrote:
> >> >>>Spam <spam@tnonline.net> said:
> >> >>>>Christer Weinigel <christer@weinigel.se> said:

[...]

> >> >>>>>2. How do we want to expose named streams?
> >> >>>>>  One suggestion is file-as-directory in some form.

> >> >>>Which is broken, as it forbids hard links to files.

> >> >>No, it forbids hard links to the directory aspect of the file-directory
> >> >>duality.

> >> >How do you distinguish a "hard link to the directory personality" from
> >> >"hard link to the file personality"?

> >> Put in (undoubtedly overly) simple terms, if you can do it to a file you
> >> can do it to the file personality, but if you currently can only do it
> >> to a directory and get an error from attempting it to a file then in the
> >> new scheme doing it to the hard link only gives the same error.

> > Let me sort this out: If it can't be done POSIXly to a directory, it can't
> > be done in Reiser4 to a file (which really is a directory too). So there
> > can be exactly _one_ hard link to a file. Way borken.

>   But you can make a hard link to a file in reiser4,

IIUC, this isn't safe (you might tie your filesystem into a knot this way,
as LWN puts it). Sure, you can do it right now (just as I was able to
create hard links to directories in some propietary Unix way back;
untangling the resulting mess, complete with segfaulting fsck, was a whole
world of fun... it just happened rarely enough that it did not matter very
much); production versions will have to forbid it (or incurr in _huge_
overheads locking everything in sight, or detecting loops).
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Alexander G. M. Smith]

Horst von Brand wrote on Tue, 07 Sep 2004 17:02:20 -0400:
> Hans Reiser <reiser@namesys.com> said:
> > Or, we can ask Alexander to help us use his deadlock detection algorithm 
> > and try to do things right....
> 
> Good luck with that one. I'd suspect if it can be made to work, it will
> have _huge_ overhead, so much that it is useless. I'd love to be proven
> wrong, but I won't hold my breath.

Depends if you consider it to be a huge overhead to lock all the objects
that are ancestors (possibly through cyclical multiple parents) of the
fildirute (file/directory/attribute/whatever) being deleted/renamed.
It could definitely be a lot if you have some weird worst case directory
layout with lots of parent directories.  But that's rare (most users
don't have that many directories anyway).

By the way, the deadlock system I was using is just a hack - a timeout
on the locking semaphores for each file.  If it fails to lock everything
it needs, it backs off, waits a while, tries again, and eventually
reports EDEADLK if it exhausts the retry count.

Well, actually it's a bit fancier - it read-locks the file nodes as it
is traversing the graph to find all ancestors.  Then when it has finished,
it write-locks the ones it needs to change.

- Alex

Re: silent semantic changes with reiser4

[Christer Weinigel]

Horst von Brand <vonbrand@inf.utfsm.cl> writes:

> > > 1. Do we want support for named streams?
> > 
> > >    I belive the answer is yes [...]
> 
> There are many people around here who disagree (that is precisely the heart
> of the discussion). I for one don't think Linux has to get $RANDOM_FEATURE
> just because $SOME_OTHER_OS has got it. Either the feature stands on its
> own _in the context of POSIX/Unix/Linux_ (possibly as an extension or
> modification of said standards) or it isn't worth it.

Ok, noted. :-)

I myself am not very interested in generic named stream support in
Linux.  But since we have support for filesystems with named streams
and people are interested in getting at those streams (if nothing else
thanq to serve streams from an NTFS file system via Samba on a dual boot
machine), we'd better do it well, instead of with ugly hacks.

> We need to sort out exactly how far it makes sense to go, by showing
> concrete, down to earth uses for whatever substructure we want. Then show
> the effect can't be easily gotten through tools for power users or faking
> it for unsuspecting users via GUI, and that overall the complexity and
> performance cost is less than the win. Note that the success of the Unix
> way is in large part due to its use of few, simple concepts that can be
> combined endlessly; and tools following the same strategy. Adding extra
> concepts that current tools can't naturally handle has to be considered
> with extreme care.

Regarding this section and all you wrote before, I definitely agree.
Simplicity wins hands down every time.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Gunnar Ritter]

Spam <spam@tnonline.net> wrote:

>   One suggestion is missed. It is to provide system calls for copy.
>   That would also solve the problem.

No, it would not. If you read the POSIX.1 specification for cp
carefully <http://www.unix.org/version3/online.html>, you will
notice that the process for copying a regular file is carefully
standardized. A POSIX.1-conforming cp implementation would not
be allowed to copy additional streams, unless either additional
options are given or the type of the file being copied is other
than S_IFREG. And cp is just one example of a standardized file
handling program.

	Gunnar

-- 
http://omnibus.ruf.uni-freiburg.de/~gritter

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> wrote:

>>   One suggestion is missed. It is to provide system calls for copy.
>>   That would also solve the problem.

> No, it would not. If you read the POSIX.1 specification for cp
> carefully <http://www.unix.org/version3/online.html>, you will
> notice that the process for copying a regular file is carefully
> standardized. A POSIX.1-conforming cp implementation would not
> be allowed to copy additional streams, unless either additional
> options are given or the type of the file being copied is other
> than S_IFREG. And cp is just one example of a standardized file
> handling program.

  It would solve the problem in Linux. However, it may not be POSIX.1
  compatible. On the other hand I read that NTFS 5.0 is POSIX.1
  compliant - and Windows uses copy system call. NTFS also has streams
  support using special character in the file names to select the
  streams.

  Surely there must be a solution in Linux that will allow things like
  streams and meta-data(meta-streams) be visible to the user.
  ~S

> 	Gunnar

Re: silent semantic changes with reiser4

[Gunnar Ritter]

Spam <spam@tnonline.net> wrote:

> > Spam <spam@tnonline.net> wrote:
> >>   One suggestion is missed. It is to provide system calls for copy.
> >>   That would also solve the problem.
> > No, it would not. If you read the POSIX.1 specification for cp
> > carefully <http://www.unix.org/version3/online.html>, you will
> > notice that the process for copying a regular file is carefully
> > standardized. A POSIX.1-conforming cp implementation would not
> > be allowed to copy additional streams, unless either additional
> > options are given or the type of the file being copied is other
> > than S_IFREG. And cp is just one example of a standardized file
> > handling program.
>   It would solve the problem in Linux. However, it may not be POSIX.1
>   compatible. On the other hand I read that NTFS 5.0 is POSIX.1
>   compliant - and Windows uses copy system call.

You should obviously take a thorough read of at least the Base
Definitions, section 2, 'Conformance' of POSIX.1 before you further
comment on this issue.

	Gunnar

Re: silent semantic changes with reiser4

[Christer Weinigel]

Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes:

> No, it would not. If you read the POSIX.1 specification for cp
> carefully <http://www.unix.org/version3/online.html>, you will
> notice that the process for copying a regular file is carefully
> standardized. A POSIX.1-conforming cp implementation would not
> be allowed to copy additional streams, unless either additional
> options are given or the type of the file being copied is other
> than S_IFREG. And cp is just one example of a standardized file
> handling program.

We can safely ignore POSIX when it is too broken.  cp could very well
be modified to copy named streams except when the option --posix is
specified or the environment variale POSIXLY_CORRECT is set.

  /Christer


-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Gunnar Ritter]

Christer Weinigel <christer@weinigel.se> wrote:

> Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes:
> > No, it would not. If you read the POSIX.1 specification for cp
> > carefully <http://www.unix.org/version3/online.html>, you will
> > notice that the process for copying a regular file is carefully
> > standardized. A POSIX.1-conforming cp implementation would not
> > be allowed to copy additional streams, unless either additional
> > options are given or the type of the file being copied is other
> > than S_IFREG. And cp is just one example of a standardized file
> > handling program.
> We can safely ignore POSIX when it is too broken.

Excuse me, but there's really nothing broken here with POSIX and cp.
You're just making an insulting talk about a part of the specification
which currently serves GNU/Linux and other Unix-like environments very
well, and has done so for about twelve years now.

> cp could very well be modified to copy named streams except when
> the option --posix is specified

Hey, you didn't ever even have a look at POSIX Shell & Utilities, did
you? Then why are you making derogatory statements about it?

> or the environment variale POSIXLY_CORRECT is set.

Cool, data loss depending upon an environment variable which is even
currently used by many programs unaware of such results. This really
sounds like good engineering to me.

	Gunnar

Re: silent semantic changes with reiser4

[Christer Weinigel]

Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes:

> Excuse me, but there's really nothing broken here with POSIX and cp.
> You're just making an insulting talk about a part of the specification
> which currently serves GNU/Linux and other Unix-like environments very
> well, and has done so for about twelve years now.

"Broken" in the sense "POSIX mandates something that users wouldn't
expect".

> > cp could very well be modified to copy named streams except when
> > the option --posix is specified
> 
> Hey, you didn't ever even have a look at POSIX Shell & Utilities, did
> you? Then why are you making derogatory statements about it?

Those derogatory statments are really all in your mind.  

> > or the environment variale POSIXLY_CORRECT is set.
> 
> Cool, data loss depending upon an environment variable which is even
> currently used by many programs unaware of such results. This really
> sounds like good engineering to me.

How would you consider cp to cause "data loss" if it _besides_ copying
the normal stream _also_ copied any named streams or xattrs belonging
to the stream?  How would it cause data loss if cp started using a
theoretical copyfile syscall?  It may not be 100% according to POSIX,
but I'd definitely say that it does what the user expects.

Lots of GNU utilities already differ from POSIX mandated behaviour
because the authors of those utilities belive that the POSIX mandated
behaviour is confusing.

http://www.wlug.org.nz/POSIXLY_CORRECT

    POSIXLY_CORRECT is an environment variable that some programs use
    to follow strict POSIX standards behaviour, where that isn't the
    default.

    Probably the most well-known example of this is that POSIX states
    that filesystem blocks are 512 bytes per block, so the GNU
    fileutils such as df(1) and GNU tar(1) use 512 if the variable
    POSIXLY_CORRECT is set, and 1024 bytes per block by default.

    Many of the GNU tools comply with POSIX by default, except for
    where the author thinks the POSIX standard is wrong or dumb. :) As
    a result, some programs also check if a variable named
    POSIX_ME_HARDER is set as an acceptable alias for
    POSIXLY_CORRECT. See Democracy Triumphs in Disk Units.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Gunnar Ritter]

Christer Weinigel <christer@weinigel.se> wrote:

> Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes:
> > Excuse me, but there's really nothing broken here with POSIX and cp.
> > You're just making an insulting talk about a part of the specification
> > which currently serves GNU/Linux and other Unix-like environments very
> > well, and has done so for about twelve years now.
> "Broken" in the sense "POSIX mandates something that users wouldn't
> expect".

Only if one breaks it by making extensions in an inappropriate way.
This is not a fault of POSIX. POSIX usually allows a lot of sane ways
to introduce extensions. There are usually valid interoperability
arguments for behavior prescribed by POSIX. It is really not one of
those standards where you want to ignore every second word because
it is obviously nothing but committee nonsense.

> > > or the environment variale POSIXLY_CORRECT is set.
> > Cool, data loss depending upon an environment variable which is even
> > currently used by many programs unaware of such results. This really
> > sounds like good engineering to me.
> How would you consider cp to cause "data loss" if it _besides_ copying
> the normal stream _also_ copied any named streams or xattrs belonging
> to the stream?

You are reversing the argument. If additional streams are introduced
inappropriately by extending the semantics of S_IFREG files, POSIX
requires cp to lose the data. Your proposal would then make this loss
of additional stream data dependent on an environment variable that
is already in wide use. If it was set by accident, the data would be
lost.

Besides, copying xattrs is usually permitted (POSIX.1-2004, XCU cp):

# If the implementation provides additional or alternate access control
# mechanisms (see the Base Definitions volume of IEEE Std 1003.1-2001,
# Section 4.4, File Access Permissions), their effect on copies of files
# is implementation-defined.

It is also permitted to add other S_IFXXX types and then let cp act
in an implementation-defined manner on them (cf. my earlier message
<413E40D1.nailFBI11XFML@pluto.uni-freiburg.de>).

The 'standardized' data loss would only occur if the standardized type
of regular file, S_IFREG, was abused. This would really not be a fault
of POSIX.

> Lots of GNU utilities already differ from POSIX mandated behaviour
> because the authors of those utilities belive that the POSIX mandated
> behaviour is confusing.

Sure, but it is not the preferred method of adding features. In
addition, most of the existing POSIXLY_CORRECT influences are
nothing but cosmetical details in comparison to copying/not
copying stream data.

	Gunnar

Re: silent semantic changes with reiser4

[Christer Weinigel]

Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes:

> Besides, copying xattrs is usually permitted (POSIX.1-2004, XCU cp):
> 
> # If the implementation provides additional or alternate access control
> # mechanisms (see the Base Definitions volume of IEEE Std 1003.1-2001,
> # Section 4.4, File Access Permissions), their effect on copies of files
> # is implementation-defined.

In <413DD5B4.nailC801GI4E2@pluto.uni-freiburg.de> you wrote:

>A POSIX.1-conforming cp implementation would not be allowed to copy
>additional streams, unless either additional options are given or the
>type of the file being copied is other than S_IFREG.

I read this as that POSIX mandates that cp can absolutely not copy
anything else but the file contents.  That is what I called broken.

If we implement named streams as xattrs and that can be accessed with
openat(..., O_XATTR) this means that cp is allowed to copy the xattrs
(well, named streans don't neccesarily have to be "alternate access
control mechanisms", but they can use the same xattr namespace).  

That's quite ok in that case.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Gunnar Ritter]

Christer Weinigel <christer@weinigel.se> wrote:

> Gunnar Ritter <Gunnar.Ritter@pluto.uni-freiburg.de> writes:
>
> > Besides, copying xattrs is usually permitted (POSIX.1-2004, XCU cp):
> > 
> > # If the implementation provides additional or alternate access control
> > # mechanisms (see the Base Definitions volume of IEEE Std 1003.1-2001,
> > # Section 4.4, File Access Permissions), their effect on copies of files
> > # is implementation-defined.
>
> In <413DD5B4.nailC801GI4E2@pluto.uni-freiburg.de> you wrote:
>
> >A POSIX.1-conforming cp implementation would not be allowed to copy
> >additional streams, unless either additional options are given or the
> >type of the file being copied is other than S_IFREG.
>
> I read this as that POSIX mandates that cp can absolutely not copy
> anything else but the file contents.  That is what I called broken.

It would be really helpful if you read the specification before you
comment on it or try to interpret my wording further.

> If we implement named streams as xattrs and that can be accessed with
> openat(..., O_XATTR) this means that cp is allowed to copy the xattrs

No. Not if the file type of such beasts remains S_IFREG.

> (well, named streans don't neccesarily have to be "alternate access
> control mechanisms", but they can use the same xattr namespace).  

POSIX does not know anything about the 'xattr namespace'. It just
allows 'additional or alternate access control mechanisms'. Which
are, in turn, well-defined terms in the standard again. You can
read about them too <http://www.unix.org/version3/>.

	Gunnar

Re: silent semantic changes with reiser4

[Hans Reiser]

Christer Weinigel wrote:

>David Masover <ninja@slaphack.com> writes:
>
>  
>
>>|>Second, there are quite a few things which I might want to do, which can
>>|>be done with this interface and without patching programs,
>>| Such as?
>>They've been mentioned.
>>    
>>
>
>  
>
>>| Haven't seen any that made sense to me, sorry.
>>Sorry if they don't make sense to you, but I don't feel like discussing
>>them now.  Either you get it or you don't, either you agree or you
>>don't.  Read the archives.
>>    
>>
>
>Great argument.  Not.  There has been so much shit thrown around here
>so that it's impossible to keep track of all examples.
>
>Could you please try summarize a few of the arguments that you find
>especially compelling?  This thread has gotten very confused since
>there are a bunch of different subjects all being intermixed here.
>
>What are we discussing?
>
>1. Do we want support for named streams?
>
>   I belive the answer is yes, since both NTFS and HFS (that's the
>   MacOS filesystem, isn't it?) supports streams we want Linux to
>   support this if possible.
>
>   Anyone disagreeing?
>  
>
No, we want files and directories that can do what streams can do.  This 
means files that are also directories, plugins that aggregate the 
contents of a directory, files that inherit stat data, maybe I forget 
something ---- it is on my website.

Streams themselves are a bad idea because they are a filesystem inside 
of a file which is double the overhead, and they fragment the namespace.

>4. What belongs in the generic VFS, what belongs in Reiser4?
>
>   Some things reiser4 do, such as files-as-directories need changes
>   to the VFS because it breaks assumptions that the VFS makes
>   (i.e. a deadlock or an oops when doing a hard link out of one).
>
>   Some other things reiser4 can do would be better if they were in
>   the VFS since other filesystems might want to support the same
>   functionality. 
>  
>
It is always better to lead by example.  These ideas are too new for the 
other fs developers, they need 5 years to get used to them.  Reiser4 
should create something that works, and let others follow when they will.

>   Or Linux may not support some of the things reiserfs at all.
>
>5. What belongs in the kernel, what belongs in userspace?
>  
>
This is the wrong question.  The right question is, what belongs in a 
unified namespace?  Then having answered that, the extent to which it is 
easier to have all aspects of name resolution together in one body of 
code is an implementation detail that can change and evolve with time.  
A rapidly evolving namespace is easier to modify if it is all one body 
of code.  Furthermore, the people doing the work should really be left 
to decide such implementation details themselves because they are more 
expert on their code than anyone else. 

Hans

Re: silent semantic changes with reiser4

[Christer Weinigel]

Hans Reiser <reiser@namesys.com> writes:

> No, we want files and directories that can do what streams can do.
> This means files that are also directories, plugins that aggregate the
> contents of a directory, files that inherit stat data, maybe I forget
> something ---- it is on my website.
> 
> Streams themselves are a bad idea because they are a filesystem inside
> of a file which is double the overhead, and they fragment the
> namespace.

There really isn't a difference between your files are directories
think and named streams that can be accessed with openat.  Both are
concepts that won't fit into the classical Unix entity "file".

As far as I can tell, windows named streams could be exposed via a
file-as-directory model.  In that case what windows sees as
foo.txt:icon would be accessed as foo.txt/icon instead.  The question
in my mind is if that is a desirable concept.  I can see that it is
very a attractive concept because it's easy to explain to users, but
at the same time, a lot of things suggest that the files-as-
directories concept is a bad idea.  I mentioned a few of the possible
problems in the grand-grand-parent of this message.

Try to address those problems.  It may be that your answer is "I don't
give a shit about existing backup applications", or "I don't care
about security in existing web-servers, they'll have to be taught
about reiser4", but in that case say so.  Or show how it's not a
problem with existing applictions.

> It is always better to lead by example.  These ideas are too new for
> the other fs developers, they need 5 years to get used to them.
> Reiser4 should create something that works, and let others follow when
> they will.

Pretty please, give up the marketing bullshit.  I already know that
you belive reiser4 is the best thing since sliced bread.  You say that
"reiser4 should create something that works".  So far you haven't
convinced me that reiser4 actually "works".  It's a nice proof of
concept, but it definitely does not feel like something that I'd like
in the mainstream kernel until those problems are addressed.

Have you looked at the VFS deadlocks that Alexander Viro pointed at?

Does tar work with files-as-directories?  Can I use tar to make a
backup of my resier4 directory.  What changes would be required to the
tar application or to the tar file format to make this work?

Can I do a cp -a /home/wingel to a NFS-mounted drive on some other
computer to make a backup of my home directory or will I lose data
that way?  What changes would be required to cp to make this work?

Please remember that reiser4 does not exist in a vaccum.  If you want
to make a fundamental change to unix concepts that have been with us
for three decades, it's really up to you to show that it doesn't break
too many existing applications.

> This is the wrong question.  The right question is, what belongs in a
> unified namespace?  Then having answered that, the extent to which it
> is easier to have all aspects of name resolution together in one body
> of code is an implementation detail that can change and evolve with
> time.  A rapidly evolving namespace is easier to modify if it is all
> one body of code.  Furthermore, the people doing the work should
> really be left to decide such implementation details themselves
> because they are more expert on their code than anyone else. 

I still haven't seen a good answer to the namespace problem.  You say
that files-as-directories is great, I'm definitely not convinced.  

The VFS is the body of code that handles most aspects of name
resolution today, so it sounds as if reiser4 is supposed to be the new
super-VFS, and you want to keep it that way and do not want to
integrate with the current VFS.  In that case resier4 really does not
belong in the mainstream kernel since it's your pet research project.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[David Lang]

so far the best answer that I've seen is a slight varient of what Hans is 
proposing for the 'file-as-a-directory'

make the base file itself be a serialized version of all the streams and 
if you want the 'main' stream open file/. (or some similar varient)

this doesn't address the hard-link issue, but it should handle the backup 
problems (your backup software just goes through the files and what it 
gets is suitable for backups).

you will ask what serializer to user, and my answer is to let one of the 
streams tell you, and have the kernel make a call out to userspace to 
execute the appropriate program (note that this means that tar is not put 
into the kernel)

in fact it may make sense to just open file/file to get at the 'main' 
stream of the file (there may be cases where the concept of a single main 
stream may not make sense)

so if this solves the tool/backup problem then we can look and figure out 
if there's a reasonable way to solve the hard-link problem

David Lang

On Tue, 8 Sep 2004, Christer Weinigel wrote:

> Date: 08 Sep 2004 00:13:12 +0200
> From: Christer Weinigel <christer@weinigel.se>
> To: Hans Reiser <reiser@namesys.com>
> Cc: Christer Weinigel <christer@weinigel.se>,
>     David Masover <ninja@slaphack.com>,
>     Horst von Brand <vonbrand@inf.utfsm.cl>, Spam <spam@tnonline.net>,
>     Tonnerre <tonnerre@thundrix.ch>, Linus Torvalds <torvalds@osdl.org>,
>     Pavel Machek <pavel@ucw.cz>, Jamie Lokier <jamie@shareable.org>,
>     Chris Wedgwood <cw@f00f.org>, viro@parcelfarce.linux.theplanet.co.uk,
>     Christoph Hellwig <hch@lst.de>, linux-fsdevel@vger.kernel.org,
>     linux-kernel@vger.kernel.org, Alexander Lyamin aka FLX <flx@namesys.com>,
>     ReiserFS List <reiserfs-list@namesys.com>
> Subject: Re: silent semantic changes with reiser4
> 
> Hans Reiser <reiser@namesys.com> writes:
>
>> No, we want files and directories that can do what streams can do.
>> This means files that are also directories, plugins that aggregate the
>> contents of a directory, files that inherit stat data, maybe I forget
>> something ---- it is on my website.
>>
>> Streams themselves are a bad idea because they are a filesystem inside
>> of a file which is double the overhead, and they fragment the
>> namespace.
>
> There really isn't a difference between your files are directories
> think and named streams that can be accessed with openat.  Both are
> concepts that won't fit into the classical Unix entity "file".
>
> As far as I can tell, windows named streams could be exposed via a
> file-as-directory model.  In that case what windows sees as
> foo.txt:icon would be accessed as foo.txt/icon instead.  The question
> in my mind is if that is a desirable concept.  I can see that it is
> very a attractive concept because it's easy to explain to users, but
> at the same time, a lot of things suggest that the files-as-
> directories concept is a bad idea.  I mentioned a few of the possible
> problems in the grand-grand-parent of this message.
>
> Try to address those problems.  It may be that your answer is "I don't
> give a shit about existing backup applications", or "I don't care
> about security in existing web-servers, they'll have to be taught
> about reiser4", but in that case say so.  Or show how it's not a
> problem with existing applictions.
>
>> It is always better to lead by example.  These ideas are too new for
>> the other fs developers, they need 5 years to get used to them.
>> Reiser4 should create something that works, and let others follow when
>> they will.
>
> Pretty please, give up the marketing bullshit.  I already know that
> you belive reiser4 is the best thing since sliced bread.  You say that
> "reiser4 should create something that works".  So far you haven't
> convinced me that reiser4 actually "works".  It's a nice proof of
> concept, but it definitely does not feel like something that I'd like
> in the mainstream kernel until those problems are addressed.
>
> Have you looked at the VFS deadlocks that Alexander Viro pointed at?
>
> Does tar work with files-as-directories?  Can I use tar to make a
> backup of my resier4 directory.  What changes would be required to the
> tar application or to the tar file format to make this work?
>
> Can I do a cp -a /home/wingel to a NFS-mounted drive on some other
> computer to make a backup of my home directory or will I lose data
> that way?  What changes would be required to cp to make this work?
>
> Please remember that reiser4 does not exist in a vaccum.  If you want
> to make a fundamental change to unix concepts that have been with us
> for three decades, it's really up to you to show that it doesn't break
> too many existing applications.
>
>> This is the wrong question.  The right question is, what belongs in a
>> unified namespace?  Then having answered that, the extent to which it
>> is easier to have all aspects of name resolution together in one body
>> of code is an implementation detail that can change and evolve with
>> time.  A rapidly evolving namespace is easier to modify if it is all
>> one body of code.  Furthermore, the people doing the work should
>> really be left to decide such implementation details themselves
>> because they are more expert on their code than anyone else.
>
> I still haven't seen a good answer to the namespace problem.  You say
> that files-as-directories is great, I'm definitely not convinced.
>
> The VFS is the body of code that handles most aspects of name
> resolution today, so it sounds as if reiser4 is supposed to be the new
> super-VFS, and you want to keep it that way and do not want to
> integrate with the current VFS.  In that case resier4 really does not
> belong in the mainstream kernel since it's your pet research project.
>
>  /Christer
>
> --
> "Just how much can I get away with and still go to heaven?"
>
> Freelance consultant specializing in device driver programming for Linux
> Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christer Weinigel]

David Lang <david.lang@digitalinsight.com> writes:

> so far the best answer that I've seen is a slight varient of what Hans
> is proposing for the 'file-as-a-directory'
> 
> make the base file itself be a serialized version of all the streams
> and if you want the 'main' stream open file/. (or some similar
> varient)

> in fact it may make sense to just open file/file to get at the 'main'
> stream of the file (there may be cases where the concept of a single
> main stream may not make sense)

So what happens if I have a text file foo.txt and add an author
attribute to it?  When I read foo.txt the next time it's supposed to
give me a serialized version with both the contents of foo.txt _and_
the author attribute?

That would definitely confuse me.

Or did I misunderstand something?

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[David Lang]

On Tue, 8 Sep 2004, Christer Weinigel wrote:

> Subject: Re: silent semantic changes with reiser4
> 
> David Lang <david.lang@digitalinsight.com> writes:
>
>> so far the best answer that I've seen is a slight varient of what Hans
>> is proposing for the 'file-as-a-directory'
>>
>> make the base file itself be a serialized version of all the streams
>> and if you want the 'main' stream open file/. (or some similar
>> varient)
>
>> in fact it may make sense to just open file/file to get at the 'main'
>> stream of the file (there may be cases where the concept of a single
>> main stream may not make sense)
>
> So what happens if I have a text file foo.txt and add an author
> attribute to it?  When I read foo.txt the next time it's supposed to
> give me a serialized version with both the contents of foo.txt _and_
> the author attribute?
>
> That would definitely confuse me.
>
> Or did I misunderstand something?
>

good point. under my scheme you would need to access foo.txt/foo.txt or 
foo.txt/. instead of just foo.txt

I guess my way would work if there is a way to know that a file has been 
extended (or if you just make it a habit of opening the file/file instead 
of just file) but not for random additions of streams to otherwise normal 
files.

Oh well, it seemed like a easy fix (and turned out to be to easy to be 
practical)

David Lang

>  /Christer
>
> --
> "Just how much can I get away with and still go to heaven?"
>
> Freelance consultant specializing in device driver programming for Linux
> Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se
>

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Horst von Brand]

David Lang <david.lang@digitalinsight.com> said:
> On Tue, 8 Sep 2004, Christer Weinigel wrote:
> > David Lang <david.lang@digitalinsight.com> writes:

[...]

> > So what happens if I have a text file foo.txt and add an author
> > attribute to it?  When I read foo.txt the next time it's supposed to
> > give me a serialized version with both the contents of foo.txt _and_
> > the author attribute?
> >
> > That would definitely confuse me.
> >
> > Or did I misunderstand something?

> good point. under my scheme you would need to access foo.txt/foo.txt or 
> foo.txt/. instead of just foo.txt

> I guess my way would work if there is a way to know that a file has been 
> extended

Out of the question if you want to use legacy tools.

>          (or if you just make it a habit of opening the file/file instead 
> of just file) but not for random additions of streams to otherwise normal 
> files.

And then a file called foo inside directory foo will have to be refered to
as foo/foo/foo...

> Oh well, it seemed like a easy fix (and turned out to be to easy to be 
> practical)

My gut feeling is that this whole concept is too messed up to be sorted
out. That something can easily be done given whatever underlying structure
there is doesn't make it worthwhile to do it. Just as it is absurdly easy
to allow hard links to directories if you look at how they are implemented,
but the consecuences aren't that easy to live with.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Gunnar Ritter]

David Lang <david.lang@digitalinsight.com> wrote:

> so far the best answer that I've seen is a slight varient of what Hans is 
> proposing for the 'file-as-a-directory'
>
> make the base file itself be a serialized version of all the streams and 
> if you want the 'main' stream open file/. (or some similar varient)

As has been said previously, all such proposals except for those
with two leading slashes would directly violate POSIX.1-2004, Base
Definitions, 4.11 'Pathname Resolution'. In particular,

# A pathname that contains at least one non-slash character and that
# ends with one or more trailing slashes shall be resolved as if a
# single dot character ( '.' ) were appended to the pathname.

A regular file name with special semantics, in contrast, would not
violate POSIX, in particular if a stat() on it would not return one
of the standard S_IFXXX types. Inappropriate operations could just
fail with EINVAL on such files, and it would be unspecified by the
standard how they were handled by 'cp' or other standardized utilities.

Having a separate S_IFXXX type for streamed files would also give
attention to portability issues easily. It might in fact be better
to have existing applications fail explicitly with them than to
make this as transparent as possible, but hope for good luck.

Utilities like 'tar' would just say 'Unknown file type' or the like
and exclude such files from the archives they create. This would lead
the user's explicit attention to the data loss. He might then choose
to ignore the error, or to get a special version of 'tar' to handle it.

It would also be clear to programmers and users that such files are
special things they don't normally need. Regular Linux installations
might totally ignore them. If streamed files are only intended for
use with CIFS or other special cases, that might be a more clean
solution for this issue than watering down the existing portable
file semantics.

	Gunnar

Re: silent semantic changes with reiser4

[Julian Blake Kongslie]

[-- Attachment #1: Type: text/plain, Size: 1597 bytes --]

On Wed, 08 Sep 2004 01:14:25 +0200, Gunnar Ritter
<Gunnar.Ritter@pluto.uni-freiburg.de> wrote:
> Having a separate S_IFXXX type for streamed files would also give
> attention to portability issues easily. It might in fact be better
> to have existing applications fail explicitly with them than to
> make this as transparent as possible, but hope for good luck.
> 
> Utilities like 'tar' would just say 'Unknown file type' or the like
> and exclude such files from the archives they create. This would lead
> the user's explicit attention to the data loss. He might then choose
> to ignore the error, or to get a special version of 'tar' to handle it.
> 
> It would also be clear to programmers and users that such files are
> special things they don't normally need. Regular Linux installations
> might totally ignore them. If streamed files are only intended for
> use with CIFS or other special cases, that might be a more clean
> solution for this issue than watering down the existing portable
> file semantics.

You could actually use two S_IFXXX types - one for stream files which
should be archived by backup programs because they contain real
information, and a different one for stream files which should not be
archived because they are merely a different "view" of data from the main
file. This simplifies the archival problem because legacy programs would
complain that the file types aren't handled - alerting the user to
potential data loss, and updated programs would automatically backup only
the attributes that are required.

-- 
-Julian Blake Kongslie
<jblake@omgwallhack.org>

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Horst von Brand]

David Lang <david.lang@digitalinsight.com> said:
> so far the best answer that I've seen is a slight varient of what Hans is 
> proposing for the 'file-as-a-directory'

> make the base file itself be a serialized version of all the streams and 
> if you want the 'main' stream open file/. (or some similar varient)

Serialized how? cpio(1), tar(1), cat(1)ed together, ...? Ship it via FTP or
something, how do you unpack on a traditional filesystem? On a wacky
filesystem?

Note that this has the same performance implications as the targzfilesystem
concept, just the other way around: Instead of unpacking on the fly it is
packing on the fly. And paying a (possibly huge) cost for "simple, everyday
operations" in some cases that can't be distinguished from ones where there
is no such cost is very wrong to me.

> this doesn't address the hard-link issue,

It has to be solved!

>                                           but it should handle the backup 
> problems (your backup software just goes through the files and what it 
> gets is suitable for backups).

But how do you recreate the original "files" later? You only get the
serialized version back. Even worse, if you unpack on top of the original
wacky file, you'd get a wacky file, where the main stream is the serialized
version of the original... Rinse and repeat. [Shudder]

> you will ask what serializer to user, and my answer is to let one of the 
> streams tell you, and have the kernel make a call out to userspace to 
> execute the appropriate program (note that this means that tar is not put 
> into the kernel)

If I want to see it serialized via tar(1), and you via cpio(1), and
somebody else via "take the icon stream, discard everything else", how do
you handle that?

> in fact it may make sense to just open file/file to get at the 'main' 
> stream of the file (there may be cases where the concept of a single main 
> stream may not make sense)

What do you do then?

> so if this solves the tool/backup problem

It makes it much worse, AFAICS.

>                                            then we can look and figure out 
> if there's a reasonable way to solve the hard-link problem

Right. And if none is found, can we drop this madness?
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[David Lang]

On Tue, 7 Sep 2004, Horst von Brand wrote:

> David Lang <david.lang@digitalinsight.com> said:
>> so far the best answer that I've seen is a slight varient of what Hans is
>> proposing for the 'file-as-a-directory'
>
>> make the base file itself be a serialized version of all the streams and
>> if you want the 'main' stream open file/. (or some similar varient)
>
> Serialized how? cpio(1), tar(1), cat(1)ed together, ...? Ship it via FTP or
> something, how do you unpack on a traditional filesystem? On a wacky
> filesystem?
>

as I said elsewhere in the origional post the owner of the file sets the 
tool to use to serialize it. if you want tar set it to use tar, if you 
want cpio set that. in both cases it would be a call out to userspace

> Note that this has the same performance implications as the targzfilesystem
> concept, just the other way around: Instead of unpacking on the fly it is
> packing on the fly. And paying a (possibly huge) cost for "simple, everyday
> operations" in some cases that can't be distinguished from ones where there
> is no such cost is very wrong to me.

this is a (huge) oppurtunity for the filesystem to provide for an 
optimization. if it is able to maintain a serialized version then this 
could turn into a simple read. no it's not always the simple obvious 
thing, but very few things are.

the problem of things having unexpected side effects in an uncontrolled 
server already are out there.

>> this doesn't address the hard-link issue,
>
> It has to be solved!

I never disputed that, in fact I pointed out that that issue remained, 
with the idea (since disproved) that this suggestion would settle the 
other issues sufficiantly to let everyone concentrate on the link issue.

>
>>                                           but it should handle the backup
>> problems (your backup software just goes through the files and what it
>> gets is suitable for backups).
>
> But how do you recreate the original "files" later? You only get the
> serialized version back. Even worse, if you unpack on top of the original
> wacky file, you'd get a wacky file, where the main stream is the serialized
> version of the original... Rinse and repeat. [Shudder]

how smart if the filesystem plugin? can the filesystem recognise compound 
filetypes? if so then it can tell what's happening and do the right thing 
when recreating the file

>> you will ask what serializer to user, and my answer is to let one of the
>> streams tell you, and have the kernel make a call out to userspace to
>> execute the appropriate program (note that this means that tar is not put
>> into the kernel)
>
> If I want to see it serialized via tar(1), and you via cpio(1), and
> somebody else via "take the icon stream, discard everything else", how do
> you handle that?

the owner of the file decides (or whoever that owner grants the 
permissions to to change it to a different system)

>> in fact it may make sense to just open file/file to get at the 'main'
>> stream of the file (there may be cases where the concept of a single main
>> stream may not make sense)
>
> What do you do then?

the backup doesn't really care. if you refrence files by file/file then 
you can't really say that file/foo is any more the 'main' stream then 
file/bar. but if you call it file/. then you do have to define what 
themain version is.

>> so if this solves the tool/backup problem
>
> It makes it much worse, AFAICS.



>>                                            then we can look and figure out
>> if there's a reasonable way to solve the hard-link problem
>
> Right. And if none is found, can we drop this madness?

solutions are known , it's just not known if any solution can be 
implemented with suitable performance (lock the entire filesystem when you 
go to do a dangerous action goes a _long_ way towards solving the 
problems, but it's not acceptable due to performance issues)

the fundamental problem with hard links and directories (maintaining a 
sane path) is the exact same problem that the *notify tools need solved so 
that a program can setup notification on a directory tree that it's 
interested in without tieing up a large number of file handles. it's a odd 
problem, but if it can be solved there are several doors that open up.

David Lang



-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 593 bytes --]

Salut,

On Tue, Sep 07, 2004 at 12:50:16AM -0500, David Masover wrote:
> Transparently?
> 
> It _works_ to do
> 	zcat file.gz > /tmp/file
> 	vim /tmp/file
> 	gzip -c /tmp/file > file.gz
> 	rm /tmp/file
> You can even do that as a script, call it zvim.  You could even do it as
> a generic script, where "vim" is replaced with "$1".  But is it as
> elegent as transparently compressed files?

...or   you    get   yourself    a   sane   editor    which   supports
gzopen/gzread/gzwrite/gzclose. You can do  that in userland, no kernel
implementation needed.

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Hans Reiser]

David Masover wrote:

>
> There are some things which can't be solved without patching.  Version
> control is one such thing.

You can do most of the version control stuff without patching, for 
instance selecting a version by name is easy (see Clearcase).  Clearcase 
users mostly do not patch user space binaries.  But your point is 
generally true that there are features that knowing about them allows 
you to better employ them.

> But then there can be more generic patches
> -- as soon as the transaction API is done, you only have to patch apps
> to use that, and have a version control reiser4 plugin.
>
> | I'd go the other way around: Get userspace to agree on a common 
> framework,
> | make it work in userspace; if (extensive, hopefully) experience 
> shows that
> | a pure userspace solution has issues that can't be solved except by 
> kernel
> | assistance, so be it.
>
> We already have such a framework -- it's called "VFS".

If doing it in the kernel is so hard, why hasn't it stopped us yet? ;-)

I am not asking other people to contibute their labor to making this 
thing they view as infeasible work, I am just asking them to get out of 
the way please, and let the users decide whether they like it.

Nothing significant about the reiserfs project was thought likely to 
work by sensible people before it worked.  I am a bit used to that.  
After all, Oracle's IFS and several similar projects proved filesystems 
on top of balanced trees cannot  perform well....;-)

Anyone who complains about kernel bloat should first consider that 
reiser4 is not by any means the largest filesystem in the kernel.

Hans

Re: silent semantic changes with reiser4

[Alan Cox]

On Maw, 2004-08-31 at 21:05, Linus Torvalds wrote:
> In a graphical environment, the "icon" stream is a good example of this.  
> It literally has _nothing_ to do with the data in the main stream. The
> only linkage is a totally non-technical one, where the user wanted to
> associate a secondary stream with the main stream _without_ altering the
> main one. THAT is where named streams make sense.

The icon doesn't belong in the document, that was a catastrophic
disaster in early MacOS (although they only use an index). Users want to
manage their icon choices, tags, tooltip notes and attached labels, and
you cannot do that in the file if you don't own the file. 

Also the icon is *not* unrelated to the file in a modern GUI, eg rox and
nautilus uses scaled versions of the content for many media types and
will show you pictures, frames from a movie etc to help you remember the
content. On top they then add user specific annotations.

The things that are more independant are:

"This file was created by OpenOffice 1.2"
"This is a text/plain file in UTF-8"
"This file has a UUID of ...."

The type has dragons because you get heirarchical typing within
documents (consider XML containing namespaces)

(UUIDs being one really useful thing we don't tag everywhere onto files
that would be a godsend on the desktop providing they moved with the
file on rename)

Alan

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Tue, Aug 31, 2004 at 01:05:40PM -0700, Linus Torvalds wrote:
>
>There's no point to having the kernel export information that is already 
>inherent in the main stream.

As said before, it can be bounced :)

You have support for streams, someone could read the inherent information
and write it to the stream. At the risk of these two streams being
unsynced temporarily, requiring some syncer.

>I've seen all these examples of exposing MP3 ID information as a "side 
>stream", and that's TOTALLY POINTLESS! The information is already there, 

No it's not pointless :)

>it's in a standard format, and exporting it as a stream buys you 
>absolutely nothing.

There is no nice standard way of finding MP3s by the ID information.

find music/ -name *mp3/..metas/id3/ | xargs grep -i "dream theater"

or something. Which reveals the metadata to find. 
The kernel doesn't have to parse the ID3 tags, that can be handled in 
userspace by a daemon, updated based on mtimes or somesuch, yes?

>Which means that normally we really don't _want_ named streams. In 99% of
>all cases we can use equally good - and _much_ simpler - tool-based
>solutions.

Sure, when someone goes through every application from ls to nautilus
and rams a tool-based patch down the app maintainer's throat and then
we're still dependent on the filesystem layout, which is easier to break
than the filesystem itself :)

>Which means that the only _real_ technical issue for supporting named
>streams really ends up being things like samba, which want named streams

So they should be implemented for samba and other guys can come up with
alternative means for using them ;)

-- 
mjt

Re: silent semantic changes with reiser4

[Neil Brown]

On Wednesday September 1, mjt@nysv.org wrote:
> On Tue, Aug 31, 2004 at 01:05:40PM -0700, Linus Torvalds wrote:
> >
> >There's no point to having the kernel export information that is already 
> >inherent in the main stream.

Having read this quote a few times in this thread I finally figured
out what was wrong with it.

It seems to imply that "iso9660" shouldn't be in the kernel.  After
all, it just exports information that is already in the underlying
device. 
It doesn't provide any "mediate multiple access" benefit as a
read-only filesystem doesn't require any mediation between users.
It might provide some caching benefit, but I somehow don't think that
is the reason that it is in the kernel.

Now I'm happy to agree that there is a case for "iso9660" but not for
"tarfs", but I don't think it is that "there's no point to having the
kernel export information that is already [there]"

NeilBrown

Re: silent semantic changes with reiser4

[Chris Wedgwood]

On Tue, Sep 07, 2004 at 04:43:11PM +1000, Neil Brown wrote:

> It seems to imply that "iso9660" shouldn't be in the kernel.  After
> all, it just exports information that is already in the underlying
> device.

Some applications do isofs/udf in userspace, DVD playback for example.

> It doesn't provide any "mediate multiple access" benefit as a
> read-only filesystem doesn't require any mediation between users.

Umm... you can have permissions for users and you need the kernel to
determine who can access what.



  --cw

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
| Pavel Machek <pavel@ucw.cz> said:
|
|>David Masover <ninja@slaphack.com> said:
|
|
| [...]
[...]
| You do need extra tools anyway, placing them in the kernel is cheating
(and
| absolutely pointless, IMHO).

Repeat after me:  plugins in kernel does NOT equal tools in kernel.

It's not hard to, for instance, imagine a generic plugin for archive
manipulation which talks to a userspace daemon/library.  The kernel
doesn't know anything other than (maybe) a list of extensions that are
archives.  All else is handled in userspace -- the idea that "this is a
zipfile" and "zipfiles can be extracted with the 'zip' command" are all
in userspace.

It's not about the kernel, it's about the interface.  And see my other mail:
	cat foo.zip/README
	less foo.zip/contents/bar.c
is a lot easier than
	lynx http://google.com/search?q=zip
	emerge zip
	man zip
	unzip foo.zip
	cat bar.c
which already assumes quite a lot of expertise.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUieXgHNmZLgCUhAQLj8Q//VEuRWg9RB/PVg/M1t36SraKIRdbrgGq6
0DYLZwwVWkfz+UKGhUkACNGQzhHG3ySn/lRjqXCsJhzMOzfgmte2CwgGxunHYqdp
W69OSZ1vv0ZLpdpzEkVdMfRhhg8RBL54Ewj6fQu7G1ZzJ0J0HUFCEZIsVoL4zJfk
+GFl4M+vvc+152HipEDsvrrPasc0dSxmryyzPbyABrfPRi3l2E3Okfewn/4MCC1E
H/L04nmvJQ2OSZYu1eSMSpnLrgQdZJbYefRuSinpqfKhjpwQXjj2rnfcPm+xc3hh
FDMlAUPA/DAlthTuhxd9+8iDPEARY57/FAMFTalYVhBEv0NWieMMn9V6R8BbnhZW
8y9EWu0Nvoowg9qElkVjWEt5M1Rbeuq4TDviyVyPYg/KcTKDGpjLuyQQbWv6hMVM
WAx6THtRK8Ji/XLCj4Kh6qDOEYP1nuwqLYSWv13BtNAJ3bVeKryOJ4Er4AmuJomA
uydBPQVsrtzTNmk7Eul2ZjrxXdESpUnUIAbu7H1Gbn51uP4rQTdlXkdPTv6s/wvG
OSOMWKs28GBFsG3+IJLnoN2wxrffpgTBEAaPwbh9TRFKLIH+pkCFSx2cIkTj93nr
RjETFObjYdJGhGygrq0xOmN5e7iprdxhxd3/+fvjQxnZGxXT3fZLUwhVX9/zhaXG
cxau2Onr3jc=
=MrR9
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> | You do need extra tools anyway, placing them in the kernel is cheating
> (and
> | absolutely pointless, IMHO).
> 
> Repeat after me:  plugins in kernel does NOT equal tools in kernel.
> 
> It's not hard to, for instance, imagine a generic plugin for archive
> manipulation which talks to a userspace daemon/library.  The kernel
> doesn't know anything other than (maybe) a list of extensions that are
> archives.  All else is handled in userspace -- the idea that "this is a
> zipfile" and "zipfiles can be extracted with the 'zip' command" are all
> in userspace.
> 
> It's not about the kernel, it's about the interface.  And see my other mail:
> 	cat foo.zip/README
> 	less foo.zip/contents/bar.c
> is a lot easier than
> 	lynx http://google.com/search?q=zip
> 	emerge zip
> 	man zip
> 	unzip foo.zip
> 	cat bar.c
> which already assumes quite a lot of expertise.

If you only want nice user interface, you can have that today. Its
done using coda, and hosted at uservfs.sf.net.
								Pavel
-- 
When do you have heart between your knees?

Re: silent semantic changes with reiser4

[Jamie Lokier]

Pavel Machek wrote:
> > It's not about the kernel, it's about the interface.  And see my other mail:
> > 	cat foo.zip/README
> > 	less foo.zip/contents/bar.c
> > is a lot easier than
> > 	lynx http://google.com/search?q=zip
> > 	emerge zip
> > 	man zip
> > 	unzip foo.zip
> > 	cat bar.c
> > which already assumes quite a lot of expertise.
> 
> If you only want nice user interface, you can have that today. Its
> done using coda, and hosted at uservfs.sf.net.

Can I do these already using uservfs?

   cd
   less foo.zip/contents/bar.c
   less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c
   grep -R obscure_label ~/RPMS

No I can't.

Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can
only do the above with convenient paths, completion etc. if I mount my
home directory and distfiles over uservfs, so that all I/O in those
directories including ordinary accesses is through uservfs, and have
the _real_ storage elsewhere.  That's quite illogical, cumbersome to
set up, and slows down normal I/O significantly.

I can do some of those things in a similar way if I do them inside
Midnight Commander, Gnome, KDE, or Emacs -- but even then a path that
works in one of those doesn't work in another, and the extensions
don't even work in all programs, plugins or scripts for those systems.

As we're adding file-as-directories to VFS, I propose it would be good
to have *hooks* in place, and non-metadata namespace reserved inside
the directories, so that uservfs-like userspace filesystems can
be auto-mounted at those places and synchronised with the file.

In other words, something that makes the next version of uservfs a
whole lot more useful.

The good news is that a lot of the hooks are falling into place already.

Kernel support is needed to handle the interface, cacheing to trigger
deletions, page cacheing so that I/O is a normal speed and mmap()
works, moveable mounts so that renames work and the cached data
followed them, and coherency so that if the underlying archive is
changed, it invalidates cached data with the appropriate degree of
synchronisation.  Note that the current dnotify cannot provide the
latter with any strength; it's too weak an interface (so is inotify).
(You can omit a lot of these things, but then you get a flaky interface).

The FUSE project looks like it will provide an efficient interface to
the page cache, avoiding round trips where possible and supporting
mmap.  If it doesn't, it should.  (Similar VFS facilities are useful
for distributed filesystems).  The presently discussed file-as-directory
semantics will provide a nice view, and force the necessary minimal
tool changes, so _all_ tools will be fine with the new uservfs.

So all I am asking for is a facility to auto-mount with
file-as-directory, and the ability for a userspace daemon to be
notified of regular file modifications synchronously.  Both can be
added later, once file-as-directory and moveable mounts are
established.  (fcntl leases and dnotify _almost_ provide this, but
they don't.  Looks like incoherent hacks keep getting added all over
the place for Samba... :).

That's all.  Kernel space needs to provide very little that it doesn't
already provide, for this rather nice and universally usable feature to work.

-- Jamie

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > > It's not about the kernel, it's about the interface.  And see my other mail:
> > > 	cat foo.zip/README
> > > 	less foo.zip/contents/bar.c
> > > is a lot easier than
> > > 	lynx http://google.com/search?q=zip
> > > 	emerge zip
> > > 	man zip
> > > 	unzip foo.zip
> > > 	cat bar.c
> > > which already assumes quite a lot of expertise.
> > 
> > If you only want nice user interface, you can have that today. Its
> > done using coda, and hosted at uservfs.sf.net.
> 
> Can I do these already using uservfs?
> 
>    cd
>    less foo.zip/contents/bar.c

less foo.zip#uzip/contents/bar.c

>    less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c

less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c

>    grep -R obscure_label ~/RPMS

I do not think you'd want this. How would you tell grep obscure label
without entering archives, then? 

> No I can't.

> Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can
> only do the above with convenient paths, completion etc. if I mount
> my

Well, you can... or at least I had version somewhere that
automagically prepended /overlay when file was not found. That way,
normal requests were fast and uservfs were used whenever it was
needed... At the price of little uglyness and slowing down -ENOENTs.

> As we're adding file-as-directories to VFS, I propose it would be good
> to have *hooks* in place, and non-metadata namespace reserved inside
> the directories, so that uservfs-like userspace filesystems can
> be auto-mounted at those places and synchronised with the file.
> 
> In other words, something that makes the next version of uservfs a
> whole lot more useful.

Yep, something less hacky would be nice.

> So all I am asking for is a facility to auto-mount with
> file-as-directory, and the ability for a userspace daemon to be
> notified of regular file modifications synchronously.  Both can be
> added later, once file-as-directory and moveable mounts are

If the userspace daemon is synchronously notified of file
modification... will not that lead to very ugly deadlocks?
								Pavel

-- 
Horseback riding is like software...
...vgf orggre jura vgf serr.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Pavel Machek wrote:
> > Can I do these already using uservfs?
> > 
> >    cd
> >    less foo.zip/contents/bar.c
> 
> less foo.zip#uzip/contents/bar.c
> 
> >    less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c
> 
> less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c
>
> >    grep -R obscure_label ~/RPMS
> 
> I do not think you'd want this. How would you tell grep obscure label
> without entering archives, then? 

Granted, this is where we need "grep --recurse-into-files" or, more
generally useful, a find option.

> > No I can't.
> 
> > Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can
> > only do the above with convenient paths, completion etc. if I mount
> > my
> 
> Well, you can... or at least I had version somewhere that
> automagically prepended /overlay when file was not found. That way,
> normal requests were fast and uservfs were used whenever it was
> needed... At the price of little uglyness and slowing down -ENOENTs.

There you go... a kernel hook to prepend /overlay works :)
These hook I keep mentioning, they are really very simple :)

> > So all I am asking for is a facility to auto-mount with
> > file-as-directory, and the ability for a userspace daemon to be
> > notified of regular file modifications synchronously.  Both can be
> > added later, once file-as-directory and moveable mounts are
> 
> If the userspace daemon is synchronously notified of file
> modification... will not that lead to very ugly deadlocks?

Not if it's written properly.

-- Jamie

Re: silent semantic changes with reiser4

[Lee Revell]

On Wed, 2004-09-01 at 17:59, Jamie Lokier wrote:
> Pavel Machek wrote:
> > > Can I do these already using uservfs?
> > > 
> > >    cd
> > >    less foo.zip/contents/bar.c
> > 
> > less foo.zip#uzip/contents/bar.c
> > 
> > >    less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c
> > 
> > less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c
> >
> > >    grep -R obscure_label ~/RPMS
> > 
> > I do not think you'd want this. How would you tell grep obscure label
> > without entering archives, then? 
> 
> Granted, this is where we need "grep --recurse-into-files" or, more
> generally useful, a find option.
> 

FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
option, enabled by default, to look inside archives.  If you tell it to
look for a driver in a given directory it will also look inside .cab 
and .zip files.  It's extremely useful, I would imagine someone who uses
XP a lot will come to expect this feature.

Of course, no idea how it's implemented, but users like it.

Lee

Re: silent semantic changes with reiser4

[Horst von Brand]

Lee Revell <rlrevell@joe-job.com> said:

[...]

> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> option, enabled by default, to look inside archives.  If you tell it to
> look for a driver in a given directory it will also look inside .cab 
> and .zip files.  It's extremely useful, I would imagine someone who uses
> XP a lot will come to expect this feature.

It is trivial to implement this by looking inside the files. I.e., the way
mc has done this for ages.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 10:25, Horst von Brand wrote:
> Lee Revell <rlrevell@joe-job.com> said:
> 
> [...]
> 
> > FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> > option, enabled by default, to look inside archives.  If you tell it to
> > look for a driver in a given directory it will also look inside .cab 
> > and .zip files.  It's extremely useful, I would imagine someone who uses
> > XP a lot will come to expect this feature.
> 
> It is trivial to implement this by looking inside the files. I.e., the way
> mc has done this for ages.

This requires a separate, MC-specific namespace.  The point is to unify
the namespace, not fragment it.

If Hans had a comprehensible web page, maybe more people would
understand this aspect of his argument.  Would you put digressions about
chaos theory and King Arthur in a man page?

Lee

Re: silent semantic changes with reiser4

[Spam]

  

> Lee Revell <rlrevell@joe-job.com> said:

> [...]

>> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
>> option, enabled by default, to look inside archives.  If you tell it to
>> look for a driver in a given directory it will also look inside .cab
>> and .zip files.  It's extremely useful, I would imagine someone who uses
>> XP a lot will come to expect this feature.

> It is trivial to implement this by looking inside the files. I.e., the way
> mc has done this for ages.

  Difference is that you can't do "locate" or "find" or "Search".. You
  would have to open the files in an archive-supporting application
  such as mc.

Re: silent semantic changes with reiser4

[Alan Cox]

On Iau, 2004-09-02 at 20:41, Spam wrote:
> > It is trivial to implement this by looking inside the files. I.e., the way
> > mc has done this for ages.
> 
>   Difference is that you can't do "locate" or "find" or "Search".. You
>   would have to open the files in an archive-supporting application
>   such as mc.

And would you rather that logic was running swappable in shared library
space or privileged and unswappable in kernel ?

Alan

Re: silent semantic changes with reiser4

[Spam]

  

> On Iau, 2004-09-02 at 20:41, Spam wrote:
>> > It is trivial to implement this by looking inside the files. I.e., the way
>> > mc has done this for ages.
>> 
>>   Difference is that you can't do "locate" or "find" or "Search".. You
>>   would have to open the files in an archive-supporting application
>>   such as mc.

> And would you rather that logic was running swappable in shared library
> space or privileged and unswappable in kernel ?

  I would rather have it as a filesystem/vfs plugin that would allow
  all my programs to use the features the plugin gives, even if it
  would mean being totally in kernel space.

> Alan

Re: silent semantic changes with reiser4

[Alan Cox]

On Iau, 2004-09-02 at 21:07, Spam wrote:
> > And would you rather that logic was running swappable in shared library
> > space or privileged and unswappable in kernel ?
> 
>   I would rather have it as a filesystem/vfs plugin that would allow
>   all my programs to use the features the plugin gives, even if it
>   would mean being totally in kernel space.

Thats not what I asked. 

Re: silent semantic changes with reiser4

[Spam]

> On Iau, 2004-09-02 at 21:07, Spam wrote:
>> > And would you rather that logic was running swappable in shared library
>> > space or privileged and unswappable in kernel ?
>> 
>>   I would rather have it as a filesystem/vfs plugin that would allow
>>   all my programs to use the features the plugin gives, even if it
>>   would mean being totally in kernel space.

> Thats not what I asked.

  I failed to understand your question then. :(

  ~S

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > > It is trivial to implement this by looking inside the files. I.e., the way
> > > mc has done this for ages.
> > 
> >   Difference is that you can't do "locate" or "find" or "Search".. You
> >   would have to open the files in an archive-supporting application
> >   such as mc.
> 
> And would you rather that logic was running swappable in shared library
> space or privileged and unswappable in kernel ?

Well, having it in kernel mean that cache gets actually shared between
different processes -- and that's usefull. With coda and similar tools
you can put most of the logic into userspace (kernel impact is 20
lines). You *will* want it to run priviledged, because otherwise
caches are useless.
								Pavel

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> >> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> >> option, enabled by default, to look inside archives.  If you tell it to
> >> look for a driver in a given directory it will also look inside .cab
> >> and .zip files.  It's extremely useful, I would imagine someone who uses
> >> XP a lot will come to expect this feature.
> 
> > It is trivial to implement this by looking inside the files. I.e., the way
> > mc has done this for ages.
> 
>   Difference is that you can't do "locate" or "find" or "Search".. You
>   would have to open the files in an archive-supporting application
>   such as mc.

You really need archive support in find. At the very least you need
option "enter archives" vs. "do not enter archives". Entering archives
automagically is seriously wrong.
								Pavel

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 15:49, Pavel Machek wrote:
> Hi!
> 
> > >> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> > >> option, enabled by default, to look inside archives.  If you tell it to
> > >> look for a driver in a given directory it will also look inside .cab
> > >> and .zip files.  It's extremely useful, I would imagine someone who uses
> > >> XP a lot will come to expect this feature.
> > 
> > > It is trivial to implement this by looking inside the files. I.e., the way
> > > mc has done this for ages.
> > 
> >   Difference is that you can't do "locate" or "find" or "Search".. You
> >   would have to open the files in an archive-supporting application
> >   such as mc.
> 
> You really need archive support in find. At the very least you need
> option "enter archives" vs. "do not enter archives". Entering archives
> automagically is seriously wrong.

But is it efficient to make every application that reads files have to
know how to get inside a tar file, just to read its contents?  That
seems like a massive duplication of effort.  Better to have the contents
accessible via a separate stream, in the same namespace.  Fix it once in
the kernel vs. fix it in umpteen apps.

The key point here is that the expressive power of the system is greatly
reduced by having a fragmented namespace.  Of course there are any
number of ways for an app to find out what is in a tar file, but
exporting all of that information in a unified namespace is nontrivial
and much more interesting.

Lee

Re: silent semantic changes with reiser4

[Alan Cox]

On Iau, 2004-09-02 at 21:01, Lee Revell wrote:
> But is it efficient to make every application that reads files have to
> know how to get inside a tar file, just to read its contents?  That
> seems like a massive duplication of effort.  Better to have the contents
> accessible via a separate stream, in the same namespace.  Fix it once in
> the kernel vs. fix it in umpteen apps.

Thats how you get yourself a non useful OS. Fix it in a library and
share it between the apps that care. Like say.. gnome-vfs2

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 717 bytes --]

Salut,

On Thu, Sep 02, 2004 at 08:16:35PM +0100, Alan Cox wrote:
> Thats how you get yourself a non useful OS. Fix it in a library and
> share it between the apps that care. Like say.. gnome-vfs2

Even KIOslave has  it. They even support sftp and  stuff just by using
shared files  in /tmp in reality.  That's a much  saner interface than
doing it all in the kernel.

I  mean,  the  kernel  is  supposed  to support  access  to  the  disk
drives. Who says that it's got  to be the uppermost VFS level? You can
be perfectly happy to build your own  VFS on top of it (or use other's
implementations, that is.)

I can  already see people moving  to FreeBSD if  this gets implemented
into the kernel...

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Spam]

  

> Salut,

> On Thu, Sep 02, 2004 at 08:16:35PM +0100, Alan Cox wrote:
>> Thats how you get yourself a non useful OS. Fix it in a library and
>> share it between the apps that care. Like say.. gnome-vfs2

> Even KIOslave has  it. They even support sftp and  stuff just by using
> shared files  in /tmp in reality.  That's a much  saner interface than
> doing it all in the kernel.

> I  mean,  the  kernel  is  supposed  to support  access  to  the  disk
> drives. Who says that it's got  to be the uppermost VFS level? You can
> be perfectly happy to build your own  VFS on top of it (or use other's
> implementations, that is.)

  Yes, we have several VFS versions. KDE and Gnome has their own, mc
  another, and so on. None is compatible with the other and all data
  is unavailable if you do not use a program specifically coded to use
  the particular VFS you stored the data with.

> I can  already see people moving  to FreeBSD if  this gets implemented
> into the kernel...

  Why? No one has to use it. you do not have to load FS/VFS plugins so
  you can extend the functionality of your filesystem with stuff like
  compression, encryption, sorting, filtering, etc etc..

  

> 			    Tonnerre

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > Thats how you get yourself a non useful OS. Fix it in a library and
> > share it between the apps that care. Like say.. gnome-vfs2
> 
> Even KIOslave has  it. They even support sftp and  stuff just by using
> shared files  in /tmp in reality.  That's a much  saner interface than
> doing it all in the kernel.
> 
> I  mean,  the  kernel  is  supposed  to support  access  to  the  disk
> drives. Who says that it's got  to be the uppermost VFS level? You can
> be perfectly happy to build your own  VFS on top of it (or use other's
> implementations, that is.)

You can not reasonably do caching when you are in shared library. And
you can not do caching across users at all.

								Pavel

Re: silent semantic changes with reiser4

[Horst von Brand]

Tonnerre <tonnerre@thundrix.ch> said:

[...]

> I can  already see people moving  to FreeBSD if  this gets implemented
> into the kernel...

If it works better for them, great! It is after all a free world. If it
turns out via *BSD/MacOS XIX/Windows 2010/... that it is really useful, we
can add it to Linux. Linux is _not_ in a feeping creaturitis race, thank
you very much.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 815 bytes --]

Salut,

On Mon, Sep 06, 2004 at 02:08:15PM -0400, Horst von Brand wrote:
> Tonnerre <tonnerre@thundrix.ch> said:
> > I can  already see people moving  to FreeBSD if  this gets implemented
> > into the kernel...
> 
> If it works better for them, great! It is after all a free world. If it
> turns out via *BSD/MacOS XIX/Windows 2010/... that it is really useful, we
> can add it to Linux. Linux is _not_ in a feeping creaturitis race, thank
> you very much.

Please  read again  what I  just  said. I'm  against implementing  it,
because it is creeping featurism.  On the contrary, I said that people
might  actually change  to a  less complicated/bloated  OS  (Whereas I
would choose NetBSD over FreeBSD, but that's another story).

So you say the wrong things at the wrong people.

			Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Horst von Brand]

Tonnerre <tonnerre@thundrix.ch> said:
> On Mon, Sep 06, 2004 at 02:08:15PM -0400, Horst von Brand wrote:
> > Tonnerre <tonnerre@thundrix.ch> said:
> > > I can  already see people moving  to FreeBSD if  this gets implemented
> > > into the kernel...

> > If it works better for them, great! It is after all a free world. If it
> > turns out via *BSD/MacOS XIX/Windows 2010/... that it is really useful, we
> > can add it to Linux. Linux is _not_ in a feeping creaturitis race, thank
> > you very much.
> 
> Please  read again  what I  just  said. I'm  against implementing  it,
> because it is creeping featurism.  On the contrary, I said that people
> might  actually change  to a  less complicated/bloated  OS  (Whereas I
> would choose NetBSD over FreeBSD, but that's another story).

Sorry, understood exactly the wrong way around. Fingers faster than the
brain, again.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Chris Wedgwood]

On Thu, Sep 02, 2004 at 04:01:17PM -0400, Lee Revell wrote:

> Better to have the contents accessible via a separate stream, in the
> same namespace.  Fix it once in the kernel vs. fix it in umpteen
> apps.

This is ridiculous.  We have shared libraries, 99% of applications
manage to use libc for example --- this isn't that different.


  --cw

Re: silent semantic changes with reiser4

[Spam]

> On Thu, Sep 02, 2004 at 04:01:17PM -0400, Lee Revell wrote:

>> Better to have the contents accessible via a separate stream, in the
>> same namespace.  Fix it once in the kernel vs. fix it in umpteen
>> apps.

> This is ridiculous.  We have shared libraries, 99% of applications
> manage to use libc for example --- this isn't that different.

  Yes if that was the case then programs wouldn't brake if libc and
  its API was fixed. Sadly, I think it isn't.

  ~S


>   --cw

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > > >> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> > > >> option, enabled by default, to look inside archives.  If you tell it to
> > > >> look for a driver in a given directory it will also look inside .cab
> > > >> and .zip files.  It's extremely useful, I would imagine someone who uses
> > > >> XP a lot will come to expect this feature.
> > > 
> > > > It is trivial to implement this by looking inside the files. I.e., the way
> > > > mc has done this for ages.
> > > 
> > >   Difference is that you can't do "locate" or "find" or "Search".. You
> > >   would have to open the files in an archive-supporting application
> > >   such as mc.
> > 
> > You really need archive support in find. At the very least you need
> > option "enter archives" vs. "do not enter archives". Entering archives
> > automagically is seriously wrong.
> 
> But is it efficient to make every application that reads files have to
> know how to get inside a tar file, just to read its contents?  That

Application does not have to know how to handle tar/zip/etc, but it
has to make distinction between "enter archives" and "do not enter
archives". See uservfs.sf.net.
								Pavel

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 16:43, Pavel Machek wrote:
> Hi!
> 
> > > > >> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> > > > >> option, enabled by default, to look inside archives.  If you tell it to
> > > > >> look for a driver in a given directory it will also look inside .cab
> > > > >> and .zip files.  It's extremely useful, I would imagine someone who uses
> > > > >> XP a lot will come to expect this feature.
> > > > 
> > > > > It is trivial to implement this by looking inside the files. I.e., the way
> > > > > mc has done this for ages.
> > > > 
> > > >   Difference is that you can't do "locate" or "find" or "Search".. You
> > > >   would have to open the files in an archive-supporting application
> > > >   such as mc.
> > > 
> > > You really need archive support in find. At the very least you need
> > > option "enter archives" vs. "do not enter archives". Entering archives
> > > automagically is seriously wrong.
> > 
> > But is it efficient to make every application that reads files have to
> > know how to get inside a tar file, just to read its contents?  That
> 
> Application does not have to know how to handle tar/zip/etc, but it
> has to make distinction between "enter archives" and "do not enter
> archives". See uservfs.sf.net.

But how do you cache the information you had to look in the archive for
in a way that other apps can use it?  How do you synchronize access to
the cache and maintain cache coherency in userspace?

Lee

Re: silent semantic changes with reiser4

[Chris Wedgwood]

On Thu, Sep 02, 2004 at 04:47:40PM -0400, Lee Revell wrote:

> But how do you cache the information you had to look in the archive
> for in a way that other apps can use it?

~/.object-cache/ or whatever

> How do you synchronize access to the cache and maintain cache
> coherency in userspace?

coherency doesn't exist for normal files for the most part anyhow

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-09-02 at 16:49, Chris Wedgwood wrote:
> On Thu, Sep 02, 2004 at 04:47:40PM -0400, Lee Revell wrote:
> 
> > But how do you cache the information you had to look in the archive
> > for in a way that other apps can use it?
> 
> ~/.object-cache/ or whatever
> 

How are permissions handled?  If root lists the contents of a tar file
that is world readable, then joeuser comes along and does the same, can
joeuser sees the cached listing?  Would it depend on root's umask?

Lee

Re: silent semantic changes with reiser4

[Chris Wedgwood]

On Thu, Sep 02, 2004 at 04:57:06PM -0400, Lee Revell wrote:

> How are permissions handled?  If root lists the contents of a tar
> file that is world readable, then joeuser comes along and does the
> same, can joeuser sees the cached listing?

everyone has their own cache i guess, works well enough elsewhere

also, am i the only person scared by the code complexity when it comes
to root and setuid code here?  sounds like a disaster waiting to
happen

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > Application does not have to know how to handle tar/zip/etc, but it
> > has to make distinction between "enter archives" and "do not enter
> > archives". See uservfs.sf.net.
> 
> But how do you cache the information you had to look in the archive for
> in a way that other apps can use it?  How do you synchronize access to
> the cache and maintain cache coherency in userspace?

Uservfs.sf.net.

Unlike alan, I do not think that "do it all in library" is good
idea. I put it in the userspace as "codafs" server, and let
applications see it as a regular filesystem.
							Pavel

Re: silent semantic changes with reiser4

[Alan Cox]

On Iau, 2004-09-02 at 21:58, Pavel Machek wrote:
> Uservfs.sf.net.
> 
> Unlike alan, I do not think that "do it all in library" is good
> idea. I put it in the userspace as "codafs" server, and let
> applications see it as a regular filesystem.

That works for me too, providing someone has fixed all the user mode fs
deadlocks with paging

Re: silent semantic changes with reiser4

[Gianni Tedesco]

On Thu, 2004-09-02 at 23:33 +0100, Alan Cox wrote:
> On Iau, 2004-09-02 at 21:58, Pavel Machek wrote:
> > Uservfs.sf.net.
> > 
> > Unlike alan, I do not think that "do it all in library" is good
> > idea. I put it in the userspace as "codafs" server, and let
> > applications see it as a regular filesystem.
> 
> That works for me too, providing someone has fixed all the user mode fs
> deadlocks with paging

Aren't the deadlock scenarios only applicable for read/write mounted
filesystems ?

-- 
// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source www.scaramanga.co.uk/scaramanga.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D

Userspace framework (was: Re: silent semantic changes with reiser4)

[Luca Ferroni]

Il Fri, 03 Sep 2004 04:22:48 +0100,  Gianni Tedesco <gianni@scaramanga.co.uk> ha scritto:

> On Thu, 2004-09-02 at 23:33 +0100, Alan Cox wrote:
> > On Iau, 2004-09-02 at 21:58, Pavel Machek wrote:
> > > Uservfs.sf.net.
> > > 
> > > Unlike alan, I do not think that "do it all in library" is good
> > > idea. I put it in the userspace as "codafs" server, and let
> > > applications see it as a regular filesystem.
> > 
> > That works for me too, providing someone has fixed all the user mode fs
> > deadlocks with paging
> 
> Aren't the deadlock scenarios only applicable for read/write mounted
> filesystems ?
> 

AFAIK deadlock arises when kernel manages buffers:
it has to free a buffer ==> choose a dirty one ==> if cleaning requires to make a call to
network server and this last is waiting for a buffer (cleaning accomplished) ==>
==> deadlock.

I think buffers are involved even in reading file system access.
I think also that NFS is threatened by deadlock, not CodaFS.

The problem with CodaFS is that it doesn't support random file access: 
you have to download a whole file in coda cache directory even
to read its first byte.

I'm writing my laurea thesis about userspace file system,
and two Interesting papers i found about this, are:

http://a5.repetae.net/~frederik/avfs/frederik_surf_paper.pdf by Frederik Eaton
and the more generic UFO paper
http://www.usenix.org/publications/library/proceedings/ana97/alexandrov.html

I think that LUFS (Linux User File System)  and FUSE (Filesystem in USErspace) 
kernel modules are solutions to be considered for adding to Linux develop branch.
They are similar, but FUSE is still maintained and improved, and it permit to create
a custom daemon for each user filesystem (unlinke lufsd daemon which loads a 
shared library). Communication kernel-user space is done through a special file descriptor,
unlike LUFS socket.
The only doubt is that FUSE is tuned for local file system performance, but I don't know
if it is possible to develop network file system with this framework.
LUFS is slower but more generic in this sense.

http://sourceforge.net/projects/avf

There is more documentation in the tarball than in the home page.

Ciao
Luca

Re: Userspace framework (was: Re: silent semantic changes with reiser4)

[Gianni Tedesco]

On Fri, 2004-09-03 at 11:24 +0200, Luca Ferroni wrote:
> Il Fri, 03 Sep 2004 04:22:48 +0100,  Gianni Tedesco <gianni@scaramanga.co.uk> ha scritto:
> 
> > On Thu, 2004-09-02 at 23:33 +0100, Alan Cox wrote:
> > > On Iau, 2004-09-02 at 21:58, Pavel Machek wrote:
> > > > Uservfs.sf.net.
> > > > 
> > > > Unlike alan, I do not think that "do it all in library" is good
> > > > idea. I put it in the userspace as "codafs" server, and let
> > > > applications see it as a regular filesystem.
> > > 
> > > That works for me too, providing someone has fixed all the user mode fs
> > > deadlocks with paging
> > 
> > Aren't the deadlock scenarios only applicable for read/write mounted
> > filesystems ?
> > 
> 
> AFAIK deadlock arises when kernel manages buffers:
> it has to free a buffer ==> choose a dirty one ==> if cleaning
> requires to make a call to
> network server and this last is waiting for a buffer (cleaning
> accomplished) ==>
> ==> deadlock.

so during page launder, we need to write to the filesystem, but if thats
in userspace there is the possibility the page launder could have been
caused in order that the filesystem daemon may run. AFAICS this problem
only arises when file is being written.

The only deadlock I can think of for read-only filesystems is if the
demon inadvertantly accesses one of the files that it is handling. That
could be avoided quite simply by preventing the demon from doing that in
the kernel.

I'm sure I'm missing something, I'd just like to know what ;)

-- 
// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source www.scaramanga.co.uk/scaramanga.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > Uservfs.sf.net.
> > 
> > Unlike alan, I do not think that "do it all in library" is good
> > idea. I put it in the userspace as "codafs" server, and let
> > applications see it as a regular filesystem.
> 
> That works for me too, providing someone has fixed all the user mode fs
> deadlocks with paging

Its okay: coda works on whole file granularity, and files are backed
by real disk, so there are no deadlocks with paging. We only block
open() and similar syscalls, and those are not in paging paths. It
should be safe.
								Pavel

Re: silent semantic changes with reiser4

[Horst von Brand]

Lee Revell <rlrevell@joe-job.com> said:
> To: Pavel Machek <pavel@ucw.cz>
> Cc: Spam <spam@tnonline.net>, Horst von Brand <vonbrand@inf.utfsm.cl>,
>         Jamie Lokier <jamie@shareable.org>, David Masover <ninja@slaphack.com>,
>         Chris Wedgwood <cw@f00f.org>, viro@parcelfarce.linux.theplanet.co.uk,
>         Linus Torvalds <torvalds@osdl.org>, Christoph Hellwig <hch@lst.de>,
>         Hans Reiser <reiser@namesys.com>, linux-fsdevel@vger.kernel.org,
>         linux-kernel <linux-kernel@vger.kernel.org>,
>         Alexander Lyamin aka FLX <flx@namesys.com>,
>         ReiserFS List <reiserfs-list@namesys.com>
> X-Mailer: Ximian Evolution 1.4.6 
> Date: Thu, 02 Sep 2004 16:01:17 -0400
> 
> On Thu, 2004-09-02 at 15:49, Pavel Machek wrote:

[...]

> > You really need archive support in find. At the very least you need
> > option "enter archives" vs. "do not enter archives". Entering archives
> > automagically is seriously wrong.

I have used find(1) for quite some time now, and have never (or very
rarely) missed this.

> But is it efficient to make every application that reads files have to
> know how to get inside a tar file, just to read its contents?

Totally ridiculous, especially if you factor in .gz, .bz2, .zip, .a,
.whatever.new.format.they.come.up.with.tomorrow. But then again, this would
presumably reside in a (shared) library, so it isn't so bad...

>                                                                That
> seems like a massive duplication of effort.

Right. tar, gzip, bunzip, et al are already around.

>                                              Better to have the contents
> accessible via a separate stream, in the same namespace.  Fix it once in
> the kernel vs. fix it in umpteen apps.

Dead wrong. It is better to fix it in userspace (via a library, if
required; could call random unpacking etc programs at will, even be
configured on a user-by-user basis through ~/.wacky-file-handling or
environment) than force this junk into the kernel. Kernel code is _always_
resident, extremely security critical, and hard to get right. Besides, not
everybody will want to carry this around, and so it will forever stay a
"weird Linux kernel configuration only" feature, i.e., useless in practice.

> The key point here is that the expressive power of the system is greatly
> reduced by having a fragmented namespace.  Of course there are any
> number of ways for an app to find out what is in a tar file, but
> exporting all of that information in a unified namespace is nontrivial
> and much more interesting.

I don't see how it is "nontrivial": "tar tf some.tar" is quite enough to
find out what is inside, in a customary format.

Placing random junk in the kernel doesn't magically make it fast, right, or
useful. Quite a lot of work is required for that, much more than for
getting the same in userland.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Jamie Lokier]

Horst von Brand wrote:
> > > You really need archive support in find. At the very least you need
> > > option "enter archives" vs. "do not enter archives". Entering archives
> > > automagically is seriously wrong.
> 
> I have used find(1) for quite some time now, and have never (or very
> rarely) missed this.

I've occasionally had the need to search all files on my system for
the one file which contains a particular phrase -- all I remember is
the phrase.

Just doing "grep -R" was a tedious job: at least half an hour.

Sometimes, I want to search all source files on my system for a
particular word, for example to search for uses of a particular system
call or library function.

That would require something that could search through all the .tar.gz
files and .zip files (nested if necessary) as well as plain files.  It
would take so long -- hours at least, maybe more than a day -- that
I've never bothered doing such a thing.

"find "that entered archives really wouldn't help (although sometimes
"locate" that entered archives would be nice).

In other words, I'd use that capability if it was magically fast, but
as we expect it to be insanely slow (just grepping gigabytes is slow)
that makes it not so useful.

However, if we ever see that search engine index thing happen, it
would be a most excellent capability if it searched inside archive
files too.  I would definitely use that.  Not often, but occasionally I would.

-- Jamie

Re: silent semantic changes with reiser4

[Alan Cox]

On Gwe, 2004-09-03 at 00:23, Jamie Lokier wrote:
> However, if we ever see that search engine index thing happen, it
> would be a most excellent capability if it searched inside archive
> files too.  I would definitely use that.  Not often, but occasionally I would.

Thats an indexer decision, the search backend (which is the performance
and complexity critical part) doesn't give a damn.

Re: silent semantic changes with reiser4

[Spam]

  

> On Gwe, 2004-09-03 at 00:23, Jamie Lokier wrote:
>> However, if we ever see that search engine index thing happen, it
>> would be a most excellent capability if it searched inside archive
>> files too.  I would definitely use that.  Not often, but occasionally I would.

> Thats an indexer decision, the search backend (which is the performance
> and complexity critical part) doesn't give a damn.

  I am just talking general now, but it seems to me that there have
  been many suggestions on user-land solutions like shared librares
  and so forth just to say there is no need for file streams and
  plugins. Many of these ideas do exist in one way or another, but
  none is truly system wide and and as application independent as 
  file streams+plugins would be. Would it not be much less effort to
  implement these in a good way, than trying to reinvent lots of new
  stuff in userland - that wouldn't be systemwide anyway?

  ~S

  

Re: silent semantic changes with reiser4

[John Stoffel]

>>>>> "Jamie" == Jamie Lokier <jamie@shareable.org> writes:

Jamie> Horst von Brand wrote:
>> > > You really need archive support in find. At the very least you need
>> > > option "enter archives" vs. "do not enter archives". Entering archives
>> > > automagically is seriously wrong.
>> 
>> I have used find(1) for quite some time now, and have never (or very
>> rarely) missed this.

Jamie> I've occasionally had the need to search all files on my system
Jamie> for the one file which contains a particular phrase -- all I
Jamie> remember is the phrase.

So use glimpse and run an indexer once a night to pick up changes.  

Jamie> Just doing "grep -R" was a tedious job: at least half an hour.

Ugh, esp since you're not caching any results of all that work either,
when you need to repeat 10 minutes down the line...

Jamie> Sometimes, I want to search all source files on my system for a
Jamie> particular word, for example to search for uses of a particular
Jamie> system call or library function.

I think we want to pull this back a bit more and define this better.
Instead of 'system' we should be thinking 'namespace' or 'hierarchy'.  

Jamie> That would require something that could search through all the
Jamie> .tar.gz files and .zip files (nested if necessary) as well as
Jamie> plain files.  It would take so long -- hours at least, maybe
Jamie> more than a day -- that I've never bothered doing such a thing.

Jamie> In other words, I'd use that capability if it was magically
Jamie> fast, but as we expect it to be insanely slow (just grepping
Jamie> gigabytes is slow) that makes it not so useful.

Jamie> However, if we ever see that search engine index thing happen,
Jamie> it would be a most excellent capability if it searched inside
Jamie> archive files too.  I would definitely use that.  Not often,
Jamie> but occasionally I would.

Is the overhead of pre-indexing all files and their contents worth it
though?  And would the use of inotify allow us to efficiently update
the index in a fairly quick time?

All the various arguements have been that we need a filesystem which
magically indexes files, does queries and does it all atomically and
without any thought on the part of the developer/user.  It could
happen, but then we'd have such a *slow* system in general, just to
make 1% of the usage simpler (for some measure of simple) that it
would be a waste.

Providing a simple, consistent set of semantics and syntax allows you
to build on top of it the layer(s) you need to provide the guarenttees
you need for your application.

I don't expect that Oracle has the same needs as does my mail spool,
or as an image store.  Sure, Oracle could do both of them, but are
the overheads of oracle worth the slowdown in the common case?

John

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 669 bytes --]

Salut,

On Wed, Sep 01, 2004 at 06:51:12PM -0400, Lee Revell wrote:
> FWIW, this is how Windows does it now.  As of XP, 'Find files' has an
> option, enabled by default, to look inside archives.  If you tell it to
> look for a driver in a given directory it will also look inside .cab 
> and .zip files.  It's extremely useful, I would imagine someone who uses
> XP a lot will come to expect this feature.
> 
> Of course, no idea how it's implemented, but users like it.

The trick  is that their tools are  aware of it, and  that the library
the programs use supports on-the-fly decompression.

Same applies to Spotlight for Tiger.

			    Tonnerre


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Gianni Tedesco]

On Wed, 2004-09-01 at 17:14 +0100, Jamie Lokier wrote:
> So all I am asking for is a facility to auto-mount with
> file-as-directory, and the ability for a userspace daemon to be
> notified of regular file modifications synchronously.  Both can be
> added later, once file-as-directory and moveable mounts are
> established.  (fcntl leases and dnotify _almost_ provide this, but
> they don't.  Looks like incoherent hacks keep getting added all over
> the place for Samba... :).

How will synchronous file modification notifications be handled?

Programs listening for the notification will want to see the changes
when they read the file but the reason this is synchronous is to prevent
other applications seeing the changes before they are reflected in these
indexing programs etc.. right?

-- 
// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source www.scaramanga.co.uk/scaramanga.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus, you are looking at this like a lieutenant instead of an HQ 
staffer, which is unusual of you.

You are saying, 1-2% simpler and better, no biggie, why work so hard to 
get it?

And we are saying, 1-2% simpler and better, times thousands of 
applications, wow! That's a lot!

Yes, changing cat to use openat() is no big deal. 1-2% additional coding 
cost for cat, who cares? But if you add 1-2% coding cost to every 
application which might access an attribute/stream/whatever, well, that 
totals more than the effort of authoring emacs. Can you see that?

Namespace simplifications and empowerments are force multipliers. They 
don't add to Linux like writing a new app adds to Linux, they add to 
Linux like adding percentage improvements to every app in Linux adds to 
Linux. HQ staffer types know that if you collect enough itty bitty 
little force multipliers, you win the war. Whether the troops have to 
spend five minutes a day polishing their shoes because their type of 
shoe needs polishing, that matters more than losing a tank battalion, 
when you are a major power. Linux is a major power.

This is a software engineering issue. We are discussing improvements 
that because they are diffused throughout the OS in their impact, seem 
like no big deal. But they are a big deal. One of the major determinants 
of an organization's efficiency is whether the management can recognize 
widely diffused inefficiencies as well as it can recognize focused 
inefficiencies of the same magnitude.

David is so very right about the usability issues. Usability is all 
about trivia. Usability battles feature no dragons, they feature armies 
of spiders. If you add one step to what a user needs to do to get to the 
data with an app, that matters a lot. That's doubling the spider army. 
Making a 2 step process to access data into a one step process to access 
the data halves the time cost of accessing the data. Lifestyle 
efficiency is mostly about reducing the cost of trivia, because trivia 
is most of what we do. Most people have as much interest in reading the 
man page for tar as you have in learning how to turn the hand crank to 
start your car. Just the look of "tar -xf" turns them away. Crypto-Geek 
gobbledy-gook is what it is. Let's value their time, there are a lot of 
them.

Hans

David Masover wrote:

>
>
> It's not about the kernel, it's about the interface. And see my other 
> mail:
> cat foo.zip/README
> less foo.zip/contents/bar.c
> is a lot easier than
> lynx http://google.com/search?q=zip
> emerge zip
> man zip
> unzip foo.zip
> cat bar.c
> which already assumes quite a lot of expertise.
>
>

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Hans Reiser wrote:
> 
> You are saying, 1-2% simpler and better, no biggie, why work so hard to 
> get it?
> 
> And we are saying, 1-2% simpler and better, times thousands of 
> applications, wow! That's a lot!

But would thousands care? Seriously?

For example, you could make just _one_ program support "openat()", and 
you'd get most of the advantages, with no possibility of _breaking_ any of 
thousands of applications..

I know, you've ignored the "runat" program (which is just a wrapper around
the openat() system call), but it _has_ been mentioned several times in 
this thread. Yes, you'd type a bit more to do

	runat file ls -l

instead of

	ls -l file/

but at least the openat/runat approach also works for directories, which 
does actually make it a lot more _generic_ than the "show in the regular 
filespace" approach. No special cases.

So your comparison isn't valid, because you're ignoring the people who
shout "runat" at you. You've also not ever actually answered about the
problems about directories with attributes.

		Linus

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Tue, 31 Aug 2004, Hans Reiser wrote:
>  
>
>>You are saying, 1-2% simpler and better, no biggie, why work so hard to 
>>get it?
>>
>>And we are saying, 1-2% simpler and better, times thousands of 
>>applications, wow! That's a lot!
>>    
>>
>
>But would thousands care? Seriously?
>  
>
>For example, you could make just _one_ program support "openat()", and 
>you'd get most of the advantages, with no possibility of _breaking_ any of 
>thousands of applications..
>
>I know, you've ignored the "runat" program (which is just a wrapper around
>the openat() system call), but it _has_ been mentioned several times in 
>this thread. Yes, you'd type a bit more to do
>
>	runat file ls -l
>
>instead of
>
>	ls -l file/
>
>but at least the openat/runat approach also works for directories, which 
>does actually make it a lot more _generic_ than the "show in the regular 
>filespace" approach. No special cases.
>  
>
So you are saying I can do runat emacs my-compound-document/stream-name? 
and in place of CTRL-X CTRL-F to open a file in emacs I type what to 
open an attribute?

I think it does not work Linus....


>So your comparison isn't valid, because you're ignoring the people who
>shout "runat" at you. You've also not ever actually answered about the
>problems about directories with attributes.
>  
>
Which problem specifically? I will list them and hope I don't miss what 
you want addressed. I won't claim to be sure I have the right answers 
for all of these (because VFS has not been my interest in the past), but 
these look feasible to me.;-)

Cycle detection:

We should either 1) make hard links only link to the file aspect of the 
file-directory duality, and persons who want to link to the directory 
aspect must use symlinks (best short term answer), or 2) ask Alexander 
Smith to help us with applying his cycle detection algorithm and gain 
the benefit of being able to hard link to directories (if it works well, 
best long term answer).

Ambiguity:

Linus said:

 - how do you handle the ambiguity of
	//attr/usr/bin/emacs/icon
   (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the 
   "emacs/icon" attribute on "/usr/bin").

If you want to avoid crowding the namespace with some piece of meta 
information that you want to be applicable to both directories and the 
files in them, and also widely used enough that you care about crowding 
the namespace, and unambiguous as to what level in the hierarchy it 
metas, then you put it into the metas subdirectory. There is no 
principled difference between a file that is a child of a directory and 
an attribute. The ambiguity is inherent in hiearchical semantics I 
think, and whether or not you have attributes you have this type of 
ambiguity.

Thus, you have either:

/usr/bin/metas/emacs/icon

or /usr/bin/emacs/metas/icon

and these are easily distinguished and unambiguous. This makes metas a 
reserved keyword, and there was a long discussion about whether to use 
"metas" or "..metas" as the reserved subdirectory name which I can 
summarize for you on request. There is also "pseudos", which contains 
files implementing special methods of special plugins. Feel free to 
invent more reserved subdirectories as needed, or to argue for fewer.


Seeing two trees, one with attrs and one without:

Linus said:

If you open a filename in some "secondary" tree (be it /proc or //attr or 
whatever) based on the filename in the primary one, you have two issues 
that you need to work out:

 - how do you handle a name change in the primary tree at the same time as 
   lookup
 - how do you handle the ambiguity of
	//attr/usr/bin/emacs/icon
   (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the 
   "emacs/icon" attribute on "/usr/bin").

The ambiguity can be handled by saying that attributes only have one
component (ie only the _last_ component of a lookup is the attribute
name). But the race between primary tree and secondary tree cannot be
handled in a normal name-space.

Are you sure that you have to have two trees in order to see two trees? 
That is, can you not refuse to follow or see a link depending on whether 
some flag is set and what the type of the link is?

Locking: 

Linus said:

let's say that you have filename "a" hard-linked to filename "b", 
and you have a directory structure of streams under there. So you have

	a/file1
	a/dir1/file2
	a/dir2/file3

and (through the hard-link with "b") you have aliases of all these same 
names available as "b/file1", "b/dir1/file2" etc).

Now, imagine that you have two processes doing

	mv a/dir1 a/dir2/newdir

and

	mv b/dir2 b/dir1/newdir

at the same time. Both of them MUST NOT SUCCEED, for pretty obvious 
reasons (you'd have moved two directories within each other, and now 
neither would be accessible any more).

How do you handle locking for this situation?

  

(Here is where my knowledge of VFS is shakiest.)  Making "hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks" seems to solve this.  If we want to use the Alexander Smith solution to cycle detection, and allow hard links to directories, then we must make sure that for the hard linked entity, there is a single lock for it somewhere (e.g. the directory inode) that can be taken.  If you create two dentries for the same object, then flag the dentries as needing to lock a lock in their shared inode and not in the dentry.  If you don't want to read in the inode from disk to do that, then create a pointer to a lock shared by them both.

Linus, is this a start on solving some of the issues?  

Re: silent semantic changes with reiser4

[Alexander G. M. Smith]

Hans Reiser wrote on Wed, 01 Sep 2004 00:22:49 -0700:
> Making "hard links only link to the file aspect of the file-directory
> duality, and persons who want to link to the directory aspect must use
> symlinks" seems to solve this.  If we want to use the Alexander Smith
> solution to cycle detection, and allow hard links to directories, then
> we must make sure that for the hard linked entity, there is a single
> lock for it somewhere (e.g. the directory inode) that can be taken.

Oddly enough that's what I did in my RAM file system.  No matter how
the file/directory node is found (the path can lead through different
parents), it has a unique ID and a corresponding single lock for it.

Rename/delete goes and locks all the nodes that might change parents
and/or children, then it does its operations (though it has to watch
out for deadlocks, which are unavoidable when two rename/deletes both
sneak up on the same file by different paths).

- Alex

Re: silent semantic changes with reiser4

[Timothy Miller]

I'm probably not the first to suggest this idea, and it's probably not a 
very good idea, but here's my idea anyhow:

You have a file "/usr/bin/emacs"
with a metadata property in the overlaid namespace 
"/usr/bin/emacs/[[..]metas/]icon"

According to some, this could cause some confusion.  Howabout instead:

You have a file "/usr/bin/emacs"
with a metadata property in a slightly separated namespace 
"/metas/usr/bin/emacs/icon"

This has the advantage of still having the metadata in the filesystem 
namespace but without the confusion of having files-as-directories, 
ambiguity of filename, backup issues, etc.  This is the reverse of 
having the namespaces overlaid with a "/nometas" view which is separate.

Furthermore, you can split things further like this:

You have a file "/usr/bin/emacs"
with an automatically-generated metadata property that you don't want to 
back up in "/autometas/usr/bin/emacs/modification_date"
and a manually generated metadata property that you MAY want to backup 
in "/staticmetas/usr/bin/emacs/icon".

This is inelegant, I know.  But if we do this, we can add the extra 
features of reiser4 without confusing existing apps or having to modify 
them to support the new functionality.

Furthermore, you can easily hide the extra features by not mounting the 
meta top-level directories (assuming they're mounted like separate 
filesystems, rather than just magically appearing there, which is okay too).

Re: silent semantic changes with reiser4

[Helge Hafting]

Timothy Miller wrote:

> I'm probably not the first to suggest this idea, and it's probably not 
> a very good idea, but here's my idea anyhow:
>
> You have a file "/usr/bin/emacs"
> with a metadata property in the overlaid namespace 
> "/usr/bin/emacs/[[..]metas/]icon"
>
> According to some, this could cause some confusion.  Howabout instead:
>
> You have a file "/usr/bin/emacs"
> with a metadata property in a slightly separated namespace 
> "/metas/usr/bin/emacs/icon"
>
And the problem with such a "solution" is
mv /usr/bin/emacs /usr/bin/old-emacs
Do this with an ordinary fs, and the /metas/usr/bin/emacs/icon
won't move with it.  Now metas might might not be an ordinary fs,
so perhaps the move happens automatically there, but if so
it will be unexpected. 

> This has the advantage of still having the metadata in the filesystem 
> namespace but without the confusion of having files-as-directories, 
> ambiguity of filename, backup issues, etc.  This is the reverse of 
> having the namespaces overlaid with a "/nometas" view which is separate.
>
> Furthermore, you can split things further like this:
>
> You have a file "/usr/bin/emacs"
> with an automatically-generated metadata property that you don't want 
> to back up in "/autometas/usr/bin/emacs/modification_date"
> and a manually generated metadata property that you MAY want to backup 
> in "/staticmetas/usr/bin/emacs/icon".
>
> This is inelegant, I know.  But if we do this, we can add the extra 
> features of reiser4 without confusing existing apps or having to 
> modify them to support the new functionality.
>
> Furthermore, you can easily hide the extra features by not mounting 
> the meta top-level directories (assuming they're mounted like separate 
> filesystems, rather than just magically appearing there, which is okay 
> too).

Having to go up to root and then down a similar but different path to 
reach a
file's metadata seems very counterintuitive to me.  And you have to 
update all
tools to do this automatically, or it'll be hopeless to actually use.

Helge Hafting

Re: silent semantic changes with reiser4

[Alan Cox]

On Mer, 2004-09-01 at 06:50, Hans Reiser wrote:
> Yes, changing cat to use openat() is no big deal. 1-2% additional coding 
> cost for cat, who cares? 

Large cost on its own. The FSF would almost certainly reject such a
change until someone had written portable emulation of the feature for 
every other platform which would mean vendor patches which would
essentially mean it wouldn't happen.

I know I rarely agree with Hans but I think he's right on this one, it
has to work with the existing open() API.

Alan

Re: silent semantic changes with reiser4

[Paul Jakma]

On Tue, 31 Aug 2004, Hans Reiser wrote:

> learning how to turn the hand crank to start your car. Just the look of "tar 
> -xf" turns them away. Crypto-Geek gobbledy-gook is what it is. Let's value 
> their time, there are a lot of them.

These kinds of people would use something like file-roller in GNOME, 
and they'd care not a jot whether file-rollers used built-in 
tar support, and/or openat() or via file-as-dir fs support.

All they care about is that they get a window with icons for their 
files that they can click on. How it's done does not matter - it only 
matters to those who must write the code.

> Hans

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
I haven't lost my mind; I know exactly where I left it.

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus, did you get the email in which I (I think....) answered all the 
problems with files as directories that you and Viro had raised?

You and Viro did not respond to it....

Hans

Re: silent semantic changes with reiser4

[viro]

On Thu, Sep 02, 2004 at 11:06:39PM -0700, Hans Reiser wrote:
> Linus, did you get the email in which I (I think....) answered all the 
> problems with files as directories that you and Viro had raised?
> 
> You and Viro did not respond to it....

Message-ID, please.  There was a lot of mails from you in this thread and
I probably have missed one that did address *any* problems.  If such mail had
ever existed, that is.

Re: silent semantic changes with reiser4

[Sriram Karra]

viro@parcelfarce.linux.theplanet.co.uk writes:

> On Thu, Sep 02, 2004 at 11:06:39PM -0700, Hans Reiser wrote:
>> Linus, did you get the email in which I (I think....) answered all
>> the problems with files as directories that you and Viro had
>> raised?
>> 
>> You and Viro did not respond to it....
>
> Message-ID, please.  There was a lot of mails from you in this
> thread and I probably have missed one that did address *any*
> problems.  If such mail had ever existed, that is.

Perhaps this is one? Message-ID: <413578C9.8020305@namesys.com>

Re: silent semantic changes with reiser4

[viro]

On Wed, Sep 08, 2004 at 01:21:45PM +0530, Sriram Karra wrote:
> Perhaps this is one? Message-ID: <413578C9.8020305@namesys.com>

OK...

One note before replying: current code deadlocks even if you make ->link()
*ALWAYS* return an error.  It doesn't get to calling the method.  No amount
of "disallow hard links to <something>" is going to help here, obviously.

<quote>
Cycle detection:

We should either 1) make hard links only link to the file aspect of the
file-directory duality, and persons who want to link to the directory
aspect must use symlinks (best short term answer), or 2) ask Alexander
Smith to help us with applying his cycle detection algorithm and gain
the benefit of being able to hard link to directories (if it works well,
best long term answer).
</quote>

... which doesn't address the problem at all.  The question is what to do
with seeing directory "aspect..." in more than one place when we have many
links to file in question.  So much for (1).  And (2) is not feasible with
on-disk fs both due to memory, CPU and IO costs _and_ due to exclusion from
hell you'll need to make it safe.

Re: ambiguity - lots and lots of handwaving on both sides.  FWIW, I agree
with Hans in one (and only one) respect here - openat() as a primary API
(and not a convenient libc function) is an atrocity.  Simply because it
doesn't address operations beyond open (unlinkat(2), anyone?).

However, I still haven't seen any strong arguments for need of this "metas"
stuff _or_ the need to export mode/ownership as files, both for regular
files and for directories.  Aside of "we can do that" [if we solve the
locking issues] and "xattrs are atrocious" [yes, they are; it doesn't make
alternative mechanism any better] there was nothing that even pretended to
be a technical reason.

Note that we also have fun issues with device nodes (Linus' "show partitions"
vs. "show metadata from hosting filesystem"), which makes it even more dubious.
We also have symlinks to deal with (do they have attributes?  where should
that be exported?).

Reserved names have one more problem: to be useful, they'd have to be
hardcoded into applications.  And that will create hell with use of
such applications on existing filesystems.  Again, no feasible scheme
to deal with that in userland code had been proposed so far, AFAICS.

Locking: see above - links to regular files would create directories seen
in many places.  With all related issues...

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 485 bytes --]

Salut,

On Tue, Aug 31, 2004 at 08:14:33PM -0500, David Masover wrote:
> It's not about the kernel, it's about the interface.  And see my other mail:
> 	cat foo.zip/README
> 	less foo.zip/contents/bar.c
> is a lot easier than
> 	lynx http://google.com/search?q=zip
> 	emerge zip
> 	man zip
> 	unzip foo.zip
> 	cat bar.c
> which already assumes quite a lot of expertise.

Most distros ship Ziv's packer by default.

telnet dict.org 2628
DEFINE * "creeping featurism"
QUIT

				Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pavel Machek wrote:
| Hi!
|
|
|>| uservfs does
|>|
|>| cd foo.deb#uar
|>cd foo.deb/ar
|>| vs.
|>| cd foo.deb#udeb
|>cd foo.deb/deb
|>
|>and why would you want that, instead of just:
|>cd foo.deb	# for the ar
|>dpkg -i foo.deb	# for the deb
|
|
| Because I want to see contents of that .deb package, nicely parsed?
|
|
|>Just want to extract the tar file?  Maybe something like
|>cat foo.tgz/gunzip
|>In which case (of course) foo.tgz/gunzip has exactly the same directory
|>contents as foo.tgz
|
|
| Yes, that would work.
|
|
|>In fact, for just about any syntax anyone could suggest, I can't really
|>see why you can't just replace all weird symbols with a slash and a
|>symbol.  Instead of
|>	foo.tgz#utar
|>you have
|>	foo.tgz/#/utar
|>Only difference is, some things which used to require special tools can
|>now be serviced by less than what's in busybox.
|
|
| That would work, too. I do not get your comment about busybox.

Busybox is a bare minimum of unix tools, yet still includes things like
tar/bz2.  With such a system, you can use the most basic (sub-GNU)
versions of tools like cp, cat, sh, ls, and so on, to do much more
complex operations.  What's more, you can see everything you can do to a
file just by looking at it.  I can certainly imagine doing
	cat foo.tgz/metas/README
rather than a google search for tgz, followed by a "man tar" and "man
gzip". Assuming I know nothing about tar/gzip files, it's a lot easier
as a new user to have one common interface for everything -- including
docs.  Fewer commands to memorize, faster to learn new stuff.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUha3gHNmZLgCUhAQKRJw/7BUEm41gKZGoVj7gOkcUVe/SRdTs9WtTL
jkjCX8Zzr/1nnAKTHLqNJEiAJ68Ndf+dRhDfL5y6dDHhOy/Dx+38SzsWz+dQnMJ0
Fiq5fFDO/DTd0UPecXYPhBrtVeCdfjyyENhgRgJud6D8F39qvLWrZFUsIkjAA3lj
nzZ/48jRPDWT/mF6JJmv0RfVvNtMWh9HUcW/i0+n4WZU1yFp6jyiF5ExCOP84z3x
yXQ4Q9hnjgt52MDJTrGsCDvfCSQscQilxvXbl0iaozquIhG9Jz1XNYpij2XUygFi
RDc1LA1k74VvbWahlkdfbTI+3P4qeMYGBblmfwXVDRwW0i4NH7X0/6xcnr1QjaK/
f/7EHBz0wnkIG4OpcGWzu9ik4WOR0WXapKrd938XC/mG+WzIluJlQsxG1A2HoRqg
R66E/mVMVvHJWnJjrW6s10WIemPiKnuXgeIVPifsw81DLXlZ3wxjlZU3iuo7xked
O6wNDFQBWQR8kv+EMDIt3gtbUdB+5KpxGR+Yli0AjxEdpdk3oZTfe5XQiNdJhcl5
xrcomVKXGvR96IlNUwHy1EsQk5LySMl101vJIS4z6i7lAIM5CQ1d6eOZrPp/DBOP
Rw0KoQaiPd3SKNMgoyWSKQL6h85i1PRUN62WH0T2NRU1fZ3EazpiRZr+fuaFqFT9
Wk2lkDvm5yw=
=ShWo
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> > One of the big potential uses for file-as-directory is to go inside
> > archive files, ELF files, .iso files and so on in a convenient way.
> 
> Arguably this belongs in userspace --- and people have put it there.
> 

uservfs.sf.net, using existing coda hooks. There are 2+
very similar projects.


Advantage is we can do ftpfs, too. Having ftp done by

> /ftp
cd /ftp/sunsite

would be too ugly for reiser4 plugins... Or not?
-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: silent semantic changes with reiser4

[Chris Wright]

* viro@parcelfarce.linux.theplanet.co.uk (viro@parcelfarce.linux.theplanet.co.uk) wrote:
> On Thu, Aug 26, 2004 at 01:11:52AM +0100, Jamie Lokier wrote:
> > Is this a problem if we treat entering a file-as-directory as crossing
> > a mount point (i.e. like auto-mounting)?
> 
> Yes - mountpoints can't be e.g. unlinked.

Could it be essentially MNT_DETACH'd?

Re: silent semantic changes with reiser4

[Hans Reiser]

viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
>  
>
>>On Wed, 25 Aug 2004, Christoph Hellwig wrote:
>>    
>>
>>>For one thing _I_ didn't decide about xattrs anyway.  And I still
>>>haven't seen a design from you on -fsdevel how you try to solve the
>>>problems with files as directories.
>>>      
>>>
>>Hey, files-as-directories are one of my pet things, so I have to side with 
>>Hans on this one. I think it just makes sense. A hell of a lot more sense 
>>than xattrs, anyway, since it allows scripts etc standard tools to touch 
>>the attributes.
>>
>>It's the UNIX way.
>>    
>>
>
>Not if you allow link(2) on them.  And not if you design and market your
>stuff as a general-purpose backdoor into kernel. 
>
What backdoor?  Please spell it out.  Plugins are not dynamically 
loadable. 

> Note how *EVERY* *DAMN*
>*OPERATION* is made possible to override by "plugins".  Which is the reason
>for deadlocks in question, BTW.
>
>Don't fool yourself - that's what Hans is selling.  Target market: ISV.
>  
>
Hunh?  No, target market is hackers who like to spend a weekend dreaming 
up funky new kinds of files.  One guy (Jason Holt, clever guy) came up 
with an idea for write only files for which even root cannot read the 
parts of the file written prior to the time root was achieved, because 
the encryption key is changed in a forward computable only direction 
with every write, and the start key is kept on another computer.  Lots 
of folks will have plugins I would never dream of.   Think of photoshop 
plugins, and what plugins did for photoshop.  Same thing will happen to 
reiser4 next year (or earlier).

>Marketed product: a set of hooks, the wider the better, no matter how
>little sense it makes.  The reason for doing that outside of core kernel:
>bypassing any review and being able to control the product being sold (see
>above).
>
>Shame that it got an actual filesystem mixed in with the marketing plans
>and general insanity...
>
>
>  
>
I am one of those free software hippy-child anarchists who thinks that 
random people should come up with ideas and contribute them.  You 
understand me well.;-)

Re: silent semantic changes with reiser4

[Matt Mackall]

On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
> 
> 
> On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> > 
> > For one thing _I_ didn't decide about xattrs anyway.  And I still
> > haven't seen a design from you on -fsdevel how you try to solve the
> > problems with files as directories.
> 
> Hey, files-as-directories are one of my pet things, so I have to side with 
> Hans on this one. I think it just makes sense. A hell of a lot more sense 
> than xattrs, anyway, since it allows scripts etc standard tools to touch 
> the attributes.
> 
> It's the UNIX way.

I thought the UNIX way is "everything's a file", not "everything's a
directory".

> Will it potentially break something? Sure. Do we care? Me, I'll take that 
> kind of extension _any_ day over xattrs, that are fundamentally flawed in 
> my opinion and totally useless.

There's always the option that they're both broken.

-- 
Mathematics is the supreme nostalgia of our time.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Wed, 25 Aug 2004, Matt Mackall wrote:
> > 
> > It's the UNIX way.
> 
> I thought the UNIX way is "everything's a file", not "everything's a
> directory".

It really was. Directories were historically largely just files too, 
although with the special "lookup" operation. 

Historic unix didn't have readdir/rmdir/mkdir/rename or really much _any_
special directory handling. Directories were just files, and you read them 
like files.

Of course, even in that early unix, "directories" were very much a 
reality even apart from the fact that they happened to be implemented 
pretty much like files. Nobody has ever claimed that the UNIX way is 
"everything is _one_ file", after all ;)

> > Will it potentially break something? Sure. Do we care? Me, I'll take that 
> > kind of extension _any_ day over xattrs, that are fundamentally flawed in 
> > my opinion and totally useless.
> 
> There's always the option that they're both broken.

Yes. Highly likely. However, something like that _does_ end up what a 
Windows fileserver wants. IOW, even if it's broken, _something_ is likely 
forced on us by that nasty thing we call "real users". Damn them.

		Linus

Re: silent semantic changes with reiser4

[Mikulas Patocka]

On Wed, 25 Aug 2004, Linus Torvalds wrote:

> On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> >
> > For one thing _I_ didn't decide about xattrs anyway.  And I still
> > haven't seen a design from you on -fsdevel how you try to solve the
> > problems with files as directories.
>
> Hey, files-as-directories are one of my pet things, so I have to side with
> Hans on this one. I think it just makes sense. A hell of a lot more sense
> than xattrs, anyway, since it allows scripts etc standard tools to touch
> the attributes.
>
> It's the UNIX way.
>
> And yes, the semantics can _easily_ be solved in very unixy ways.
>
> One way to solve it is to just realize that a final slash at the end
> implies pretty strongly that you want to treat it as a directory. So what
> you do is:
>
>  - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
>  - with the slash, it won't open _without_ O_DIRECTORY (EISDIR)
>
> Problem solved. Very user-friendly, and very intuitive.

Stupid question: who will use it? And why?

Anyone can write an userspace library, that implements function
set_attribute(char *file, char *attribute, char *value), that creates
directory ".attr/file" in file's directory and stores attribute there.
(and you can get list of attributes from shell too:
ls `echo "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/'`
). There's no need to add extra functionality to kernel and filesystem.

Advantage:
- you don't add bloat to kernel or filesystem
- you don't need to teach tar/cp -a/mc about attributes
- you won't lose attributes after editing file in vim (it creates another
file and renames it over original one)

> Will it potentially break something? Sure. Do we care? Me, I'll take that
> kind of extension _any_ day over xattrs, that are fundamentally flawed in
> my opinion and totally useless. The argument that applications like "tar"
> won't understand the file-as-directory thing is _flawed_, since legacy
> apps won't understand xattrs either.

The only way xattrs are useful is that backup/restore software doesn't
have to know about every filesystem with it's specific attributes and
every magic ioctl for setting them. Instead it can save/restore
filesystem-specific attributes without understanding what do they mean.
However there's no need why application should use them. And no
application does.

I can't imagine anyone shipping an application with "this app requires
reiser4" prerequisite. Why should anyone use it if he can store attributes
in ".attr" directory or whereever and make the application work on any OS
and any filesystem?

Mikulas

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Mikulas Patocka wrote:
> 
> Stupid question: who will use it? And why?
> 
> Anyone can write an userspace library, that implements function
> set_attribute(char *file, char *attribute, char *value), that creates
> directory ".attr/file" in file's directory and stores attribute there.
> (and you can get list of attributes from shell too:
> ls `echo "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/'`
> ). There's no need to add extra functionality to kernel and filesystem.

..and the above is, roughly, what I understand samba etc falls back on.

The problem ends up being that the above isn't in any way safe from people 
moving files around (oops, where did those attributes go?) nor does it 
have any consistency guarantees. So it only works well if _one_ 
application does this, and that application follows all the locking rules.

Is it enough? It may have to be.

> The only way xattrs are useful is that backup/restore software doesn't
> have to know about every filesystem with it's specific attributes and
> every magic ioctl for setting them. Instead it can save/restore
> filesystem-specific attributes without understanding what do they mean.
> However there's no need why application should use them. And no
> application does.

If no application does, then why back them up? Why implement them in the 
first place?

In other words - some apps obviously do want to use the. Sadly.

		Linus

Re: silent semantic changes with reiser4

[Mikulas Patocka]

> > The only way xattrs are useful is that backup/restore software doesn't
> > have to know about every filesystem with it's specific attributes and
> > every magic ioctl for setting them. Instead it can save/restore
> > filesystem-specific attributes without understanding what do they mean.
> > However there's no need why application should use them. And no
> > application does.
>
> If no application does, then why back them up? Why implement them in the
> first place?
>
> In other words - some apps obviously do want to use the. Sadly.

You can add more functionality to filesystem and use xattrs to control it.
For example:
- acls
- compress file
- encrypt file (copy user's password into task_struct and use it to
encrypt his files)
- preallocate file in 4MB contignuous chunks, becuase it needs real time
multimedia access
- sync/append-only/immutable
etc.
However there's no need why an application should care whether the file is
compressed, whether it has acls, or so. And applications don't.

And I think this is the only legitimate use for xattrs. Who else uses them
except samba? I don't see how reiser4's hybrids would help.

Mikulas

Re: silent semantic changes with reiser4

[Hans Reiser]

Streams are quite ugly.  However, if you decompose streams into all of 
the little pieces that are needed to emulate them, the pieces are quite 
nice.

For instance, inheriting stat data from a common parent is nice, and 
inheritance is nice, and being able to cat dirname/pseudos/cat and get a 
concatenation of all of the files is nice, and being able to cat 
dirname/pseudos/tar and get an archive of the directory is nice, and, 
well, if you decompose all of the features of streams into little 
features you get a bunch of fun little features much nicer than streams.

Hans

Re: silent semantic changes with reiser4

[Jamie Lokier]

Hans Reiser wrote:
> being able to cat dirname/pseudos/cat and get a 
> concatenation of all of the files is nice, and being able to cat 
> dirname/pseudos/tar and get an archive of the directory is nice

Yes.  Being able to cd into filename.tar.gz and filename.iso is also
nice, but all of these features should be supported by the VFS
generically, not in any specific filesystem, and there should be a
hook to invoke the various fun filesystem-independent handlers by name.

-- Jamie

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 11:53:30AM +0100, Jamie Lokier wrote:
> Hans Reiser wrote:
> > being able to cat dirname/pseudos/cat and get a 
> > concatenation of all of the files is nice, and being able to cat 
> > dirname/pseudos/tar and get an archive of the directory is nice
> 
> Yes.  Being able to cd into filename.tar.gz and filename.iso is also
> nice, but all of these features should be supported by the VFS
> generically, not in any specific filesystem, and there should be a
> hook to invoke the various fun filesystem-independent handlers by name.

It doesn't belong into the kernel at all.  If at all it belongs into a
userspace filesystems, but even in that case the magic detection of
which one to use is kinda hard.  You absoutely don't want to hardcode
file formats in the kernel.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> > > being able to cat dirname/pseudos/cat and get a 
> > > concatenation of all of the files is nice, and being able to cat 
> > > dirname/pseudos/tar and get an archive of the directory is nice
> > 
> > Yes.  Being able to cd into filename.tar.gz and filename.iso is also
> > nice, but all of these features should be supported by the VFS
> > generically, not in any specific filesystem, and there should be a
> > hook to invoke the various fun filesystem-independent handlers by name.
> 
> It doesn't belong into the kernel at all.  If at all it belongs into a
> userspace filesystems, but even in that case the magic detection of
> which one to use is kinda hard.  You absoutely don't want to hardcode
> file formats in the kernel.

Oh, I agree userspace should be involved.

-- Jamie

Re: silent semantic changes with reiser4

[Spam]

> On Thu, Aug 26, 2004 at 11:53:30AM +0100, Jamie Lokier wrote:
>> Hans Reiser wrote:
>> > being able to cat dirname/pseudos/cat and get a 
>> > concatenation of all of the files is nice, and being able to cat 
>> > dirname/pseudos/tar and get an archive of the directory is nice
>> 
>> Yes.  Being able to cd into filename.tar.gz and filename.iso is also
>> nice, but all of these features should be supported by the VFS
>> generically, not in any specific filesystem, and there should be a
>> hook to invoke the various fun filesystem-independent handlers by name.

> It doesn't belong into the kernel at all.  If at all it belongs into a
> userspace filesystems, but even in that case the magic detection of
> which one to use is kinda hard.  You absoutely don't want to hardcode
> file formats in the kernel.

  Do  you  mean  user-level  file  system  as  a  VFS  handled by user
  applications,  or  a  intermediate  file  system  layer  between any
  application  and  the  real  file  system?  The latter would be good
  enough as it would still be transparent to the applications.

  ~S
  

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Mikulas Patocka wrote:
> On Wed, 25 Aug 2004, Linus Torvalds wrote:

> > One way to solve it is to just realize that a final slash at the end
> > implies pretty strongly that you want to treat it as a directory. So what
> > you do is:
> 
> Stupid question: who will use it? And why?

I've got a stupid question too.  How do you back up these
things ?

If your backup program reads them as a file and restores
them as a file, you might lose your directory-inside-the-file
magic.

If your backup program dives into the file despite stat()
saying it's a file and you restore your backup, how are the
"file is a file" semantics preserved ?

Obviously this is something that needs to be sorted out at
the VFS layer.  A filesystem specific backup and restore
program isn't desirable, if only because then there'd be
no way for Hans's users to switch to reiser5 in 2010 ;)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Helge Hafting]

Rik van Riel wrote:

>On Thu, 26 Aug 2004, Mikulas Patocka wrote:
>  
>
>>On Wed, 25 Aug 2004, Linus Torvalds wrote:
>>    
>>
>
>  
>
>>>One way to solve it is to just realize that a final slash at the end
>>>implies pretty strongly that you want to treat it as a directory. So what
>>>you do is:
>>>      
>>>
>>Stupid question: who will use it? And why?
>>    
>>
>
>I've got a stupid question too.  How do you back up these
>things ?
>
>If your backup program reads them as a file and restores
>them as a file, you might lose your directory-inside-the-file
>magic.
>
>If your backup program dives into the file despite stat()
>saying it's a file and you restore your backup, how are the
>"file is a file" semantics preserved ?
>
>  
>
>Obviously this is something that needs to be sorted out at
>the VFS layer.  A filesystem specific backup and restore
>program isn't desirable, if only because then there'd be
>no way for Hans's users to switch to reiser5 in 2010 ;)
>
>  
>
Sure, this sort of thing must be sorted out at the VFS layer.
And a backup program working on such a filesystem
will need to know that something can be a file, a directory - or both.

So an old "tar" won't get this right as it will assume that an object
is either file or directory.  The change to get it right won't be
that big - just notice that an object is both, then backup the
ordinary file contents as usual, before recursing into the
directory it also provides and backup stuff there as usual.

The resulting .tar can of course only be unpacked properly
on a fs supporting file-as-directory, similiar to how a .tar of
a fs with links only will unpack properly on a fs supporting links.

I don't see much problems for userland.  Old apps will keep working,
as the new features is a superset.  Those who care about
file-as-directory extras will provide patches for "tar" and friends,
after that the extras become useable.

Helge Hafting

Re: silent semantic changes with reiser4

[Paul Jackson]

Helge wrote:
> So an old "tar" won't get this right as it will assume that an object
> is either file or directory.

There are many backup apps, not just one.  I've written a few myself,
none of which will ever be worthy of notice.  The sourceforge
Topic.System.Archiving.Backup lists 335 projects at present.

I find the idea that most backup tools and scripts will silently
stop working correctly to be pretty scary.  

And then there's archiving, installation, distribution, administration,
emulation, file system and partition managers, and on and on.

===

I wonder if we can make this "modal" somehow.

The one consistency I see is that apps that want the "enhanced" view
need to ask for it, somehow.  It is the new views of the data that are
being added - let the app announce to the kernel (usually via
specialized code in some shared library that the app is using to get the
alternate views) that either per-task, or per-file descriptor, it is to
see the "enhanced" view, as a side affect of trying to access it.

Old stuff, or even new stuff that is content to work with the "classic"
view that a file is a single data stream, and that directories only
have pathnames, not data, would by default see that view, and see
_all_ the data, presented somehow in that view, perhaps as additional
files with magic names.

This still leaves the breakage that such tools don't know, and don't
preserve, the magic linkage between such magic files.  But that is
much less of an issue, in my view.  Programs such as backups that are
manipulating the files of apps they know nothing about already have
to presume that all the files are important in inscrutable ways, and
just be careful to preserve or copy or backup all of them.

Yeah - I realize that there will be a few followups denouncing modal
architectures.  I might even agree with some of them.

If this were easy, it would have been done years ago.

The onus should be on the new stuff to request the enhanced view,
rather than on the old dogs to learn new tricks.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Jamie Lokier]

Paul Jackson wrote:
> I find the idea that most backup tools and scripts will silently
> stop working correctly to be pretty scary.  

Too late.  We have xattrs already; many programs don't store them.

-- Jamie

Re: silent semantic changes with reiser4

[Paul Jackson]

> Too late.  We have xattrs already; many programs don't store them.

If by your "too late" you mean we can stop worrying about any more
breakage of file system utilities, because there exists an example
in which some were already broken, then you are absolutely wrong.

Just because we caused some breakage doesn't give us license to cause
even more.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Helge Hafting]

Paul Jackson wrote:

>Helge wrote:
>  
>
>>So an old "tar" won't get this right as it will assume that an object
>>is either file or directory.
>>    
>>
>
>There are many backup apps, not just one.  I've written a few myself,
>none of which will ever be worthy of notice.  The sourceforge
>Topic.System.Archiving.Backup lists 335 projects at present.
>
>I find the idea that most backup tools and scripts will silently
>stop working correctly to be pretty scary.  
>  
>
They won't stop working, they will merely not support
the new features.  That is only a problem if
you actually use those features.  If all you have
is plain files and plain directories - no problem.  Not
if files-as-directories are implemented right.

Helge Hafting

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>
>
>If your backup program dives into the file despite stat()
>saying it's a file and you restore your backup, how are the
>"file is a file" semantics preserved ?
>
>Obviously this is something that needs to be sorted out at
>the VFS layer.
>
It needs to be sorted out, whether it is sorted out at the VFS layer is 
unimportant.

>  A filesystem specific backup and restore
>program isn't desirable, if only because then there'd be
>no way for Hans's users to switch to reiser5 in 2010 ;)
>
>  
>
It might be that we need a filenameA/metas/backup method for all of our 
file plugins, which if cat'd gives a set of instructions which if 
executed are adequate for restoring filenameA.

Hans

Re: silent semantic changes with reiser4

[Lars Marowsky-Bree]

On 2004-08-26T01:40:29,
   Hans Reiser <reiser@namesys.com> said:

> >Obviously this is something that needs to be sorted out at
> >the VFS layer.
> >
> It needs to be sorted out, whether it is sorted out at the VFS layer is 
> unimportant.

So what exactly is wrong with sorting it out at the VFS layer, and why
do you _insist_ on sorting it in the reiserfs4 core? I'm missing
something, please fill me in on the details.


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering	   \\\  /// 
SUSE Labs, Research and Development \honk/ 
SUSE LINUX AG - A Novell company     \\// 

Re: silent semantic changes with reiser4

[Paul Jackson]

Rik van Riel writes:
> If your backup program reads them as a file and restores
> them as a file, you might lose your directory-inside-the-file
> magic.

Encode the magic in the names, by stealing a bit of the existing
filename space to encode it.

Such works pretty well as part of the magic to map long filenames
into DOS 8.3 names on my FAT partitions.

Apps linked with the appropriate Windows library see nice fancy
long names.

The rest of the world, including DOS apps and my Unix backup
scripts, see the primitive 8.3 names, including one or a few
extra files per directory, which are nothing special to them.

So long as these other apps don't presume to know that they can
keep some of the files in an apps directory, and drop others, then
it works well enough.  And no self-respecting general purpose
backup program is going to presume such knowledge anyway.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Wed, 25 Aug 2004, Christoph Hellwig wrote:
>  
>
>>For one thing _I_ didn't decide about xattrs anyway.  And I still
>>haven't seen a design from you on -fsdevel how you try to solve the
>>problems with files as directories.
>>    
>>
>
>Hey, files-as-directories are one of my pet things, so I have to side with 
>Hans on this one. I think it just makes sense. A hell of a lot more sense 
>than xattrs, anyway, since it allows scripts etc standard tools to touch 
>the attributes.
>
>It's the UNIX way.
>
>And yes, the semantics can _easily_ be solved in very unixy ways.
>
>One way to solve it is to just realize that a final slash at the end 
>implies pretty strongly that you want to treat it as a directory. So what 
>you do is:
>
> - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
> - with the slash, it won't open _without_ O_DIRECTORY (EISDIR)
>
>Problem solved. Very user-friendly, and very intuitive.
>
>Will it potentially break something? Sure. Do we care? Me, I'll take that 
>kind of extension _any_ day over xattrs, that are fundamentally flawed in 
>my opinion and totally useless. The argument that applications like "tar" 
>won't understand the file-as-directory thing is _flawed_, since legacy 
>apps won't understand xattrs either.
>
>Oh, add a O_NOXATTRS flag to force a path lookup to only use regular
>directories, the same way we have O_NOFOLLOW and friends. That allows
>people to see the difference, if they care (ie a file server might decide
>that it doesn't want to expose things like this).
>  
>
I think we should require people to care enough to supply an O_NOMETAS 
flag to see the difference.

>I never liked the xattr stuff. It makes little sense, and is totally 
>useless for 99.9999% of everything. I still don't see the point of it, 
>except for samba. Ugly.
>
>		Linus
>
>
>  
>

Re: silent semantic changes with reiser4

[Michael Halcrow]

[-- Attachment #1: Type: text/plain, Size: 5340 bytes --]

If I may chime in here...

On Wed, Aug 25, 2004 at 01:22:55PM -0700, Linus Torvalds wrote:
> On Wed, 25 Aug 2004, Christoph Hellwig wrote:
> > 
> > For one thing _I_ didn't decide about xattrs anyway.  And I still
> > haven't seen a design from you on -fsdevel how you try to solve
> > the problems with files as directories.
> 
> Hey, files-as-directories are one of my pet things, so I have to
> side with Hans on this one. I think it just makes sense. A hell of a
> lot more sense than xattrs, anyway, since it allows scripts etc
> standard tools to touch the attributes.
> 
> It's the UNIX way.

This is an issue that directly affects work I am doing in extended
cryptfs:

http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55
http://halcrow.us/~mhalcrow/ols2004.pdf
http://halcrow.us/~mhalcrow/ols_cryptfs.sxi

The basic idea is that the cryptographic context for every file is
correlated with the individual file via xattr's.  A file is a unit of
data that should, as it stands, contain all the information requisite
for the encrypting filesystem layer to transparently decrypt (and
encrypt, when the file is written to).  This allows for a key->file
granularity, as opposed to a key->block device (dm-crypt) or a
key->mount point (CFS) granularity.

My grand vision is to have a policy that determines whether or not the
encrypted version of the file or the decrypted version of the file is
read, dependent on whether or not the file is leaving the security
domain (the storage device under the control of the currently running
kernel).  For example, if the ``cp'' command is copying a file from a
filesystem mounted from /dev/hda1 to a filesystem mounted from
/dev/fd0, then the policy would indicate that (unless otherwise noted
in the .cryptfsrc file in the root of the filesystem mounted from
/dev/fd0, which might also contain the default security context for
that filesystem or directory - like whose public keys should be used
to encrypt the symmetric key for data) the file is leaving the
security domain, and the encrypted contents of the file should be
given to cp.  Same with mutt reading an email attachment (as opposed
to, say, .muttrc, where, more likely than not, the unencrypted version
is wanted).

The goal is to enable an ``encrypted by default'' policy, in which
files on the storage devices are independent encrypted units that
remain encrypted until an application that actually needs to see the
decrypted contents opens them.  Then the encryption and decryption is
done transparently by the fs layer, as long as the user has the right
keys.  Extended attributes seem like a natural way to store this
context.

Once you consider that you can have a crypto context for each file,
you can start doing other neat tricks, like keyed hashes over extents
within the file, to allow for dynamic integrity verification during
the read.  If an offset of 1.5 gigabytes into a 2-gigabyte has been
tampered with, then that tampering will be caught when that portion of
the file is read; you don't have to verify the hash of the entire
2-gigabyte file at the time of the open.  Of course, this would very
rapidly overrun the available xattr storage size.  And so to
realistically implement something like this, some new underlying file
format is in order.

In any case, the issue of userspace applications supporting extended
attributes is key to the viability of this approach.  If cp, uuencode,
tar, or what not do not preserve the extended attributes, then the
crypto context is lost, and the file is unreadable.  So the $64,000
question is, just how committed is the community to this whole concept
of extended attributes?  From this point, should I assume that good
xattr support is forthcoming, or should I abandon the idea of using
xattr's for this altogether?

One solution I've been kicking around is to make cryptfs
GnuPG-compatible.  Not only would this eliminate the need to store
some of the crypto context in the xattr set, but it would also
preserve the crypto context with apps that don't know about xattr's,
and it would be possible for users who are not running cryptfs to read
the files with gpg.  Keyed hashes over extents would be doable if
GnuPG allowed for opaque data blobs in the file that gpg would just
ignore when decrypting the file (gnupg-dev list had technical issues
last time I tried to post these ideas to it - any gpg guys around that
can comment on this?).

> I never liked the xattr stuff. It makes little sense, and is totally 
> useless for 99.9999% of everything. I still don't see the point of it, 
> except for samba. Ugly.

If xattr's wind up getting supported by a certain critical mass of
applications, then they are somewhat useful for me, although, as
currently implemented, are insufficient for what I really need (keyed
hashes over extents require too much space).

BTW, early this week I migrated cryptfs over to use David Howell's new
keyring, which is working out nicely.

Mike
.___________________________________________________________________.
                         Michael A. Halcrow                          
       Security Software Engineer, IBM Linux Technology Center       
GnuPG Fingerprint: 05B5 08A8 713A 64C1 D35D  2371 2D3C FDDA 3EB6 601D

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Hans Reiser]

Michael Halcrow wrote:

> This is an issue that directly affects work I am doing in extended
>
>cryptfs:
>
>http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55
>http://halcrow.us/~mhalcrow/ols2004.pdf
>http://halcrow.us/~mhalcrow/ols_cryptfs.sxi
>
>  
>
Reiser4 has an encryption plugin that will ship sometime this year.  You 
might want to talk to edward@namesys.com about it.

Re: silent semantic changes with reiser4

[Alan Cox]

On Mer, 2004-08-25 at 21:22, Linus Torvalds wrote:
>  - without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
>  - with the slash, it won't open _without_ O_DIRECTORY (EISDIR)
> 
> Problem solved. Very user-friendly, and very intuitive.

Breaks bash and almost all other file completion 8)

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Wed, Aug 25, 2004 at 10:08:59PM +0200, Christoph Hellwig wrote:
> On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote:
> > You ignored everything I said during the discussion of xattrs about how 
> > there is no need to have attributes when you can just have files and 
> > directories, and that xattrs reflected a complete ignorance of name 
> > space design principles.
> 
> Actually in most of the discussion you simply didn't participate.  While
> xattrs might not be the nicest interface they have the advantag of not
> breaking the SuS assumption of what directories vs files are, and they
> do not break the Linux O_DIRECTORY semantics that are defined and need
> to solve real-world races either.

Reiser4 may have a mount option for whoose who like or have to use traditional
O_DIRECTORY semantics.  There would be no /metas under non-directories, if user
wants that.

[ ... ]

-- 
Alex.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 12:35:17AM +0400, Alex Zarochentsev wrote:
> Reiser4 may have a mount option for whoose who like or have to use traditional
> O_DIRECTORY semantics.  There would be no /metas under non-directories, if user
> wants that.

Again, O_DIRECTORY was added to solve a real-world race, not just for
the sake of it.  If you really want to add that option I'll research the
CAN number for you so you can named it after that - or just call it -o
insecure directly.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> > Reiser4 may have a mount option for whoose who like or have to use
> > traditional O_DIRECTORY semantics.  There would be no /metas under
> > non-directories, if user wants that.
> 
> Again, O_DIRECTORY was added to solve a real-world race, not just for
> the sake of it.  If you really want to add that option I'll research the
> CAN number for you so you can named it after that - or just call it -o
> insecure directly.

man open(2) explains that O_DIRECTORY is used by opendir() to prevent
blocking when opening pipes and certain devices*, and should only by
used by opendir (of course it isn't only used by opendir, as it's a
handy optimisation).

In fact O_DIRECTORY is also used by Glibc to optimise away stat()
before and fstat() after calls.

An O_NODEVICE flag would be equally secure, and more generally useful.

It's important that device nodes cannot be opened when O_DIRECTORY is
set.  This is compatible with reiser4 file-as-directory semantics, but
I don't know if reiser4 actually implements this.  If it does (and it
should) then there is no device blocking problem.

That leaves only the optimisation of fstat() in opendir().

But that begs the question: do we want opendir() to succeed on a reiser4 file?

It's up to us to decide if we like that semantic, or prefer a
different one such as the one I described before (path search enters
file-as-directory, but opendir() directly on it fails), or the Sun
syscalls someone mentioned.

-- Jamie


* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on
many systems.  It's best if you can avoid opening devices at all though.

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>Christoph Hellwig wrote:
>  
>
>>>Reiser4 may have a mount option for whoose who like or have to use
>>>traditional O_DIRECTORY semantics.  There would be no /metas under
>>>non-directories, if user wants that.
>>>      
>>>
>>Again, O_DIRECTORY was added to solve a real-world race, not just for
>>the sake of it.  If you really want to add that option I'll research the
>>CAN number for you so you can named it after that - or just call it -o
>>insecure directly.
>>    
>>
>
>man open(2) explains that O_DIRECTORY is used by opendir() to prevent
>blocking when opening pipes and certain devices*, and should only by
>used by opendir (of course it isn't only used by opendir, as it's a
>handy optimisation).
>
>In fact O_DIRECTORY is also used by Glibc to optimise away stat()
>before and fstat() after calls.
>
>An O_NODEVICE flag would be equally secure, and more generally useful.
>
>It's important that device nodes cannot be opened when O_DIRECTORY is
>set.  This is compatible with reiser4 file-as-directory semantics, but
>I don't know if reiser4 actually implements this.  If it does (and it
>should) then there is no device blocking problem.
>
>That leaves only the optimisation of fstat() in opendir().
>
>But that begs the question: do we want opendir() to succeed on a reiser4 file?
>  
>
Yes, we do.  Why not?  Not is more complex.....

>It's up to us to decide if we like that semantic, or prefer a
>different one such as the one I described before (path search enters
>file-as-directory, but opendir() directly on it fails), or the Sun
>syscalls someone mentioned.
>
>-- Jamie
>
>
>* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on
>many systems.  It's best if you can avoid opening devices at all though.
>
>
>  
>

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 12:54:09AM +0100, Jamie Lokier wrote:
> Christoph Hellwig wrote:
> > > Reiser4 may have a mount option for whoose who like or have to use
> > > traditional O_DIRECTORY semantics.  There would be no /metas under
> > > non-directories, if user wants that.
> > 
> > Again, O_DIRECTORY was added to solve a real-world race, not just for
> > the sake of it.  If you really want to add that option I'll research the
> > CAN number for you so you can named it after that - or just call it -o
> > insecure directly.
> 
> man open(2) explains that O_DIRECTORY is used by opendir() to prevent
> blocking when opening pipes and certain devices*, and should only by
> used by opendir (of course it isn't only used by opendir, as it's a
> handy optimisation).
> 
> In fact O_DIRECTORY is also used by Glibc to optimise away stat()
> before and fstat() after calls.
> 
> An O_NODEVICE flag would be equally secure, and more generally useful.

It would prevent the open device issue but you'd need another syscall.
Anyways, this is an existing API and chaging it now would suddenly
making everything that relies on it for that purpose insecure, so it's
completely out of option.

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>On Thu, Aug 26, 2004 at 12:35:17AM +0400, Alex Zarochentsev wrote:
>  
>
>>Reiser4 may have a mount option for whoose who like or have to use traditional
>>O_DIRECTORY semantics.  There would be no /metas under non-directories, if user
>>wants that.
>>    
>>
>
>Again, O_DIRECTORY was added to solve a real-world race, not just for
>the sake of it. 
>
Can you supply more details and we will try to reply concretely?  Thanks.

> If you really want to add that option I'll research the
>CAN number for you so you can named it after that - or just call it -o
>insecure directly.
>
>
>
>  
>

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 01:43:59AM -0700, Hans Reiser wrote:
> >Again, O_DIRECTORY was added to solve a real-world race, not just for
> >the sake of it. 
> >
> Can you supply more details and we will try to reply concretely?  Thanks.

It's really userland programming 101 :)

 you need to fstat after open to really check that no one swtiched the
 path below you to a fifo or device, O_DIRECTORY gets that directly in
 the open call, with the additional benefit of making sure you don't
 get any of the sideeffects at all that would happen if you're opening
 a device file.

 the current reiser4 semantics break that and as soon as you're having a
 world-writeable (e.g. /tmp) dir on it and someone is doing an opendir
 on it he's lost.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
>  you need to fstat after open to really check that no one swtiched the
>  path below you to a fifo or device, O_DIRECTORY gets that directly in
>  the open call, with the additional benefit of making sure you don't
>  get any of the sideeffects at all that would happen if you're opening
>  a device file.
> 
>  the current reiser4 semantics break that and as soon as you're having a
>  world-writeable (e.g. /tmp) dir on it and someone is doing an opendir
>  on it he's lost.

How does the current reiser4 semantics break that?

In a reiser4 filesystem, a file _is_ a directory.
opendir() is supposed to succeed on it.

There's bound to be some security issue, but I'm not sure what you're
getting at with /tmp.  What sort of sort of security problem arises
with a world-writeable directory such as /tmp, that cannot arise with
the standard fs semantics?

-- Jamie

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 02:48:12PM +0100, Jamie Lokier wrote:
> >  the current reiser4 semantics break that and as soon as you're having a
> >  world-writeable (e.g. /tmp) dir on it and someone is doing an opendir
> >  on it he's lost.
> 
> How does the current reiser4 semantics break that?
> 
> In a reiser4 filesystem, a file _is_ a directory.
> opendir() is supposed to succeed on it.
> 
> There's bound to be some security issue, but I'm not sure what you're
> getting at with /tmp.  What sort of sort of security problem arises
> with a world-writeable directory such as /tmp, that cannot arise with
> the standard fs semantics?

Actually you are right on that issue because it would open the
device/fifo as directory and not device/fifo (in fact I'd had to look at
the code again to see whether they actually do this only for files or
also for special files)

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> > There's bound to be some security issue, but I'm not sure what you're
> > getting at with /tmp.  What sort of sort of security problem arises
> > with a world-writeable directory such as /tmp, that cannot arise with
> > the standard fs semantics?
> 
> Actually you are right on that issue because it would open the
> device/fifo as directory and not device/fifo (in fact I'd had to look at
> the code again to see whether they actually do this only for files or
> also for special files)

Are you saying that with reiser4, you can open a device or fifo with
O_DIRECTORY?

-- Jamie

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 05:06:38PM +0100, Jamie Lokier wrote:
> Christoph Hellwig wrote:
> > > There's bound to be some security issue, but I'm not sure what you're
> > > getting at with /tmp.  What sort of sort of security problem arises
> > > with a world-writeable directory such as /tmp, that cannot arise with
> > > the standard fs semantics?
> > 
> > Actually you are right on that issue because it would open the
> > device/fifo as directory and not device/fifo (in fact I'd had to look at
> > the code again to see whether they actually do this only for files or
> > also for special files)
> 
> Are you saying that with reiser4, you can open a device or fifo with
> O_DIRECTORY?

That's what I thought, but as far as I can follow the code this is not
actually true.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Christoph Hellwig wrote:
>
> > Are you saying that with reiser4, you can open a device or fifo with
> > O_DIRECTORY?
> 
> That's what I thought, but as far as I can follow the code this is not
> actually true.

It should be possible to do, though. There's nothing really different in 
making the "default" (unnamed) fork be a special device or a fifo.

And it would be perfectly ok for O_DIRECTORY to open such a file, as long
as it opens the directory branch, not the special device.

I advocated (long ago) something like this for /dev handling, just because
I think it would make sense to have

	/dev/hda	<- special file
	/dev/hda/part1	<- partition 1 (aka /dev/hda1)

which just seems like a very obvious and intuitive interface to me. Of 
course, we have so much legacy in /dev that there's no real point to doing 
this, but it's still an appealing approach, I think.

But I do take Al's concerns seriously. I like the notion of supporting
"containers", but there are undoubtedly serious issues in the notion. I
don't strictly know exactly _how_ to implement it sanely (I can talk about
using the vfsmnt structure all I like, but the fact is, it's a different
thing from a normal mount, and there may be serious problems indeed
there).

Still, I really do like the idea of merging the notion of file and
directory into one notion of "container". I absolutely _detest_ files with
internal structure that tools have to know about (ie I hate seeing all
those embedded formats that I can't use "grep" on - MIME being one case).
I'd much rather see a "group of files"  and a "file with a grouping of
information".

(Now, flattening that "group of files" is obviously needed for serial 
protocols, so I think MIME/tar/xxxx are fine for _transporting_ data, but 
I'm saying that outside of transport I really prefer a "collection of 
files" approach).

		Linus

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> And it would be perfectly ok for O_DIRECTORY to open such a file, as long
> as it opens the directory branch, not the special device.

What about:

    cd /dev/into_directory_branch/
    run_setuid_program

        -> calls pwd
           pwd opens("."), (".."), ("../..") etc.
 
        -> the setuid program thus ends up opening a device or fifo,
           when it does pwd's path walk.  Yes it could use the getcwd
           syscall, but some programs do their own path walk.

> I advocated (long ago) something like this for /dev handling, just because
> I think it would make sense to have
> 
> 	/dev/hda	<- special file
> 	/dev/hda/part1	<- partition 1 (aka /dev/hda1)
> 
> which just seems like a very obvious and intuitive interface to me. Of 
> course, we have so much legacy in /dev that there's no real point to doing 
> this, but it's still an appealing approach, I think.

It also fits the container idea very well:

        /dev/hda/part1/ <- the filesystem inside partition 1

That's not a contrivance, it's what I'd expect to happen if
/dev/hda is an ordinary file containing a disk image:

 	image.bin	 <- hard disk image (a regular file)
 	image.bin/part1	 <- partition 1
 	image.bin/part1/ <- the filesystem inside partition 1

That's assuming there's a format handler which recognises that image.

-- Jamie

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Jamie Lokier wrote:
>
>     run_setuid_program
> 
>         -> calls pwd
>            pwd opens("."), (".."), ("../..") etc.

Ehh.. Not only does pwd not do that..

(hint: there's a getcwd() system call)

>         -> the setuid program thus ends up opening a device or fifo,
>            when it does pwd's path walk.  Yes it could use the getcwd
>            syscall, but some programs do their own path walk.

.. but even if it did that, it should use O_DIRECTORY when it did so. If 
it doesn't, it's broken.

So no, it would _not_ open the device or fifo when it did so.

The fact is, anything that expects to open a directory should already be 
opening it with O_DIRECTORY.

That said, ".." and "." are special already inside the kernel, and it 
migth be worth making them automatically imply O_DIRECTORY, since nothing 
else makes sense anyway. That would fix the case where somebody uses ".." 
_without_ using O_DIRECTORY. 

> It also fits the container idea very well:
> 
>         /dev/hda/part1/ <- the filesystem inside partition 1

I don't think you can do that. The kernel has no idea how to mount the
filesystem.

If it's already mounted somewhere else, that's a different issue.  
Although it might be mounted in several places (as a bind mount) with
different writability, I guess, so even then it might be "interesting".

		Linus

Re: silent semantic changes with reiser4

[viro]

On Thu, Aug 26, 2004 at 10:29:45AM -0700, Linus Torvalds wrote:
> >         /dev/hda/part1/ <- the filesystem inside partition 1
> 
> I don't think you can do that. The kernel has no idea how to mount the
> filesystem.

... and what's more, fundamental thing for mount is fs type, what to
do with device(s) (if any) and any other arguments is the filesystem's
business.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> >         -> the setuid program thus ends up opening a device or fifo,
> >            when it does pwd's path walk.  Yes it could use the getcwd
> >            syscall, but some programs do their own path walk.
> 
> .. but even if it did that, it should use O_DIRECTORY when it did so. If 
> it doesn't, it's broken.

Didn't someone just say that O_DIRECTORY will succeed on a device,
precisely because opendir() is supposed to succeed on the device?

I'm not sure.  Let me refresh my memory.

Linus Torvalds wrote:
> And it would be perfectly ok for O_DIRECTORY to open such a file, as long
> as it opens the directory branch, not the special device.

Oh, I see.  You're right.  ".." sees the path of directory branches.

> > It also fits the container idea very well:
> > 
> >         /dev/hda/part1/ <- the filesystem inside partition 1
> 
> I don't think you can do that. The kernel has no idea how to mount the
> filesystem.

It is not the kernel which decides.  The filesystem containing
/dev/hda/part1 opens "the directory branch".  If the filesystem is
suitably configured, it's exactly the same as opening a .zip or ELF
file: it asks userspace for help.  Userspace helps by either mounting
it, or creating a container virtual directory -- in exactly the same
way it helps with .zip and ELF files by creating a container virtual
directory for those.  It would mount it if the kernel supports the fs,
and create a virtual directory if the kernel doesn't support the fs
and userspace does.

This is exactly the same as when you cd into a filesystem image: the
helper mounts (this time with a lookback file mount) if the kernel
supports the image fs, otherwise the helper may be able to decode it
in userspace.

By the way, these mechanisms could allow some of the old filesystems
that nobody uses any more to be removed from the kernel altogether and
kept in an archive of userspace-only supported filesystems.  What do
you think of that idea?

> If it's already mounted somewhere else, that's a different issue.  
> Although it might be mounted in several places (as a bind mount) with
> different writability, I guess, so even then it might be "interesting".

The obvious implementation has the userspace helper just mounting it,
end of story.  If the mount command fails, it fails.  Much like autofs.

-- Jamie

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Jamie Lokier wrote:
>
> > .. but even if it did that, it should use O_DIRECTORY when it did so. If 
> > it doesn't, it's broken.
> 
> Didn't someone just say that O_DIRECTORY will succeed on a device,
> precisely because opendir() is supposed to succeed on the device?

It will succeed on the _name_ of the device, but it won't open the device 
node. It would open the _directory_ node associated with that name.

Think of it this way: a pathname always points to one "container". That 
container is either a directory or a single node - and O_DIRECTORY ends up 
being the thing that chooses between them.

> > I don't think you can do that. The kernel has no idea how to mount the
> > filesystem.
> 
> It is not the kernel which decides.  The filesystem containing
> /dev/hda/part1 opens "the directory branch".

But that filesystem cannot know what the _other_ filesystem configurations 
are. And that's what you'd have to have to mount.

> The obvious implementation has the userspace helper just mounting it,
> end of story.  If the mount command fails, it fails.  Much like autofs.

Yes, that would work, but it's of questionable use. If you want autofs, 
then just _use_ autofs.

		Linus

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> > It is not the kernel which decides.  The filesystem containing
> > /dev/hda/part1 opens "the directory branch".
> 
> But that filesystem cannot know what the _other_ filesystem configurations 
> are. And that's what you'd have to have to mount.

At which point, userspace comes in.  Who knows what userspace chooses.

> > The obvious implementation has the userspace helper just mounting it,
> > end of story.  If the mount command fails, it fails.  Much like autofs.
> 
> Yes, that would work, but it's of questionable use. If you want autofs, 
> then just _use_ autofs.

I don't mean to imply it's useful behaviour: only that it seems to
happen by default if you don't do anything special to prevent it, if
you see what I mean.

It does seem useful to have _files_ mounted automagically (using
loopback devices): for example, cd into a .iso file and it would be
nice to see the contents.  At this point, deciding on the preferred
-EBUSY or not behaviour becomes relevant, as does deciding whether the
.iso in this example is viewed using the kernel's isofs code, or using
a userspace helper in the same way as we'd use one to browse a .zip file.

-- Jamie

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Thu, Aug 26, 2004 at 10:29:45AM -0700, Linus Torvalds wrote:
> 
> 
> On Thu, 26 Aug 2004, Jamie Lokier wrote:
> >
> >     run_setuid_program
> > 
> >         -> calls pwd
> >            pwd opens("."), (".."), ("../..") etc.
> 
> Ehh.. Not only does pwd not do that..
> 
> (hint: there's a getcwd() system call)
> 
> >         -> the setuid program thus ends up opening a device or fifo,
> >            when it does pwd's path walk.  Yes it could use the getcwd
> >            syscall, but some programs do their own path walk.
> 
> .. but even if it did that, it should use O_DIRECTORY when it did so. If 
> it doesn't, it's broken.
> 
> So no, it would _not_ open the device or fifo when it did so.
> 
> The fact is, anything that expects to open a directory should already be 
> opening it with O_DIRECTORY.

reiser4 files-as-dirs do not depend on that.

A file behaves as directory if one calls i_op->lookup() or f_op->readdir().  I
am not sure that O_DIRECTORY should be a switch. 

> That said, ".." and "." are special already inside the kernel, and it 
> migth be worth making them automatically imply O_DIRECTORY, since nothing 
> else makes sense anyway. That would fix the case where somebody uses ".." 
> _without_ using O_DIRECTORY. 
> 
> > It also fits the container idea very well:
> > 
> >         /dev/hda/part1/ <- the filesystem inside partition 1
> 
> I don't think you can do that. The kernel has no idea how to mount the
> filesystem.
> 
> If it's already mounted somewhere else, that's a different issue.  
> Although it might be mounted in several places (as a bind mount) with
> different writability, I guess, so even then it might be "interesting".
> 
> 		Linus

-- 
Alex.

Re: silent semantic changes with reiser4

[Giuliano Pochini]

On 26-Aug-2004 Linus Torvalds wrote:

> I advocated (long ago) something like this for /dev handling, just because
> I think it would make sense to have
>
>       /dev/hda        <- special file
>       /dev/hda/part1  <- partition 1 (aka /dev/hda1)

This breaks the r4 semantics if I understood it correctly. Because
/partN are not simply associated to the file, they are part of the
file. ie. when I modify /dev/hda I also change /dev/hda/partN and
vice-versa. I don't see any pratical problem, though.


> Still, I really do like the idea of merging the notion of file and
> directory into one notion of "container". I absolutely _detest_ files with
> internal structure that tools have to know about (ie I hate seeing all
> those embedded formats that I can't use "grep" on - MIME being one case).
> I'd much rather see a "group of files"  and a "file with a grouping of
> information".

You're actually looking for a database with a legacy fs-like interface :)


--
Giuliano.

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Fri, Aug 27, 2004 at 11:18:55AM +0200, Giuliano Pochini wrote:
> 
> On 26-Aug-2004 Linus Torvalds wrote:
> 
> > I advocated (long ago) something like this for /dev handling, just because
> > I think it would make sense to have
> >
> >       /dev/hda        <- special file
> >       /dev/hda/part1  <- partition 1 (aka /dev/hda1)
> 
> This breaks the r4 semantics if I understood it correctly. 

Reiser4 currently links /metas/ directory under each reiser4 fs object. 
Reiser4 "silently" allows regular file to be a directory for that.

That is only an example how an fs object may link its internal hierarchical
structure to the fs tree.  /dev/hda/-as-a-dir is another example, but, I am
afraid, it is not a reiser4 business to look into the devices.  Devfs or its
successor may do it. 

> Because
> /partN are not simply associated to the file, they are part of the
> file. ie. when I modify /dev/hda I also change /dev/hda/partN and
> vice-versa. I don't see any pratical problem, though.
> 
> 
> > Still, I really do like the idea of merging the notion of file and
> > directory into one notion of "container". I absolutely _detest_ files with
> > internal structure that tools have to know about (ie I hate seeing all
> > those embedded formats that I can't use "grep" on - MIME being one case).
> > I'd much rather see a "group of files"  and a "file with a grouping of
> > information".
> 
> You're actually looking for a database with a legacy fs-like interface :)
> 
> 
> --
> Giuliano.

-- 
Alex.

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Thu, Aug 26, 2004 at 06:13:03PM +0200, Christoph Hellwig wrote:
> On Thu, Aug 26, 2004 at 05:06:38PM +0100, Jamie Lokier wrote:
> > Christoph Hellwig wrote:
> > > > There's bound to be some security issue, but I'm not sure what you're
> > > > getting at with /tmp.  What sort of sort of security problem arises
> > > > with a world-writeable directory such as /tmp, that cannot arise with
> > > > the standard fs semantics?
> > > 
> > > Actually you are right on that issue because it would open the
> > > device/fifo as directory and not device/fifo (in fact I'd had to look at
> > > the code again to see whether they actually do this only for files or
> > > also for special files)
> > 
> > Are you saying that with reiser4, you can open a device or fifo with
> > O_DIRECTORY?
> 
> That's what I thought, but as far as I can follow the code this is not
> actually true.

All reiser4 inodes have i_ops->lookup != NULL, so open(..., O_DIRECTORY) would
succeed on them (thanks Nikita for reminding me that).

It may be better to pass the control of that to ->i_op->permission() through 
explicit MAY_LOOKUP flag.  It can be possible to eliminate the race by some
more complex logic (strict ownership check, for example) in ->permission().

-- 
Alex.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote:
> 
> You ignored everything I said during the discussion of xattrs about how 
> there is no need to have attributes when you can just have files and 
> directories, and that xattrs reflected a complete ignorance of name 
> space design principles.  When I said we should just add some nice 
> optional features to files and directories so that they can do 
> everything that attributes can do if they are used that way, you just 
> didn't get it.  You instead went for the quick ugly hack called xattrs.  
> You then got that ugly hack done first, because quick hacks are, well, 
> quick.  I then went about doing it the right way for Reiser4, and got 
> DARPA to fund doing it.  I was never silent about it.

I don't want to comment on any of the technical issues about VFS etc. as
I would be completely out of my depth, however I do want to say 2 things. Firstly,
this is a feature that Samba users have been needing for many years to maintain
compatibility with NTFS and Windows clients. Microsoft no longer sell any servers
or clients without support for multiple data streams per file, and their latest
XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad
that Hans and Namesys have created something we can use to match this
functionality - soon we will need it in order to be able to exist in
a Microsoft client-dominated world.

My second point is the following. Hans - did you *really* have to reinvent
the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001
for Solaris) ?

http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5
http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C
http://bama.ua.edu/cgi-bin/man-cgi?openat+2

I'm complaining here as someone who will have to write portable code
to try and work on all these "files with streams" systems.

Jeremy.

Re: silent semantic changes with reiser4

[Hans Reiser]

Jeremy Allison wrote:

>On Wed, Aug 25, 2004 at 12:53:28PM -0700, Hans Reiser wrote:
>  
>
>>You ignored everything I said during the discussion of xattrs about how 
>>there is no need to have attributes when you can just have files and 
>>directories, and that xattrs reflected a complete ignorance of name 
>>space design principles.  When I said we should just add some nice 
>>optional features to files and directories so that they can do 
>>everything that attributes can do if they are used that way, you just 
>>didn't get it.  You instead went for the quick ugly hack called xattrs.  
>>You then got that ugly hack done first, because quick hacks are, well, 
>>quick.  I then went about doing it the right way for Reiser4, and got 
>>DARPA to fund doing it.  I was never silent about it.
>>    
>>
>
>I don't want to comment on any of the technical issues about VFS etc. as
>I would be completely out of my depth, however I do want to say 2 things. Firstly,
>this is a feature that Samba users have been needing for many years to maintain
>compatibility with NTFS and Windows clients. Microsoft no longer sell any servers
>or clients without support for multiple data streams per file, and their latest
>XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad
>that Hans and Namesys have created something we can use to match this
>functionality - soon we will need it in order to be able to exist in
>a Microsoft client-dominated world.
>  
>
I agree that your work is important without agreeing that MS client 
domination will last.;-)  It is indeed my desire to give you every 
single feature you need to emulate MS streams within files, but doing it 
using directories that are files.  I would like to support you in 
emulating windows faster than windows.

>My second point is the following. Hans - did you *really* have to reinvent
>the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001
>for Solaris) ?
>  
>
I interviewed for the file system architect job at Sun in, I think, 
1999, and they offered me the job conditional on my giving up on my 
Linux work.  (After much trying and failing to convince them that it 
would be okay for me to work on Linux also, I declined the job, much to 
my fiscal loss and work satisfaction.)

They do not do a pure job of implementing attributes in the file 
namespace though.  There are far more distinctions between files and 
attributes than are necessary that are described in these man pages 
below, and those distinctions cause a loss of closure.  I can say more 
on that if asked.

>http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5
>http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C
>http://bama.ua.edu/cgi-bin/man-cgi?openat+2
>
>I'm complaining here as someone who will have to write portable code
>to try and work on all these "files with streams" systems.
>
>Jeremy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Hans Reiser wrote:

> I agree that your work is important without agreeing that MS client 
> domination will last.;-)  It is indeed my desire to give you every 
> single feature you need to emulate MS streams within files, but doing it 
> using directories that are files.  I would like to support you in 
> emulating windows faster than windows.

1) how do you back up and restore files with streams inside ?

2) how do standard unix utilities handle them ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 09:27:52AM -0400, Rik van Riel wrote:
>
>1) how do you back up and restore files with streams inside ?

With a program that supports files with streams inside.

>2) how do standard unix utilities handle them ?

Most likely they don't ;) That is, until they are fixed or replaced.
I've heard of people who want xattrs to be backed up so they use star, 
not gnu tar, already.

-- 
mjt

Re: silent semantic changes with reiser4

[Paul Jackson]

Markus wrote:
> >2) how do standard unix utilities handle them ?
> 
> Most likely they don't ;) That is, until they are fixed or replaced.

Not good, in my view.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>On Thu, 26 Aug 2004, Hans Reiser wrote:
>
>  
>
>>I agree that your work is important without agreeing that MS client 
>>domination will last.;-)  It is indeed my desire to give you every 
>>single feature you need to emulate MS streams within files, but doing it 
>>using directories that are files.  I would like to support you in 
>>emulating windows faster than windows.
>>    
>>
>
>1) how do you back up and restore files with streams inside ?
>
>2) how do standard unix utilities handle them ?
>
>  
>
To repeat, I think it would be nice to implement a 
filename/pseudos/backup method for all the plugins.

Guys, we just have the beginnings in place.  One plugin method at a time 
it will all fall into place.

What we have now is useful now.  The more methods come into existence, 
the more compelling it becomes --- standard network economic theory 
applies here.

Hans

Re: silent semantic changes with reiser4

[Chris Mason]

On Wed, 2004-08-25 at 15:53, Hans Reiser wrote:
> I had not intended to respond to this because I have nothing positive to 
> say, but Andrew said I needed to respond and suggested I should copy 
> Linus. Sigh.
> 
> Dear Christoph,
> 
> Let me see if I can summarize what you and your contingent are saying, 
> and if I misconstrue anything, let me know.;-)
> 
Just for fun why don't we look at the way things are today:

1) reiser4 has semantics that do belong at the VFS level.  They weren't
implemented at the VFS level for a variety of reasons, none of which
really matter right now.

2) new kernel patches that fragment the application developers between
apis are a bad thing.  There does need to be one interface here, and it
is in Hans' best interest to unify his work by working with people to
introduce new kernel wide apis.

This starts with exactly what Christoph described in writing a short
summary of how you want things to work today.  Since we can't resist,
we'll also go ahead and rehash all the old flame wars over this, but try
to include some new ideas about where you want to see the reiser4
interfaces in 6 months as well.

-chris

Re: silent semantic changes with reiser4

[Hans Reiser]

Chris Mason wrote:

>
>
>This starts with exactly what Christoph described in writing a short
>summary of how you want things to work today.  Since we can't resist,
>we'll also go ahead and rehash all the old flame wars over this, but try
>to include some new ideas about where you want to see the reiser4
>interfaces in 6 months as well.
>
>-chris
>
>
>
>
>
>  
>
Did Christoph read the www.namesys.com URL that has for several years 
been hiding from him? ;-)

Hans

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 01:42:50AM -0700, Hans Reiser wrote:
> Did Christoph read the www.namesys.com URL that has for several years 
> been hiding from him? ;-)

Yes, and I'm sick of your marketing bullshit Hans.

Re: silent semantic changes with reiser4

[Chris Friesen]

Hans Reiser wrote:

> Making files into directories caused only two applications out of the
> entire OS to notice the change, and that was because of a bug in what
> error code we returned that we are going to fix.  You think that was a
> disaster; I think it was a triumph.

>  Since we have such a performance lead, Namesys is about to change its
> focus from the storage layer to semantics, look at
> www.namesys.com/whitepaper.html for details.  Semantic enhancements are
> the important stuff, and finally Namesys is where we have all the
> storage layer prerequisites done right, and the real work can begin. 

Just curious about your comments on Jamie Lokier's suggestions for enabling 
files-as-directories semantics without breaking existing apps.

Chris

Re: silent semantic changes with reiser4

[Andrew Morton]

Hans Reiser <reiser@namesys.com> wrote:
>
> I had not intended to respond to this because I have nothing positive to 
> say, but Andrew said I needed to respond and suggested I should copy 
> Linus.

Yes, but I didn't say "flame Christoph and ignore the issues" ;)

There are lots of little things to do with implementation, coding style,
module exports, deadlocks, what code goes where, etc.  These are all normal
daily kernel business and we should set them aside for now and concentrate
on the bigger issues.

And as I see it, there are two big issues:

a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not
   anticipate and 

b) it does this within the context of just a single filesystem.

I see three possible responses:

a) accept the reiser4-only extensions as-is (possibly with post-review
   modifications, of course) or

b) accept the reiser4-only extensions with a view to turning them into
   kernel-wide extensions at some time in the future, so all filesystems
   will offer the extensions (as much as poss) or

c) reject the extensions.


My own order of preference is b) c) a).  The fact that one filesystem will
offer features which other filesystems do not and cannot offer makes me
queasy for some reason.

b) means that at some time in the future we need to hoist the reiser4
extensions (at a conceptual level) (and probably with restrictions) up into
the VFS.  This will involve much thought, argument and work.


To get us started on this route it would really help me (and, probably,
others) if you could describe what these API extensions are in a very
simple way, without referring to incomprehehsible web pages, and without
using terms which non-reiser4 people don't understand.

It would be best if each extension was addressed in a separate email
thread.

We also need to discuss what a reiser4 "module" is, what its capabilities
are, and what licensing implications they have.

Then, we can look at each one and say "yup, that makes sense - we want
Linux to do that" and we can also think about how we would implement it at
the VFS level.

If we follow the above route I believe we can make progress in a technical
direction and not get deadlocked on personal/political stuff.


Now, an alternative to all the above is to just merge reiser4 as-is, after
addressing all the lower-level coding issues.  And see what happens.  That
may be a thing which Linus wishes to do.  I'm easy.

Re: silent semantic changes with reiser4

[Spam]

> And as I see it, there are two big issues:

> a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not
>    anticipate and 

> b) it does this within the context of just a single filesystem.

> I see three possible responses:

> a) accept the reiser4-only extensions as-is (possibly with post-review
>    modifications, of course) or

> b) accept the reiser4-only extensions with a view to turning them into
>    kernel-wide extensions at some time in the future, so all filesystems
>    will offer the extensions (as much as poss) or

> c) reject the extensions.


> My own order of preference is b) c) a).  The fact that one filesystem will
> offer features which other filesystems do not and cannot offer makes me
> queasy for some reason.

  This last sentence makes me wonder. Where is Linux heading? The idea
  that   a  FS  cannot  contain  features that no other FS has is very
  scary.

  I  am  all  for  uniformity, but not at the expense of shutting down
  advanced progress that Linux is so badly needing.

  This talk about old UNIX seems like people want to still live in the
  70'ies and not look forward. Please wake up!

  ~S

Re: silent semantic changes with reiser4

[Christoph Hellwig]

>   This last sentence makes me wonder. Where is Linux heading? The idea
>   that   a  FS  cannot  contain  features that no other FS has is very
>   scary.
> 
>   I  am  all  for  uniformity, but not at the expense of shutting down
>   advanced progress that Linux is so badly needing.
> 
>   This talk about old UNIX seems like people want to still live in the
>   70'ies and not look forward. Please wake up!

Just because semantics are at the VFS layer doesn't mean every
filesystem has to implement them.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Spam wrote:
> 
> > My own order of preference is b) c) a).  The fact that one filesystem will
> > offer features which other filesystems do not and cannot offer makes me
> > queasy for some reason.
> 
>   This last sentence makes me wonder. Where is Linux heading? The idea
>   that   a  FS  cannot  contain  features that no other FS has is very
>   scary.

That's not what Andrew said or meant.

Note the "cannot offer". As in "there is no way to offer them even if the 
filesystem could support it otherwise". 

We have tons of filesystems that do things other filesystems cannot do.  
Most filesystems support writing to a file - despite the fact that some
filesystems (iso9600 being an obvious one) cannot. The infrastructure is
there in the VFS layer, and it becomes a _choice_ for the filesystem
whether it offers certain capabilities.

So look at what Andrew said, again: his top choice would be (b). Let's see 
what that was again, shall we?

> b) accept the reiser4-only extensions with a view to turning them into
>    kernel-wide extensions at some time in the future, so all filesystems
>    will offer the extensions (as much as poss) or

In other words, if reiserfs does something special, we should make 
standard interfaces for doing that special thing, so that everybody can 
do it without stepping on other peoples toes.

That doesn't mean that we'd _force_ everybody to do it. The same way we 
don't force iso9660 to write to a CD-ROM.

		Linus

Re: silent semantic changes with reiser4

[Spam]

> On Thu, 26 Aug 2004, Spam wrote:
>> 
>> > My own order of preference is b) c) a).  The fact that one filesystem will
>> > offer features which other filesystems do not and cannot offer makes me
>> > queasy for some reason.
>> 
>>   This last sentence makes me wonder. Where is Linux heading? The idea
>>   that   a  FS  cannot  contain  features that no other FS has is very
>>   scary.

> That's not what Andrew said or meant.

> Note the "cannot offer". As in "there is no way to offer them even if the
> filesystem could support it otherwise". 

> We have tons of filesystems that do things other filesystems cannot do.
> Most filesystems support writing to a file - despite the fact that some
> filesystems (iso9600 being an obvious one) cannot. The infrastructure is
> there in the VFS layer, and it becomes a _choice_ for the filesystem
> whether it offers certain capabilities.

> So look at what Andrew said, again: his top choice would be (b). Let's see
> what that was again, shall we?

>> b) accept the reiser4-only extensions with a view to turning them into
>>    kernel-wide extensions at some time in the future, so all filesystems
>>    will offer the extensions (as much as poss) or

> In other words, if reiserfs does something special, we should make 
> standard interfaces for doing that special thing, so that everybody can
> do it without stepping on other peoples toes.

  Agreed  that would be the best. But how much time and effort will it
  be,  and  how much of the original ideas would be lost on the way to
  implement  them  in  the  VFS? Especially with new and very advanced
  FS's like Reiser4.

  Isn't  the  line  between the actual file system and the virtual one
  very hair thin? Where would the separation lay in Reiser4?
  
> That doesn't mean that we'd _force_ everybody to do it. The same way we
> don't force iso9660 to write to a CD-ROM.

>       Linus

  I got caught in the moment of flame war. My appologies.

  ~S

Re: silent semantic changes with reiser4

[Andrew Morton]

Spam <spam@tnonline.net> wrote:
>
> > In other words, if reiserfs does something special, we should make 
> > standard interfaces for doing that special thing, so that everybody can
> > do it without stepping on other peoples toes.
> 
>   Agreed  that would be the best. But how much time and effort will it
>   be

Zero.

We can add these new features tomorrow, as reiser4-only features, with a
plan in hand to generalise them later.

-->>__if__<<-- we think these are features which Linux should offer.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Aug 25, 2004 at 04:32:25PM -0700, Andrew Morton wrote:
> 
> We can add these new features tomorrow, as reiser4-only features, with a
> plan in hand to generalise them later.
> 
> -->>__if__<<-- we think these are features which Linux should offer.

Multiple-data-stream files are something we should offer, definately (IMHO).
I don't care how we do it, but I know it's something we need as application
developers.

Jeremy.

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Jeremy Allison wrote:
> Multiple-data-stream files are something we should offer, definately (IMHO).
> I don't care how we do it, but I know it's something we need as application
> developers.

Aside from samba, is there any other application that has a use for
them? 

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
> Previously Jeremy Allison wrote:
> > Multiple-data-stream files are something we should offer, definately (IMHO).
> > I don't care how we do it, but I know it's something we need as application
> > developers.
> 
> Aside from samba, is there any other application that has a use for
> them? 
> 

Anything that currently stores a file's metadata in another file really
wants this right now. Things like image thumbnails, document summaries,
digital signatures, etc.

As to how to do it, I think the Solaris interface is reasonably decent.
The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view

(An important detail for those who want to access their
multiple-data-streams from non-MDS aware apps is the runat shell
command, which basically does a chdir into the specified file's
attribute directory and then runs a command. i.e. 'runat ~/blah ls' will
list the ~/blah's attributes.)

The only real problem I have with their design is the calling them
attributes and using "at" everywhere. 

"Attributes",  because it will get confused with the current Linux xattr
implementation (which is still useful for things that actually are file
attributes, like security labels, ACLs, weird attributess that
FAT/NTFS/whatever have, etc.).

I don't like "at" because the API changes don't have anything to do with
the actual attributes. It's a general set of changes to allow paths
relative to a fd instead of the cwd, and doesn't really have anything
specifically to do with attributes (with the exception of the O_XATTR
flag). 

Replace "at" with "rel" and O_XATTR with O_FORK or O_MULTI or something,
and it's all good.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Nicholas Miell wrote:
> Anything that currently stores a file's metadata in another file really
> wants this right now. Things like image thumbnails, document summaries,
> digital signatures, etc.

Additionally, all of those things you describe should be deleted if
the file is modified -- to indicate that they're no longer valid and
should be regenerated if needed.

Whereas there are some other kinds of metadata which should not be
deleted if the file is modified.

-- Jamie

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Wed, 2004-08-25 at 18:03, Jamie Lokier wrote:
> Nicholas Miell wrote:
> > Anything that currently stores a file's metadata in another file really
> > wants this right now. Things like image thumbnails, document summaries,
> > digital signatures, etc.
> 
> Additionally, all of those things you describe should be deleted if
> the file is modified -- to indicate that they're no longer valid and
> should be regenerated if needed.
> 
> Whereas there are some other kinds of metadata which should not be
> deleted if the file is modified.
> 
> -- Jamie

Presumably the app which uses the metadata will be smart enough to
compare the st_mtime of the MDS/stream/attribute/whatever (can we choose
a name for these things now?) to the st_mtime of the file and do the
right thing.

thumbnail - regenerate it
summary - keep it, it's relatively independent of the file's exact
contents
signature - always verify it
etc.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Nicholas Miell wrote:
> > Additionally, all of those things you describe should be deleted if
> > the file is modified -- to indicate that they're no longer valid and
> > should be regenerated if needed.
> > 
> > Whereas there are some other kinds of metadata which should not be
> > deleted if the file is modified.
> > 
> > -- Jamie
> 
> Presumably the app which uses the metadata will be smart enough to
> compare the st_mtime of the MDS/stream/attribute/whatever (can we choose
> a name for these things now?) to the st_mtime of the file and do the
> right thing.

[This is straying off the topic of files-as-directories].

st_mtime tests are weak.  They break sometimes, and are not suitable
for strong data models such as transparently caching generated data
from a file's contents.

They're especially breakable if you change a file and then read it
within a second.  Sometimes, more than a second.

As has been raised before, nanosecond timestamps (a) don't solve this
unless they're stored in the filesystem; (b) even then they will fail
one day, on a sufficiently fast box; (c) don't work when there are
changes to wall clock time.

They're fine where you don't care if the generated data is wrong sometimes.

If you want the _equivalent_ behaviour to opening and parsing the
file, but fast, then they're no good.

Modification serial numbers would work, though.

-- Jamie

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Wed, 2004-08-25 at 18:53, Jamie Lokier wrote:
> Nicholas Miell wrote:
> > > Additionally, all of those things you describe should be deleted if
> > > the file is modified -- to indicate that they're no longer valid and
> > > should be regenerated if needed.
> > > 
> > > Whereas there are some other kinds of metadata which should not be
> > > deleted if the file is modified.
> > > 
> > > -- Jamie
> > 
> > Presumably the app which uses the metadata will be smart enough to
> > compare the st_mtime of the MDS/stream/attribute/whatever (can we choose
> > a name for these things now?) to the st_mtime of the file and do the
> > right thing.
> 
> [This is straying off the topic of files-as-directories].
> 
> st_mtime tests are weak.  They break sometimes, and are not suitable
> for strong data models such as transparently caching generated data
> from a file's contents.
> 
> They're especially breakable if you change a file and then read it
> within a second.  Sometimes, more than a second.
> 
> As has been raised before, nanosecond timestamps (a) don't solve this
> unless they're stored in the filesystem; (b) even then they will fail
> one day, on a sufficiently fast box; (c) don't work when there are
> changes to wall clock time.
> 
> They're fine where you don't care if the generated data is wrong sometimes.
> 
> If you want the _equivalent_ behaviour to opening and parsing the
> file, but fast, then they're no good.
> 
> Modification serial numbers would work, though.
> 
> -- Jamie

Remember that the Solaris attribute model is just another filesystem
tree rooted in a hidden directory associated with a regular file. If you
can come up with semantics that work in general for all files, that's
fine.

inodes that delete themselves when other inodes are changed creep me
out.

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>Nicholas Miell wrote:
>  
>
>>Anything that currently stores a file's metadata in another file really
>>wants this right now. Things like image thumbnails, document summaries,
>>digital signatures, etc.
>>    
>>
>
>Additionally, all of those things you describe should be deleted if
>the file is modified -- to indicate that they're no longer valid and
>should be regenerated if needed.
>
>Whereas there are some other kinds of metadata which should not be
>deleted if the file is modified.
>
>-- Jamie
>
>
>  
>
Yes, I agree.

Actually we plan to have a whole link taxonomy, and one expected feature 
is that some links don't count towards the refcount needed to keep an 
object in existence (For instance, the links between key words and text 
documents, you don't want to have to explicitly unlink every keyword in 
order to delete a document).

Re: silent semantic changes with reiser4

[Giuseppe Bilotta]

Jamie Lokier wrote:
> Nicholas Miell wrote:
> > Anything that currently stores a file's metadata in another file really
> > wants this right now. Things like image thumbnails, document summaries,
> > digital signatures, etc.
> 
> Additionally, all of those things you describe should be deleted if
> the file is modified -- to indicate that they're no longer valid and
> should be regenerated if needed.

In principle, not all of them. For example, a document summary 
for a text document or a long (textual) description of a video 
clip might remain the same when the user is only working on the 
finishing details.

Maybe the metadata needs an attribute to determine how 
'immutable' it should be wrt changes on the file? (Can you 
spell meta-meta-data <g>)

-- 
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
> 
> Anything that currently stores a file's metadata in another file really
> wants this right now. Things like image thumbnails, document summaries,
> digital signatures, etc.

Which is exactly how Windows apps have started to use streams.

> As to how to do it, I think the Solaris interface is reasonably decent.
> The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view

I agree. This is an interface Samba can live with I think. I
was thinking of implementing it anyway, just so I could piss
off the Linux kernel developers by saying "oh if you need proper
Windows semantics on Samba you have to use an advanced OS like
Solaris, Linux doesn't cut it" :-) :-).

> The only real problem I have with their design is the calling them
> attributes and using "at" everywhere. 

Yep - they're different from xattrs. The easiest way to remember
this is that file streams are seekable and get a fd, xattrs aren't
and don't :-).

Jeremy.

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Jeremy Allison wrote:
> Which is exactly how Windows apps have started to use streams.

Interesting I can not find a way using the .NET framework, which is
supposed to be The Future, to use streams.

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Tierra]

On Thu, 26 Aug 2004 13:01:41 +0200, Wichert Akkerman <wichert@wiggy.net> wrote:
> Previously Jeremy Allison wrote:
> > Which is exactly how Windows apps have started to use streams.
>
> Interesting I can not find a way using the .NET framework, which is
> supposed to be The Future, to use streams.

That's because you don't even need .NET...
http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/q105/7/63.asp&NoWebContent=1&NoWebContent=1

http://tinyurl.com/45mfa

Bryan

Re: silent semantic changes with reiser4

[Matt Mackall]

On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
> On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
> > Previously Jeremy Allison wrote:
> > > Multiple-data-stream files are something we should offer, definately (IMHO).
> > > I don't care how we do it, but I know it's something we need as application
> > > developers.
> > 
> > Aside from samba, is there any other application that has a use for
> > them? 
> > 
> 
> Anything that currently stores a file's metadata in another file really
> wants this right now. Things like image thumbnails, document summaries,
> digital signatures, etc.

That is _highly_ debatable. I would much rather have my cp and grep
and cat and tar and such continue to work than have to rewrite every
tool because we've thrown the file-is-a-stream-of-bytes concept out
the window. Never mind that I've got thumbnails, document summaries,
and digital signatures already.

While the number of annoying properties of files with forks is
practically endless, the biggest has got to be utter lack of
portability. How do you stick the thing in an attachment or on an ftp
site? Well you can't because it's NOT A FILE. 

A file is a stream of bytes.

-- 
Mathematics is the supreme nostalgia of our time.

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Wed, 2004-08-25 at 21:44, Matt Mackall wrote:
> On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
> > On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
> > > Previously Jeremy Allison wrote:
> > > > Multiple-data-stream files are something we should offer, definately (IMHO).
> > > > I don't care how we do it, but I know it's something we need as application
> > > > developers.
> > > 
> > > Aside from samba, is there any other application that has a use for
> > > them? 
> > > 
> > 
> > Anything that currently stores a file's metadata in another file really
> > wants this right now. Things like image thumbnails, document summaries,
> > digital signatures, etc.
> 
> That is _highly_ debatable. I would much rather have my cp and grep
> and cat and tar and such continue to work than have to rewrite every
> tool because we've thrown the file-is-a-stream-of-bytes concept out
> the window. Never mind that I've got thumbnails, document summaries,
> and digital signatures already.
> 
> While the number of annoying properties of files with forks is
> practically endless, the biggest has got to be utter lack of
> portability. How do you stick the thing in an attachment or on an ftp
> site? Well you can't because it's NOT A FILE. 
> 
> A file is a stream of bytes.

"OMG! It breaks tar and email!!!" argument doesn't fly. Things break all
the time and are fixed. It's called progress.

cp, grep, cat, and tar will continue to work just fine on files with
multiple streams.

tar and cp will lose the extra streams until somebody fixes them, but
they lose ACLs and xattrs right now, and I don't hear anybody suggesting
that ACLs and xattrs be removed from the kernel because of this.

Fixing programs that do recursive filesystem traversals is a matter of a
glibc patch to nftw(3) and a modification to their option processing.

Holding back a useful feature because you don't want to upgrade
coreutils is just plain dumb.

(BTW, for email, multipart/parallel is a start, but a specific multipart
content type for multi-stream file attachments would probably be more
appropriate.)

Re: silent semantic changes with reiser4

[James Morris]

On Wed, 25 Aug 2004, Nicholas Miell wrote:

> (BTW, for email, multipart/parallel is a start, but a specific multipart
> content type for multi-stream file attachments would probably be more
> appropriate.)

I don't understand why something like this is so important that it needs 
underlying VFS support.  Multipart messages seem to have survived fine up 
until now.  Where is the justification?


- James
-- 
James Morris
<jmorris@redhat.com>

Re: silent semantic changes with reiser4

[Matt Mackall]

On Wed, Aug 25, 2004 at 10:09:08PM -0700, Nicholas Miell wrote:
> On Wed, 2004-08-25 at 21:44, Matt Mackall wrote:
> > On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
> > > On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
> > > > Previously Jeremy Allison wrote:
> > > > > Multiple-data-stream files are something we should offer, definately (IMHO).
> > > > > I don't care how we do it, but I know it's something we need as application
> > > > > developers.
> > > > 
> > > > Aside from samba, is there any other application that has a use for
> > > > them? 
> > > > 
> > > 
> > > Anything that currently stores a file's metadata in another file really
> > > wants this right now. Things like image thumbnails, document summaries,
> > > digital signatures, etc.
> > 
> > That is _highly_ debatable. I would much rather have my cp and grep
> > and cat and tar and such continue to work than have to rewrite every
> > tool because we've thrown the file-is-a-stream-of-bytes concept out
> > the window. Never mind that I've got thumbnails, document summaries,
> > and digital signatures already.
> > 
> > While the number of annoying properties of files with forks is
> > practically endless, the biggest has got to be utter lack of
> > portability. How do you stick the thing in an attachment or on an ftp
> > site? Well you can't because it's NOT A FILE. 
> > 
> > A file is a stream of bytes.
> 
> "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all
> the time and are fixed. It's called progress.

What it breaks is the concept of a file. In ways that are ill-defined,
not portable, hard to work with, and needlessly complex. Along the
way, it breaks every single application that ever thought it knew what
a file was.

Progress? No, this has been done before. Various dead operating
systems have done it or similar and regretted it. Most recently MacOS,
which jumped through major hurdles to begin purging themselves of
resource forks when they went to OS X. They're still there, but
heavily deprecated.

> cp, grep, cat, and tar will continue to work just fine on files with
> multiple streams.

Find some silly person with an iBook and open a shell on OS X. Use cp
to copy a file with a resource fork. Oh look, the Finder has no idea
what the new file is, even though it looks exactly identical in the
shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
fork. How is that ever going to work?

I like cat < a > b. You can keep your progress.

-- 
Mathematics is the supreme nostalgia of our time.

Re: silent semantic changes with reiser4

[Denis Vlasenko]

> > > A file is a stream of bytes.
> >
> > "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all
> > the time and are fixed. It's called progress.
>
> What it breaks is the concept of a file. In ways that are ill-defined,
> not portable, hard to work with, and needlessly complex. Along the
> way, it breaks every single application that ever thought it knew what
> a file was.
>
> Progress? No, this has been done before. Various dead operating
> systems have done it or similar and regretted it. Most recently MacOS,
> which jumped through major hurdles to begin purging themselves of
> resource forks when they went to OS X. They're still there, but
> heavily deprecated.
>
> > cp, grep, cat, and tar will continue to work just fine on files with
> > multiple streams.
>
> Find some silly person with an iBook and open a shell on OS X. Use cp
> to copy a file with a resource fork. Oh look, the Finder has no idea
> what the new file is, even though it looks exactly identical in the
> shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
> fork. How is that ever going to work?
>
> I like cat < a > b. You can keep your progress.

cat <a >b does not preserve following file properties even on standard
UNIX filesystems: name,owner,group,permissions.
-- 
vda

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Denis Vlasenko wrote:

> > I like cat < a > b. You can keep your progress.
> 
> cat <a >b does not preserve following file properties even on standard
> UNIX filesystems: name,owner,group,permissions.

Losing permissions is one thing.  Annoying, mostly.

However, actual losing file data during such a copy is
nothing short of a disaster, IMHO.  

In my opinion we shouldn't merge file-as-a-directory
semantics into the kernel until we figure out how to
fix the backup/restore problem and keep standard unix
utilities work.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 3556 bytes --]

Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:

> > > I like cat < a > b. You can keep your progress.
> > 
> > cat <a >b does not preserve following file properties even on standard
> > UNIX filesystems: name,owner,group,permissions.
> 
> Losing permissions is one thing.  Annoying, mostly.
> 
> However, actual losing file data during such a copy is
> nothing short of a disaster, IMHO.  
> 
> In my opinion we shouldn't merge file-as-a-directory
> semantics into the kernel until we figure out how to
> fix the backup/restore problem and keep standard unix
> utilities work.

Well, again, what about xattrs and ACLs...?

Actually, reiser4 doesn't currently implement storage of arbitrary user
data under a file.

It's just some sysfs-like information that can be retrieved, some
properties can be changed. If you copy a file the worst thing that can
happen right now is that you lose the information whether that file
should be encrypted or compressed. You don't lose data.

In case some people are wondering, this is what you can find in the
reiser4 metas:

leto:/home/chtephan/.muttrc/metas > la
insgesamt 1
dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 .
-rwx------  1 chtephan users 290 12. Jan 2004  ..
-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
--w-------  1 chtephan users   0 26. Aug 13:52 new
-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid
dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 plugin
-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid

Some of them are reiser4 specific things (items, key, locality, oid)
which can only be queried anyway.

The rest is what you can also get using other VFS calls.

leto:/home/chtephan/.muttrc/metas/plugin > la
insgesamt 0
dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 .
dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 ..
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd

Here you can change reiser4 specific things. Here you can change some
properties how the file is stored in the reiser4 tree. For example you
can activate compression, encryption or authentication or set some hints
to optimize speed.

You can't create or remove these pseudo-files, just like in sysfs.

I don't know, but if someone wants to store user data I think you would
have to implement a directory "userdata" or something where you can
store it.

Not all of the information does make a lot sense inside a modified tar.
Who is interested in the bmap information of the file or some other
read-only information?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Yury Umanets]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:
>
>  
>
>>>>I like cat < a > b. You can keep your progress.
>>>>        
>>>>
>>>cat <a >b does not preserve following file properties even on standard
>>>UNIX filesystems: name,owner,group,permissions.
>>>      
>>>
>>Losing permissions is one thing.  Annoying, mostly.
>>
>>However, actual losing file data during such a copy is
>>nothing short of a disaster, IMHO.  
>>
>>In my opinion we shouldn't merge file-as-a-directory
>>semantics into the kernel until we figure out how to
>>fix the backup/restore problem and keep standard unix
>>utilities work.
>>    
>>
>
>Well, again, what about xattrs and ACLs...?
>
>Actually, reiser4 doesn't currently implement storage of arbitrary user
>data under a file.
>
>It's just some sysfs-like information that can be retrieved, some
>properties can be changed. If you copy a file the worst thing that can
>happen right now is that you lose the information whether that file
>should be encrypted or compressed. You don't lose data.
>
>In case some people are wondering, this is what you can find in the
>reiser4 metas:
>
>leto:/home/chtephan/.muttrc/metas > la
>insgesamt 1
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 .
>-rwx------  1 chtephan users 290 12. Jan 2004  ..
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
>--w-------  1 chtephan users   0 26. Aug 13:52 new
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 plugin
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid
>
>Some of them are reiser4 specific things (items, key, locality, oid)
>which can only be queried anyway.
>
>The rest is what you can also get using other VFS calls.
>
>leto:/home/chtephan/.muttrc/metas/plugin > la
>insgesamt 0
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 .
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 ..
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd
>
>Here you can change reiser4 specific things. Here you can change some
>properties how the file is stored in the reiser4 tree. For example you
>can activate compression, encryption or authentication or set some hints
>to optimize speed.
>  
>
I suppose that you are talking about the prinsiple ability to change 
these things. Because in real life it will also need some kind of 
convertion from one look (say with old plugin) to another one. Just like 
tail conversion rebuilds file's body after tail policy plugin gets changed.





-- 
umka

Re: silent semantic changes with reiser4

[Yury Umanets]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:
>
>  
>
>>>>I like cat < a > b. You can keep your progress.
>>>>        
>>>>
>>>cat <a >b does not preserve following file properties even on standard
>>>UNIX filesystems: name,owner,group,permissions.
>>>      
>>>
>>Losing permissions is one thing.  Annoying, mostly.
>>
>>However, actual losing file data during such a copy is
>>nothing short of a disaster, IMHO.  
>>
>>In my opinion we shouldn't merge file-as-a-directory
>>semantics into the kernel until we figure out how to
>>fix the backup/restore problem and keep standard unix
>>utilities work.
>>    
>>
>
>Well, again, what about xattrs and ACLs...?
>
>Actually, reiser4 doesn't currently implement storage of arbitrary user
>data under a file.
>
>It's just some sysfs-like information that can be retrieved, some
>properties can be changed. If you copy a file the worst thing that can
>happen right now is that you lose the information whether that file
>should be encrypted or compressed. You don't lose data.
>
>In case some people are wondering, this is what you can find in the
>reiser4 metas:
>
>leto:/home/chtephan/.muttrc/metas > la
>insgesamt 1
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 .
>-rwx------  1 chtephan users 290 12. Jan 2004  ..
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
>--w-------  1 chtephan users   0 26. Aug 13:52 new
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid
>dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 plugin
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
>-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
>-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid
>
>Some of them are reiser4 specific things (items, key, locality, oid)
>which can only be queried anyway.
>
>The rest is what you can also get using other VFS calls.
>
>leto:/home/chtephan/.muttrc/metas/plugin > la
>insgesamt 0
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 .
>dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 ..
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
>-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd
>
>Here you can change reiser4 specific things. Here you can change some
>properties how the file is stored in the reiser4 tree. For example you
>can activate compression, encryption or authentication or set some hints
>to optimize speed.
>  
>
I suppose that you are talking about the principle ability to change
these things. Because in real life it will also need some kind of
conversion from one look (say with old plugin) to another one. Just like
tail conversion rebuilds file's body after tail policy plugin gets changed.





-- 
umka

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> > cat <a >b does not preserve following file properties even on standard
> > UNIX filesystems: name,owner,group,permissions.
> 
> Losing permissions is one thing.  Annoying, mostly.

I find losing mtime annoying.  That's why I don't use Mozilla's
download manager (and Navigator's before it) because they presumed
that mtime isn't worth copying.  ncftp and wget do the right thing,
but it's annoying that I can't use the fancy tools and get the
metadata _I_ want transferred by them.

> However, actual losing file data during such a copy is
> nothing short of a disaster, IMHO.  

Note that file-as-directory doesn't imply that you can store just
anything into those directories.

Is it a problem to decree that "file data MUST NOT be stored in a
metadata directory; only non-essential metadata and data computed from
the file data may be stored"?

> In my opinion we shouldn't merge file-as-a-directory
> semantics into the kernel until we figure out how to
> fix the backup/restore problem and keep standard unix
> utilities work.

Same as xattrs, for data that should be stored.

But, unlike xattrs, while permissions and C-source-encoding should be
stored, some of this metadata should _not_ be stored when making an
archive and perhaps not when making a backup either.

Personally I wouldn't want an archive containing a file to save and
restore recomputable metadata like cached, extracted-on-demand MP3 id
tags and email headers-as-metadata.  I'd rather they were recalculated
when needed, implying a stronger guarantee about their relationship to
the data from which they're extracted - and implying that a user
program might not be allowed to set them to inconsistent values.

-- Jamie

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Jamie Lokier wrote:
> I find losing mtime annoying.  That's why I don't use Mozilla's
> download manager (and Navigator's before it) because they presumed
> that mtime isn't worth copying.

It's free software, you could fix that :)

> Is it a problem to decree that "file data MUST NOT be stored in a
> metadata directory; only non-essential metadata and data computed from
> the file data may be stored"?

That would put us right back at the samba-and-other-apps-want-streams
point in the discussion, but it certainly would not be a bad starting
point. From there we could always add streams later on if the API is
right.

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Grzegorz Jaśkiewicz]

Jamie Lokier wrote:

>Note that file-as-directory doesn't imply that you can store just
>anything into those directories.
>
>Is it a problem to decree that "file data MUST NOT be stored in a
>metadata directory; only non-essential metadata and data computed from
>the file data may be stored"?
>  
>

That's exactly what folks seem to lost in this debate.
We (developers) want to have files as dirs to:
* be able to extract/modify file part on fly. For instance, simple .iso 
file, without need to mount it over loop (btw, if someone is going to 
redesign VFS, can we handle that kinda case too please?), .tar, or 
unzipped (kernel can handle zips) "streams". File is a stream, why 
shouldn't I have a chance to grab a stream out of archive/image than ?
* be able to store metadata, that doesn't matter on copy, and even 
should not be copied sometimes. Things like thumbnails, metainformation 
(used for search). All of these extracted from acctual file (either by 
VFS 'plugins', or by userspace application), and saved there so that it 
can be used later on. So we save some time on extracting it.


--
GJ

Re: silent semantic changes with reiser4

[Frank van Maarseveen]

On Fri, Sep 03, 2004 at 06:36:41AM +0200, Grzegorz Ja??kiewicz wrote:
> Jamie Lokier wrote:
> 
> >Is it a problem to decree that "file data MUST NOT be stored in a
> >metadata directory; only non-essential metadata and data computed from
> >the file data may be stored"?
> 
> That's exactly what folks seem to lost in this debate.

And that's not even the most important part: How would you educate all
the developers _not_ on this list: application developers? I think this
is just a disaster waiting for us to occur.

> * be able to extract/modify file part on fly.
> * be able to store metadata, that doesn't matter on copy

those may be orthogonal.

-- 
Frank

Re: silent semantic changes with reiser4

[Horst von Brand]

=?UTF-8?B?R3J6ZWdvcnogSmHFm2tpZXdpY3o=?= <gj@pointblue.com.pl> said:
> Jamie Lokier wrote:
> >Note that file-as-directory doesn't imply that you can store just
> >anything into those directories.

> >Is it a problem to decree that "file data MUST NOT be stored in a
> >metadata directory; only non-essential metadata and data computed from
> >the file data may be stored"?

> That's exactly what folks seem to lost in this debate.
> We (developers) want to have files as dirs to:
> * be able to extract/modify file part on fly. For instance, simple .iso 
> file, without need to mount it over loop (btw, if someone is going to 
> redesign VFS, can we handle that kinda case too please?), .tar, or 
> unzipped (kernel can handle zips) "streams". File is a stream, why 
> shouldn't I have a chance to grab a stream out of archive/image than ?

ISO9660 is designed for read-only, i.e., modifying something in the middle
is probably as hard to do as just unpack/modify/repack. Same for tar, cpio,
and compressed versions of same. No gain, for a major hassle implementing
this.

> * be able to store metadata, that doesn't matter on copy, and even 
> should not be copied sometimes. Things like thumbnails, metainformation 
> (used for search). All of these extracted from acctual file (either by 
> VFS 'plugins', or by userspace application), and saved there so that it 
> can be used later on. So we save some time on extracting it.

You save iff the resources spent updating this stuff is less than
recomputing on the fly each time. Also, placing random stuff in the kernel
is _not_ "zero cost", it is _very_ expensive (kernel memory is always
resident, kernel code is _extremely_ critical, once it is in the kernel it
is very hard to change interfaces). Besides, this kind of "metadata" will
most probably be different for each user, and a user should also be able to
associate it to files she has no write right on. I.e., keep it in a
per-user directory, managed by whatever GUI tools she uses.

What you (developer) wish for and what kernel hackers are able to deliver
aren't the same. As it stands this is a badly-thought-through proposal,
with very complex ramifications that few here are aware of (see Al Viro's
and Linus' posts about locking; it breaks the fundamental principle that
something is a file or a directory, so this would break many applications),
and no clean solutions as of now.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Grzegorz Jaśkiewicz]

Oh, and if I may notice something more. Just my opinion.
It's going to be 'samba uses only' thing, as long as it's not in VFS.
xattrs are not so much needed as file streams there. So argument that 
xattrs exists already in VFS, and are not used, so why do we need 
streams there, and that it wouldn't change anything is just silly.

Btw, reiserfs4 on my laptop is providing very nice base for KDE source 
tree (XFS sucked so much, I had to replace it). Saving me some time on 
recompilation, and cvs up.

Thanks.

--
GJ

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 07:42, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Denis Vlasenko wrote:
> 
> > > I like cat < a > b. You can keep your progress.
> > 
> > cat <a >b does not preserve following file properties even on standard
> > UNIX filesystems: name,owner,group,permissions.
> 
> Losing permissions is one thing.  Annoying, mostly.
> 

cat <a >b does preserve permissions - the permissions of b.  I always
considered this a feature.

Lee

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>
>
>In my opinion we shouldn't merge file-as-a-directory
>semantics into the kernel until we figure out how to
>fix the backup/restore problem and keep standard unix
>utilities work.
>
>  
>
One incremental piece at a time our solution will fall into place.  
Pieces should go in when they are individually ready.  Warning users 
that some pieces might be changed out from under them up until when 2.8 
ships is ok with me though.

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote:
>
>What it breaks is the concept of a file. In ways that are ill-defined,
>not portable, hard to work with, and needlessly complex. Along the
>way, it breaks every single application that ever thought it knew what
>a file was.

It breaks the concept of a file. In ways that offer more versatility,
challenge the imagination to make even better progress and keeps
Linux competing with competitors who are implementing this stuff
as we speak.

I for one would truly welcome the coming of thumbnails and descriptions
in picture files, because I have a real-life project going on where
that would be extremely handy to have in the actual file.
Were I any richer, I'd pay Namesys to have this work for me :)

>Find some silly person with an iBook and open a shell on OS X. Use cp
>to copy a file with a resource fork. Oh look, the Finder has no idea
>what the new file is, even though it looks exactly identical in the
>shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
>fork. How is that ever going to work?

Then I guess OS X ships a broken implementation of cp, yes?

On the cat example, what if cat < a > b simply copies the "main stream"
and not the metadata, as a feature. The key being, "as a feature"

The metadata streams could get file descriptors of their own OR
another program, streamcat or something, could be written to compensate.

>I like cat < a > b. You can keep your progress.

With all due respect, I hope not too many people agree with you :)

-- 
mjt

Re: silent semantic changes with reiser4

[Jan Harkes]

On Thu, Aug 26, 2004 at 10:53:48AM +0300, Markus   Törnqvist wrote:
> On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote:
> >What it breaks is the concept of a file. In ways that are ill-defined,
> >not portable, hard to work with, and needlessly complex. Along the
> >way, it breaks every single application that ever thought it knew what
> >a file was.
> 
> It breaks the concept of a file. In ways that offer more versatility,
> challenge the imagination to make even better progress and keeps
> Linux competing with competitors who are implementing this stuff
> as we speak.

Sigh, it really isn't all that groundbreaking. The only special feature
is the fact that it introduces directory hardlinks.

Just turn the phrase 'file-as-dir' around and you might see the point.
As far as userspace is concerned the implementation would behave 99%
identical if we provided a 'dir-as-file' semantics. i.e. opening a
directory without 'O_DIRECTORY' would return a filehandle to a default
file within the directory. Something like open("/path/to/dir", O_RDONLY)
would be translated to open('/path/to/dir/_contents', O_RDONLY).

So I could simply replace all my files with directories and move the
original file contents into a '_contents' file. Applications can still
open them by the same name, although the attributes would look a bit
different. Poof, we have the same file-as-dir concept, but now we play
fair with the existing VFS and don't introduced races/deadlocks and
other issues. Also existing backup tools will be able to back up and
restore these 'streams'.

(btw. the same could be implemented completely in userspace, for
instance in glibc. Whenever an open call gets an EISDIR error, simply
retry the open, but this time with /_contents appended).

> I for one would truly welcome the coming of thumbnails and descriptions
> in picture files, because I have a real-life project going on where
> that would be extremely handy to have in the actual file.

Ehhh, that already exists and doesn't require meta-data streams,

    http://www.exif.org/

> >Find some silly person with an iBook and open a shell on OS X. Use cp
> >to copy a file with a resource fork. Oh look, the Finder has no idea
> >what the new file is, even though it looks exactly identical in the
> >shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
> >fork. How is that ever going to work?
> 
> Then I guess OS X ships a broken implementation of cp, yes?

No cp is working just fine, there is no copy syscall in the VFS, so any
file system specific features such meta data streams will have to be
handled by the application.

So at some point 'cp' will have to be taught about macos resource forks,
posix acls, xattrs, afs acls, reiserfs file-as-dir, freebsd attributes,
samba/ntfs streams. And possibly even learn the semantical differences
between these so that it can copy metadata between the various file
systems types.

Jan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Jan Harkes wrote:
> 
> (btw. the same could be implemented completely in userspace, for
> instance in glibc. Whenever an open call gets an EISDIR error, simply
> retry the open, but this time with /_contents appended).

Yes, and no - just to make it obvious before people jump on this, a lot of 
things can be prototyped in user space with things like this, but once you 
have to deal with races and mixed tool environments, user space suddenly 
doesn't work so well any more.

I think Jan understands this distinction, I just wanted to make sure 
everybody else is aware of the _one_ thing that kernel land does well:

 - safely synchronize globally visible data structures

That's quite fundamental. 99% of what a kernel does is exactly that. TCP
would be in user space too, if it wasn't for _exactly_ this issue. A lot
of people think that kernels are about hardware access, and yes, that's
the other 99% of the picture (I see the _big_ picture, remember?), but the 
"safe access to common data" is really very fundamental.

The kernel is literally the thing that makes sure that you don't have -
and _cannot_ have - user programs that confuse each other by modifying 
data unsynchronized.

For example, a filesystem is really nothing but a way to access a disk in 
a controlled manner - it's not so much about hardware access, as it is 
about maintaining a coherent view of how some shared data (disk or 
whatever) is maintained. 

Same goes for caches. We could cache things in user space, but if you want 
to _share_ your caches (so that you don't have to re-load them for every 
new application), you need some entity that manages those shared data 
structures in a secure manner. In other words, you need the kernel.

The same goes for something like a "container file". Whether you see it as
"dir-as-file" or "file-as-dir" (and I agree with Jan that the two are
totally equivalent), the point of having the capability in the kernel is
not that the operations cannot be done in user space - the point is that
they cannot be done in user space _safely_. The kernel is kind of the
thing that guarantees that everybody follows the rules.

Imagine the security problems if a set-uid program were to (unwittingly) 
depend on a user-space library that implements what Jan's prototype 
library would do? Races galore, since a user-space implementation wouldn't 
have _any_ way to do tests like the above atomically. 

		Linus

Re: silent semantic changes with reiser4

[Jan Harkes]

I trimmed/adjusted the CC list somewhat in the hope of getting a more
productive discussion going.

On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote:
> On Thu, 26 Aug 2004, Jan Harkes wrote:
> > 
> > (btw. the same could be implemented completely in userspace, for
> > instance in glibc. Whenever an open call gets an EISDIR error, simply
> > retry the open, but this time with /_contents appended).
> 
> Yes, and no - just to make it obvious before people jump on this, a lot of 
> things can be prototyped in user space with things like this, but once you 
> have to deal with races and mixed tool environments, user space suddenly 
> doesn't work so well any more.

Agreed.

> The same goes for something like a "container file". Whether you see it as
> "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are
> totally equivalent), the point of having the capability in the kernel is

Only to some point. I was mostly trying to make the point that similar
functionality is already achievable within the current VFS framework.

However, files and directories still have some semantical differences,
and these are probably the points that we would have to address to get
Hans's efforts work well in cooperation with the Linux VFS.

We cannot atomically unlink a dir-as-file, but can unlink a file-as-dir.
In this case, the dcache seems to have no problems unlinking/unhashing a
non-empty directory, so I'm assuming that the unlink problem is mostly
enforced to avoid unreachable, but still allocated files in a file
system. (I'm sure that Viro can probably come up with other reasons why
unlinking a non-empty directory can be evil).

Also, we cannot create hardlinks to a directory (-as-file), but I guess
reiserfs4 allows hardlinks to files (-as-directory). Luckily as far as
hardlinked directories are concerned the file-as-dir implementation
should be a relatively benign subset. I assume that we are not allowed
to rename or link arbitrary objects from the regular fs into it, Coda
returns EXDEV in such situations. As a result there will be no cycles
and this simple premise should get rid of at least a couple of the
possible deadlock scenarios that I've seen mentioned.

Jan

Re: silent semantic changes with reiser4

[Tom Vier]

On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote:
>  - safely synchronize globally visible data structures
> That's quite fundamental. 99% of what a kernel does is exactly that. TCP
> would be in user space too, if it wasn't for _exactly_ this issue. A lot

What about microkernels? They do tcp in userspace. So did winsock, iirc. As
long as a trusted process keeps data such as free ports, what's the problem?
Perhaps a process could even simply hand off proto:port handles and let a
library do the rest, but i know little about exokernels. (There's probably
some catches.)

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Mon, 30 Aug 2004, Tom Vier wrote:

> On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote:
> >  - safely synchronize globally visible data structures
> > That's quite fundamental. 99% of what a kernel does is exactly that. TCP
> > would be in user space too, if it wasn't for _exactly_ this issue. A lot
> 
> What about microkernels? They do tcp in userspace.

No they don't. They do TCP in a separate address space from user space, 
that just also happens to be separate from the "microkernel address 
space".

So a microkernel will have _more_ address spaces, and they won't be "user
space". They'll be "server deamon space" or something. Now, that's also
why they tend to have performance problems - because you need to copy the
data between different address spaces, and switch the CPU context etc
around.

Not user space. They may be "ring 3" from a CPU standpoint, but they 
aren't user space from a _user_ standpoint - it's still very much a 
separate address space, with domain protection.

> So did winsock, iirc.

Now that is a different case. Things like the PalmOS TCP stack (and, I
believe, Winsock) are true "user space" TCP stacks, in that they really
do live as libraries in the same address space as the user app.

It sucks. Exactly because now the data structures are _not_ protected, and
they are _not_ shared. So your library basically ends up being a "single
client" library, with no protection between clients (or no sharing: you
can have "protected" single-stream TCP, but then you won't share the TCP
state that needs to be shared like listen queues etc).

This works in an environment like Palm or Win-3.0, which are really just
single-client _anyway_, without any protection. But notice how Windows
started doing TCP in the kernel, and notice how you can actually use it as
a server these days?

In short: you _need_ to have a separate address space (either kernel, or
"TCP server" or whatever) if you want to have reliable, secure and
generally usable TCP.


> As long as a trusted process keeps data such as free ports, what's the
> problem?

None - because it's not user space any more. 

Well, performance might still suck, of course. And it does. 

		Linus

Re: silent semantic changes with reiser4

[Jeff Garzik]

Linus Torvalds wrote:
> 
> On Mon, 30 Aug 2004, Tom Vier wrote:
> 
> 
>>On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote:
>>
>>> - safely synchronize globally visible data structures
>>>That's quite fundamental. 99% of what a kernel does is exactly that. TCP
>>>would be in user space too, if it wasn't for _exactly_ this issue. A lot
>>
>>What about microkernels? They do tcp in userspace.
> 
> 
> No they don't. They do TCP in a separate address space from user space, 
> that just also happens to be separate from the "microkernel address 
> space".
> 
> So a microkernel will have _more_ address spaces, and they won't be "user
> space". They'll be "server deamon space" or something. Now, that's also
> why they tend to have performance problems - because you need to copy the
> data between different address spaces, and switch the CPU context etc
> around.


Man, this thread came a long way.  I peek in and see a pet topic...


_My_ own definition of microkernel, which differs from traditional CS, 
is where I would love to see Linux go:  move as much as humanly possible 
to userspace, such that, the kernel only contains pagecache operations, 
driver fast paths, and security checks.  Move slow paths, including ACPI 
probing, PCI bus walking, driver init/reset paths, some of the ioctl 
handling, to userspace.  Be willing to accept extra context switches as 
a cost in all but the fast paths.

What you have left after you move all the slow paths to userspace is my 
version of a microkernel.  The kernel is still monolithic and a single 
address space, but a lot smaller.

Now this (a) is likely just a pipe dream and (b) will increase 
complexity because each driver will _require_ a userspace component, but 
hey... you gotta have goals in life.

	Jeff

Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Tue, Aug 31, 2004 at 12:38:14AM -0400, Jeff Garzik wrote:
> _My_ own definition of microkernel, which differs from traditional CS, 
> is where I would love to see Linux go:  move as much as humanly possible 
> to userspace, such that, the kernel only contains pagecache operations, 
> driver fast paths, and security checks.  Move slow paths, including ACPI 
> probing, PCI bus walking, driver init/reset paths, some of the ioctl 
> handling, to userspace.  Be willing to accept extra context switches as 
> a cost in all but the fast paths.

	http://www.dragonflybsd.org

DragonFly BSD, the only remotely functional open source BSD project on this
planet, has plans in place to push certain kernel components like their VFS
layer into userspace for easier debugging, testing and other things that go
with developing file systems easily. If they back it with something like C++
for doing constructor style type conversion on top of overloaded operators
to back VFS data structure translation, could easily import stuff like most
Linux file systems without major restructuring, say, if they had their
translation library written. In this case, userspace kernel systems have
some serious programming advantages over traditional kernels.

They're also pushing an async syscall model to support a non-1:1 threading
system for userspace unlike what Linux has done with futexes. It'll allow
for them to plug in various userspace schedulers at will for whatever
scheduling policy you want dynamically, even while the program is running.

This isn't mentioning their single system image clustering goals that have
abstracted various kernel systems like their VFS stuff to a messaging model
which should get them proccess migration as an intrinsic part of their OS
design. They've made a lot of progress over the last year and it's largely
because of Matt Dillon's ability to create a non-overly political group that
creates trust. He's a true leader, not some random loud mouth(s) unlike
other BSD efforts. It's much like the Linux community in this regard with
Linus.

bill

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Jeff Garzik]

Bill Huey (hui) wrote:
> DragonFly BSD, the only remotely functional open source BSD project on this
> planet, has plans in place to push certain kernel components like their VFS
> layer into userspace for easier debugging, testing and other things that go

That violates Jeff's First Rule, put the fast path stuff in the kernel.


> with developing file systems easily. If they back it with something like C++
> for doing constructor style type conversion on top of overloaded operators
> to back VFS data structure translation, could easily import stuff like most
> Linux file systems without major restructuring, say, if they had their
> translation library written. In this case, userspace kernel systems have
> some serious programming advantages over traditional kernels.

Sounds like security would be a pain in the ass :)


> They're also pushing an async syscall model to support a non-1:1 threading
> system for userspace unlike what Linux has done with futexes. It'll allow

messaging passing is also known as "really slow C function calls"

My PCI bus passes messages all the time.  A message is in the eye of the 
beholder.

	Jeff

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Tue, Aug 31, 2004 at 01:56:45AM -0400, Jeff Garzik wrote:
> Bill Huey (hui) wrote:
> >DragonFly BSD, the only remotely functional open source BSD project on this
> >planet, has plans in place to push certain kernel components like their VFS
> >layer into userspace for easier debugging, testing and other things that go
> 
> That violates Jeff's First Rule, put the fast path stuff in the kernel.

It's not for native file system, but one that are coming around. I kind
of misspoke when I said that the VFS layer is going to be pushed into
userspace. It's not. Access to the VFS layer is going to be exported to
userspace. The system is interesting because of its potential programming
flexibility.

> >with developing file systems easily. If they back it with something like 
> >C++
> >for doing constructor style type conversion on top of overloaded operators
> >to back VFS data structure translation, could easily import stuff like most
> >Linux file systems without major restructuring, say, if they had their
> >translation library written. In this case, userspace kernel systems have
> >some serious programming advantages over traditional kernels.
> 
> Sounds like security would be a pain in the ass :)

Don't know. But if you can get away from the typical C programming errors, then
it could be a win.

> >They're also pushing an async syscall model to support a non-1:1 threading
> >system for userspace unlike what Linux has done with futexes. It'll allow
> 
> messaging passing is also known as "really slow C function calls"
> 
> My PCI bus passes messages all the time.  A message is in the eye of the 
> beholder.

Message passing is used as a method of concurrency in that system outside
of the typical use of function calles. It makes access to kernel facilities,
remote or local, transparent. They also don't use a mutex for their locking,
but something called a token which is per-CPU kind of locking primitive. It's
definitely an interesting system with a lot of potential.

bill

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Tue, Aug 31, 2004 at 03:56:13PM +0200, Diego Calleja wrote:
> El Tue, 31 Aug 2004 01:56:45 -0400 Jeff Garzik <jgarzik@pobox.com> escribi?:
> > messaging passing is also known as "really slow C function calls"
> > 
> > My PCI bus passes messages all the time.  A message is in the eye of the 
> > beholder.
> 
> There're some numbers for that:
> 
> http://lists.freebsd.org/pipermail/freebsd-hackers/2003-August/002426.html

As you can see the numbers are very fast for a general purpose system
like that. Add that with their XIO framework for data propagation and
you have a really amazingly fast system that combines both aggressive
design and optimized low level facilities for both remote and local
commuication.

Matt's one of the very few open source engineers on this planet that
can do both well.

bill

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Diego Calleja]

El Tue, 31 Aug 2004 13:52:11 -0700 Bill Huey (hui) <bhuey@lnxw.com> escribió:

> As you can see the numbers are very fast for a general purpose system
> like that. Add that with their XIO framework for data propagation and

For the syscall case, I remember that they would be able to "batch" syscalls,
so this framework could have even better performance in some cases.
The good thing is that they didn't do it just for the sake of doing it
(like microkernel people) but to achieve their goals (SSI, etc)

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Thu, Sep 02, 2004 at 07:20:58PM +0200, Diego Calleja wrote:
> El Tue, 31 Aug 2004 13:52:11 -0700 Bill Huey (hui) <bhuey@lnxw.com> escribi?:
> 
> > As you can see the numbers are very fast for a general purpose system
> > like that. Add that with their XIO framework for data propagation and
> 
> For the syscall case, I remember that they would be able to "batch" syscalls,
> so this framework could have even better performance in some cases.
> The good thing is that they didn't do it just for the sake of doing it
> (like microkernel people) but to achieve their goals (SSI, etc)

For what they're doing, they have minimize the overhead cross boundaries
as much as possible for usage in both user and kernel spaces. If dfBSD
achieves what they intend to do, they'll have on of the best SSI systems
on the planet and it'll be able to use the VFS layer in ways that other
systems can't even touch with iSCSI drivers, etc...

They've got process suspend and restore already in order to support
serialization across machines. Having it VFS driven, VM objects, etc...
makes the concept of migration a function of the VFS layer. It's quite
clever and I'm sure that it's driven by Matt's experience with distributed
databases, specifically Backplane.

All locking in his system is CPU local, no mutexes of any sort, so
migration as a function of messaging, scheduling and other things are
done in per-CPU chunks. It's quite clever make handling in per-process
resources chunks.

	http://twiki.im.ufba.br/pub/MAT154/WebHome/duality78.pdf

Talks about part of this and how "procedural" is isomorphic to "message-passing"
concurrency system. One can be transformed to another.

It's a project that's move much faster in the area of SMP than FreeBSD 5.x
(Solaris style SMP) and really the only BSD project on this planet that has
the talent and social chemistry to make it happen.

bill

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Tue, Aug 31, 2004 at 03:56:13PM +0200, Diego Calleja wrote:
> There're some numbers for that:
> 
> http://lists.freebsd.org/pipermail/freebsd-hackers/2003-August/002426.html

Link about XIO and things hijacked from FreeBSD-current.

http://leaf.dragonflybsd.org/mailarchive/kernel/2004-03/msg00469.html

bill

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 841 bytes --]

Salut,

On Mon, Aug 30, 2004 at 10:30:55PM -0700, Bill Huey wrote:
> DragonFly BSD, the only remotely functional open source BSD project on this
> planet, has plans in place to push certain kernel components like their VFS
> layer into userspace for easier debugging, testing and other things that go
> with developing file systems easily. If they back it with something like C++
> for doing constructor style type conversion on top of overloaded operators
> to back VFS data structure translation, could easily import stuff like most
> Linux file systems without major restructuring, say, if they had their
> translation library written. In this case, userspace kernel systems have
> some serious programming advantages over traditional kernels.

Uh, what about enhancing lufs code in this regard and porting all fs'es to it?

				Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Userspace file systems

[Luca Ferroni]

Il Tue, 31 Aug 2004 11:01:58 +0200,  Tonnerre <tonnerre@thundrix.ch> ha scritto:

> Uh, what about enhancing lufs code in this regard and porting all fs'es to it?
> 

I think that porting all file systems to LUFS is a mad idea :), 
but providing a facility in the kernel for user-space file system 
implementations could be great.

I am working on PackageFS (http://packagefs.sourceforge.net)
which is an user-space file system to manage packages:
it relies on LUFS, but I plan to port it soon to FUSE ( http://sourceforge.net/projects/avf)
that, at present, is maintained by Miklos Szeredi and works with latest kernels.

PackageFS relies also on existent package managers, it will be transparent to all
package formats. I. e.: you can install a package by copying, 
and see package informations by a simple cat.
Now it works only in Debian distro, and it is far from complete (it is 0.09 version :) ).

So, what do you think about including a kernel module to make a user able 
to develop, install and use an user-space file system ?

Maybe Miklos Szeredi will write more details about FUSE which is the interesting part
for kernel inclusion.

Bye
Luca

-- 
-----------------------------------------------
They'll never stole us our.... FREEDOM!!!
Luca Ferroni
ICQ #317977679
www.cs.unibo.it/~fferroni/
-----------------------------------------------

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

[Bill Huey]

On Tue, Aug 31, 2004 at 11:01:58AM +0200, Tonnerre wrote:
> Salut,
> 
> On Mon, Aug 30, 2004 at 10:30:55PM -0700, Bill Huey wrote:
...
> > translation library written. In this case, userspace kernel systems have
> > some serious programming advantages over traditional kernels.
> 
> Uh, what about enhancing lufs code in this regard and porting all fs'es to it?

I wasn't aware that your project has gotten this far. I'll tell Hiten
about this to see if some of this can be used in their project.

Thanks

bill

Re: silent semantic changes with reiser4

[Ryan Breen]

On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote:
> 
> Man, this thread came a long way.
> 

You said it -- from Reiser to microkernel.  If we can only figure out
a way to get a BitKeeper discussion going, we'll have the Grand
Unified Flamewar.

Re: silent semantic changes with reiser4

[Jeff Garzik]

Ryan Breen wrote:
> On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote:
> 
>>Man, this thread came a long way.
>>
> 
> 
> You said it -- from Reiser to microkernel.  If we can only figure out
> a way to get a BitKeeper discussion going, we'll have the Grand
> Unified Flamewar.


If you can find a link between the reiserfs thread and the webcam driver 
thread, you've gone all the way from filesystems to Nazis, thus invoking 
Godwin's Law ;-)

	Jeff


[1] http://en.wikipedia.org/wiki/Godwin's_law

Re: silent semantic changes with reiser4

[Clay Haapala]

On Tue, 31 Aug 2004, Ryan Breen verbalised:
> On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote:
>> 
>> Man, this thread came a long way.
>> 
> 
> You said it -- from Reiser to microkernel.  If we can only figure out
> a way to get a BitKeeper discussion going, we'll have the Grand
> Unified Flamewar.

OK, I'll byte: consider file versioning of an SCM implemented as a
ReiserFS 4 plug-in, as in:

diff -u ultrastore.c/delta/3.2.3 ultrastore.c/delta/3.2.4

(Brrr!  Just had a VMS flash-back shiver.)
-- 
Clay Haapala (chaapala@cisco.com) Cisco Systems Storage +1 763-398-1056
   6450 Wedgwood Rd, Suite 130 Maple Grove MN 55311 PGP: C89240AD
 The mysterious blob seen in satellite photos off the east coast of Florida
 during the passing of Hurricane Charley has now been positively identified
 as a large cloud of Democrat absentee-ballot applications and voter lists.

Re: silent semantic changes with reiser4

[Yury Umanets]

Clay Haapala wrote:

>On Tue, 31 Aug 2004, Ryan Breen verbalised:
>  
>
>>On Tue, 31 Aug 2004 00:38:14 -0400, Jeff Garzik <jgarzik@pobox.com> wrote:
>>    
>>
>>>Man, this thread came a long way.
>>>
>>>      
>>>
>>You said it -- from Reiser to microkernel.  If we can only figure out
>>a way to get a BitKeeper discussion going, we'll have the Grand
>>Unified Flamewar.
>>    
>>
>
>OK, I'll byte: consider file versioning of an SCM implemented as a
>ReiserFS 4 plug-in, as in:
>
>diff -u ultrastore.c/delta/3.2.3 ultrastore.c/delta/3.2.4
>
>(Brrr!  Just had a VMS flash-back shiver.)
>  
>
hehe ;) Seems to develop this extention NAMESYS should stop use 
BitKeeper. Or should buy it.

-- 
umka

Re: silent semantic changes with reiser4

[Alan Cox]

On Maw, 2004-08-31 at 05:08, Linus Torvalds wrote:
> > What about microkernels? They do tcp in userspace.
> 
> No they don't. They do TCP in a separate address space from user space, 
> that just also happens to be separate from the "microkernel address 
> space".

Several do TCP in user space. The only thing you need in kernel for
TCP/IP is enough decode to decide who gets the packet. Even some non
microkernel embedded OS's do this in order to keep kernel size down.

Alan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Alan Cox wrote:
> 
> Several do TCP in user space. The only thing you need in kernel for
> TCP/IP is enough decode to decide who gets the packet.

Only thing? I don't think so. 

You also want to make sure that regular users cannot send "impossible" 
packets. Think about the old "ping of death" kind of thing, where a normal 
mis-behaving (and I'm not saying intentionally so: it might be a small bug 
that just overwrites some data) program should _not_ be able to cause 
problems on the network.

Admins absolutely _hate_ that. They will ban an OS if it sends out packets
that cause troublem. You should remember that - we used to do strange
things on the net (long long time ago), and we brought down servers by
mistake, and nobody ever considered it a server bug: it was a Linux bug
that it wouldn't do the right thing.

Things like not sending FIN-packets when a program suddenly goes away is 
NOT acceptable behaviour! Neither is it acceptable behaviour to allow user 
programs to make up their own packets.

> Even some non microkernel embedded OS's do this in order to keep kernel
> size down.

..and I'm not disagreeing that it doesn't happen. I explicitly mentioned 
PalmOS, I bet it happens in other cases too. But I'd strongly argue that 
it's a bug, not a feature.

It's a bug that people tend to accept in a "single-client" environment. 

NOTE! This is totally ignoring the fact that you can't be called "UNIX" 
any more. You _need_ to have sequence numbers etc be shared between 
multiple programs that all write to the stream. Again, that _does_ mean 
that you have another protection domain (aka "kernel" or "TCP deamon") 
that keeps track of the sequence number. 

		Linus

Re: silent semantic changes with reiser4

[Frank van Maarseveen]

On Tue, Aug 31, 2004 at 10:15:25AM -0700, Linus Torvalds wrote:
> 
> Admins absolutely _hate_ that. They will ban an OS if it sends out packets
> that cause troublem. You should remember that - we used to do strange
> things on the net (long long time ago), and we brought down servers by
> mistake, and nobody ever considered it a server bug: it was a Linux bug
> that it wouldn't do the right thing.
> 
> Things like not sending FIN-packets when a program suddenly goes away is 
> NOT acceptable behaviour! Neither is it acceptable behaviour to allow user 
> programs to make up their own packets.

The user/kernel distinction is not always (heck, maybe almost never)
present in the embedded world (a large world). The notion of a "regular
user" does not apply at all in such a case. This is not a bug but merely
the state of technology. It will slowly go away I think because embedded
software becomes more complex and hardware becomes cheaper.

To implement multiple TCP clients _and_ the TCP/IP stack in one space is
perfectly possible and it's actually done in practice. It has advantages
(speed, when done properly) and disadvantages (complexity/bug-prone).

> 
> NOTE! This is totally ignoring the fact that you can't be called "UNIX" 
> any more. You _need_ to have sequence numbers etc be shared between 
> multiple programs that all write to the stream. Again, that _does_ mean 
> that you have another protection domain (aka "kernel" or "TCP deamon") 
> that keeps track of the sequence number. 

There is nothing in the networking or UNIX standards that prescibe another
protection domain for this. Would be insane to leave that out in a hosted
environment but it _can_ be done without.

-- 
Frank

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Tue, 31 Aug 2004, Frank van Maarseveen wrote:
> 
> There is nothing in the networking or UNIX standards that prescibe another
> protection domain for this. Would be insane to leave that out in a hosted
> environment but it _can_ be done without.

My point is that TCP _does_ have a lot of state that needs to be handled 
in a safe manner by a proper operating system.

The fact that there are OS's out there that are crap doesn't change that 
matter. There are lots of embedded OS's out there that still do 
multitasking in a purely cooperative way. I don't think it's a valid model 
for anything but toys. Same goes for putting TCP in user space. It's 
doable, but it's not an "OS". It's a program loader.

		Linus

Re: silent semantic changes with reiser4

[Toon van der Pas]

On Mon, Aug 30, 2004 at 09:08:07PM -0700, Linus Torvalds wrote:
> 
> 
> On Mon, 30 Aug 2004, Tom Vier wrote:
> 
> > On Thu, Aug 26, 2004 at 09:48:04AM -0700, Linus Torvalds wrote:
> > >  - safely synchronize globally visible data structures
> > > That's quite fundamental. 99% of what a kernel does is exactly
> > > that. TCP would be in user space too, if it wasn't for
> > > _exactly_ this issue. A lot
> > 
> > What about microkernels? They do tcp in userspace.
> 
> No they don't. They do TCP in a separate address space from user
> space, that just also happens to be separate from the "microkernel
> address space".

The HP NonStop Server (formerly called Tandem) has an MPP architecture
and a message passing OS (the NonStop Kernel). On this OS the TCP/IP
stack is implemented as a library against which the application
software is linked. The ethernet controllers/drivers maintain a
routing table for routing incoming traffic to the right process.
So the TCP/IP stack is implemented in userspace on the NonStop
Server, I would say.

Regards,
Toon.
-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Tom Vier]

On Mon, Aug 30, 2004 at 09:08:07PM -0700, Linus Torvalds wrote:
> > What about microkernels? They do tcp in userspace.
> 
> No they don't. They do TCP in a separate address space from user space, 
> that just also happens to be separate from the "microkernel address 
> space".

Well, I'd call that userspace (if it has a pid and it's own addr space, like
every other proc).

> So a microkernel will have _more_ address spaces, and they won't be "user
> space". They'll be "server deamon space" or something. Now, that's also
> why they tend to have performance problems - because you need to copy the
> data between different address spaces, and switch the CPU context etc
> around.

Just out of curiousity, what do you think of L4? I don't remember the
numbers, but it wasn't much slower than linux (iirc), even on x86. I think
k42 has msg passing close to the speed of a syscall. (Bulk data could use
shared mem, maybe.)

Anyway, it would be neat to have a tcp/ip daemon. You could run experimental
code w/o fear of it clobbering other things. Somethings, of course, a
microkernel can't help (if the rootfs's fs daemon dies, you're screwed). A
little bit more containment from bad kernel code would be nice.

(BTW, for those who don't know, mach optimized away msg passing and used a
single addr space, once code was "trusted". You could choose speed over
protection. That wasn't a hard choice since Mach msg passing sucks. Mach
gave microkernels a bad name.)

> Not user space. They may be "ring 3" from a CPU standpoint, but they 
> aren't user space from a _user_ standpoint - it's still very much a 
> separate address space, with domain protection.

How are they different from regular user procs, other then being trusted to
manage certain resources?
> In short: you _need_ to have a separate address space (either kernel, or
> "TCP server" or whatever) if you want to have reliable, secure and
> generally usable TCP.

Exokernels are another topic. 8)

> > As long as a trusted process keeps data such as free ports, what's the
> > problem?
> 
> None - because it's not user space any more. 

Yes it is, it's a user process.

> Well, performance might still suck, of course. And it does. 

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 2 Sep 2004, Tom Vier wrote:
>
> > Not user space. They may be "ring 3" from a CPU standpoint, but they 
> > aren't user space from a _user_ standpoint - it's still very much a 
> > separate address space, with domain protection.
> 
> How are they different from regular user procs, other then being trusted to
> manage certain resources?

Ehh, they are separate the same way "inetd" is separate. It's not a _user_ 
proc, it's a system proc. The user can't actually do anything about it.

In many ways UNIX _is_ a microkernel. It does nonessential stuff in "user
space".  Anything that is critical for performance or the working of the
machine is in kernel space.

The big difference between UNIX and what people _call_ "microkernels" is 
that UNIX has a very functional and sane partitioning of what is a 
critical thing.

But from a kernel _protection_ angle, the only part that is important is 
that the services be in some protected domain. That was what started this 
discussion: 99% of what the kernel does is protecting shared data. Whether 
it does so by passing it on to some trusted third party or not is an 
implementation issue, and is totally pointless from a user standpoint, 
since the user won't see it anyway.

		Linus

Re: silent semantic changes with reiser4

[Timothy Miller]

Linus Torvalds wrote:

> 
> The same goes for something like a "container file". Whether you see it as
> "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are
> totally equivalent), the point of having the capability in the kernel is
> not that the operations cannot be done in user space - the point is that
> they cannot be done in user space _safely_. The kernel is kind of the
> thing that guarantees that everybody follows the rules.
> 

Well, it CAN be done safely if every client has to go through the kernel 
which does all the appropriate locks/semaphors and then passes the 
request down to the daemon.  (Isn't NFS implemented this way?)

This is very micro-kernel-ish, but it's a reasonable idea to do it this 
way in cases where things are non-critical.

Say there's a way to cd into a tgz file to look around.  If the access 
methods through the kernel get routed back to a user-space process 
(which probably does some amount of caching in memory and on disk of 
uncompressed bits of the archive), it could be a bit slower than if it 
were all in-kernel.  The thing is that the processing time in the daemon 
is probably quite high compared to the overhead of the context switches, 
so it doesn't cost much.  (And if it CAN be done in userspace without 
being horribly convoluted or unsafe, then it SHOULD be done in 
userspace.)  Besides, even if it were a LOT slower to access a tgz file 
without extracting it first, I would STILL think it was wonderful AND 
use it a LOT.

Heh... now, the next thing is to be able to do some sort of union-mount 
like thing where the tgz is really read-only but if you WRITE to it, it 
gets stored somewhere else.  This way, you can compile a kernel tree 
without extracting it first.

Taking this a bit further, the union-mount could be done away with if 
the daemon were to lazily rebuild the tgz file from its cache when the 
tgz is modified.  That is, writes have little or no penalty because they 
just get stored in the daemon's cache, and the daemon rebuilds the 
archive in the background.

The VFS layer would have to recognize that a particular file has been 
accessed in such a way that it must go through the daemon for ANY access 
to it, which means that if someone tries to access the original archive 
as a file, requests have to go through the daemon.  This makes it 
coherent, although with an expected performance penalty.

One issue to consider is that of shutdowns, both intentional and 
crashes.  This is a non-issue with an underlying journaling file system. 
  As long as the tgz daemon comes up early enough, it would see its 
cache (which is on-disk anyway), recognize which archive it belongs to, 
and continue where it left off.  This way, the daemon doesn't have to go 
through a long process of rebuilding every archive you've ever modified 
on the disk on shut-down.  If, however, you needed to access the archive 
without the daemon, that could be a problem (but what you'd get is 
probably the archive before it was modified).  Also, some sort of sync 
command would be needed to cause the daemon to rebuild all archives 
immediately.

I really dig the idea of being able to access archives without 
extracting them first, regardless of any performance penalty.

Re: silent semantic changes with reiser4

[Elladan]

On Fri, Sep 10, 2004 at 03:22:59PM -0400, Timothy Miller wrote:
> 
> Linus Torvalds wrote:
> 
> >The same goes for something like a "container file". Whether you see it as
> >"dir-as-file" or "file-as-dir" (and I agree with Jan that the two are
> >totally equivalent), the point of having the capability in the kernel is
> >not that the operations cannot be done in user space - the point is that
> >they cannot be done in user space _safely_. The kernel is kind of the
> >thing that guarantees that everybody follows the rules.
> 
> Well, it CAN be done safely if every client has to go through the kernel 
> which does all the appropriate locks/semaphors and then passes the 
> request down to the daemon.  (Isn't NFS implemented this way?)
> 
> This is very micro-kernel-ish, but it's a reasonable idea to do it this 
> way in cases where things are non-critical.
> 
> Say there's a way to cd into a tgz file to look around.  If the access 
> methods through the kernel get routed back to a user-space process 
> (which probably does some amount of caching in memory and on disk of 
> uncompressed bits of the archive), it could be a bit slower than if it 
> were all in-kernel.  The thing is that the processing time in the daemon 
> is probably quite high compared to the overhead of the context switches, 
> so it doesn't cost much.  (And if it CAN be done in userspace without 
> being horribly convoluted or unsafe, then it SHOULD be done in 
> userspace.)  Besides, even if it were a LOT slower to access a tgz file 
> without extracting it first, I would STILL think it was wonderful AND 
> use it a LOT.

This is only safe for some limited definitions of safety.

Leaving aside the obvious complexities of making sure the user space
daemon doesn't just do something crazy, you have a number of problems:

* Which user does the daemon run as?  If it runs as root, it needs to
  enforce strict security requirements in terms of VFS operations coming
  in, and also, it has all the problems of a SUID application running on
  an arbitrary user file.  I don't know about you, but I don't trust
  joebob's tarfsd to be suid when running on some script kiddie tarball.

  If it runs as the user making the request, then what happens when two
  users try to open the same file at the same time.  Do they see two
  different views of the file?  Is one of them locked out?  How can they
  possibly synchronize if the views are writable?

* If the daemon runs as a user, then what happens if you try to run a
  suid program inside of the view?  What if root tries to walk into a
  tar file?  The conservative view is that the kernel would need to
  always return ENOENT for any process with elevated capabilities, but
  even this is not safe.

* Consider what happens if foo.tar/blah_blah is automatically bound to
  enter a tarfsd view of foo.tar.  What happens if I point the web
  server at foo.tar/blah?  The web server runs as httpd or something, so
  presumably httpd ends up running some sort of tarfsd view on the
  file.  But if the tarball was made by a malicious person, presumably
  it can obtain httpd user access now by exploiting a bug in tarfsd.

* Even if you assume the view is read-only and the kernel coerces all
  permissions and ownership and such, there's the possibility of tarfsd
  presenting unexpected syscall results - weird error codes, short
  reads, file data changing underneath mmap, etc. that user applications
  don't expect and may be susceptible to.

* Besides all these security issues, if this scheme is writable it has
  all the resource problems that loopback network filesystems suffer.
  What if the kernel is short on memory and tries to flush dirty buffers
  to reclaim it.  If those buffers are running against the user FS
  daemon, then that daemon must wake up to clear dirty buffers.  If it
  tries to allocate memory, deadlock in the kernel.


Probably the security problems can be solved to some degree by being
very paranoid in the kernel (at an associated loss in utility), and the
resource issues can be solved by restricting dirty buffers for loopback
mounts (at an associated loss in performance), but it's hardly simple.

-J

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

I'm talking about uservfs.sf.net, aka podfuk, which implements tarfs
(and more).

> Leaving aside the obvious complexities of making sure the user space
> daemon doesn't just do something crazy, you have a number of problems:
> 
> * Which user does the daemon run as?  If it runs as root, it needs to
>   enforce strict security requirements in terms of VFS operations coming
>   in, and also, it has all the problems of a SUID application running on
>   an arbitrary user file.  I don't know about you, but I don't trust
>   joebob's tarfsd to be suid when running on some script kiddie tarball.

It needs to run as root. If there's bug in tar, that script kiddie is going
to get your users anyway. Audit the code.

> * If the daemon runs as a user, then what happens if you try to run a

It does not.

> * Consider what happens if foo.tar/blah_blah is automatically bound to
>   enter a tarfsd view of foo.tar.  What happens if I point the web
>   server at foo.tar/blah?  The web server runs as httpd or something, so
>   presumably httpd ends up running some sort of tarfsd view on the
>   file.  But if the tarball was made by a malicious person, presumably
>   it can obtain httpd user access now by exploiting a bug in tarfsd.

If there's a bug in tar, I can do a lot of interesting things. Audit tar.

> * Even if you assume the view is read-only and the kernel coerces all
>   permissions and ownership and such, there's the possibility of tarfsd
>   presenting unexpected syscall results - weird error codes, short
>   reads, file data changing underneath mmap, etc. that user applications
>   don't expect and may be susceptible to.

FUD. Fix tarfsd if its problem.

> * Besides all these security issues, if this scheme is writable it has

There are no security issues. Yes, uservfs is security-sensitive code.

>   all the resource problems that loopback network filesystems suffer.
>   What if the kernel is short on memory and tries to flush dirty buffers
>   to reclaim it.  If those buffers are running against the user FS
>   daemon, then that daemon must wake up to clear dirty buffers.  If it
>   tries to allocate memory, deadlock in the kernel.

Coda solves this: all writable things are file-backed on file in /tmp =>
bad performance if you only need piece of very large file,
but no deadlocks.

> Probably the security problems can be solved to some degree by being
> very paranoid in the kernel (at an associated loss in utility), and the
> resource issues can be solved by restricting dirty buffers for loopback
> mounts (at an associated loss in performance), but it's hardly simple.

See sources. It is <1000 lines of code.
-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: silent semantic changes with reiser4

[Pavel Machek]

Hi!

> Say there's a way to cd into a tgz file to look around.  If the 
> access methods through the kernel get routed back to a user-space 
> process (which probably does some amount of caching in memory and on 
> disk of uncompressed bits of the archive), it could be a bit slower 
> than if it were all in-kernel.  The thing is that the processing time 

Exactly. See uservfs.sf.net. It is using coda hooks into the kernel.

> be done in userspace.)  Besides, even if it were a LOT slower to 
> access a tgz file without extracting it first, I would STILL think it 
> was wonderful AND use it a LOT.

Go ahead :-).


-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

Re: silent semantic changes with reiser4

[Grzegorz Jaśkiewicz]

Markus Törnqvist wrote:

>On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote:
>  
>
>>Find some silly person with an iBook and open a shell on OS X. Use cp
>>to copy a file with a resource fork. Oh look, the Finder has no idea
>>what the new file is, even though it looks exactly identical in the
>>shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
>>fork. How is that ever going to work?
>>    
>>
>
>Then I guess OS X ships a broken implementation of cp, yes?
>
>  
>
Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that acctually care about cp being broken.

--
GJ 

Re: silent semantic changes with reiser4

[viro]

On Fri, Sep 03, 2004 at 06:43:21AM +0200, Grzegorz Ja??kiewicz wrote:
> >Then I guess OS X ships a broken implementation of cp, yes?
> >
> Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that 
> acctually care about cp being broken.

Gotta love the Mac logics: "Is $FOO broken?" - "Nope, $BAR works. Very few
care about $FOO being broken".

Re: silent semantic changes with reiser4

[Grzegorz Jaśkiewicz]

viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Fri, Sep 03, 2004 at 06:43:21AM +0200, Grzegorz Ja??kiewicz wrote:
>  
>
>>>Then I guess OS X ships a broken implementation of cp, yes?
>>>
>>>      
>>>
>>Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that 
>>acctually care about cp being broken.
>>    
>>
>
>Gotta love the Mac logics: "Is $FOO broken?" - "Nope, $BAR works. Very few
>care about $FOO being broken".
>  
>
I don't represent Mac logic ;) IMO it's wrong, but it's still a fact.

--
GJ

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 1109 bytes --]

Salut,

On Fri, Sep 03, 2004 at 06:43:21AM +0200, Grzegorz Ja??kiewicz wrote:
> >Then I guess OS X ships a broken implementation of cp, yes?
> 
> Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that 
> acctually care about cp being broken.

Actually, there is a (non-GNU) free implementation of the fileutils on
MacOS/X  that treats  and  copies  the .DS_Store  entries  (or say  is
metadata aware).

And  even  though  I might  agree  on  that  the principle  of  having
per-directory  database files is  somewhat hacky,  it appears  to work
like a charm (As long as userland programs do care about it).

On Linux we have a solution that actually works just the same and is a
lot cooler. It's called POSIX extended attributes...

			    Tonnerre

PS. If you want to talk further  on the subject of how MacOS/X does it
and how I would do it using extended attributes, discuss it in private
mail with me, since MacOS/X isn't really Linux. And no, I don't want a
single daemon do  that, but I want it to  be a standard implementation
that libc and fileutils support.

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Helge Hafting]

Matt Mackall wrote:
[...]

>What it breaks is the concept of a file. In ways that are ill-defined,
>not portable, hard to work with, and needlessly complex. Along the
>way, it breaks every single application that ever thought it knew what
>a file was.
>
>Progress? No, this has been done before. Various dead operating
>systems have done it or similar and regretted it. Most recently MacOS,
>which jumped through major hurdles to begin purging themselves of
>resource forks when they went to OS X. They're still there, but
>heavily deprecated.
>
>  
>
>>cp, grep, cat, and tar will continue to work just fine on files with
>>multiple streams.
>>    
>>
>
>Find some silly person with an iBook and open a shell on OS X. Use cp
>to copy a file with a resource fork. Oh look, the Finder has no idea
>what the new file is, even though it looks exactly identical in the
>shell. Isn't that _wonderful_? 
>
It is what I'd expect.  Now, use cp -R to copy  the file
_with its directory_, and see if that fares better.  If not - bad
implementation of fs and/or cp.  The way I see file-as -directory
is that _file_ operations (like the reads issued by cat) only
work on the file part.  You want the directory part?  Use
directory operations such as those "cp -R" use.

>Now try cat < a > b on a file with a
>fork. How is that ever going to work?
>  
>
It is going to copy the _file_ part, of course.  I wouldn't
expect anything else - "cat" doesn't deal with directories.

This also raise the question of when to use file-as-directory.
A usage where you need everything to follow the file, even
when using "cat" calls out for an application that puts everything
into one file.  Directory-as-file is the wrong tool for that job, so don't
worry about such problems.

Sticking thumbnails in a file-as-dir is another story though.  Move the
file (with "mv a b", not  "cat a>b;rm a") moves the image file _and_ the
thumbnail.  No time wasted on regenerating thumbnails, no disk space
or cleanup time wasted on dangling thumbs.  Use the image file
for its intended purposes (view it, mail it off, serve it on the web,
edit it, print it) and the thumbnail doesn't get in the way
because it isn't embedded in the file format.  Embedding it
in the file might work for jpeg which support generic embedded data,
it surely won't work for every image format out there.

This is my idea of how file-as-directory should work.

Helge Hafting


Helge Hafting

Re: silent semantic changes with reiser4

[Horst von Brand]

Helge Hafting <helge.hafting@hist.no> said:
> Matt Mackall wrote:

[...]

> >Find some silly person with an iBook and open a shell on OS X. Use cp
> >to copy a file with a resource fork. Oh look, the Finder has no idea
> >what the new file is, even though it looks exactly identical in the
> >shell. Isn't that _wonderful_? 

> It is what I'd expect.

Great. That means that all the tools of the trade stop working. Sounds like
show-killer feature to me.

>                         Now, use cp -R to copy  the file
> _with its directory_,

Either it is a file or a directory. Make up your mind. If you have no clear
distinction, you'll only get messed up. Badly.

>                       and see if that fares better.  If not - bad
> implementation of fs and/or cp.  The way I see file-as -directory
> is that _file_ operations (like the reads issued by cat) only
> work on the file part.  You want the directory part?  Use
> directory operations such as those "cp -R" use.

Excuse me, I must grab my sickness bag here.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Helge Hafting]

On Thu, Aug 26, 2004 at 02:17:03PM -0400, Horst von Brand wrote:
> Helge Hafting <helge.hafting@hist.no> said:
> > Matt Mackall wrote:
> 
> [...]
> 
> > >Find some silly person with an iBook and open a shell on OS X. Use cp
> > >to copy a file with a resource fork. Oh look, the Finder has no idea
> > >what the new file is, even though it looks exactly identical in the
> > >shell. Isn't that _wonderful_? 
> 
> > It is what I'd expect.
> 
> Great. That means that all the tools of the trade stop working. Sounds like
> show-killer feature to me.
>
All wrong.  Feel free to dislike the concept all you want, but don't provide 
false arguments.  I am sure you can think up some _good_ ones, if
you feel so strongly about this.

_No_ tool "stop working" with such a fs.  Of course the new
feature (file-as-directory) may not be supported - which is a problem
only for files that use the new feature.  Stick a lot of ordinary
files and directories in such a fs, and the tools works as before
with no breakage.  People obviously won't deploy new features before
they are useable - the fs have to come first.  Then the tools,
and widespread use only after that.  This is similiar to other extensions 
like ACLs which many tools don't understand.  They're still useable though.

Of course file-as-directory can have pilot uses like the often
mentioned thumbnails - because it doesn't matter if those aren't
backed up or get lost at random during the experimental phase.

 
> >                         Now, use cp -R to copy  the file
> > _with its directory_,
> 
> Either it is a file or a directory. Make up your mind. 

Wrong again.  The file-as-directory concept means that
a filename refers to something that is a file, or a directory,
or both.  That's the point of it.  Being both isn't
possible with current filesystems (except reiser4) of course, 
but file-as-directory is a proposed _extension_.  

> If you have no clear
> distinction, you'll only get messed up. Badly.
>
The distinction is clear enough - three options instead of two.
 
> Excuse me, I must grab my sickness bag here.

Oh, very impressive.  Please take a look at what we're talking
about before attacking.   

Helge Hafting

Re: silent semantic changes with reiser4

[Tom Vier]

What's wrong with ~/.thumbcache or a daemon that manages system wide cache?

I have no problem if a plugin wants to parse a file and split it inside the
file-as-dir. However, multiple file streams are a cross-platform nighmare.
Anyone who's used mac os knows this.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: silent semantic changes with reiser4

[Helge Hafting]

Tom Vier wrote:

>What's wrong with ~/.thumbcache or a daemon that manages system wide cache?
>
>  
>
Moving a file doen't move the associated thumbnail, and then you
notice something is missing, or don't find the file, or have to wait
for regeneration when the app notices a file without a tumb. 
That could take some time if you moved a directory full of postscript
files, for example.

Helge Hafting

Re: silent semantic changes with reiser4

[Jan Dittmer]

Zitat von Helge Hafting <helge.hafting@hist.no>:
> Tom Vier wrote:
> 
> >What's wrong with ~/.thumbcache or a daemon that manages system wide
> cache?
> >
> >  
> >
> Moving a file doen't move the associated thumbnail, and then you
> notice something is missing, or don't find the file, or have to wait
> for regeneration when the app notices a file without a tumb. 
> That could take some time if you moved a directory full of postscript
> files, for example.

Use hash + filename in ~/.thumbcache and be smart when trying to find a
thumbnail. That really can all be done in userspace.

Jan 

Re: silent semantic changes with reiser4

[Erik Hensema]

Jan Dittmer (jdittmer@ppp0.net) wrote:
> Zitat von Helge Hafting <helge.hafting@hist.no>:
>> Tom Vier wrote:
>> 
>> >What's wrong with ~/.thumbcache or a daemon that manages system wide
>> cache?
>> >
>> >  
>> >
>> Moving a file doen't move the associated thumbnail, and then you
>> notice something is missing, or don't find the file, or have to wait
>> for regeneration when the app notices a file without a tumb. 
>> That could take some time if you moved a directory full of postscript
>> files, for example.
>
> Use hash + filename in ~/.thumbcache and be smart when trying to find a
> thumbnail. That really can all be done in userspace.

Yes, it can be done. It's slow, ineffecient and dangerous (rename
a file, lose the thumbnail... so you'd have to do some garbage
collection on the thumbnail cache).

Thumbnails are actually an *excelent* example of a multistream
file. Renaming/moving a file leaves the image<->thumbnail link
intact. Removing the image removes the thumbnail. Copying the
file to another multistream-file capable machine also copies the
thumbnail, saving you the time to regenerate the thumbnail.
When the target machine doesn't understand a multistream file,
you simply lose the thumbnail, no harm done.

-- 
Erik Hensema <erik@hensema.net>

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 467 bytes --]

Salut,

On Fri, Sep 03, 2004 at 10:07:00AM +0200, Helge Hafting wrote:
> Moving a file doen't move the associated thumbnail, and then you
> notice something is missing, or don't find the file, or have to wait
> for regeneration when the app notices a file without a tumb. 
> That could take some time if you moved a directory full of postscript
> files, for example.

That's why the userland file utilities must be aware of the feature..

			    Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Helge Hafting]

Tonnerre wrote:

>Salut,
>
>On Fri, Sep 03, 2004 at 10:07:00AM +0200, Helge Hafting wrote:
>  
>
>>Moving a file doen't move the associated thumbnail, and then you
>>notice something is missing, or don't find the file, or have to wait
>>for regeneration when the app notices a file without a tumb. 
>>That could take some time if you moved a directory full of postscript
>>files, for example.
>>    
>>
>
>That's why the userland file utilities must be aware of the feature..
>  
>
That won't ever happen.  There are many ways of implementing
thumbnails, and people don't want to slow down the operation
of "mv" by having it search for _all_ the known thumbnails schemes.
(Yes - several may be in use simultaneosly because the user uses several
file managers!)  The waste of time is even more interesting for the many
directories that doesn't have thumbs at all.

A file-as-dir implementation of thumbnails avoids this particular
problem.  There are other uses, so I hope people won't see
file-as-dir as a mechanism for "file attributes" _only_.

Helge Hafting

Re: silent semantic changes with reiser4

[Spam]

>> cp, grep, cat, and tar will continue to work just fine on files with
>> multiple streams.

> Find some silly person with an iBook and open a shell on OS X. Use cp
> to copy a file with a resource fork. Oh look, the Finder has no idea
> what the new file is, even though it looks exactly identical in the
> shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
> fork. How is that ever going to work?

  It  should  work  on  the main file, the extra streams and meta data
  won't be copied.  Just as it is now with special attributes etc. Cat
  reads the content of a file stream, not the file container.

> I like cat < a > b. You can keep your progress.

Re: silent semantic changes with reiser4

[Joseph Fannin]

[-- Attachment #1: Type: text/plain, Size: 2248 bytes --]

On Thu, Aug 26, 2004 at 12:32:00AM -0500, Matt Mackall wrote:

> What it breaks is the concept of a file. In ways that are ill-defined,
> not portable, hard to work with, and needlessly complex. Along the
> way, it breaks every single application that ever thought it knew what
> a file was.
> 
> Progress? No, this has been done before. Various dead operating
> systems have done it or similar and regretted it. Most recently MacOS,
> which jumped through major hurdles to begin purging themselves of
> resource forks when they went to OS X. They're still there, but
> heavily deprecated.
> 
> > cp, grep, cat, and tar will continue to work just fine on files with
> > multiple streams.
> 
> Find some silly person with an iBook and open a shell on OS X. Use cp
> to copy a file with a resource fork. Oh look, the Finder has no idea
> what the new file is, even though it looks exactly identical in the
> shell. Isn't that _wonderful_? Now try cat < a > b on a file with a
> fork. How is that ever going to work?

   I've been known to be silly now and then, for sure.  Apple says
this about the next version of OS X, due next year:

"Tiger provides a standard, Darwin-level API for managing resource
forks, filesystem metadata, security information, properties and other
attributes in a consistent, cross-platform manner. For example, common
UNIX utilities such as cp, tar and rsync can properly handle HFS+
resource forks."

    That's marketing copy, but I think it's worth noting.  It seems to
be that resource forks aren't deprecated across the board,
just... changed.

    Apple has found a way to make this work, at least partially.  I
don't know if it's a good way, but there it is.

    I'd like to be able to keep my Macintosh stuff on my Linux
fileserver, without losing the resource forks.  That would require
that the network fs support it of course, and whether I could use tar
to back up my linux box would depend on how streams might be
implemented there.

> I like cat < a > b. You can keep your progress.

    Can't we have both?

-- 
Joseph Fannin
jhf@rivenstone.net

"Bull in pure form is rare; there is usually some contamination by data."
    -- William Graves Perry Jr.

[-- Attachment #2: Type: application/pgp-signature, Size: 186 bytes --]

Re: silent semantic changes with reiser4

[Paul Jackson]

> "OMG! It breaks tar and email!!!" argument doesn't fly. Things break all
> the time and are fixed. It's called progress.

Yes - we break things all the time.  But we take notice, when we are
breaking things, of how long standing and deeply embedded they are.

The deeper the roots, the more respect we show it, and the harder
it will be to change.

Heaping scorn on someone who is reluctant to change something as
deeply embedded as "a file is a byte stream" does not further the
discussion.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Spam]

> cp, grep, cat, and tar will continue to work just fine on files with
> multiple streams.

> tar and cp will lose the extra streams until somebody fixes them, but
> they lose ACLs and xattrs right now, and I don't hear anybody suggesting
> that ACLs and xattrs be removed from the kernel because of this.

  I  thought  that  the  underlaying  FS/VFS would make sure the extra
  streams  would  be  preserved. Normal applications that doesn't have
  support  for  them  should  not  destroy the extra streams (and meta
  data,attributes etc).

  Of course tar is different, but any probgram that uses the normal OS
  API to copy, write and read files should not affect the extra info.

  ~S

> Fixing programs that do recursive filesystem traversals is a matter of a
> glibc patch to nftw(3) and a modification to their option processing.

> Holding back a useful feature because you don't want to upgrade
> coreutils is just plain dumb.

> (BTW, for email, multipart/parallel is a start, but a specific multipart
> content type for multi-stream file attachments would probably be more
> appropriate.)

Re: silent semantic changes with reiser4

[Jamie Lokier]

Matt Mackall wrote:
> > Anything that currently stores a file's metadata in another file really
> > wants this right now. Things like image thumbnails, document summaries,
> > digital signatures, etc.
> 
> That is _highly_ debatable. I would much rather have my cp and grep
> and cat and tar and such continue to work than have to rewrite every
> tool because we've thrown the file-is-a-stream-of-bytes concept out
> the window. Never mind that I've got thumbnails, document summaries,
> and digital signatures already.
> 
> While the number of annoying properties of files with forks is
> practically endless, the biggest has got to be utter lack of
> portability. How do you stick the thing in an attachment or on an ftp
> site? Well you can't because it's NOT A FILE. 
> 
> A file is a stream of bytes.

I couldn't agree more.  Metadata which is easily lost is a terrible
place to put a "document summary".

However, it's not a bad place to put thumbnails which are generated
from the file contents, if they can be regenerated after transporting.
It's slightly better than a ".thumbnails" directory because the latter
won't follow renames and things like that.

It's a very good place to put metadata which is semantically bound to
the file in special ways not intended for transport, such as security
attributes, auto-invalidated digests of the file's contents, an
auto-unpacked view of an archive, a virtual tree of an XML or other
structured file, or an auto-generated textual representation of a
binary file.

The key concept is "not intended for transport".

Generally that means permissions metadata, and things which are
re-generated from the file's contents on demand.*

It's a very bad place to put an abstract of a text document, or the
names of authors, or the MIME content type, or the character encoding
of the document, or the name of the shell to run the file as a script,
unless these things are also deducible from the file's contents.
However, sometimes the content doesn't give any clue, and then
something like the character encoding of a text file is usefully
stored in the metadata simply because that's better than nothing.

Of course the facility can be abused.  What does Windows XP do?  Does
it only store the appropriate kind of metadata in the alternate
streams?  Even Windows XP users expect files transfered over HTTP or
WebDAV to work, don't they?

-- Jamie

* - By my definition, modification time arguably shouldn't be metadata.
    Personally I like downloads to be stamped with their source's
    modification time, which is why I don't use Mozilla's download
    manager but cut-and-paste URLs to "wget -N" instead.

Re: silent semantic changes with reiser4

[Helge Hafting]

Matt Mackall wrote:

>On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
>  
>
>>On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
>>    
>>
>>>Previously Jeremy Allison wrote:
>>>      
>>>
>>>>Multiple-data-stream files are something we should offer, definately (IMHO).
>>>>I don't care how we do it, but I know it's something we need as application
>>>>developers.
>>>>        
>>>>
>>>Aside from samba, is there any other application that has a use for
>>>them? 
>>>
>>>      
>>>
>>Anything that currently stores a file's metadata in another file really
>>wants this right now. Things like image thumbnails, document summaries,
>>digital signatures, etc.
>>    
>>
>
>That is _highly_ debatable. I would much rather have my cp and grep
>and cat and tar and such continue to work than have to rewrite every
>tool because we've thrown the file-is-a-stream-of-bytes concept out
>the window. Never mind that I've got thumbnails, document summaries,
>and digital signatures already.
>  
>
Utilities that works on files only, such as cat, should keep
working.  No problem there.  If you cat a file that also is
a directory, then the file contents is all you get - by design.

Utilities that _only_ traverse the directory tree, such as find,
should keep working too.  Perhaps with a very minor update
so they don't mistake a file-directory for a file only.  I.e.
find must recurse into anything that support directory semantics.

Something that both recurse and operate on files (cp -a, tar, grep, ...
will need minor updating.  They already knows how to handle
files and directories, now they will need an update so
they're open for objects that are both.  I.e. let grep scan the file
as usual, then recurse into its directory part in the usual way too.
Performance problems should be avoided by not supporting
directory operations on files with no directory content, which I
believe will be many of them.

The "file-as-directory" thing will not be that useful before the tools
gets these relatively simple updates.  Till then it'll be a toy, which 
shouldn't
stop it from getting into the VFS.  Updating the tools will be a task
for the file-as-dir fans.

(I don't know wether reiser4 does things this way - it is certainly
they way I would want file-as-directory though.)

>While the number of annoying properties of files with forks is
>practically endless, the biggest has got to be utter lack of
>portability. How do you stick the thing in an attachment or on an ftp
>site? Well you can't because it's NOT A FILE. 
>  
>
Stick the file in an attachment and you get the file only. 
No problem, it is designed that way.  An app that really
wants everything in a single file should use a file structured
for that, not file-as-dir.  File-as-dir attach stuff to a file in a
more loose way.

If you want to attach the directory contents too, do what you usually do
when you want to mail someone a directory tree.  You can't stick a 
directory
in an attachment because it is not a file.  So you either attach every 
file in
the tree, or use tar.  In this case, an updated tar.

The ftp server shouldn't be a problem.  It supports both files and
directories already.  It may need a minor update in order to
not mistake directory for file or vice-versa when someone
request an operation.

ftp> get filename  #Will get you the contents of the file part only - by 
design.
ftp> cd filename    #Will change into the directory (if the file indeed 
provides one.)

>A file is a stream of bytes.
>  
>
Sure.  And a file with a directory support both directory and file 
operations.
You can get the stream of bytes as usual - that's the file part.  Or you
can cd into the directory as usual. There isn't much overlap between
file operations and directory operations, so there is little conflict 
the way
I see it.  Merely letting the tools know that being a file no longer rule
out the possibilities of directory operations.  Getting the VFS right
is another matter of course, but I don't worry about userland tools.

Helge Hafting

Re: silent semantic changes with reiser4

[Spam]

> On Wed, Aug 25, 2004 at 05:42:21PM -0700, Nicholas Miell wrote:
>> On Wed, 2004-08-25 at 16:46, Wichert Akkerman wrote:
>> > Previously Jeremy Allison wrote:
>> > > Multiple-data-stream files are something we should offer, definately (IMHO).
>> > > I don't care how we do it, but I know it's something we need as application
>> > > developers.
>> > 
>> > Aside from samba, is there any other application that has a use for
>> > them? 
>> > 
>> 
>> Anything that currently stores a file's metadata in another file really
>> wants this right now. Things like image thumbnails, document summaries,
>> digital signatures, etc.

> That is _highly_ debatable. I would much rather have my cp and grep
> and cat and tar and such continue to work than have to rewrite every
> tool because we've thrown the file-is-a-stream-of-bytes concept out
> the window. Never mind that I've got thumbnails, document summaries,
> and digital signatures already.

  In  Windows,  the  extra file streams are not lost or removed if you
  use  a  program that doesn't support them. They are only lost if you
  move the file to a file system that doesn't support the streams.

  Even RAR support the NTFS file streams.

> While the number of annoying properties of files with forks is
> practically endless, the biggest has got to be utter lack of
> portability. How do you stick the thing in an attachment or on an ftp
> site? Well you can't because it's NOT A FILE. 

> A file is a stream of bytes.

Re: silent semantic changes with reiser4

[Spam]

> Previously Jeremy Allison wrote:
>> Multiple-data-stream files are something we should offer, definately (IMHO).
>> I don't care how we do it, but I know it's something we need as application
>> developers.

> Aside from samba, is there any other application that has a use for
> them?

  Yes,  for  example  documents,  image  files  etc. The multiple data
  streams  can  contain thumbnails, info about who is editing the file
  (useful for networked files) etc. Could be used for version handling
  and much more.

  Also, just because we do not see all the benefits right now, doesn't
  mean there won't be any.

  ~S

> Wichert.

Re: silent semantic changes with reiser4

[Andrew Morton]

Spam <spam@tnonline.net> wrote:
>
>    Yes,  for  example  documents,  image  files  etc. The multiple data
>    streams  can  contain thumbnails, info about who is editing the file
>    (useful for networked files) etc. Could be used for version handling
>    and much more.

All of which can be handled in userspace library code.

What compelling reason is there for doing this in the kernel?

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> wrote:
>>
>>    Yes,  for  example  documents,  image  files  etc. The multiple data
>>    streams  can  contain thumbnails, info about who is editing the file
>>    (useful for networked files) etc. Could be used for version handling
>>    and much more.

> All of which can be handled in userspace library code.

> What compelling reason is there for doing this in the kernel?


  Because  having user space tools and code will make it not work with
  everything. Keeping stuff in the kernel should make the new features
  transparent to the applications.

  Applications  that support the new features will benefit, all others
  will continue to work without destroying data.

  

Re: silent semantic changes with reiser4

[Andrew Morton]

Spam <spam@tnonline.net> wrote:
>
> 
> 
> > Spam <spam@tnonline.net> wrote:
> >>
> >>    Yes,  for  example  documents,  image  files  etc. The multiple data
> >>    streams  can  contain thumbnails, info about who is editing the file
> >>    (useful for networked files) etc. Could be used for version handling
> >>    and much more.
> 
> > All of which can be handled in userspace library code.
> 
> > What compelling reason is there for doing this in the kernel?
> 
> 
>   Because  having user space tools and code will make it not work with
>   everything. Keeping stuff in the kernel should make the new features
>   transparent to the applications.
> 
>   Applications  that support the new features will benefit, all others
>   will continue to work without destroying data.

Sorry, but that all sounds a bit fluffy.   Please provide some examples.

(Generally, getting all of userspace to agree on a particular library is
socially hard [*], but I don't see that as a reason for putting the
functionality into the kernel)

[*] Example: where's the library to manipulate /etc/whatever.conf? [**]

[**] yes, I know about gconf.

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> wrote:
>>
>> 
>> 
>> > Spam <spam@tnonline.net> wrote:
>> >>
>> >>    Yes,  for  example  documents,  image  files  etc. The multiple data
>> >>    streams  can  contain thumbnails, info about who is editing the file
>> >>    (useful for networked files) etc. Could be used for version handling
>> >>    and much more.
>> 
>> > All of which can be handled in userspace library code.
>> 
>> > What compelling reason is there for doing this in the kernel?
>> 
>> 
>>   Because  having user space tools and code will make it not work with
>>   everything. Keeping stuff in the kernel should make the new features
>>   transparent to the applications.
>> 
>>   Applications  that support the new features will benefit, all others
>>   will continue to work without destroying data.

> Sorry, but that all sounds a bit fluffy.   Please provide some examples.

  We  already had the examples with cp and mv. Both should continue to
  work and the files will still be copied. The same with Konqueror and
  Nautilus.  Files  and  their  meta-files/streams/attributes  will be
  retained as long as applications are using the OS API.

  However,  if  things  are  to  implemented  in  user-space, then old
  applications  will not work correctly and there is risk that all the
  extra information will be lost or corrupted.

  You  said  it  would be socially hard. I think it would be very much
  close to impossible to get it right. Imagine that Gnome and Nautilus
  would  implement  support  for  these. I doubt that cp, mv, KDE, mc,
  app-xyz  would  implement  this anytime soon and in the meantime the
  data is at risk.

  If  we  want  things  to  work  from  the  start, including with old
  applications  then  these  features  should  be at below-application
  level.  If  it  is  in VFS or FS doesn't matter to me, as long as it
  doesn't hinder or delay progress to long.

> (Generally, getting all of userspace to agree on a particular library is
> socially hard [*], but I don't see that as a reason for putting the
> functionality into the kernel)

> [*] Example: where's the library to manipulate /etc/whatever.conf? [**]

> [**] yes, I know about gconf.

Re: silent semantic changes with reiser4

[Andrew Morton]

Spam <spam@tnonline.net> wrote:
>
> 
> 
> > Spam <spam@tnonline.net> wrote:
> >>
> >> 
> >> 
> >> > Spam <spam@tnonline.net> wrote:
> >> >>
> >> >>    Yes,  for  example  documents,  image  files  etc. The multiple data
> >> >>    streams  can  contain thumbnails, info about who is editing the file
> >> >>    (useful for networked files) etc. Could be used for version handling
> >> >>    and much more.
> >> 
> >> > All of which can be handled in userspace library code.
> >> 
> >> > What compelling reason is there for doing this in the kernel?
> >> 
> >> 
> >>   Because  having user space tools and code will make it not work with
> >>   everything. Keeping stuff in the kernel should make the new features
> >>   transparent to the applications.
> >> 
> >>   Applications  that support the new features will benefit, all others
> >>   will continue to work without destroying data.
> 
> > Sorry, but that all sounds a bit fluffy.   Please provide some examples.
> 
>   We  already had the examples with cp and mv. Both should continue to
>   work and the files will still be copied. The same with Konqueror and
>   Nautilus.  Files  and  their  meta-files/streams/attributes  will be
>   retained as long as applications are using the OS API.
> 
>   However,  if  things  are  to  implemented  in  user-space, then old
>   applications  will not work correctly and there is risk that all the
>   extra information will be lost or corrupted.
> 
>   You  said  it  would be socially hard. I think it would be very much
>   close to impossible to get it right. Imagine that Gnome and Nautilus
>   would  implement  support  for  these. I doubt that cp, mv, KDE, mc,
>   app-xyz  would  implement  this anytime soon and in the meantime the
>   data is at risk.
> 

No.  All of the applications which you initially identified can be
implemented by putting the various bits of data into a single file and
getting applications to agree on the format of that file.

For example, some image file formats already support embedded metadata, do
they not?

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> wrote:
>>
>> 
>> 
>> > Spam <spam@tnonline.net> wrote:
>> >>
>> >> 
>> >> 
>> >> > Spam <spam@tnonline.net> wrote:
>> >> >>
>> >> >>    Yes,  for  example  documents,  image  files  etc. The multiple data
>> >> >>    streams  can  contain thumbnails, info about who is editing the file
>> >> >>    (useful for networked files) etc. Could be used for version handling
>> >> >>    and much more.
>> >> 
>> >> > All of which can be handled in userspace library code.
>> >> 
>> >> > What compelling reason is there for doing this in the kernel?
>> >> 
>> >> 
>> >>   Because  having user space tools and code will make it not work with
>> >>   everything. Keeping stuff in the kernel should make the new features
>> >>   transparent to the applications.
>> >> 
>> >>   Applications  that support the new features will benefit, all others
>> >>   will continue to work without destroying data.
>> 
>> > Sorry, but that all sounds a bit fluffy.   Please provide some examples.
>> 
>>   We  already had the examples with cp and mv. Both should continue to
>>   work and the files will still be copied. The same with Konqueror and
>>   Nautilus.  Files  and  their  meta-files/streams/attributes  will be
>>   retained as long as applications are using the OS API.
>> 
>>   However,  if  things  are  to  implemented  in  user-space, then old
>>   applications  will not work correctly and there is risk that all the
>>   extra information will be lost or corrupted.
>> 
>>   You  said  it  would be socially hard. I think it would be very much
>>   close to impossible to get it right. Imagine that Gnome and Nautilus
>>   would  implement  support  for  these. I doubt that cp, mv, KDE, mc,
>>   app-xyz  would  implement  this anytime soon and in the meantime the
>>   data is at risk.
>> 

> No.  All of the applications which you initially identified can be
> implemented by putting the various bits of data into a single file and
> getting applications to agree on the format of that file.

> For example, some image file formats already support embedded metadata, do
> they not?

  Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
  these  with  an application that doesn't support the extensions then
  you will loose them.

  Also,  you  are thinking _very_ narrowly now. There are thousands of
  file  formats.  Implementing  the  support  for  meta-data/ streams/
  attributes  in  the  kernel  will  make  it  possible  to  use  this
  generically for all files.

  I  use  this  in  Windows  quite much. I put information description
  fields  for  lots  of  different  files. These descriptions are then
  searchable etc. I could use the command prompt to copy the files and
  the descriptions would still be there.

  Secondly, do you expect file managers like Nautilus and Konqueror to
  support  every piece of file format on the planet so they could read
  information directly from the documents?

  ~S

Re: silent semantic changes with reiser4

[Andrew Morton]

Spam <spam@tnonline.net> wrote:
>
>    Secondly, do you expect file managers like Nautilus and Konqueror to
>    support  every piece of file format on the planet so they could read
>    information directly from the documents?

Sure.  You're proposing that we implement a single, golden multi-stream file
format in the kernel.

We could just as well do that in libmultistreamfileformat.so.

But I'll grant that one cannot go adding new metadata to, say, C files this
way.  I don't know how useful such a thing is though.

Remember that my main point is that there's a lack of coordination in
userspace.  Hell, there's none.  Putting it in-kernel forces that
coordination, and may be the way to go, but it's pretty sad.

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Andrew Morton wrote:
> But I'll grant that one cannot go adding new metadata to, say, C files this
> way.  I don't know how useful such a thing is though.

That is actually one of the few places where a bit of metadata would be
very useful. Right now there is no way to indicate in what encoding a
source is written: ascii, utf-8, ucs16, etc. are all possible. But a
compiler or interpreter has no good way to figure that out.

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Tom Vier]

On Thu, Aug 26, 2004 at 01:29:38PM +0200, Wichert Akkerman wrote:
> That is actually one of the few places where a bit of metadata would be
> very useful. Right now there is no way to indicate in what encoding a
> source is written: ascii, utf-8, ucs16, etc. are all possible. But a
> compiler or interpreter has no good way to figure that out.

Make it a cmd line option. The makefile can contain info on what files are
what and invoke the right options.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: silent semantic changes with reiser4

[Timothy Miller]

Wichert Akkerman wrote:
> Previously Andrew Morton wrote:
> 
>>But I'll grant that one cannot go adding new metadata to, say, C files this
>>way.  I don't know how useful such a thing is though.
> 
> 
> That is actually one of the few places where a bit of metadata would be
> very useful. Right now there is no way to indicate in what encoding a
> source is written: ascii, utf-8, ucs16, etc. are all possible. But a
> compiler or interpreter has no good way to figure that out.
> 


[NOTE:  I am 5000 messages behind.  Please forgive any redundancy.]

This reminds me of a paper someone wrote on how HFS(+) stored the file 
type (actually, application that knows how to use the file) as metadata, 
separate from the filename.  He was lamenting the fact that the Mac was 
being 'corrupted' by the PC's broken philosophy of including as part of 
the filename something which should not be.  He also mentioned that 
Windows' feature of hiding the extension doesn't cut it.  One benefit, I 
recall, was that you can't change the association accidentally when 
changing the filename.  Another thing was that file name and file type 
are not semantically related, so they shouldn't be squished together.

I don't remember this well enough, so I can't argue the point, but 
having the file type as metadata separate from the filename has SOME 
amount of elegant appeal to me.

Re: silent semantic changes with reiser4

[Horst von Brand]

Andrew Morton <akpm@osdl.org> said:
> Spam <spam@tnonline.net> wrote:
> >
> >    Secondly, do you expect file managers like Nautilus and Konqueror to
> >    support  every piece of file format on the planet so they could read
> >    information directly from the documents?
> 
> Sure.  You're proposing that we implement a single, golden multi-stream file
> format in the kernel.

Which won't be able to cater for all needs, I'm afraid. Just look at /etc,
and contemplate why there are so many different configuration file formats.
Sure, some are just random coder preferences set in stone, but others do
make sense being different. The same will happen with file formats. Heck,
it is already that way, with file formats for C source code, ELF shared
libraries, various graphics and audio and video formats, ...

> We could just as well do that in libmultistreamfileformat.so.

Right.

> But I'll grant that one cannot go adding new metadata to, say, C files this
> way.  I don't know how useful such a thing is though.

You can never know. Perhaps annotations in swahili? Alternative versions
for different architectures? In object format(s)? Copyright info? 
Changelog?

> Remember that my main point is that there's a lack of coordination in
> userspace.  Hell, there's none.  Putting it in-kernel forces that
> coordination, and may be the way to go, but it's pretty sad.

I think that forcing coordination is wrong.  For example, I do value having
ar/tar/cpio/zip doing essentially the same thing (and being able to define
and use my own way of doing it if it pleases me), and I'd strongly oppose
the kernel forcing one way of doing it down my throat.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Spam wrote:
>   Also,  you  are thinking _very_ narrowly now. There are thousands of
>   file  formats.  Implementing  the  support  for  meta-data/ streams/
>   attributes  in  the  kernel  will  make  it  possible  to  use  this
>   generically for all files.

And how many different formats will be used for thumbnails? How many
different names to indicate a summary for a document?

Having the ability to add attributes standardizes a way to access that
data, but not the format and naming of those attributes. It could be
a step closer to a more flexible way of handling storage, but you have
to realize that a lot (if not most) of the work still needs to be done
in userspace.

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Spam wrote:

>   Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
>   these  with  an application that doesn't support the extensions then
>   you will loose them.

OK, so we've got a choice.

Either we will lose the extensions when modifying the file
with an unaware program, or you lose the extensions when
copying (or restoring from backup) using an unaware program.

Personally I'd prefer to keep the file intact when not
modifying it...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Spam]

> On Thu, 26 Aug 2004, Spam wrote:

>>   Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
>>   these  with  an application that doesn't support the extensions then
>>   you will loose them.

> OK, so we've got a choice.

> Either we will lose the extensions when modifying the file
> with an unaware program, or you lose the extensions when
> copying (or restoring from backup) using an unaware program.

> Personally I'd prefer to keep the file intact when not
> modifying it...

  Backup to me is a special case, not simply a copy of files, but also
  retaining all the extra data, info, attributes, etc, that comes with
  the file.

  Enabling  support  to  have  all  of  this  extra  stuff  below  the
  application  level  it  will  be  possible to retain everything even
  though applications do not support them.

  Backup tools, however, must know about these new features to be able
  to  backup  them.  And so will every tool that directly accesses the
  filesystem instead of using the OS API for it.

  And  from  I  learned  in an earlier message, tools like cp actually
  work  on  the  fs level and themselves move the data to the new file
  instead   of  letting  the OS handle it. This seem to me as it could
  be  dangerous and could also prevent any kind of enhancements to the
  FS unless every tool like cp is patched.
  

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 15:59, Spam wrote:

>   Backup to me is a special case, not simply a copy of files, but also
>   retaining all the extra data, info, attributes, etc, that comes with
>   the file.
>
>   Enabling  support  to  have  all  of  this  extra  stuff  below  the
>   application  level  it  will  be  possible to retain everything even
>   though applications do not support them.
>
>   Backup tools, however, must know about these new features to be able
>   to  backup  them.  And so will every tool that directly accesses the
>   filesystem instead of using the OS API for it.

Thus, this kind of metadata API must be made accessible through the VFS layer, 
if we want to make it FS-independent (not tied to Reiser4). If the real FS 
does not support metadata operations, just ignore them (much like in ISO9660 
ignoring write operations).

>   And  from  I  learned  in an earlier message, tools like cp actually
>   work  on  the  fs level and themselves move the data to the new file
>   instead   of  letting  the OS handle it. This seem to me as it could
>   be  dangerous and could also prevent any kind of enhancements to the
>   FS unless every tool like cp is patched.

I still don't understand why a copy_file() syscall is still missing.

Re: silent semantic changes with reiser4

[Horst von Brand]

Spam <spam@tnonline.net> said:
> Andrew Morton <akpm@osdl.org> said:

[...]

> > For example, some image file formats already support embedded metadata, do
> > they not?

>   Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
>   these  with  an application that doesn't support the extensions then
>   you will loose them.

As you will each time you muck around with your files-are-directories.
Nothing new there.

>   Also,  you  are thinking _very_ narrowly now. There are thousands of
>   file  formats.  Implementing  the  support  for  meta-data/ streams/
>   attributes  in  the  kernel  will  make  it  possible  to  use  this
>   generically for all files.

So the _kernel_ has to know about thousands of formats, just in case it
some blue day it comes across a strange file? Better leave that to the
applications.

You will _not_ be able to define a single format for extra data about the
file, there will be differing extra data for different users (do you
suggest a root-only file with special writable pouches for "graphical icon
for the file" for each user on the system?!), so the idea in itself is
doomed from the start.

>   I  use  this  in  Windows  quite much.

Then use it and be happy. No need to screw up Linux for that.

>                                          I put information description
>   fields  for  lots  of  different  files. These descriptions are then
>   searchable etc. I could use the command prompt to copy the files and
>   the descriptions would still be there.

The descriptions might make sense to _you_, _now_. No guarantee they make
any sense (or are in the least useful) for other users on your system, and
I might want them in arabic or some such. The descriptions might make no
sense to you in a couple of years.

>   Secondly, do you expect file managers like Nautilus and Konqueror to
>   support  every piece of file format on the planet so they could read
>   information directly from the documents?

That's their (self-selected) job, yes.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote:
>
>So the _kernel_ has to know about thousands of formats, just in case it
>some blue day it comes across a strange file? Better leave that to the
>applications.

Or come up with a way to export some interface to userspace.

Or make a standard description format in something like XML that can be
used for arbitrary files and have userspace interface with that metadata
stream that is the XML.
This is, however, dangerously close to what we have now and does. not. work,
so the only added bonus we might have is the fact that it's endorsed by the
kernel.
I don't like this idea except as a last fallback solution...

>>   I  use  this  in  Windows  quite much.
>Then use it and be happy. No need to screw up Linux for that.

Linux does not need to be screwed up by that.
It can be done right.

>The descriptions might make sense to _you_, _now_. No guarantee they make
>any sense (or are in the least useful) for other users on your system, and

They also make perfect sense to me, and I would like this to have
had this yesterday. or last year ;)

>I might want them in arabic or some such. The descriptions might make no
>sense to you in a couple of years.

So what do you do if you use someone else's system and he doesn't have
Arabic man pages but you want them in Arabic?

And a description that doesn't make sense in many years is badly formed.

>>   Secondly, do you expect file managers like Nautilus and Konqueror to
>>   support  every piece of file format on the planet so they could read
>>   information directly from the documents?
>That's their (self-selected) job, yes.

Sure, but they store it differently in different places. Having them in
the actual file makes it universal.

I think the biggest issue is that you are arguing against this out of
some principle. There is a certain want for these new features among
the users, there are people who are willing to implement them, there
is Linus who seems to agree and accept.

What do you, and people who agree with you, lose if this gets implemented?
Drive space when you have the slightly-grown kernel sources downloaded?

-- 
mjt

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Andrew Morton <akpm@osdl.org> said:

> [...]

>> > For example, some image file formats already support embedded metadata, do
>> > they not?

>>   Yes, JPEG, TIFF and PNG files for example. But, if you modify any of
>>   these  with  an application that doesn't support the extensions then
>>   you will loose them.

> As you will each time you muck around with your files-are-directories.
> Nothing new there.

>>   Also,  you  are thinking _very_ narrowly now. There are thousands of
>>   file  formats.  Implementing  the  support  for  meta-data/ streams/
>>   attributes  in  the  kernel  will  make  it  possible  to  use  this
>>   generically for all files.

> So the _kernel_ has to know about thousands of formats, just in case it
> some blue day it comes across a strange file? Better leave that to the
> applications.

  No,  not  at all. The only think the kernel would need to support is
  the actual meta files and streams - not the contents of the streams.
  That is up to the applications to know and use.

  The  idea  with  plugins  to reiser4 would allow someone to create a
  plugin that would export the thumbnails from TIFF and JPEG images as
  separate streams.

> You will _not_ be able to define a single format for extra data about the
> file, there will be differing extra data for different users (do you
> suggest a root-only file with special writable pouches for "graphical icon
> for the file" for each user on the system?!), so the idea in itself is
> doomed from the start.

  We  do  not need to define a single format. You are locking yourself
  to  the contents of streams and meta files.

  If  someone  likes  the  idea of a plugin that would allow for icons
  then  that  may  be  it. It would be up to application developers to
  know  about  this and use it. Perhaps they could agree on a standard
  even.
>>   I  use  this  in  Windows  quite much.

> Then use it and be happy. No need to screw up Linux for that.

  This  wouldn't  screw anything up besides allowing for extending the
  functionality.  There  is noone that forces you to enable and instll
  plugins,  use  extra attributes or meta info. All of this always was
  optional (as I see it).

>>                                          I put information description
>>   fields  for  lots  of  different  files. These descriptions are then
>>   searchable etc. I could use the command prompt to copy the files and
>>   the descriptions would still be there.

> The descriptions might make sense to _you_, _now_. No guarantee they make
> any sense (or are in the least useful) for other users on your system, and
> I might want them in arabic or some such. The descriptions might make no
> sense to you in a couple of years.

  So? The same may be for your own word documents etc. It doesn't mean
  I do not find it useful now, or that anyone else might not. Shall we
  not  include  features  because "someone" on the same system may not
  use it? Seems very backward to me.

  If   I   include   a filestream, description.xml, then that wouldn't
  affect  anyone. I would know how to read the file, and that would be
  enough.

>>   Secondly, do you expect file managers like Nautilus and Konqueror to
>>   support  every piece of file format on the planet so they could read
>>   information directly from the documents?

> That's their (self-selected) job, yes.

  Yes  and  no.  I  used  them  as example but there are lots of other
  programs  that  can access files. Midnight commander, cp, tar, email
  programs and lots of others.

  besides, I think it would be easier for them to 1) write plugins and
  2)  just  use normal file access method to provide the extra data to
  the users.

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 889 bytes --]

Salut,

On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote:
> >   Also,  you  are thinking _very_ narrowly now. There are thousands of
> >   file  formats.  Implementing  the  support  for  meta-data/ streams/
> >   attributes  in  the  kernel  will  make  it  possible  to  use  this
> >   generically for all files.
> 
> So the _kernel_ has to know about thousands of formats, just in case it
> some blue day it comes across a strange file? Better leave that to the
> applications.

I'd  do  that  quite  differently:  Collect metadata  about  files  in
extended attributes (which are  rather universally useable), and do it
from  an userspace daemon/application/file  manager plugin  which uses
dynamically   loadable   plugins   for   the  different   file   types
differentiated by libmagic.

This has nothing to do in the kernel.

			    Tonnerre


[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Alexander G. M. Smith]

Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200:
> On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote:
> > So the _kernel_ has to know about thousands of formats, just in case it
> > some blue day it comes across a strange file? Better leave that to the
> > applications.
> 
> I'd  do  that  quite  differently:  Collect metadata  about  files  in
> extended attributes (which are  rather universally useable), and do it
> from  an userspace daemon/application/file  manager plugin  which uses
> dynamically   loadable   plugins   for   the  different   file   types
> differentiated by libmagic.
> 
> This has nothing to do in the kernel.

I mostly agree, like I was saying earlier, file types are needed!  The kernel doesn't have to know about all of them, just some of them.  It should be possible to attach a file type to everything so you know what kind of thing it is, not just block or character device or file, but something like a MIME type.  If space is a concern, add to the inode a unique 8 byte character code for each popular type and a global translation table to get back the MIME string.

The kernel (includes file systems and plug-ins) would have to know some standard file types and how to compare them for indexing purposes.  For example, an integer attribute like the year a music album was released would be stored as a 4 byte attribute "file" with a type like "primitive/int32", which tells the kernel to use 32 bit integer number comparison for indexing order.  Another type the kernel could be aware of are the file types used for security attributes.  A third kind it might need to know about are the computed attributes, which get set whenever related attributes they are computed from change.  For any unknown types, the kernel just stores the data and doesn't do any extra processing.

Optionally the kernel could also maintain the global list of all file types and their properties (such as which ones are indexed, which are computed), though that could also be done in userland if you aren't doing indexing or computed attributes or other fancy operations.

The determination of the types would be in userland (smart programs would write the type when they created the file).  Like you said, a daemon could classify unknown files in the background (BeOS does this).  Most of the uses of the file type would also be in userland, such as deciding which program can handle a given file, or what icon to display, or to hint that a file/directory/attribute thing should be interpreted as an old style directory to the user, or to mark a file thing as being worth backing up (don't need to back up the computed ones).

- Alex

Re: silent semantic changes with reiser4

[Spam]

  

> Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200:
>> On Fri, Aug 27, 2004 at 07:58:40PM -0400, Horst von Brand wrote:
>> > So the _kernel_ has to know about thousands of formats, just in case it
>> > some blue day it comes across a strange file? Better leave that to the
>> > applications.
>> 
>> I'd  do  that  quite  differently:  Collect metadata  about  files  in
>> extended attributes (which are  rather universally useable), and do it
>> from  an userspace daemon/application/file  manager plugin  which uses
>> dynamically   loadable   plugins   for   the  different   file  types
>> differentiated by libmagic.
>> 
>> This has nothing to do in the kernel.

> I mostly agree, like I was saying earlier, file types are needed!
> The kernel doesn't have to know about all of them, just some of
> them.  It should be possible to attach a file type to everything so
> you know what kind of thing it is, not just block or character
> device or file, but something like a MIME type.  If space is a
> concern, add to the inode a unique 8 byte character code for each
> popular type and a global translation table to get back the MIME
> string.

  The  kernel  doesn't  have anything to know about normal files.It is
  up to applications to determine and know the contents. Isn't it?

  Exception  would  be  of  course  special  meta  files  that  define
  attributes and permissions for the filesystem.


> The kernel (includes file systems and plug-ins) would have to
> know some standard file types and how to compare them for indexing
> purposes.  For example, an integer attribute like the year a music
> album was released would be stored as a 4 byte attribute "file" with
> a type like "primitive/int32", which tells the kernel to use 32 bit
> integer number comparison for indexing order.  Another type the
> kernel could be aware of are the file types used for security
> attributes.  A third kind it might need to know about are the
> computed attributes, which get set whenever related attributes they
> are computed from change.  For any unknown types, the kernel just
> stores the data and doesn't do any extra processing.

> Optionally the kernel could also maintain the global list of all
> file types and their properties (such as which ones are indexed,
> which are computed), though that could also be done in userland if
> you aren't doing indexing or computed attributes or other fancy
> operations.

> The determination of the types would be in userland (smart
> programs would write the type when they created the file).  Like you
> said, a daemon could classify unknown files in the background (BeOS
> does this).  Most of the uses of the file type would also be in
> userland, such as deciding which program can handle a given file, or
> what icon to display, or to hint that a file/directory/attribute
> thing should be interpreted as an old style directory to the user,
> or to mark a file thing as being worth backing up (don't need to
> back up the computed ones).

> - Alex

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 1351 bytes --]

Salut,

On Sun, Aug 29, 2004 at 08:39:25PM -0400, Alexander G. M. Smith wrote:
> I mostly  agree, like I was  saying earlier, file  types are needed!
> The kernel  doesn't have  to know  about all of  them, just  some of
> them.  It should be possible to  attach a file type to everything so
> you  know what  kind of  thing it  is, not  just block  or character
> device or file, but something like a MIME type.

Then store the value computed  by libmagic into an extended attribute,
if you  like. I still  don't see why  the kernel should even  care. It
shouldn't read things from files anyway, as a kernel shall be bootable
on a busybox  system (and *Step) as  well, so we don't rely  on the fs
layout in any way.

> Optionally the  kernel could  also maintain the  global list  of all
> file types  and their  properties (such as  which ones  are indexed,
> which are computed),  though that could also be  done in userland if
> you  aren't doing  indexing or  computed attributes  or  other fancy
> operations.

Quel horreur!

Do it in userland, really.

If I  get the time,  I'll write you  a small daemon based  on libmagic
which  stores  the  file  attributes  in xattrs,  or  if  they're  not
supported, in some MacOS/Xish per-directory files. Even a file manager
("finder") can do that, there's not even the need for a daemon.

				Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Tonnerre wrote:
> Quel horreur!
> Do it in userland, really.

I'm amazed that after all this discussion where all the realistic
implementations are done in userland with kernel support for calling
out to it, there are people who think the kernel is supposed to decode
MP3 files or whatever.

Nobody is advocating that!

> If I  get the time,  I'll write you  a small daemon based  on libmagic
> which  stores  the  file  attributes  in xattrs,  or  if  they're  not
> supported, in some MacOS/Xish per-directory files. Even a file manager
> ("finder") can do that, there's not even the need for a daemon.

How are you going to do the part where the xattr changes when the file
is modified?

(For example, if I edit an HTML file which is encoded in iso-8859-1,
change it to utf-8 and indicate that in a META element, and save it
under the same name, the full content-type should change from
"text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)

I don't see how you can do that without kernel support.

Don't say dnotify or inotify, because neither would work.

-- Jamie

Re: silent semantic changes with reiser4

[Jamie Lokier]

Tonnerre wrote:
> > > I'll write you a small daemon based on libmagic which stores the
> > > file attributes in xattrs, or if they're not supported, in some
> > > MacOS/Xish per-directory files. Even a file manager ("finder") can
> > > do that, there's not even the need for a daemon.

Jamie Lokier wrote:
> > (For example, if I edit an HTML file which is encoded in iso-8859-1,
> > change it to utf-8 and indicate that in a META element, and save it
> > under the same name, the full content-type should change from
> > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
> > 
> > I don't see how you can do that without kernel support.

Dave Kleikamp wrote:
> Your html editor should do that.

My html editor is "vi".  If I am supposed to manually set the
content-type attribute after exiting vi, doesn't that rather
invalidate the idea of a "small daemon based on libmagic" which sets
it for me?

That was the sole point of my statement.

-- Jamie

Re: silent semantic changes with reiser4

[Alan Cox]

On Mer, 2004-09-01 at 21:16, Jamie Lokier wrote:
> (For example, if I edit an HTML file which is encoded in iso-8859-1,
> change it to utf-8 and indicate that in a META element, and save it
> under the same name, the full content-type should change from
> "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
> 
> I don't see how you can do that without kernel support.
> 
> Don't say dnotify or inotify, because neither would work.

inotify done right is useful here as well as in a lot of other desktop
cases where dnotify doesn't really scale. Its enough to let me

	- Find the new file
	- Virus scan it
	- Classify its possible type heirarchies
	- Index it

Alan

Re: silent semantic changes with reiser4

[Jamie Lokier]

Alan Cox wrote:
> On Mer, 2004-09-01 at 21:16, Jamie Lokier wrote:
> > (For example, if I edit an HTML file which is encoded in iso-8859-1,
> > change it to utf-8 and indicate that in a META element, and save it
> > under the same name, the full content-type should change from
> > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
> > 
> > I don't see how you can do that without kernel support.
> > 
> > Don't say dnotify or inotify, because neither would work.
> 
> inotify done right is useful here as well as in a lot of other desktop
> cases where dnotify doesn't really scale. Its enough to let me
> 
> 	- Find the new file
> 	- Virus scan it
> 	- Classify its possible type heirarchies
> 	- Index it

Indeed it does, but it fails for the example I was commenting on to
which you replied..

      1. The file /var/www/site/index.html's written in vi.
      2. "The daemon" (that's what I objected to) receives inotify.
         blocks waiting for scheduler, however...
      3. Seeing that vi is now finished, I phone person on other side
         of world and say the updated file is now available.
      4. Person fetches http://site/index.html
         ...oh!  They receive the wrong content-type (bollocks!)
      5. Eventually after paging & scheduling deems it appropriate,
         "The daemon" gets to run, looks at file, updates content-type.

In other words I was criticising Tonnerre's idea of a daemon which
updates the xattrs by looking at file contents...  by definition, a
daemon runs in the background, and the whole point of it updating a
content-type xattr was obviously so that programs like httpd could
just _use_ that xattr.

That doesn't work, even if that daemon uses inotify.

Basically, anytime where you want an ordering guarantee that something
will be recalculated in the interval after a file is modified and
before the calculated result is used, you have to be very careful
about exactly which code is using dnotify or inotify to achieve that.
Single threaded programs can use it easily, but a multi-threaded file
server has to be very careful about ordering if it's to avoid glitches.

It is exactly a real problem I have faced in a multi-threaded HTTP
server which uses dnotify to detect changes to prerequisite files and
also changes to the path walked and permissions on it, and thus
invalidate cached generated pages, thus giving strong cache
guarantees.  Performance is great, if you ignore the race conditions...

(One of the more popular small web servers, lighttpd (PHP people seem
to like it), also uses dnotify.  Apparently it provides a big
performance boost, just by avoiding stat() calls.  So I'm not alone in
using dnotify for such things.)

-- Jamie

Re: silent semantic changes with reiser4

[Christer Weinigel]

Jamie Lokier <jamie@shareable.org> writes:

> (For example, if I edit an HTML file which is encoded in iso-8859-1,
> change it to utf-8 and indicate that in a META element, and save it
> under the same name, the full content-type should change from
> "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
>
> I don't see how you can do that without kernel support.

charset_cache = dbm(os.getenv('HOME') + '/.charset_cache')

def get_charset(path):
    file_mtime = get_mtime(path)
    cache_tuple = dbm.get(path, (None, None))
    if file_mtime != cache_tuple[0]:
        cache_tuple = (file_mtime, figure_out_charset(path))
        dbm.put(path, cache_tuple)
    return cache_tuple[1]

This code is guaranteed to always give you an up to date charset for a
file, provided that the mtime is guaranteed to change every time the
file changes.

If the file can change during the call to get_charset you'll have to
lock the file while you are working with it or handle it some other
way (comparing mtime before and after and retrying if they differ),
but that is true even if you do it all in the kernel.  This has the
added advantage that the cache is updated lazily, so the charset is
never calculated unless someone really needs it.

If the disk gets full, it is time to shrink the cache and preferably
stale entries are evicted first.  If that's not good enough, a
notifier such as dnotify or inotify can be used to invalidate the
cache immediately after the file has changed.

If this gets popular, the cache management can be moved to a daemon
(naturally with all the security aspects in mind). 

> Don't say dnotify or inotify, because neither would work.

Why not?  The approach above works on any filesystem, even without
dnotify or inotify but will be more efficient with them.

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christer Weinigel wrote:
> Jamie Lokier <jamie@shareable.org> writes:
> 
> > (For example, if I edit an HTML file which is encoded in iso-8859-1,
> > change it to utf-8 and indicate that in a META element, and save it
> > under the same name, the full content-type should change from
> > "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
> >
> > I don't see how you can do that without kernel support.
> 
> [code which runs in the program which needs to read the content-type]
> 
> This code is guaranteed to always give you an up to date charset for a
> file, provided that the mtime is guaranteed to change every time the
> file changes.

...which it isn't, but ignoring that.

Yes we all know that code will work, turning a blind eye to occasional
mtime failures.

What I objected to isn't that, but Tonerre's idea of a daemon (i.e. in
the background, that's what daemons do) updating xattrs.  Your code
will work.  Tonerre's has a serious race condition.

> > Don't say dnotify or inotify, because neither would work.
> 
> Why not?  The approach above works on any filesystem, even without
> dnotify or inotify but will be more efficient with them.

The above approach is in fact better with dnotify/inotify, when they
area available, because then it doesn't have mtime problems.

My objection is that dnotify & inotify don't fix the problems of a
"daemon" (i.e. in the background) updating an xattr - because the
daemon might not run scheduled between the write and when the file
needs to be used.

That said, yours has practical problems too.  The intended purpose (*)
of storing content-type in an xattr was so that programs which need to
read content without understanding it may just do so (e.g. mail agents,
httpd), and are distinct from the programs which compute and set it
(e.g. programs which understand specific content formats).

(*) I don't necessarily agree with that purpose.

-- Jamie

Re: silent semantic changes with reiser4

[Jamie Lokier]

Andrew Morton wrote:
> No.  All of the applications which you initially identified can be
> implemented by putting the various bits of data into a single file and
> getting applications to agree on the format of that file.
> 
> For example, some image file formats already support embedded metadata, do
> they not?

Same for MP3s and their metadata tags.  _That_ kind of data should be
part of the contents of the file: it is intended to be transported.

It is useful to auto-extract it from the file's data stream, using
format-specific programs, and offer that metadata in separate files
where it is easy to use e.g. by shell scripts and indexing programs,
which can then work independently of file formats.

That kind of metadata should be _derived_ from the file's data stream,
and ideally the extraction should be automatic, with userspace
assistance.  It may be useful for the filesystem to record what's
extracted, for fast access and indexing.

The same applies to thumbnails.  The only difference is that these may
take longer to compute.  But they're the same thing: something
calculated from the file's data stream, which may (or may not) be
stored in the filesystem so it doesn't have to be recomputed each
time.  It might even be worth expiring the metadata from disk if it
hasn't been used for a while and can be recomputed.

These are the same as looking into archives or structured files.  The
only difference is that the unpacked views might not be stored on the
filesystem (but they might be, if the filesystem knows how to cache
views for faster future access).[*]

Those are all examples of data derived from the data stream of a flat
file.  They are useful applications, and all work fine with FTP, cp,
cat <a >b, and email attachments, _no application changes required_.

There's another kind of metadata which isn't intended for transport.
That would be things like the classic attributes: permissions,
timestamps.  It would also include things like author information when
the file format cannot carry it.

-- Jamie


[*] - A fancier variant, which is hard without good fs support, it
writing to structured files through a view.  I.e. modifying files
inside a .tar.gz, or (this is useful) inside an
{MS,Open,Gnome,K}Office container file or *SQL database table.  To do
this efficiently, the file's main data stream is regenerated on demand
from the view's data.  As I said, it requires good, special fs support
to do it efficiently (fancy caching).  With appropriate userspace
libraries you can make apps work with container files on all other
OSes (just like .tar.gz and *Office containers at the moment) but are
significantly faster to work with on Linux (no need to serialise and
(de-)compress except on demand) and more visible to the standard unix
scripting tools.

Re: silent semantic changes with reiser4

[James Bruce]

Andrew Morton wrote:

>No.  All of the applications which you initially identified can be
>implemented by putting the various bits of data into a single file and
>getting applications to agree on the format of that file.
>  
>

So in order to avoid breaking backup and file utilities, we'd instead 
break every application that reads files?  That way surely lies madness 
:)  This /could/ possibly work with lots of cooperation from glibc, but 
we'd also need a way to insert and delete bytes in a file... moving 1MB 
of attributes/streams just to change a 20 character author attribute 
seems a bit silly.  In addition, there are little things, such as how 
adding a thumbnail image shouldn't change the modification time of the 
original picture IMO, but it does if you embed it in the file.

If it should be in userspace, we could take the OS-X approach of using 
directories for everything, and the "data" that would be in a hybrid 
directory-file is a specially named file under that directory.  Programs 
for instance are actually directories, with the elf file underneath it.  
When you click on the "directory-program" in the gui it runs the 
associated elf file rather than opening the directory.  I think that 
approach is promising, but so few unix programs have any indirection for 
file access it'd be hell to teach them all how it works.  Of course with 
emminent breakage of some sort, maybe now is the time...

 - Jim Bruce

Re: silent semantic changes with reiser4

[Jamie Lokier]

James Bruce wrote:
> moving 1MB of attributes/streams just to change a 20 character
> author attribute seems a bit silly.

That's what we do now.  Try editing an MP3's id tags.

> If it should be in userspace, we could take the OS-X approach of using 
> directories for everything, and the "data" that would be in a hybrid 
> directory-file is a specially named file under that directory.  Programs 
> for instance are actually directories, with the elf file underneath it.  
> When you click on the "directory-program" in the gui it runs the 
> associated elf file rather than opening the directory.  I think that 
> approach is promising, but so few unix programs have any indirection for 
> file access it'd be hell to teach them all how it works.

The interfaces we've been talking about are very close to that.  The
indirection is automatic: instead of a document (or program) looking
like a directory, it'll look like a file, and reading (or executing)
the file will do the indirection, with kernel control (and so
consistency and performance) via an appropriate flat<->structured
relation helper.

-- Jamie

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 18:58, Jamie Lokier wrote:
> James Bruce wrote:
> > moving 1MB of attributes/streams just to change a 20 character
> > author attribute seems a bit silly.
>
> That's what we do now.  Try editing an MP3's id tags.

What's the problem? Let's say a file has three streams/forks/whatever: the 
unnamed one, and two additional named. The filesystem can treat these three 
streams like three independent (but related) files on disk so, if the author 
attribute belongs to the first named stream, only the first named stream (and 
its corresponding file on the backend) would need to be modified. No need to 
move MBs of data around.

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrew Morton wrote:

>
>No.  All of the applications which you initially identified can be
>implemented by putting the various bits of data into a single file and
>getting applications to agree on the format of that file.
>
>For example, some image file formats already support embedded metadata, do
>they not?
>
>-
>  
>
and that leads to vasts numbers of file formats, all of which require 
special code to parse them, and apps that don't duplicate that code 
can't access the data, and it is a software engineering mistake.  
Uniform interfaces greatly reduce the cost of an OS and increase its 
expressive power.  Closure is the most important and least understood 
principle of OS design.

By contrast, suppose everything was stored in files and directories.  
Every app can afford the coding cost to learn about files and 
directories.  Data can freely bounce from one object to another in the 
OS because the APIs for the objects are all the same.  That's increased 
expressive power.  The number of connections between objects determines 
the expressive power of the OS, not the number of objects in it.  
Unified namespaces do a lot for an OS.

Hans

Re: silent semantic changes with reiser4

[Bernd Petrovitsch]

[ shortened Cc: of reiser- and other folks since it is no longer
reiser-specific - sorry, if it was too much ]
[ fixed missing attribution ]

On Thu, 2004-08-26 at 12:51 +0200, Spam wrote:
> On Thu, 2004-08-26 at 03:24 -0700, Andrew Morton wrote:
> > Spam <spam@tnonline.net> wrote:
[...]
> >>   Applications  that support the new features will benefit, all others
> >>   will continue to work without destroying data.
> 
> > Sorry, but that all sounds a bit fluffy.   Please provide some examples.

It is too fluffy.

>   We  already had the examples with cp and mv. Both should continue to
>   work and the files will still be copied. The same with Konqueror and

... after they are patched to support streams. As stated each stream has
an own fd, so - from the user-space perspective - one has to open *all*
streams of this file, read them and write them into the destination.

>   Nautilus.  Files  and  their  meta-files/streams/attributes  will be
>   retained as long as applications are using the OS API.

The OS-API is *one* file descriptor where you can read, write, mmap, ...
on. Therefore stream selection must happen with open() since open()
returns a fd (which uniquely identify a stream). Voila.
And actually this makes transition actually possible: Old apps simply
ignore streams (yes, there will be a default stream, which is used id
nothing else is specified) and throw them away (think of e.g. patch
which opens a new file, writes into it, unlink's the old one and mv's
the new one on the old place) until they are modified to support them.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: silent semantic changes with reiser4

[Hans Reiser]

What is important is that it all be one packaged solution from the same 
maintainer.  That is necessary for coherency of solution.

There is also benefit to it being one coherent body of code, but this is 
harder to articulate.  It is always hard to articulate why functionality 
belongs in the same program rather than split into multiple programs.  
Filesystems should provide primitives for ordering, structuring, and 
naming objects.  It is hard to say why we should do all of that, but on 
some level I just know it is our task.

Hans

Spam wrote:

>
>
>  You  said  it  would be socially hard. I think it would be very much
>  close to impossible to get it right. Imagine that Gnome and Nautilus
>  would  implement  support  for  these. I doubt that cp, mv, KDE, mc,
>  app-xyz  would  implement  this anytime soon and in the meantime the
>  data is at risk.
>
>
>  
>
>  
>

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Spam wrote:
>   Because  having user space tools and code will make it not work with
>   everything. Keeping stuff in the kernel should make the new features
>   transparent to the applications.

But having it in the kernel has the same problem. If you read the
Solaris documentation you will see that a bunch of utilties had to
get a new commandline option to be able to access the metadata and
a special utility was added for other applications. If you look
at windows you will see that you need to use a filename like
<realfilename>:<streamname>:$DATA (which obviously does not work for
single-character filenames).

Ignoring samba for a bit which just needs streams to stay compatible
with windows I see few reasons for using streams:

* files are more complex these days and tend to include multiple
  different things: images with thumbnails and exif data, 'office'
  documents containing both text and images

* standard way to add common metadata to a file which can be used for
  searching tools (author, copyright, keywords, etc.)

But both can already be done in userland (modern image formats can store
thumbnails and exif data internally, applications use tar or zip-like
files for documents, etc.). The metadata part is a lot more complicated
as well since the behaviour of attributes might need to be complex:
if I change an image using an application that is not stream-aware, what
should happen to its thumbnail? 

The only common benefits I can see are standardisation and optimization:
instead of every file format or application defining a way to specify 
metadata for a file you get a common API defined by the OS (but you'll
still need to standardize on attribute names and formats, so plenty of
room that will still not help), and instead of parsing different files
or XML streams you can directly access a bit of metadata.

So far I'm not convinced that streams are worth the effort. Not that my
opinion is all that relevant here, but still :)

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Hans Reiser]

Wichert Akkerman wrote:

>Previously Spam wrote:
>  
>
>>  Because  having user space tools and code will make it not work with
>>  everything. Keeping stuff in the kernel should make the new features
>>  transparent to the applications.
>>    
>>
>
>But having it in the kernel has the same problem. If you read the
>Solaris documentation you will see that a bunch of utilties had to
>get a new commandline option to be able to access the metadata and
>a special utility was added for other applications.
>
They did it wrong.

> If you look
>at windows you will see that you need to use a filename like
><realfilename>:<streamname>:$DATA (which obviously does not work for
>single-character filenames).
>  
>
again, they did it wrong.  See Rob Pike's "The Hideous Name" for why 
that is wrong.

>Ignoring samba for a bit which just needs streams to stay compatible
>with windows I see few reasons for using streams:
>
>* files are more complex these days and tend to include multiple
>  different things: images with thumbnails and exif data, 'office'
>  documents containing both text and images
>
>* standard way to add common metadata to a file which can be used for
>  searching tools (author, copyright, keywords, etc.)
>
>But both can already be done in userland (modern image formats can store
>thumbnails and exif data internally, applications use tar or zip-like
>files for documents, etc.). The metadata part is a lot more complicated
>as well since the behaviour of attributes might need to be complex:
>if I change an image using an application that is not stream-aware, what
>should happen to its thumbnail? 
>
>The only common benefits I can see are standardisation and optimization:
>instead of every file format or application defining a way to specify 
>metadata for a file you get a common API defined by the OS (but you'll
>still need to standardize on attribute names and formats, so plenty of
>room that will still not help), and instead of parsing different files
>or XML streams you can directly access a bit of metadata.
>
>So far I'm not convinced that streams are worth the effort. Not that my
>opinion is all that relevant here, but still :)
>
>Wichert.
>
>  
>

Re: silent semantic changes with reiser4

[Christer Weinigel]

Spam <spam@tnonline.net> writes:

>   Keeping stuff in the kernel should make the new features
>   transparent to the applications.

No, it will make just one special case, rename within the same
filesystem, work.  (Well, two special cases, rm will also delete
the other forks). 

Unless we add a new copy(2) syscall (which would be nice for other
reasons) all applicatons that copy files will fail to copy the
streams.  So no working cp command, no nautilus nor konqueror without
changes to the application.  And if you have to change the
applications anyway, isn't it much better to agree on a shared library
in userspace that everyone uses?  Which has the added bonus that it
can be made to work on top of Linux, Windows, Ultrix and VMS?

  /Christer

-- 
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <christer@weinigel.se>  http://www.weinigel.se

Re: silent semantic changes with reiser4

[Spam]

> Spam <spam@tnonline.net> writes:

>>   Keeping stuff in the kernel should make the new features
>>   transparent to the applications.

> No, it will make just one special case, rename within the same
> filesystem, work.  (Well, two special cases, rm will also delete
> the other forks). 

> Unless we add a new copy(2) syscall (which would be nice for other
> reasons) all applicatons that copy files will fail to copy the
> streams.

  How  so?  The whole idea is that the underlaying OS that handles the
  copying  should  also  know  to  copy  everything, otherwise you can
  implement  everything  into  applications  and  just  skip the whole
  filesystem part.

  Of  course,  it  won't  work if applications themselves are actually
  transferring  the  data  and  create the new files directly onto the
  filesystem. This just seem so wrong.

  ~S

>   So no working cp command, no nautilus nor konqueror without
> changes to the application.  And if you have to change the
> applications anyway, isn't it much better to agree on a shared library
> in userspace that everyone uses?  Which has the added bonus that it
> can be made to work on top of Linux, Windows, Ultrix and VMS?

>   /Christer

Re: silent semantic changes with reiser4

[Wichert Akkerman]

Previously Spam wrote:
>   How  so?  The whole idea is that the underlaying OS that handles the
>   copying  should  also  know  to  copy  everything, otherwise you can
>   implement  everything  into  applications  and  just  skip the whole
>   filesystem part.

UNIX doesn't have a copy systemcall, applications copy the data
manually.

Wichert.

-- 
Wichert Akkerman <wichert@wiggy.net>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

Re: silent semantic changes with reiser4

[Spam]

> Previously Spam wrote:
>>   How  so?  The whole idea is that the underlaying OS that handles the
>>   copying  should  also  know  to  copy  everything, otherwise you can
>>   implement  everything  into  applications  and  just  skip the whole
>>   filesystem part.

> UNIX doesn't have a copy systemcall, applications copy the data
> manually.

  Oh, this is very unfortunate and should be a bigger issue to fix.

> Wichert.

Re: silent semantic changes with reiser4

[Hans Reiser]

Bernd Petrovitsch wrote:

>On Thu, 2004-08-26 at 13:43 +0200, Spam wrote:
>  
>
>>>Previously Spam wrote:
>>>      
>>>
>>>>  How  so?  The whole idea is that the underlaying OS that handles the
>>>>  copying  should  also  know  to  copy  everything, otherwise you can
>>>>  implement  everything  into  applications  and  just  skip the whole
>>>>  filesystem part.
>>>>        
>>>>
>>>UNIX doesn't have a copy systemcall, applications copy the data
>>>manually.
>>>      
>>>
>>  Oh, this is very unfortunate and should be a bigger issue to fix.
>>    
>>
>
>Then you have to rewrite POSIX und SuSv3.
>
>	Bernd
>  
>
Spam is right.  Posix is a standard, not a vision, and the future is 
always a vision not a standard.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 01:05:02AM -0700, Hans Reiser wrote:
> Spam is right.  Posix is a standard, not a vision, and the future is 
> always a vision not a standard.

An old friend of mine always said "if someone has visions he needs to
stop somking mushrooms.."

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 09:41:07AM +0200, Bernd Petrovitsch wrote:
> > > UNIX doesn't have a copy systemcall, applications copy the data
> > > manually.
> > 
> >   Oh, this is very unfortunate and should be a bigger issue to fix.
> 
> Then you have to rewrite POSIX und SuSv3.

They don't say 'you must now have a copy syscall'.  Having one that's
actually used by system tools would be a great optimization for many
network or distributed filesystems.

Re: silent semantic changes with reiser4

[Bernd Petrovitsch]

On Fri, 2004-08-27 at 10:49 +0200, Christoph Hellwig wrote:

> On Fri, Aug 27, 2004 at 09:41:07AM +0200, Bernd Petrovitsch wrote:
> > > > UNIX doesn't have a copy systemcall, applications copy the data
> > > > manually.
> > > 
> > >   Oh, this is very unfortunate and should be a bigger issue to fix.
> > 
> > Then you have to rewrite POSIX und SuSv3.
> 
> They don't say 'you must now have a copy syscall'.  Having one that's
> actually used by system tools would be a great optimization for many
> network or distributed filesystems.

ACK. But actually spam apparently assumes that the kernel - userspace
interface is at the wrong abstraction level. And this is conceptually
bound IMHO to POSIX.
And it does not solve the problem, that the read() and write() sys-calls
will not vanish. Even if a copy() sys-call exists.

	Bernd
-- 
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services

Re: silent semantic changes with reiser4

[Hans Reiser]

Wichert Akkerman wrote:

>Previously Spam wrote:
>  
>
>>  How  so?  The whole idea is that the underlaying OS that handles the
>>  copying  should  also  know  to  copy  everything, otherwise you can
>>  implement  everything  into  applications  and  just  skip the whole
>>  filesystem part.
>>    
>>
>
>UNIX doesn't have a copy systemcall, applications copy the data
>manually.
>
>Wichert.
>
>  
>
See sys_reiser4()..... ;-)  you can go "A<-B".  I have hopes of getting 
drive vendors to support us in doing single disk copies without even 
leaving the disk drive.....

Re: silent semantic changes with reiser4

[Andreas Dilger]

[-- Attachment #1: Type: text/plain, Size: 1124 bytes --]

On Aug 26, 2004  16:56 -0700, Hans Reiser wrote:
> Wichert Akkerman wrote:
> >UNIX doesn't have a copy systemcall, applications copy the data
> >manually.
>
> See sys_reiser4()..... ;-)  you can go "A<-B".  I have hopes of getting 
> drive vendors to support us in doing single disk copies without even 
> leaving the disk drive.....

Yes, we've wanted to have a "copy" syscall for a while now also, so that
Lustre (or other network FS for that matter) can do remote copies of
large files without having to go through the VM of the client.  The work
on sendfile() in 2.6 is a step in that direction, I eagerly await the
day when "cp" will try to do a sys_sendfile() call first, before doing
the read+write looping it does now.

Consider for a network filesystem we are limited to 1/2 of the client
network bandwidth (say 50-75MB/s) for copying instead of the server-local
or multiple server-server bandwidth for copies (400-600MB/s or more).

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/             http://members.shaw.ca/golinux/


[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Stelian Ionescu]

On Thu, Aug 26, 2004 at 01:33:32PM +0200, Wichert Akkerman wrote:
>Previously Spam wrote:
>>   How  so?  The whole idea is that the underlaying OS that handles the
>>   copying  should  also  know  to  copy  everything, otherwise you can
>>   implement  everything  into  applications  and  just  skip the whole
>>   filesystem part.
>
>UNIX doesn't have a copy systemcall, applications copy the data
>manually.
couldn't sendfile(2) be used for that ?
perhaps a flag parameter could be added to sendfile(2) which specifies
whether to copy _all_ metadata if both fd's correspond to files, something
like either O_METADATA or O_NOMETADATA

-- 
Stelian Ionescu aka fe[nl]ix

Quidquid latine dictum sit, altum viditur.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christer Weinigel wrote:
> all applicatons that copy files will fail to copy the streams.  So
> no working cp command, no nautilus nor konqueror without changes to
> the application.

No true.  A lot of the fancy stuff depends on metadata which is merely
another view of the serialised contents in the flat file.  Nautilus &
Konqueror thumbnails are an example of that (I gave a few more in
other mails).  cp works fine with that, and so does ftp.  The metadata
is just recalculated when it's needed from the target file.

> And if you have to change the applications anyway, isn't it much
> better to agree on a shared library in userspace that everyone uses?
> Which has the added bonus that it can be made to work on top of
> Linux, Windows, Ultrix and VMS?

Ideally yes, a shared library _or_ at least an agreed representation.

However, with filesystem support you can improve performance by
avoiding unneed serialisation (write to your huge *Office
presentation, read it from another program, no need to wait for the
slow tar+compress stage yet it's _as if_ those were done), retain
computed metadata with coherency guarantees (e.g. real time search
indexes, crypto digests and such), and let the filesystem decide when
best to prune computed metadata or convert representations.

All of these things can work with a combination of userspace plugins
(not the same as reiser4 plugins), and filesystem support.

Without the filesystem support, you can still use the userspace
plugins -- so apps would still be portable -- but you don't get the
extras mentioned above.

-- Jamie

Re: silent semantic changes with reiser4

[Denis Vlasenko]

On Thursday 26 August 2004 14:16, Christer Weinigel wrote:
> Spam <spam@tnonline.net> writes:
> >   Keeping stuff in the kernel should make the new features
> >   transparent to the applications.
>
> No, it will make just one special case, rename within the same
> filesystem, work.  (Well, two special cases, rm will also delete
> the other forks).

In traditional UNIX filesystems, copy loses some info too
(owner,group,mode bits). cp and tar was taught about this
and they now can preserve that info.

However, I do not think that hacking on cp, tar, etc everytime
we need to add something to our filesystems (xattrs, ACLs,
whatever) is a terribly attractive idea.

I think reiserfs is going to handle that by giving uniform access
to such ancillary data. It will be visible as file/meta/*
filelets ('small files') or something like that and can be saved/restored
by e.g. tar *without* needing to know what data is stored there, and
in what format.

I think Hans is not planning turning old "file is a stream of bytes"
into eight-stream octopus. One stream will remain as a 'main' one,
which contains actual data. Others will keep metadata, etc...

Note that even today's files aren't simple "streams of bytes"
because they also have names,mode bits etc.
--
vda

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Denis Vlasenko wrote:

> I think Hans is not planning turning old "file is a stream of bytes"
> into eight-stream octopus. One stream will remain as a 'main' one,
> which contains actual data. Others will keep metadata, etc...

This is exactly what the Samba people want, though. 

Office suites can store a document with embedded images
and spread sheets "easily" by putting the text, the
images and spread sheets all in different file streams.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1259 bytes --]

Am Donnerstag, den 26.08.2004, 10:12 -0400 schrieb Rik van Riel:

> > I think Hans is not planning turning old "file is a stream of bytes"
> > into eight-stream octopus. One stream will remain as a 'main' one,
> > which contains actual data. Others will keep metadata, etc...
> 
> This is exactly what the Samba people want, though. 
> 
> Office suites can store a document with embedded images
> and spread sheets "easily" by putting the text, the
> images and spread sheets all in different file streams.

Ouch.

I think Hans' idea is (I don't know if it is a good idea nor if it is
doable, but at least it sounds interesting) to have special compound
files where you can do something like this:

cp text.txt test.compound/test.txt
cp image.jpg test.compound/image.jpg

And if you read test.compound (the main stream) you get a special format
that contains all the components. You can copy that single stream of
bytes to another (reiser4) fs and then access test.compound/test.txt
again.

The only thing I'm worrying about with this approach is what happens if
someone tries to simultaneously open test.compound and
test.compound/test.txt.

Hans has an example somewhere where he does something like that
with /etc/passwd.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christophe Saout wrote:
> I think Hans' idea is (I don't know if it is a good idea nor if it is
> doable, but at least it sounds interesting) to have special compound
> files where you can do something like this:
> 
> cp text.txt test.compound/test.txt
> cp image.jpg test.compound/image.jpg
> 
> And if you read test.compound (the main stream) you get a special format
> that contains all the components. You can copy that single stream of
> bytes to another (reiser4) fs and then access test.compound/test.txt
> again.

(To Rik especially), this is the design which more or less satisfies
lots of different goals at once.

I wrote:
> Properly implemented metadata can:
> 
>   (1) operate in both modes simultaneously;
>   (2) work with unaware applications;
>   (3) provide performance enhancements to aware applications;
>   (4) provide storage enhancements to both;
>   (5) provide useful features that work with standard unmodified unix tools,

(1) is by design.

You can copy these "files" safely using cp or cat or ftp.
You can email them.  They're flat files.  That's (2).

You can poke around inside them.

Applications which do poke around inside get performance gains,
because they don't have to serialise the flat format every time they
"load" or "save" a file.  For example, if you have a huge OpenOffice
document containing lots of sub-documents and pictures.  We can
imagine a version of OpenOffice which just opens the sub-documents,
and writes the ones which are changed after editing.  Also, another
program such as a viewer, post-processor or search indexing program
only needs to read the parts which have changed; it doesn't have to
decode the whole thing after a change.  That's (3).

We already know how useful this is with big mailboxes, hence people
switching to maildir.  Now imagine the attachments in each message are
also viewable, cat'able etc., and that the filesystem stores each
message compressed automatically, or in aggregate (your choice) and
that you can use both mbox and maildir tools on the same mailbox. That's (4).

You can ls, cp, grep, less, vi, gqview etc. the pieces of files, and
every supported file format needs just one tool to provide the
interface to its contents.  That's (5).

While I'm here I might as well add:

  (6) improve support for real time search indexing.

The "view coherency" mechanism which is needed for efficient
userspace VFS support has just the right properties to keep informing
an indexer of changes.  They could even enforce operation orderings if
we cared about _perfect_ real-timeness, although we probably don't.

  (7) yet another way of doing distros.

Why not just install .tgz or .zip files of entire binary packages, and
let the filesystem unpack the bits that are needed when they're
needed.  Some platforms (i.e. Java) do this already with .jar files
and it's quite popular.  This would extend it in a versatile way.

Rik, I hope I've explained how those 5 properties are satisfied by
Hans Reiser's brilliant design (ahem, great minds think alike, and
remember Linus likes files-as-directories too :).  But if I haven't
please do pick holes.

> The only thing I'm worrying about with this approach is what happens if
> someone tries to simultaneously open test.compound and
> test.compound/test.txt.

It is possible to implement it so that every change to one file
immediately affects the other in the appropriate way, _as if_ the
other file is regenerated every time before reading it.

This is what I mean by coherent views.  It can be done even when
userspace is doing the work, and the obvious mechanisms just happen to
have the property of being useful in other ways (like a better
dnotify, but I digress...).  They're just an ordinary cache coherency
protocols.

However, if you are writing a few bytes to test.compound, then the
coherent view in test.compound/test.txt is probably a corrupt or
non-existent file, because test.compound's intermediate state is a
truncated and/or corrupt archive file, until you've written some more.

That doesn't look nice, but it's exactly the same problem as you get
when you're writing to a file that another program is reading.  If you
do that on a live http server, for example editing a .html file, very
occasionally someone will see a truncated file.  Even if your editor
uses renames to update atomically, sometimes you need to update more
than one file atomically.  It can be done, but rarely is.

Is there a solution to this?  The obvious one is when you're writing
to test.compound, don't read from test.compound/test.txt.  That's _no
different_ from the advice which says when you're saving a file from
OpenOffice, don't load it into a document viewer during the time you're
writing it, and don't serve it.

The other advice is to save to a different file and call rename().
Again, you get pretty much the same results as now: it atomically
changes, and currently open references to the old contents will
continue to reference the old contents.

There's another solution: transactions.  reiser4 offers some kind of
transaction facility, doesn't it?  This is a fine example of when
might want to declare that a sequence of writes needs to be a single
transaction, visible only when the whole sequence is complete.  Perfect!

I don't know reiser4's interface to that, but it would make a lot of
sense to have a simple shell tool called "atomic", to wrap a sequence
of operations.  That would be useful for writing files _and_ useful
for those pesky "update several files on the web server at once"
moments.  (Yes I know you can build a hard link tree and rename the
root directory, although even that doesn't always work).

-- Jamie

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Jamie Lokier wrote:
> Christophe Saout wrote:

> > And if you read test.compound (the main stream) you get a special format
> > that contains all the components. You can copy that single stream of
> > bytes to another (reiser4) fs and then access test.compound/test.txt
> > again.
> 
> (To Rik especially), this is the design which more or less satisfies
> lots of different goals at once.

And if an unaware application reads the compound file
and then writes it out again, does the filesystem
interpret the contents and create the other streams ?

Unless I overlook something (please tell me what), the
scheme just proposed requires filesystems to look at
the content of files that is being written out, in
order to make the streams work.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1291 bytes --]

Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel:

> > > And if you read test.compound (the main stream) you get a special format
> > > that contains all the components. You can copy that single stream of
> > > bytes to another (reiser4) fs and then access test.compound/test.txt
> > > again.
> > 
> > (To Rik especially), this is the design which more or less satisfies
> > lots of different goals at once.
> 
> And if an unaware application reads the compound file
> and then writes it out again, does the filesystem
> interpret the contents and create the other streams ?
> 
> Unless I overlook something (please tell me what), the
> scheme just proposed requires filesystems to look at
> the content of files that is being written out, in
> order to make the streams work.

Yes, the main compound stream unaware applications would see would just
be a writable view of its contents. Probably not trivial to implement
but doable.

It should be a simple format. Something simliar to tar. The worst thing
that can happen is people start writing plugins for every existing
compound format out there. It should be the other way around, agree on a
simple compound format and encourage applications to use this one if the
want to use this advantage.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 06:04:42PM +0200, Christophe Saout wrote:
> It should be a simple format. Something simliar to tar. The worst thing
> that can happen is people start writing plugins for every existing
> compound format out there. It should be the other way around, agree on a
> simple compound format and encourage applications to use this one if the
> want to use this advantage.

or placing this into a userspace helper that the kernel can invoke
transparently.  If you're doing funny thing like that for your swap
device deadlocks are the last of your worries :)

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> or placing this into a userspace helper that the kernel can invoke
> transparently.

Yes, exactly this for most file formats.
Ideally, with _some_ user control over the handlers for their own
files a bit like Hurd.

Not just archive files: metadata pulled from audio files, images,
distro packages, etc.  In other words much the same as Gnome and KDE's
things, but with more emphasis on metadata extractors and efficient
on-disk cache interaction, and a sane page-cache coherency interface
(possibly FUSE).

Metadata extraction is useful for search and indexing purposes.
(That, and a way to hook change notifications up with indexers --
again comes with any proper coherency interface).

For new applications using a container, it's tempting to want one
format which is optimised for this.  But really many applications want
to write XML or compressed XML, or something else that isn't
especially efficient but is easy to work with on many OSes.  We should
support applications whatever format they need to work with, if
providing an interior view as a directory is useful for them.  (This
implies applications can introduce their own formats for files they
need to operate on.)  By the way, lazy updating of containers is an
even bigger advantage if you have to use XML as the flat format,
because the savings from not serialising when you don't need to are
more pronounced.

For some applications the "content" isn't structured in a way which
maps naturally to a plain archive file.  Hans' /etc/passwd is a simple
example: at least one of the "flat" representations, when there are
enough entries, probably ought to be a DB file for faster lookup.
(Though, it would be great if /etc/passwd could automagically get fast
lookup due to an automagic underlying representation, while still
appearing to be a flat file when you cat it).

Also, imagine "cd" into a MySQL table file.  It might just work.

By the way, we don't need containers to waste huge amounts of space
like the Microsoft Word "fast save" documents did.  Because our
containers are created on demand, there is no performance advantage in
them having unused holes, unlike the case were Word was saving changes
periodically.

-- Jamie

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1



Christophe Saout wrote:

| Yes, the main compound stream unaware applications would see would just
| be a writable view of its contents. Probably not trivial to implement
| but doable.

Not the main stream, I think, but we do need a way to make "eight-stream
octopus" files portable.  But there needs to be a copy(2) syscall, and
apps need to start supporting it.  After all, copy-on-write was planned
for Reiser4, now 4.1*.  And don't anyone dare suggest "userland library".

| It should be a simple format. Something simliar to tar. The worst thing

Simple, yes.  Similar to tar, no.

How about a serializer plugin?  foo.mp3/serialize (say) should return
some single stream which contains all of foo.mp3.  It could work for
directories as well, recursively, at least on the same filesystem --
you'd back up by doing

cp /serialize /backup/20040826

This has two potential problems.  One is security -- I want to be able
to serialize foo.mp3 without being root, but root doesn't want me to be
able to serialize /etc and thus get access to shadow.

The second problem is that in order to actually be used for backup, we
need snapshots, which are not done yet*.  Currently, it'd have to lock
the file/directory (recursively), which you don't want to do to entire
live filesystems...  So, no massive backups of live systems yet, but
definitely useful for sending around mp3s and such.

| that can happen is people start writing plugins for every existing
| compound format out there. It should be the other way around, agree on a
| simple compound format and encourage applications to use this one if the
| want to use this advantage.

For the format, I vote for whatever format we're using at the storage
layer.  It'd be a lot faster that way, both in execution time (I think)
and in development time (I'm sure).  Plus, imagine the restore:

cat /backup/20040826 > /dev/hda5
resizefs_reiser4 /dev/hda5


* Disclaimer -- all this is "I think, last I checked".


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQS4uhngHNmZLgCUhAQKTaw//WCHt0Umgaf0GaMUpEr0PLhiXhFTYQqKV
MKgpVt6SjP7lC9k3RbuCnPCLyOJGb9lIqo+Wd+B3b4TcQAQd4OYsSI6WiMPQjxXY
OdNnB7/xYkkd/ULw7/7IGcokyc4FRDjhTyOMe4KscQS/6uhdX1dCriCCzVq7iELx
u3HLulunHv1bwHbnZ6CyEZdQEr3j8XUu62+IXPTc6hWnfRRCvskV8Y4qyyVd8wFY
Kp44TxbJJSzDeWmuI/YYkxVJsPmSoxRLxmw9/B8D72KGNg1SYLnM4ueIc5IV8RH0
kG/o8tzwTRxE/eqfcHP+UBt6yXZR/i84DDogAmJFKP73g9X+U+J5o/kONOBQODhY
qY5LUwQu4UjlzXttMxC4NmZPESdUw85DgC/GhTQlsBMYBcmQHmpLAK67qt83+Kg0
1QqGPqJm4HL4weHLNknJ0DukLNKH7Bw5CVg335IwruaoiKUG7HDgthUhA319FCab
VvYa+bEfeBEw9RorSE/gWM9Hg73luM7eoSzKc9zHgx++cACTi8SBsTFWqI2FU6DX
UyWG5sNPybOiczpPsFu5S5FJBBNM3fLjkfeYgfqxUIjxGTNeUn/g/9I5ShUrCF1o
T+3nd1jFxfoehYGkfGB2YLpdeNVBv4EdKK16j39+iducfJrXnRlL1yjdo/Kte2uz
tTQZw3uTnhU=
=7bBF
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[David Lang]

On Thu, 26 Aug 2004, David Masover wrote:

> Christophe Saout wrote:
>
> | Yes, the main compound stream unaware applications would see would just
> | be a writable view of its contents. Probably not trivial to implement
> | but doable.
>
> Not the main stream, I think, but we do need a way to make "eight-stream
> octopus" files portable.  But there needs to be a copy(2) syscall, and
> apps need to start supporting it.  After all, copy-on-write was planned
> for Reiser4, now 4.1*.  And don't anyone dare suggest "userland library".
>
> | It should be a simple format. Something simliar to tar. The worst thing
>
> Simple, yes.  Similar to tar, no.
>
> How about a serializer plugin?  foo.mp3/serialize (say) should return
> some single stream which contains all of foo.mp3.  It could work for
> directories as well, recursively, at least on the same filesystem --
> you'd back up by doing
>
> cp /serialize /backup/20040826
>
> This has two potential problems.  One is security -- I want to be able
> to serialize foo.mp3 without being root, but root doesn't want me to be
> able to serialize /etc and thus get access to shadow.
>
> The second problem is that in order to actually be used for backup, we
> need snapshots, which are not done yet*.  Currently, it'd have to lock
> the file/directory (recursively), which you don't want to do to entire
> live filesystems...  So, no massive backups of live systems yet, but
> definitely useful for sending around mp3s and such.
>
> | that can happen is people start writing plugins for every existing
> | compound format out there. It should be the other way around, agree on a
> | simple compound format and encourage applications to use this one if the
> | want to use this advantage.
>
> For the format, I vote for whatever format we're using at the storage
> layer.  It'd be a lot faster that way, both in execution time (I think)
> and in development time (I'm sure).  Plus, imagine the restore:
>
> cat /backup/20040826 > /dev/hda5
> resizefs_reiser4 /dev/hda5
>

one of the virtual metatags should be something about the method to 
serialize the data

if you are viewing a tarfile as a directory the VFS/Filesystem shoud know 
that the link between the whole thing and the contents is tar, and it 
should be possible for userspace programs to find this out by reading teh 
appropriate thing inside the virtual directory (and eventually change it 
as well, which would trigger regeneration of the base object, or at least 
noting that the current version is wrong and needs to be regenerated)

David Lang

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[David Lang]

On Thu, 26 Aug 2004, Christophe Saout wrote:

> Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel:
>
>>>> And if you read test.compound (the main stream) you get a special format
>>>> that contains all the components. You can copy that single stream of
>>>> bytes to another (reiser4) fs and then access test.compound/test.txt
>>>> again.
>>>
>>> (To Rik especially), this is the design which more or less satisfies
>>> lots of different goals at once.
>>
>> And if an unaware application reads the compound file
>> and then writes it out again, does the filesystem
>> interpret the contents and create the other streams ?
>>
>> Unless I overlook something (please tell me what), the
>> scheme just proposed requires filesystems to look at
>> the content of files that is being written out, in
>> order to make the streams work.
>
> Yes, the main compound stream unaware applications would see would just
> be a writable view of its contents. Probably not trivial to implement
> but doable.
>
> It should be a simple format. Something simliar to tar. The worst thing
> that can happen is people start writing plugins for every existing
> compound format out there. It should be the other way around, agree on a
> simple compound format and encourage applications to use this one if the
> want to use this advantage.
>

I also don't see why the VFS/Filesystem can't decide that (for example) 
this tar.gz is so active that instead of storing it as a tar.gz and 
providing a virtual directory of the contents that it instead stores the 
directory of the contents and makes the tar.gz virtual (regenerating it as 
needed or as extra system resources are available)

implementation wise I see headaches in doing this, but conceptually this 
is just an optimization that could take place in the future if we fine 
that it's needed.

David Lang

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Jamie Lokier]

David Lang wrote:
> I also don't see why the VFS/Filesystem can't decide that (for example) 
> this tar.gz is so active that instead of storing it as a tar.gz and 
> providing a virtual directory of the contents that it instead stores the 
> directory of the contents and makes the tar.gz virtual (regenerating it as 
> needed or as extra system resources are available)

Absolutely.  It could keep both views, if they're both actively being
used.  Or more than both, if there are more.  (You could think of
compression being an alternate view, and both compressed and
uncompressed may as well remain on disk if there's space and it's
being actively accessed.

> implementation wise I see headaches in doing this, but conceptually this 
> is just an optimization that could take place in the future if we fine 
> that it's needed.

untarring a big source tree takes ages.  I wouldn't like to do it
after every reboot, if there was a .tar.bz2 tree I looked at often.
That's why I have things like glibc untarred in my home directory.

With good cacheing, I could cd into the .tar.bz2 files and
_effectively_ have the performance of untarred source trees for the
ones I look at often on my disk -- automatically cleaned if the space
if needed for something else, too.  It would be quite nice.

-- Jamie

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Jamie Lokier wrote:
> 
> With good cacheing, I could cd into the .tar.bz2 files and
> _effectively_ have the performance of untarred source trees for the
> ones I look at often on my disk -- automatically cleaned if the space
> if needed for something else, too.  It would be quite nice.

What you're talking about is just a caching filesystem. Then you can have 
any deamon do whatever it wants to fill it - whether from tar-files or 
over the network or anything else.

I think you should ask David Howells about it..

		Linus

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 23:05, David Lang wrote:

> I also don't see why the VFS/Filesystem can't decide that (for example)
> this tar.gz is so active that instead of storing it as a tar.gz and
> providing a virtual directory of the contents that it instead stores the
> directory of the contents and makes the tar.gz virtual (regenerating it as
> needed or as extra system resources are available)

Because that would mean the kernel should "talk" the tar format, which is, 
IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to 
perform this task, instead.

Re: silent semantic changes with reiser4

[Chris Dukes]

On Fri, Aug 27, 2004 at 12:30:19AM +0200, Felipe Alfaro Solana wrote:
> On Thursday 26 August 2004 23:05, David Lang wrote:
> 
> > I also don't see why the VFS/Filesystem can't decide that (for example)
> > this tar.gz is so active that instead of storing it as a tar.gz and
> > providing a virtual directory of the contents that it instead stores the
> > directory of the contents and makes the tar.gz virtual (regenerating it as
> > needed or as extra system resources are available)
> 
> Because that would mean the kernel should "talk" the tar format, which is, 
> IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to 
> perform this task, instead.

Wasn't HURD supposed to do this for us?
-- 
Chris Dukes
Warning: Do not use the reflow toaster oven to prepare foods after
it has been used for solder paste reflow. 
http://www.stencilsunlimited.com/stencil_article_page5.htm

Re: silent semantic changes with reiser4

[David Lang]

On Fri, 27 Aug 2004, Felipe Alfaro Solana wrote:

> On Thursday 26 August 2004 23:05, David Lang wrote:
>
>> I also don't see why the VFS/Filesystem can't decide that (for example)
>> this tar.gz is so active that instead of storing it as a tar.gz and
>> providing a virtual directory of the contents that it instead stores the
>> directory of the contents and makes the tar.gz virtual (regenerating it as
>> needed or as extra system resources are available)
>
> Because that would mean the kernel should "talk" the tar format, which is,
> IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to
> perform this task, instead.
>

the kernel will definantly need the ability to use user-space code to do 
the transformations from one version to the other (if nothing else think 
of the thumbnail version of images, we don't want the image manipulation 
code in the kernel and we definantly want this sort of option available)

the interesting issue is going to be defining the kernel->user-space 
interface for doing the extractions.

David Lang

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> And if an unaware application reads the compound file
> and then writes it out again, does the filesystem
> interpret the contents and create the other streams ?

Yes, exactly that.  The streams are created on demand of course, and
by userspace helpers when that's appropriate which I suspect it almost
always is.

> Unless I overlook something (please tell me what), the
> scheme just proposed requires filesystems to look at
> the content of files that is being written out, in
> order to make the streams work.

Yes.  Hence the idea of coherent views between two files: writing to
one affects the content of the other, although the calcalation is only
done on demand (or when the fs wants to migrate the representation --
for example, creating the flat container prior to deleting the
regeneratable pieces in order to save space).

I haven't seen anything from Namesys that says they'll do that.  I
have the impression the streams are just generated in memory on the
fly, not stored on disk with a cacheing policy, but that's just an
impression.  (We've all seen the Namesys white papers, they're not
_that_ revealing). :)

I'm just pointing out how to do it _right_.  I think it will turn out
like this eventually, either next year or some time over the next
decade after some iterations.  Inevitable.  Mark my words, etc. :)

-- Jamie

Re: silent semantic changes with reiser4

[John Stoffel]

Jamie> Rik van Riel wrote:
>> And if an unaware application reads the compound file
>> and then writes it out again, does the filesystem
>> interpret the contents and create the other streams ?

Jamie> Yes, exactly that.  The streams are created on demand of
Jamie> course, and by userspace helpers when that's appropriate which
Jamie> I suspect it almost always is.

So how would a program that converts between a JPEG file (with exif
data) and a PNG work, such as ImageMagick?  Are we proposing to teach
the VFS (or worse yet each filesystem) how to do this?  

I've been following this discussion a bit and I'm not sure that I've
actually seen any concrete examples of where this is a *good* thing to
have.  People talk about only having to modify 20 bytes at a time
instead of reading and writing 1mb of data.  Isn't that what mmap()
does?

Now I can sorta understand the idea that having a directory look like
a file is neat, and certainly simplifies some aspects, but I think
that going all the way down to the logical conclusion here is a bit
silly.

To use the principle of blowing things up to make them very large or
very small, what happens if I decide that the best idea is to make all
files just be directories which contain single byte files?  Isn't that
the logical extreme here?  So my 1mb JPEG file is not just some image
data and header info in multiple files, but it's really just 1
million (ok 1024 * 1024) individual files that the VFS knows how to
put together.  Seems like the logical extreme.  Oh wait, maybe we
should be exposing a single file per bit instead! 

>From my point of view, there lies madness.  As Rik pointed out, how do
backup and restore tools work with this stuff?  Most people could care
less about how their data is organized, but they certainly care when
they can't restore it from backups.  

I'd really like to see a concrete example from Hans or other
proponents about why this makes things easier/faster/better to do.
Mostly, I've just seen "proof by vigorous handwaving" that it's a good
thing.

In alot of ways, I think people are going in the wrong direction, you
want to excapsulate and hide the details more, not expose them.
That's what a good API does, it hides the details while giving you a
rich set of semantics to manage your data.  

God knows I'm not smart enough or driven enough to actually come up
with my own ideas, but I certainly haven't seen anyone else (even
Linus) come up with an earth shattering arguement to say why this is
the right move to make.

As Linus says, most of the OS's job is to mediate access to
objects/data.  Why do we want to expose such low level data then?

John

   John Stoffel - Senior Unix Systems Administrator - Lucent Technologies
	 stoffel@lucent.com - http://www.lucent.com - 978-952-7548

Re: silent semantic changes with reiser4

[Denis Vlasenko]

On Thursday 26 August 2004 22:32, John Stoffel wrote:
> Jamie> Rik van Riel wrote:
> >> And if an unaware application reads the compound file
> >> and then writes it out again, does the filesystem
> >> interpret the contents and create the other streams ?
>
> Jamie> Yes, exactly that.  The streams are created on demand of
> Jamie> course, and by userspace helpers when that's appropriate which
> Jamie> I suspect it almost always is.
>
> So how would a program that converts between a JPEG file (with exif
> data) and a PNG work, such as ImageMagick?  Are we proposing to teach
> the VFS (or worse yet each filesystem) how to do this?
>
> I've been following this discussion a bit and I'm not sure that I've
> actually seen any concrete examples of where this is a *good* thing to
> have.  People talk about only having to modify 20 bytes at a time
> instead of reading and writing 1mb of data.  Isn't that what mmap()
> does?
>
> Now I can sorta understand the idea that having a directory look like
> a file is neat, and certainly simplifies some aspects, but I think
> that going all the way down to the logical conclusion here is a bit
> silly.
>
> To use the principle of blowing things up to make them very large or
> very small, what happens if I decide that the best idea is to make all
> files just be directories which contain single byte files?  Isn't that
> the logical extreme here?  So my 1mb JPEG file is not just some image
> data and header info in multiple files, but it's really just 1
> million (ok 1024 * 1024) individual files that the VFS knows how to
> put together.  Seems like the logical extreme.  Oh wait, maybe we
> should be exposing a single file per bit instead!

It is doable. It doesn't mean we shall do this.

I think some reiser4 supporters try to do too much too soon.
It is more sensible to do it in small steps.

I think we can start small and make all file metadata to be accessible
via file/meta/{uid,gid,mode,mtime,atime,...}.

Normal tools will be unable to see file/meta/ because file is not
a directory and file/ is not a directory (i.e. open(O_DIRECTORY)
will fail on them). Hardlinking to file/meta/* is not allowed.

However, file/. _is_ a directory. We can make a tar-like
tool which can do its work and save/restore metadata along with data
by just tarring/untarring file/meta/*. Notice how this 'new tar'
does not need to know about exact kind of metadata supported by fs.
You can add ACLs to the fs, and this 'new tar' will be able to save/restore
ACLs. Without any modufications.

Contrast this with existing tar which has hardcoded knowlendge about
uid,gid,mode,etc...
--
vda

Re: silent semantic changes with reiser4

[Jamie Lokier]

John Stoffel wrote:
> Jamie> Yes, exactly that.  The streams are created on demand of
> Jamie> course, and by userspace helpers when that's appropriate which
> Jamie> I suspect it almost always is.
> 
> So how would a program that converts between a JPEG file (with exif
> data) and a PNG work, such as ImageMagick?  Are we proposing to teach
> the VFS (or worse yet each filesystem) how to do this?  

No.  (1) That's what "userspace helpers" are for.  (2) What does image
format conversion have to do with viewing files in their component
parts?  (3) I suspect someone will write a plugin that does indeed
convert virtually every known image format to a common format
(probably PNG and something "raw") at some point.  Why not?  It's just
a small script (it would run ImageMagick!).

> I've been following this discussion a bit and I'm not sure that I've
> actually seen any concrete examples of where this is a *good* thing to
> have.  People talk about only having to modify 20 bytes at a time
> instead of reading and writing 1mb of data.  Isn't that what mmap()
> does?

Sure, if you think mmap() is an easy substitute for "parse the author
& license tags from this [unknown format] audio, video or font file".

> Now I can sorta understand the idea that having a directory look like
> a file is neat, and certainly simplifies some aspects, but I think
> that going all the way down to the logical conclusion here is a bit
> silly.
> 
> To use the principle of blowing things up to make them very large or
> very small, what happens if I decide that the best idea is to make all
> files just be directories which contain single byte files?  Isn't that
> the logical extreme here?  So my 1mb JPEG file is not just some image
> data and header info in multiple files, but it's really just 1
> million (ok 1024 * 1024) individual files that the VFS knows how to
> put together.  Seems like the logical extreme.  Oh wait, maybe we
> should be exposing a single file per bit instead! 

If you decide to do that, you are welcome to write the userspace
helper which creates that view in your directory.

Why not?  It's very silly, but it's no sillier then writing a program
to read your 1mb JPEG and write it out one file per bit, which you're
welcome to write right now.

> >From my point of view, there lies madness.  As Rik pointed out, how do
> backup and restore tools work with this stuff?  Most people could care
> less about how their data is organized, but they certainly care when
> they can't restore it from backups.  

Generated views are something which should _not_ be backed up and
restored.  That's not what they are for.

Auxiliary metadata, such as author info and signatures which cannot be
stored in the main file for some reason -- that should be backed up.

Permissions, that should be backed up.

But not views which are computed from the main file.  You don't need
to back them up, and they don't need to take any real space.  They're
virtual, just like an infinitely deep directory on a web server can be
virtual.  You can make those very silly too, if you want to.

(Some help from the filesystem with storing temporarily cached values
is fine, for performance, but we shouldn't pretend that generated
views are anything other than virtual).

> I'd really like to see a concrete example from Hans or other
> proponents about why this makes things easier/faster/better to do.
> Mostly, I've just seen "proof by vigorous handwaving" that it's a good
> thing.
> 
> In alot of ways, I think people are going in the wrong direction, you
> want to excapsulate and hide the details more, not expose them.
> That's what a good API does, it hides the details while giving you a
> rich set of semantics to manage your data.  

Would you like to propose an alternate API?  I was under the
impression that we were discussing one, right here in this thread.
It's called POSIX with an extension where you can cd into a file.  It
has a rich set of semantics to manage your data, and works with a lot
of familiar programs.

A better one would be welcome, if you have an idea.

> God knows I'm not smart enough or driven enough to actually come up
> with my own ideas, but I certainly haven't seen anyone else (even
> Linus) come up with an earth shattering arguement to say why this is
> the right move to make.

Come up with a better one otherwise it's right ;)

> As Linus says, most of the OS's job is to mediate access to
> objects/data.  Why do we want to expose such low level data then?

Eh?  In this case the OS is mediating by providing a uniform interface
between programs that access the data and programs that handle file
formats.

What is this "low level" data you speak of?  I would say something
like the "designer" of a font, or the "from address" and "subject" of
an email are very high level abstractions, and that's among the sorts
of things which are to be exposed by these interfaces.

-- Jamie

Re: silent semantic changes with reiser4

[Alex Zarochentsev]

On Thu, Aug 26, 2004 at 05:53:51PM +0100, Jamie Lokier wrote:
> Rik van Riel wrote:
> > And if an unaware application reads the compound file
> > and then writes it out again, does the filesystem
> > interpret the contents and create the other streams ?
> 
> Yes, exactly that.  The streams are created on demand of course, and
> by userspace helpers when that's appropriate which I suspect it almost
> always is.
> 
> > Unless I overlook something (please tell me what), the
> > scheme just proposed requires filesystems to look at
> > the content of files that is being written out, in
> > order to make the streams work.
> 
> Yes.  Hence the idea of coherent views between two files: writing to
> one affects the content of the other, although the calcalation is only
> done on demand (or when the fs wants to migrate the representation --
> for example, creating the flat container prior to deleting the
> regeneratable pieces in order to save space).
> 
> I haven't seen anything from Namesys that says they'll do that.  I
> have the impression the streams are just generated in memory on the
> fly, not stored on disk with a cacheing policy, but that's just an
> impression.  (We've all seen the Namesys white papers, they're not
> _that_ revealing). :)

would it be a tautology if I say that the streams implementation depends on
(reiser4 now, may be vfs later) plugins? All proposals including user-level
helpers, shadow caching/indexing may be implemented, why not?

> I'm just pointing out how to do it _right_.  I think it will turn out
> like this eventually, either next year or some time over the next
> decade after some iterations.  Inevitable.  Mark my words, etc. :)
> 
> -- Jamie

-- 
Alex.

Re: silent semantic changes with reiser4

[Hans Reiser]

Alex Zarochentsev wrote:

>On Thu, Aug 26, 2004 at 05:53:51PM +0100, Jamie Lokier wrote:
>  
>
>>Rik van Riel wrote:
>>    
>>
>>>And if an unaware application reads the compound file
>>>and then writes it out again, does the filesystem
>>>interpret the contents and create the other streams ?
>>>      
>>>
>>Yes, exactly that.  The streams are created on demand of course, and
>>by userspace helpers when that's appropriate which I suspect it almost
>>always is.
>>
>>    
>>
>>>Unless I overlook something (please tell me what), the
>>>scheme just proposed requires filesystems to look at
>>>the content of files that is being written out, in
>>>order to make the streams work.
>>>      
>>>
>>Yes.  Hence the idea of coherent views between two files: writing to
>>one affects the content of the other, although the calcalation is only
>>done on demand (or when the fs wants to migrate the representation --
>>for example, creating the flat container prior to deleting the
>>regeneratable pieces in order to save space).
>>
>>I haven't seen anything from Namesys that says they'll do that.
>>
We do that, err, it is our design to do that. Hmm, the plugins to do 
that seem to all be in the next to implement list and not on the 
implemented already list.....

However, with respect to streams, those are just files in a 
file-directory object in our scheme. If you want them unified, well, 
then you cat compound-document/pseudos/glued, or you link 
compound-document/pseudos/glued to compound-document/pseudos/body, where 
body is our term for the default stream, and then you just cat 
compound-document

If you have the impression I am being vaporwarish, well, you are 
right.... glued is not yet on the already implemented list.

Hans

Re: silent semantic changes with reiser4

[Diego Calleja]

El Thu, 26 Aug 2004 11:54:51 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió:

> And if an unaware application reads the compound file
> and then writes it out again, does the filesystem
> interpret the contents and create the other streams ?

If the old utils doesn't support streams there's no way to make them create
streams. There's not a real solution for that except fixing the old tools.

Why instead of being "compound" you just make test.compound to
behave as a real directory? ie:

# echo "foo" > streamfile
# echo "fake metadata" > streamfile/somemetadata
# cat streamfile
foo
# cat streamfile/somemetadata
fake metadata

Now the old backup utils could see "streamfile" as a directory, ie:
old apps could see that "streamfile" is a directory and then make
their backups with the following contents:
streamfile/somemetadata
streamfile/defaultstream

The defaultstream would be what its name means. There could be more
default streams like "streamfile/compoundstream"

All this looks like reinventing the file/directory concept wheel. Instead of
adding support for streams and "use files as directories", why not orientate
it to "use directories as files? Streams could very well be provided
by directories containing files, with the addon of being able to cat
a directory (which would cat the output of a designed file inside
every directory like "defaultstream", then there could be other
"special files" like "compoundstream", although the idea of "fixed name-
special files" doesn't sounds very cool) All this could be very well
provided done userspace, but then couldn't userspace also supply the whole VFS 
functionality and still it lives in the kernel...?

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Diego Calleja wrote:

> All this looks like reinventing the file/directory concept wheel.
> Instead of adding support for streams and "use files as directories",
> why not orientate it to "use directories as files? Streams could very
> well be provided by directories containing files,

So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

This is getting surreal.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Rik van Riel wrote:
>
> On Thu, 26 Aug 2004, Diego Calleja wrote:
> 
> > All this looks like reinventing the file/directory concept wheel.
> > Instead of adding support for streams and "use files as directories",
> > why not orientate it to "use directories as files? Streams could very
> > well be provided by directories containing files,
> 
> So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

No no. The stream you get from a directory is totally _independent_ of the 
contents of the directory. Anything else would be pointless.

		Linus

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> 
> No no. The stream you get from a directory is totally _independent_ of the 
> contents of the directory. Anything else would be pointless.

Surely it depends on the directory?

The stream from a "classic directory", i.e. something made with
mkdir(), mode drwxr-xr-x filled with files, might well be a reproducible
representation of its contents.

-- Jamie

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Jamie Lokier wrote:
> Linus Torvalds wrote:
> > 
> > No no. The stream you get from a directory is totally _independent_ of the 
> > contents of the directory. Anything else would be pointless.
> 
> Surely it depends on the directory?

Well, you _could_ make it do a "tar", but what's the point? That really 
doesn't add any value as far as user space is concerned.

Think of it this way: if the directory and the "default stream" associated 
with it aren't independent, then user space already has the information, 
and the stream is pointless. In fact, the stream is _worse_ than 
pointless, because it hides the fact that there is no independent 
information.

See my point? 

		Linus

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> Well, you _could_ make it do a "tar", but what's the point? That really 
> doesn't add any value as far as user space is concerned.
> 
> Think of it this way: if the directory and the "default stream" associated 
> with it aren't independent, then user space already has the information, 
> and the stream is pointless. In fact, the stream is _worse_ than 
> pointless, because it hides the fact that there is no independent 
> information.
> 
> See my point? 

I see your point, but I agree 50%.

(1) One of the uses of "streams" is to have extra information which
    accompanies a file, and is separate from the flat file's information.

    For example, metadata like "this C source file is in UTF-8",
    "the author is Jamie (nowhere to put it in the data file)" and
    "Jamie's signature".

(2) Filesystem settings like "compress this", and perhaps "permissions".

(3) Filesystem information like "this file has been compressed by 59%,
    uses 5 1k blocks".

(4) Another use, which is more akin to containers, is to _present_ the
    information of the flat file in a different way.

    For example, the id tags and comments of any media file, the component
    parts of any media file, license (of course :), and expanded or
    decoded views such as inside an archive file, filesystem image, or
    other structured storage file (.doc :).

    Those alternate views are useful just because they can be accessed
    using ordinary unix tools, no matter what the obscure codec format
    provided it's supported.  With a bit of effort a standard
    presentation for certain kinds of data may arise.

(5) There is also data which is _calculated_ from the flat file or its
    extra metadata, and which the kernel guarantees is either
    consistent with the file, not set at all, or at least is
    invalidated if the file is changed.  It may be calculated on demand
    through the kernel interface, or applications may simply have to
    calculate it themselves when they don't have an up to date value.

    For example MD5 or SHA1 digests of the flat file, or of its parts.
    The key thing is that these are always up to date, not dependent
    on weak tests like mtime.

    Partial checksums for speeding up rsync and other delta algorithms.
    Computed thumbnails, MP3 average power levels, character encoding
    extracted from parsing old HTML META tags.  Anything really that
    can be calculated and usefully cached.

    This is similar to (4), but I'm emphasising things that
    may take a while to compute, may be cached with the filesystem
    itself helping to decide when to evict, and writing them won't
    affect the data they're calculated from, if writing is allowed at all.

    The only really important of these are MD5 or SHA1 digests, or
    some other way of tracking modifications like a strong serial number.

    All other computed cached data can be maintained by applications
    elsewhere, if they know when the data has been modified since the
    last time.  (Though there are advantages to making the data
    disappear if the file is deleted, or move if it's renamed).


The thing with these different categories is that we want different
behaviour when serialising the data for a backup or to copy it.

We probably always (1) to be copied.

(2) is questionable: it's like unix file permissions, you might want
to copy it; you might not.

(3) Shouldn't be copied or stored except for special reasons.  It's a
bit like /proc: it's not meant to be stored on your backup CD and it's
definitely not meant to be written during a restore. :)

(4) Should never be copied if you're copying the flat file, because
it's exactly the same information, just accessed differently.  You
might want to copy it by itself for some reason though, and you
certainly would like to work with it from scripts and applications.
These streams are special in another way: they might themselves
decompose into further streams.  Just think of a AVI file containing
an OGG file containing a FLAC containing metadata and data streams, or
something like that.  They happen.

(5) shouldn't be copied if you're transmitting a file, as a matter of
data integrity among other things.  You might want it on the backup,
then again maybe only some of the values.  Probably never the crypto digests.


To summarise: most data inside a directory-as-file should _not_
usually be copied and transported.  A little of it, (1), should be
copied if the serialised form will have it, and some of the other
metadata should be copied under special circumstances.

I've suggested that we could simply not have (1), and recommend that
only reproducible, non-essential data appears inside the directory.
Everything essential would be part of the serialised, flat file.

However, you seem to like (1), metadata which should be serialised and
copied around, and which is not part of the flat file.

Note that (1) will only be copied around on systems which handle
"extra forks" anyway -- things like Samba and MacOS compatibility.
FTP and HTTP obviously won't copy the metadata of (1), although HTTP
WebDAV might allow it to be visible as properties.

All that suggests a common structure for the directory inside a
"container" -- one where the metadata to be copied by default is
clearly distinguished from the metadata which should never by copied,
or which is cached, or which is a view.  Possibly it should be
distinguished by name, although permission bits or (not sure I like
this) meta-metadata could serve the same role without being name
dependent.

-- Jamie

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> 
> No no. The stream you get from a directory is totally _independent_ of the 
> contents of the directory. Anything else would be pointless.

It's a relief to know that nobody's taking my humorous
suggestion seriously, but now we still have the "standard
Unix tools can't manipulate files" problem...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Rik van Riel wrote:
> 
> It's a relief to know that nobody's taking my humorous
> suggestion seriously, but now we still have the "standard
> Unix tools can't manipulate files" problem...

I disagree. They can manipulate the files a whole lot better than they can 
manipulate xattr's.

For example, you _could_ probably (but hey, maybe "tar" tries to strip
slashes off the end of filenames, so this might not work due to silly
reasons like that) back up a compound file with

	tar cvf file.tar file file/

although unpacking it would require that tar be taught about the thing. 
And you definitely could write a script to do the thing, ie even with an 
unmodified tar you could do

	tar cvf file-archive.tar file
	cd file
	tar cvf ../attribute-archive.tar .

which is a hell of a lot better than what you can do with the fsattr 
interfaces and unmodified legacy applications.

So one of the advantages of "dir-as-file/file-as-dir" is exactly that you
_can_ manipulate the data with legacy tools. Sure, things that traverse a
directory tree might need some (likely fairly trivial) modifications if
they really want to take advantage of the subfiles, but that's still
likely to be _much_ less of an issue than with fsattr's that have a 
totally different model entirely.

		Linus

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> For example, you _could_ probably (but hey, maybe "tar" tries to strip
> slashes off the end of filenames, so this might not work due to silly
> reasons like that) back up a compound file with
> 
> 	tar cvf file.tar file file/

So you'd have both a file and a directory that just happen
to have the same name ?  How would this work in the dcache?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Rik van Riel wrote:
> 
> So you'd have both a file and a directory that just happen
> to have the same name ?  How would this work in the dcache?

There would be only one entry in the dcache. The lookup will select 
whether it opens the file or the directory based on O_DIRECTORY (and 
usage, of course - if it's in the middle of a path, it obviously needs to 
be opened as a directory regardless).

That's not the problem. The problem from a dcache standpoint ends up being 
when the file has a link, and you have two paths to the same sub-file 
through two different ways:

	.. create file 'x' with named stream 'y' ...
	ln x z
	ls -l x/y z/y	/* it's the same attribute!! */

but this is actually exactly the same thing that we already have with 
mounts, ie it is equivalent (from a dentry standpoint) to

	.. create directory 'x' with file 'y' ..
	mkdir z
	mount --bind x z
	ls -l x/y z/y	/* It's the same file!! */

so none of this is really anything "new" from a dcache standpoint.

Except for all the details, of course ;)

		Linus

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:
> On Thu, 26 Aug 2004, Rik van Riel wrote:
> > 
> > So you'd have both a file and a directory that just happen
> > to have the same name ?  How would this work in the dcache?
> 
> There would be only one entry in the dcache. The lookup will select 
> whether it opens the file or the directory based on O_DIRECTORY (and 
> usage, of course - if it's in the middle of a path, it obviously needs to 
> be opened as a directory regardless).

Hmmm, I just straced  "cp /bin/bash /tmp".
One line stood out as a potential problem:

open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4

What do we do with O_CREAT ?

Do we always allow both a directory and a file to be created with
the same name ?

Does this create a new class of "symlink attack" style security
holes ?


-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 673 bytes --]

Am Donnerstag, den 26.08.2004, 14:59 -0400 schrieb Rik van Riel:

> open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4
> 
> What do we do with O_CREAT ?
> 
> Do we always allow both a directory and a file to be created with
> the same name ?

I would say that the directory under a file is implicit. There are still
the pseudo-files a filesystem might want to provide or the VFS if there
should be an agreement to generally provide file/pseudo/uid or possibly
some security hooks. Some filesystem might want to provide those pseudo
files but doesn't have the technical possibility to store non-pseudeo
files from the user under such a directory.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Rik van Riel wrote:
> 
> Hmmm, I just straced  "cp /bin/bash /tmp".
> One line stood out as a potential problem:
> 
> open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4
> 
> What do we do with O_CREAT ?
> 
> Do we always allow both a directory and a file to be created with
> the same name ?

Either I am confused, or you are.

To me, a filesystem that allows this thing doesn't really _have_ the 
concept of "directory vs file". It's just a "filesystem object", and it 
can act as _both_ a directory and a file.

So when you create "/tmp/bash" - assuming /tmp supports the file-as-dir 
semantics at all, you're just creating a new inode - the same you always 
have. When you write to that inode, it writes to the default stream. 
There's no special cases here.

Now, after you have your regular /tmp/bash, you can then start adding 
named streams to it, ie you can do

	open("/tmp/bash/icon", O_WRONLY|O_CREAT|O_LARGEFILE, 0755);

and that will create the "icon" named stream. See? 

So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
contains both a standard data stream (the "file" part) _and_ pointers to
other named streams (the "directory" part).

Hey, think of it as a wave-particle duality. Both "modes" exist at the
same time, and cannot be separated from each other. Which one you see
depends entirely on your "experiment", ie how you open the file.

> Does this create a new class of "symlink attack" style security
> holes ?

I don't believe so. 

		Linus

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
> contains both a standard data stream (the "file" part) _and_ pointers to
> other named streams (the "directory" part).

OK, that makes sense.  

> Hey, think of it as a wave-particle duality. Both "modes" exist at the
> same time, and cannot be separated from each other. Which one you see
> depends entirely on your "experiment", ie how you open the file.

Guess I'm scared again now.  We need to make sure that
backup programs don't fall victim to the uncertainty
principle ;)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 526 bytes --]

Am Donnerstag, den 26.08.2004, 15:44 -0400 schrieb Rik van Riel:

> > Hey, think of it as a wave-particle duality. Both "modes" exist at the
> > same time, and cannot be separated from each other. Which one you see
> > depends entirely on your "experiment", ie how you open the file.
> 
> Guess I'm scared again now.  We need to make sure that
> backup programs don't fall victim to the uncertainty
> principle ;)

At least you can be sure that the backup will go into a defined state as
soon as you observe it.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Linus Torvalds wrote:

> So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
> contains both a standard data stream (the "file" part) _and_ pointers to
> other named streams (the "directory" part).

Thinking about it some more, how would file managers and
file chosers handle this situation ?

Currently the user browses the directory tree and when
the user clicks on something, one of the following 
happens:

1) if it is a directory, the file manager/choser changes
   into that directory

2) if it is a file, the file is opened

Now how do we present things to users ?

How will users know when an object can only be chdired
into, or only be opened ?

For objects that do both, how does the user choose ?

Do we really want to have a file paradigm that's different
from the other OSes out there ?

What happens when users want to transfer data from Linux
to another system ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1317 bytes --]

Am Donnerstag, den 26.08.2004, 16:10 -0400 schrieb Rik van Riel:

> > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
> > contains both a standard data stream (the "file" part) _and_ pointers to
> > other named streams (the "directory" part).
> 
> Thinking about it some more, how would file managers and
> file chosers handle this situation ?

I would say they don't. That's where metadata and pseudo-files sit. It's
up to the applications to do something useful with the data.

> Do we really want to have a file paradigm that's different
> from the other OSes out there ?

Some other OSes have similar things.

> What happens when users want to transfer data from Linux
> to another system ?

No one should store really important data inside file/ that makes the
file completely useless. I just love these postscript fonts on the Mac.
The actual main file stream is 0 bytes in size. The actual font is
inside the resource fork. When I want to convert it the Mac user needs
to pack it into a compound file using a special tool and then I can
decode it using another tool to access the PFB file. Crap.

It's useful for metadata like keywords, ACLs and these things. If you
can pack it using an aware backup program, fine. If you don't, you still
have the data.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christophe Saout wrote:
> I just love these postscript fonts on the Mac.  The actual main file
> stream is 0 bytes in size. The actual font is inside the resource
> fork. When I want to convert it the Mac user needs to pack it into a
> compound file using a special tool and then I can decode it using
> another tool to access the PFB file. Crap.

I thought it was a cunning way to stop me from copying those nice
fonts onto my Linux box.  The original DRM.  Thanks for the tip :)

-- Jamie

Re: silent semantic changes with reiser4

[Stephen Wille Padnos]

Rik van Riel wrote:

>On Thu, 26 Aug 2004, Linus Torvalds wrote:
>  
>
>>So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
>>contains both a standard data stream (the "file" part) _and_ pointers to
>>other named streams (the "directory" part).
>>    
>>
>Thinking about it some more, how would file managers and
>file chosers handle this situation ?
>
>Currently the user browses the directory tree and when
>the user clicks on something, one of the following 
>happens:
>
>1) if it is a directory, the file manager/choser changes
>   into that directory
>  
>
How does the file manager / chooser decide whether you're trying to move 
into a directory, or the meta-data-directory for a directory?
It's not just files that should have metadata - directories need* them 
too.  Making it possible to see attributes as a directory under a file 
is great, but you'd still need an O_META flag for accessing directory 
metadata (since there are already files under a directory).

>2) if it is a file, the file is opened
>
>Now how do we present things to users ?
>
>How will users know when an object can only be chdired
>into, or only be opened ?
>
>For objects that do both, how does the user choose ?
>
>Do we really want to have a file paradigm that's different
>from the other OSes out there ?
>  
>
MacOS does, Be did (sort of).  I'm not sure it would be the end of the 
world, as long as the data can be extracted.

>What happens when users want to transfer data from Linux
>to another system ?
>  
>
That would depend on the other system.  Data is easy, metadata is hard.
It would be possible to create an XML schema for metadata, and if 
requested (O_EVERYTHING?), the file data is returned with all metadata 
in XML tags.  (not advocating this, just an idea :)
- Steve

* I say need in the same way as one *needs* to upgrade their 2GHz 
computer - it would be nice :)

Re: silent semantic changes with reiser4

[Claus Färber]

Stephen Wille Padnos <spadnos@sover.net> schrieb/wrote:
> How does the file manager / chooser decide whether you're trying to
> move into a directory, or the meta-data-directory for a directory?
> It's not just files that should have metadata - directories need* them
> too.  Making it possible to see attributes as a directory under a file
> is great, but you'd still need an O_META flag for accessing directory
> metadata (since there are already files under a directory).

A simple convention that meta data files start with, say ".$", would be  
enough.

Claus
-- 
http://www.faerber.muc.de

Re: silent semantic changes with reiser4

[Alan Cox]

On Maw, 2004-08-31 at 12:01, Claus Färber wrote:
> A simple convention that meta data files start with, say ".$", would be  
> enough.

POSIX/SuS don't permit this. The only "free" namespace is that starting
"//" (and not as some desktops seem to think foo://). Remember always
send GUI desktop users files called http://read.me  .. its fun 8)

The // doesn't really help because you don't want extensions at the path
top. 

Another interesting question btw with substreams of a file is what the
semantics of fchown, fstat, fchmod, fchdir etc are, or of mounting a
file over a substream.

Alan

Re: silent semantic changes with reiser4

[Claus Färber]

Alan Cox <alan@lxorguk.ukuu.org.uk> schrieb/wrote:
> On Maw, 2004-08-31 at 12:01, Claus Färber wrote:
>> A simple convention that meta data files start with, say ".$", would
>> be enough.

> POSIX/SuS don't permit this. The only "free" namespace is that starting
> "//" (and not as some desktops seem to think foo://). Remember always
> send GUI desktop users files called http://read.me  .. its fun 8)

Well, the problem is that one does want the metadata to be in the
ordinary namespace so that they are accessible with POSIX tools.
(The current version of) POSIX does not have a "unified" namespace so a
unified namespace can't be POSIX.

I wonder if it makes sense to have an environment variable (maybe
POSIXLY_CORRECT) to switch between POSIX and the unified namespace
model. Of course, this would mean that all of this is handled in
userspace (e.g. libc), whereas the kernel and filesystems export more
"traditional" (e.g. with openat) interfaces. Accessing metadata as
ordinary files would then work with other filesystems, too.

Claus
-- 
http://www.faerber.muc.de

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Rik van Riel wrote:
> 
> Thinking about it some more, how would file managers and file chosers
> handle this situation ?

The same way they already handle it on other platforms that support it? By 
taking advantage of it..

People literally use this for icon and preview information, so some of the
stuff shows up very much in file managers. And I'd assume that a normal
double-click would just open the main file, and you can right-click for 
management information, including opening the file.

If you want an entity that acts as a directory, you create a directory. 

Directories don't go away - you still have S_ISDIR() and S_ISREG(), and 
they still return information. A file that has associated information is 
still a _file_, and people should treat it that way, it's just that it 
also has a list of named sub-streams. You can open it as a directory, but 
the stat information clearly says it is a file, and the "directory" view 
is very much associated with _that_ file.

I definitely don't think you want the file manager to act as if a named 
stream is exactly the same as a stand-alone file. They have all the same 
operations, but there's no question that there are differences too. 
Especially on a conceptual level - but for most filesystems there are 
likely real technical differences too.

For example, it's likely that most filesystems would _not_ allow linking 
of a named stream anywhere else. And you might not be able to change the 
permissions or date on the named stream either, since it may or may not 
have a separate date/permission thing from the container.

So don't believe that just because the named streams are _named_ like real 
files, that they suddenly have any existence beyond the container that 
they are part of.

There may be other limitations too - again depending on how the filesystem
actually implements named streams. It might not support more than one
level of naming, for example - so you might not be able to create a
directory structure within the named streams, for example.

In short: the fact that the VFS layer exposes the _capability_ to see the 
named streams as a full POSIX filesystem of its own does _not_ mean that 
the low-level filesystem necessarily allows the full possible semantics. 
So you shouldn't design your apps that know about named streams to think 
they are normal directories.

The directory thing is just a very powerful naming scheme, and one that 
fits into the existing UNIX model.

> Do we really want to have a file paradigm that's different
> from the other OSes out there ?

Different from what other OSes?

Last I looked, Windows, Solaris, and OSX all supported named streams. What 
other OS's are out there that you care about? 

In other words, this is _not_ a different paradigm from what others do.  
The discussion is whether we want to implement it at all, and more
importantly about syntactic issues (ie we clearly already implement
extended attributes, just with a much more limited syntactic power).

We don't have to go all the way. Solaris has "openat()", which is kind of 
a half-way there - not really directly available in the same namespace, 
but at least the result is available as a real file interface (as opposed 
to the Linux "xattr" interfaces that are _totally_ special-cased system 
calls).

In other words - the paradigm is already there. It's just that currently
it's pretty much unusable under Linux, because it requires so much
specialized knowledge that it's not worth it to modify existing apps. And
the interfaces are really very limited, so even if you _do_ end up using 
the specialized Linux interfaces, you can't actually do a lot of what you 
might want to do.

			Linus

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>
>
>For example, it's likely that most filesystems would _not_ allow linking 
>of a named stream anywhere else.
>
Why?  It is just a file that inherits its stat data and is referenced by 
a concatenation method of its parent directory.  The link will also 
inherit its stat data from the original parent directory.

> And you might not be able to change the 
>permissions or date on the named stream either, since it may or may not 
>have a separate date/permission thing from the container.
>  
>
You should be able to change the permission and data, but when you 
change it, you change it for the whole container.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Hans Reiser wrote:
>
> Linus Torvalds wrote:
> >
> >For example, it's likely that most filesystems would _not_ allow linking 
> >of a named stream anywhere else.
>
> Why?  It is just a file that inherits its stat data and is referenced by 
> a concatenation method of its parent directory.  The link will also 
> inherit its stat data from the original parent directory.

In reiser4, maybe. Not in general. Look at what other filesystems do for 
attributes..

> > And you might not be able to change the 
> >permissions or date on the named stream either, since it may or may not 
> >have a separate date/permission thing from the container.
>
> You should be able to change the permission and data, but when you 
> change it, you change it for the whole container.

Why? I'd much rather disallow it, than allow it and then have "nonlocal" 
effects.

I can also see a filesystem validly having _extra_ permissions (ie apart
from the container) on it too.

Not all the world is reiser4. 

		Linus

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>
>  
>
>>>And you might not be able to change the 
>>>permissions or date on the named stream either, since it may or may not 
>>>have a separate date/permission thing from the container.
>>>      
>>>
>>You should be able to change the permission and data,
>>
^data^date

>> but when you 
>>change it, you change it for the whole container.
>>    
>>
>
>Why? I'd much rather disallow it, than allow it and then have "nonlocal" 
>effects.
>  
>
Sometimes you want the nonlocal effects and sometimes you don't, and by 
decomposing streams into smaller primitives we can let users choose as 
they want.

Re: silent semantic changes with reiser4

[Hans Reiser]

viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Thu, Aug 26, 2004 at 04:47:39PM -0700, Hans Reiser wrote:
>
>  
>
>>Sometimes you want the nonlocal effects and sometimes you don't, and by 
>>decomposing streams into smaller primitives we can let users choose as 
>>they want.
>>    
>>
>
>Right.  Now, would you kindly post the detailed technical analysis of your
>wonderful design that handles that stuff safely?  With proof of correctness,
>please.  Since we are expected to take your code and use it as "Uber"
>replacement for the existing one, it surely would not be too much to expect,
>would it?
>
>And no, "we have a nice presentation somewhere on namesys" does not
>qualify.
>
>I apologize for the obvious posting in a thread already full of noise,
>but I would like to make *very* sure that lack of ripping you another
>one does *not* mutate into "no objections from viro" in a lovely thread
>on SlashDot in a week or so.
>
>
>  
>
Over the next 6-18 months, one piece at a time, it will all fall into 
place. 

Reiser4 as it is shipping today, consists of close to the minimal 
functionality necessary to replace reiser3, albeit done according to our 
plugin model, and with vastly higher performance.  The full solution is 
not yet written, wait for it.  I understand why you think deadlocks are 
everything, but avoiding them is only a small, albeit necessary, piece 
of the task.

Thanks though for pointing out issues with unlinking files that have 
been cd'd to.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> For objects that do both, how does the user choose ?
> 
> Do we really want to have a file paradigm that's different
> from the other OSes out there ?

What does MacOS X do?  Someone said that documents are directories in
it; it must know how to handle that.

What does Windows do when you click on a .zip file and WinZIP is
installed?  It opens the .zip file and lets you explore inside.  When
you click on a .doc file, though, it opens a viewer or editor -- you
don't get the option to look inside.

> What happens when users want to transfer data from Linux
> to another system ?

This is why I favour storing all essential metadata (about the file's
content) inside the file's contents, the primary stream.

We have another problem: what happens when users want to transfer data
from Windows (with secondary streams) and MacOS (with resource forks)?

-- Jamie

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Thu, Aug 26, 2004 at 09:48:41PM +0100, Jamie Lokier wrote:
> 
> This is why I favour storing all essential metadata (about the file's
> content) inside the file's contents, the primary stream.
> 
> We have another problem: what happens when users want to transfer data
> from Windows (with secondary streams) and MacOS (with resource forks)?

I can tell you that - right now we (Samba) has to drop all but the primary stream,
because the Linux filesystems don't have data stream support. Oh look,
we're losing user data (that's a *bad* thing :-).

Jeremy.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 01:52:18PM -0700, Jeremy Allison wrote:
> On Thu, Aug 26, 2004 at 09:48:41PM +0100, Jamie Lokier wrote:
> > 
> > This is why I favour storing all essential metadata (about the file's
> > content) inside the file's contents, the primary stream.
> > 
> > We have another problem: what happens when users want to transfer data
> > from Windows (with secondary streams) and MacOS (with resource forks)?
> 
> I can tell you that - right now we (Samba) has to drop all but the primary stream,
> because the Linux filesystems don't have data stream support. Oh look,
> we're losing user data (that's a *bad* thing :-).

Maybe you should learn from netatalk.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Fri, Aug 27, 2004 at 10:19:56AM +0100, Christoph Hellwig wrote:
> 
> Maybe you should learn from netatalk.

I am well aware of what netatalk does. However netatalk
isn't as widely used as Samba. Things they can get away
with would cause our user community to flay us alive.

We need a proper solution, not a nasty hack.

That's like me telling you to "learn from *BSD". You have
different user constituencies, you have to serve yours,
I have to serve mine.

Jeremy.

Re: silent semantic changes with reiser4

[Brad Boyer]

On Fri, Aug 27, 2004 at 08:56:29PM -0700, Jeremy Allison wrote:
> I am well aware of what netatalk does. However netatalk
> isn't as widely used as Samba. Things they can get away
> with would cause our user community to flay us alive.
> 
> We need a proper solution, not a nasty hack.
> 
> That's like me telling you to "learn from *BSD". You have
> different user constituencies, you have to serve yours,
> I have to serve mine.

While I do think that there are some hacks in netatalk, the
hacks there are not nearly as bad as the hacks that made it
into the kernel to support exporting a native Mac filesystem.
Take a look at the fork code in the hfs filesystem from 2.4
or before for exporting to netatalk.  It's disgusting, and
mostly because there is no easy way to export the resource
fork and extra metadata that is essential to the client. The
2.6 code doesn't have that anymore, but I don't think it
actually works with netatalk either. Perhaps someone can
correct me on that, but I don't see how it could work.

I would hope that something comes out of this whole discussion
that will allow a much better interface between the hfs/hfsplus
code in the kernel and netatalk.

	Brad Boyer
	flar@allandria.com

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 22:10, Rik van Riel wrote:

> Do we really want to have a file paradigm that's different
> from the other OSes out there ?
>
> What happens when users want to transfer data from Linux
> to another system ?

I think it depends on the transport being used: some transports could allow 
for metadata (i.e. a MIME-compatible transport) while some others don't. Of 
course, the remote side needs to also support metadata (i,e. using a 
MIME-compatible transport against a remote host that doesn't understand 
metadata is certainly impossible).

For example, with FTP transports, I guess the only possible option is to 
transfer the unnamed/default stream. NFS should allow metadata, but this 
probably needs some kind of extensions to the NFS protocol.

Re: silent semantic changes with reiser4

[Horst von Brand]

Rik van Riel <riel@redhat.com> said:
> On Thu, 26 Aug 2004, Linus Torvalds wrote:
> > So "/tmp/bash" is _not_ two different things. It is _one_ entity, that
> > contains both a standard data stream (the "file" part) _and_ pointers to
> > other named streams (the "directory" part).

> Thinking about it some more, how would file managers and
> file chosers handle this situation ?
> 
> Currently the user browses the directory tree and when
> the user clicks on something, one of the following 
> happens:
> 
> 1) if it is a directory, the file manager/choser changes
>    into that directory
> 
> 2) if it is a file, the file is opened

And now you have a mess. Is it "really" a file, or a directory? Why not
just keep both well apart, and stay happy? I just fail to see what could be
gained by having directories that really aren't, and files that aren't
either. Use a directory if you want one, use a file elsewhere.

> Now how do we present things to users ?
> 
> How will users know when an object can only be chdired
> into, or only be opened ?

Easy: It is a directory, or it isn't.

> For objects that do both, how does the user choose ?

Don't give silly choices.

> Do we really want to have a file paradigm that's different
> from the other OSes out there ?

I vote "no". There are/have been OSes with weird "files", none of them
survived to get anywhere as popular as Unix and "file == stream of
bytes". Even with much simpler variants like files as sequences of
records. For a good reason: The Unix way is simple, and extremely flexible,
as my proggie can define at its own whim how to handle what's inside. If a
single stream isn't enough, we have directories. No need to innovate there.

> What happens when users want to transfer data from Linux
> to another system ?

Or between Linux systems with different kernels that happen to implement
different views/metadata on files.

Please do remember devfs: It sounded like a cool idea, got into the kernel
just to be thrown out later because nobody used it. Much heat was
generated, nothing of permanent value. This looks the same: A very vocal
tiny minority is clamoring for something completely non-Unix for totally
bogus reasons. What happened to "code talks, bullshit walks"? There is _no_
code (== real-world, user programs that can't be done efficiently enough
without this), so this nonsense should just be thrown out, and everybody go
back to real hacking.
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Jamie Lokier]

Horst von Brand wrote:
> Please do remember devfs: It sounded like a cool idea, got into the kernel
> just to be thrown out later because nobody used it.

Are you kidding?  Lots of distros and many embedded Linuxes use devfs.

And udev _still_ doesn't create device nodes properly.  (Hint: I have
to run two modprobe commands before pppd works.  I have to run
modprobe before openvpn or bochs work.)

> What happened to "code talks, bullshit walks"?

devfs is a fine example of why code isn't enough.  With devfs the code
came first, the >1 year of strategic bullshit politics from the "it's
not traditional unix" crowd came later, then it went in, then lots of
people used it, then it was replaced by something which still doesn't
work as well as 2.4 does with or without devfs, and people are still
using it despite it's faults.

-- Jamie

Re: silent semantic changes with reiser4

[Olaf Hering]

 On Wed, Sep 01, Jamie Lokier wrote:

> And udev _still_ doesn't create device nodes properly.  (Hint: I have
> to run two modprobe commands before pppd works.  I have to run
> modprobe before openvpn or bochs work.)

Just because it is supposed to work that way.

-- 
USB is for mice, FireWire is for men!

sUse lINUX ag, nÜRNBERG

Re: silent semantic changes with reiser4

[Greg KH]

On Wed, Sep 01, 2004 at 11:45:13PM +0100, Jamie Lokier wrote:
> Horst von Brand wrote:
> > What happened to "code talks, bullshit walks"?
> 
> devfs is a fine example of why code isn't enough.  With devfs the code
> came first, the >1 year of strategic bullshit politics from the "it's
> not traditional unix" crowd came later, then it went in, then lots of
> people used it, then it was replaced by something which still doesn't
> work as well as 2.4 does with or without devfs, and people are still
> using it despite it's faults.

What is udev's faults that have an issue with?

Yes, we don't do module autoloading when opening a device node, but
that's well known, documented, and the way the kernel has evolved to
anyway.

thanks,

greg k-h

Re: silent semantic changes with reiser4

[Grzegorz Jaśkiewicz]

On Fri, 3 Sep 2004 10:22:57 +0200, Greg KH <greg@kroah.com> wrote:
> On Wed, Sep 01, 2004 at 11:45:13PM +0100, Jamie Lokier wrote:
> > Horst von Brand wrote:
> > > What happened to "code talks, bullshit walks"?
> >
> > devfs is a fine example of why code isn't enough.  With devfs the code
> > came first, the >1 year of strategic bullshit politics from the "it's
> > not traditional unix" crowd came later, then it went in, then lots of
> > people used it, then it was replaced by something which still doesn't
> > work as well as 2.4 does with or without devfs, and people are still
> > using it despite it's faults.
> 
> What is udev's faults that have an issue with?
> 
> Yes, we don't do module autoloading when opening a device node, but
> that's well known, documented, and the way the kernel has evolved to
> anyway.

devfs was very natural, and simple solution. But to have it right, it
would have to be the only /dev filesystem.
But no, we like choices, so we have chaos. 
Udev is just another thing adding to that chaos.

Someone was numbering things that are good in BSD design, in that
thread. One of those things was going for devfs. No cheap solutions.
One fs for /dev. And it works great.

Sorry for bit of trolling.

-- 
GJ

Re: silent semantic changes with reiser4

[Helge Hafting]

Grzegorz Jaśkiewicz wrote:

>
>devfs was very natural, and simple solution. But to have it right, it
>would have to be the only /dev filesystem.
>But no, we like choices, so we have chaos. 
>Udev is just another thing adding to that chaos.
>
>Someone was numbering things that are good in BSD design, in that
>thread. One of those things was going for devfs. No cheap solutions.
>One fs for /dev. And it works great.
>
>Sorry for bit of trolling.
>  
>
Devfs was a ver good idea.  The implementation of it
was a problem, and after some time nobody maintained it.
No surprise it had to go.  Now udev+tmpfs can do the same
job, and more.

Helge Hafting

Re: silent semantic changes with reiser4

[Grzegorz Jaśkiewicz]

On Fri, 03 Sep 2004 10:54:26 +0200, Helge Hafting <helge.hafting@hist.no> wrote:
> Grzegorz Jaśkiewicz wrote:
> 
> >
> >devfs was very natural, and simple solution. But to have it right, it
> >would have to be the only /dev filesystem.
> >But no, we like choices, so we have chaos.
> >Udev is just another thing adding to that chaos.
> >
> >Someone was numbering things that are good in BSD design, in that
> >thread. One of those things was going for devfs. No cheap solutions.
> >One fs for /dev. And it works great.
> >
> >Sorry for bit of trolling.
> >
> >
> Devfs was a ver good idea.  The implementation of it
> was a problem, and after some time nobody maintained it.
> No surprise it had to go.  Now udev+tmpfs can do the same
> job, and more.

udef is a one big mistake, having need for userspace tool to use FS is
at least silly.
I can understeand need for some things in kernel to have userspace
daemon. But FS is out of question the least one.

I am supprised noone wanted to maintain devfs. Maybe because people
didn't want to go to devfs only. But still to have classic /dev. It's
also silly, because person writing driver needs to choose between, or
implement all. That's more than bad. Once I have loads of time, and no
work in KDE, I can take over devfs happily :-)

-- 
GJ

Re: silent semantic changes with reiser4

[Miquel van Smoorenburg]

In article <2f4958ff04090302141bc222e5@mail.gmail.com>,
Grzegorz JaÅ kiewicz  <gryzman@gmail.com> wrote:
>I am supprised noone wanted to maintain devfs. Maybe because people
>didn't want to go to devfs only.

It's because disk was spelled with a 'c' (disc).

Mike.
-- 
"In times of universal deceit, telling the truth becomes
 a revolutionary act." -- George Orwell.

Re: silent semantic changes with reiser4

[Grzegorz Jaśkiewicz]

Miquel van Smoorenburg wrote:

>In article <2f4958ff04090302141bc222e5@mail.gmail.com>,
>Grzegorz JaÅ kiewicz  <gryzman@gmail.com> wrote:
>  
>
>>I am supprised noone wanted to maintain devfs. Maybe because people
>>didn't want to go to devfs only.
>>    
>>
>
>It's because disk was spelled with a 'c' (disc).
>  
>

That suppose to be a joke ?

--
GJ

Re: silent semantic changes with reiser4

[Alan Cox]

On Gwe, 2004-09-03 at 16:18, Grzegorz Jaśkiewicz wrote:
> >It's because disk was spelled with a 'c' (disc).
> That suppose to be a joke ?

The international joke symbol (8-)) is only mandatory in en_US locales.

Re: silent semantic changes with reiser4

[Grzegorz Jaśkiewicz]

Alan Cox wrote:

>On Gwe, 2004-09-03 at 16:18, Grzegorz Jaśkiewicz wrote:
>  
>
>>>It's because disk was spelled with a 'c' (disc).
>>>      
>>>
>>That suppose to be a joke ?
>>    
>>
>
>The international joke symbol (8-)) is only mandatory in en_US locales.
>
>  
>
Ok, Planty of times I did asked Brits and Americans if there's any 
difference. Neither did say that either disc or disk is more correct.
And I am using both all the time, noone has any objections ( I live in 
UK for 2.5years so far ). So.... me is confused :-0

--
GJ

Re: silent semantic changes with reiser4

[Paul Jackson]

> Ok, Planty of times I did asked Brits and Americans if there's any 
> difference. Neither did say that either disc or disk is more correct.

>From http://en.wikipedia.org/wiki/Disk_or_disc

    Disk or Disc?

    The divergence in spelling is due in part to the way in which
    the words originated. Disk came into the English language in
    the mid-17th century, and was modelled on words such as whisk;
    disc arose some time later, and was based on the original Latin
    root discus. In the 19th century, disc became the conventional
    spelling for audio recordings made on a flat plate, such as
    the gramophone record; this usage gave rise to the modern term
    disc jockey. Early BBC technicians differentiated between disks
    (in-house transcription records) and discs (the colloquial term
    for commercial gramophone records, or what the BBC dubbed CGRs).

    By the 20th century, the c-spelling was more popular in
    British English, while the k-spelling was preferred in American
    English. In the 1940s, when the American company IBM pioneered
    the first hard disk storage devices, the k-spelling was used. In
    1979 the European company Philips, along with Sony, developed the
    compact disc medium; here, the c-spelling was chosen, possibly
    because of the predominating British spelling, or because the
    compact disc was seen as a successor to the analogue disc record.

    Whatever their heritage, in computer jargon today it is common
    for the k-spelling to refer mainly to magnetic storage devices,
    while the c-spelling is customary for optical media such as the
    compact disc and similar technologies. Even in the computing
    field, however, the terms are used inconsistently; software
    documentation often uses the k-spelling exclusively.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Greg KH]

On Fri, Sep 03, 2004 at 11:14:51AM +0200, Grzegorz Ja??kiewicz wrote:
> On Fri, 03 Sep 2004 10:54:26 +0200, Helge Hafting <helge.hafting@hist.no> wrote:
> > Grzegorz Ja??kiewicz wrote:
> > 
> > >
> > >devfs was very natural, and simple solution. But to have it right, it
> > >would have to be the only /dev filesystem.
> > >But no, we like choices, so we have chaos.
> > >Udev is just another thing adding to that chaos.
> > >
> > >Someone was numbering things that are good in BSD design, in that
> > >thread. One of those things was going for devfs. No cheap solutions.
> > >One fs for /dev. And it works great.
> > >
> > >Sorry for bit of trolling.
> > >
> > >
> > Devfs was a ver good idea.  The implementation of it
> > was a problem, and after some time nobody maintained it.
> > No surprise it had to go.  Now udev+tmpfs can do the same
> > job, and more.
> 
> udef is a one big mistake, having need for userspace tool to use FS is
> at least silly.

I have never heard of a program called "udef".  Any pointers to it?  :)

If it's such a big mistake, then don't use it.  No one is forcing you
to.

> I can understeand need for some things in kernel to have userspace
> daemon. But FS is out of question the least one.

It's not a daemon (although to make things a bit easier on itself, it
does provide one by default, but some distros don't use it.)  It just
creates device nodes when the kernel finds a new device, with a name
that you pick.  Which is something that an in-kernel devfs can not do.

> I am supprised noone wanted to maintain devfs. Maybe because people
> didn't want to go to devfs only. But still to have classic /dev. It's
> also silly, because person writing driver needs to choose between, or
> implement all. That's more than bad. Once I have loads of time, and no
> work in KDE, I can take over devfs happily :-)

Well, until you do that, baseless criticism and trolling like this will
be ignored.

greg k-h

Re: silent semantic changes with reiser4

[Helge Hafting]

Grzegorz Jaśkiewicz wrote:

>On Fri, 03 Sep 2004 10:54:26 +0200, Helge Hafting <helge.hafting@hist.no> wrote:
>  
>
>>Grzegorz Jaśkiewicz wrote:
>>
>>    
>>
>>>devfs was very natural, and simple solution. But to have it right, it
>>>would have to be the only /dev filesystem.
>>>But no, we like choices, so we have chaos.
>>>Udev is just another thing adding to that chaos.
>>>
>>>Someone was numbering things that are good in BSD design, in that
>>>thread. One of those things was going for devfs. No cheap solutions.
>>>One fs for /dev. And it works great.
>>>
>>>Sorry for bit of trolling.
>>>
>>>
>>>      
>>>
>>Devfs was a ver good idea.  The implementation of it
>>was a problem, and after some time nobody maintained it.
>>No surprise it had to go.  Now udev+tmpfs can do the same
>>job, and more.
>>    
>>
>
>udef is a one big mistake, having need for userspace tool to use FS is
>at least silly.
>  
>
Well, devfs had devfsd - a userspace tool . . .

>I can understeand need for some things in kernel to have userspace
>daemon. But FS is out of question the least one.
>
>I am supprised noone wanted to maintain devfs. 
>
I believe it had soemthing to do with the design - in order
to fix it you had to rewrite it almost from scratch.  People
work on whatever they want to, and devfs wasn't it.

>Maybe because people
>didn't want to go to devfs only. But still to have classic /dev. It's
>also silly, because person writing driver needs to choose between, or
>implement all. That's more than bad. Once I have loads of time, and no
>work in KDE, I can take over devfs happily :-)
>  
>
Go ahead!  Perhaps you get it right.  Then you'll have to
convince users of udev (or plain old /dev) that your way is better.

Helge Hafting

Re: silent semantic changes with reiser4

[Tonnerre]

[-- Attachment #1: Type: text/plain, Size: 657 bytes --]

Salut,

On Wed, Sep 08, 2004 at 11:09:06AM +0200, Helge Hafting wrote:
> >udef is a one big mistake, having need for userspace tool to use FS is
> >at least silly.
> 
> Well, devfs had devfsd - a userspace tool . . .

devfsd was not a requirement, it was only a tool for people who wanted
their old device names for devfs devices. I was once doing a miniature
environment for  a package installation  system which used  pure devfs
(i.e.   no  old  dev names),  and  it  was  doing super  well  without
devfs. sysinstall  was the only  process running, and I  could install
Linux on any writeable device without using devfsd.

			   Tonnerre

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>
>
>To me, a filesystem that allows this thing doesn't really _have_ the 
>concept of "directory vs file". It's just a "filesystem object", and it 
>can act as _both_ a directory and a file.
>
>  
>
I agree.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> What do we do with O_CREAT ?
> 
> Do we always allow both a directory and a file to be created with
> the same name ?

I haven't analysed it thoroughly.  I suspect it's fine as long as:

(1) O_CREAT creates something with "file-like
    attributes", meaning stat() says it's a regular file.

(2) File-like means it can be unlinked, linked and renamed, even if
    someone has something inside it open.  Nothing that can be created
    inside it will prevent it from being unlinked (unlike a
    directory-like object, where rmdir() will refuse if it's not empty).

    File-like also means that programs like "cp from to_my_new_file"
    won't do anything so silly as to write _inside_, the way that
    "cp from to_my_new_file/from" would.

    Basic utilities will need to be checked to make sure they don't
    try appending "/" as their method of deciding if the target object
    is a directory and should be entered.

> Does this create a new class of "symlink attack" style security
> holes ?

Yes, but they don't need O_CREAT.

An adversary creates a symlink to metadata inside your file.  You
write to it: it has interesting effects that weren't anticipated, such
as either modifying another (virtual) file, or altering permissions or
other parameters which writing doesn't normally do.

This is very difficult to prevent.

We are creating a way for scripts and classic unix tools to have easy
access to fancy attributes which may affect the security: things like
being able to change a file's permissions just by writing to the
appropriate path, or reading characteristics of files that shouldn't
be visible.  In Hans Reiser's example of expanded /etc/passwd, atimes
and mtimes of individual passwd entries is security information that
perhaps shouldn't be exposed.

In a way, these holes are similar to /proc, in that just writing has
side effects which might not be acceptable.

The solution is the same as for /proc (I hope): make sure the read
permissions on all metadata inside a directory branch are restricted
to the permissions of the file branch, and write permissions even more
restricted at least by default.

For the paranoid, one way is to make new files "not readable or
executable as directories".  It's possible for the file and directory
branches to have different permission bits, though I could see that
getting a bit annoying.  Other possibilities abound.

-- Jamie

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2037 bytes --]

Am Donnerstag, den 26.08.2004, 20:29 +0100 schrieb Jamie Lokier:

> (1) O_CREAT creates something with "file-like
>     attributes", meaning stat() says it's a regular file.
> 
> (2) File-like means it can be unlinked, linked and renamed, even if
>     someone has something inside it open.  Nothing that can be created
>     inside it will prevent it from being unlinked (unlike a
>     directory-like object, where rmdir() will refuse if it's not empty).

I would say hat the file and all of its directory gets invalidated and
deleted as soon as the last opener of the file's main stream or some of
its directory content disappears. How would that be?

> > Does this create a new class of "symlink attack" style security
> > holes ?
> 
> Yes, but they don't need O_CREAT.
> 
> An adversary creates a symlink to metadata inside your file.  You
> write to it: it has interesting effects that weren't anticipated, such
> as either modifying another (virtual) file, or altering permissions or
> other parameters which writing doesn't normally do.

Hmm.

> This is very difficult to prevent.

Right. But to change some properties you usually need to be the owner of
that file anyway, or root.

> In Hans Reiser's example of expanded /etc/passwd, atimes
> and mtimes of individual passwd entries is security information that
> perhaps shouldn't be exposed.

Only assuming mtime/ctime/atime/rwx/uid/gid of file contents can
actually be independant of those from the main stream. I would say that
the rights (read and write) for non-pseudo sub-files should the same as
the file, uid and gid 644 (only owner can change these), something like
that. mtime, atime and ctime always show the same as the file itself.
Hmm.

> The solution is the same as for /proc (I hope): make sure the read
> permissions on all metadata inside a directory branch are restricted
> to the permissions of the file branch, and write permissions even more
> restricted at least by default.

That's what I was thinking.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Herbert Poetzl]

On Thu, Aug 26, 2004 at 11:55:07AM -0700, Linus Torvalds wrote:
> 
> 
> On Thu, 26 Aug 2004, Rik van Riel wrote:
> > 
> > So you'd have both a file and a directory that just happen
> > to have the same name ?  How would this work in the dcache?
> 
> There would be only one entry in the dcache. The lookup will select 
> whether it opens the file or the directory based on O_DIRECTORY (and 
> usage, of course - if it's in the middle of a path, it obviously needs to 
> be opened as a directory regardless).

hmm, that might be interesting for applications
like apache which allow to append arguments by
'building' a 'virtual' path ...

http://www.kernel.org/list.cgi/some/more/args

where the web server actually decides where the
script ends and the arguments start by checking
the filesystem ...

> That's not the problem. The problem from a dcache standpoint ends up being 
> when the file has a link, and you have two paths to the same sub-file 
> through two different ways:
> 
> 	.. create file 'x' with named stream 'y' ...
> 	ln x z
> 	ls -l x/y z/y	/* it's the same attribute!! */
> 
> but this is actually exactly the same thing that we already have with 
> mounts, ie it is equivalent (from a dentry standpoint) to
> 
> 	.. create directory 'x' with file 'y' ..
> 	mkdir z
> 	mount --bind x z
> 	ls -l x/y z/y	/* It's the same file!! */
> 
> so none of this is really anything "new" from a dcache standpoint.
> 
> Except for all the details, of course ;)

if the file is removed, but some attributes are
'locked' by access, will the 'other' attributes
remain visible or disappear ... leaving a partial
'view' of the contents? or did I get the idea
completely wrong?

TIA,
Herbert

> 		Linus
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: silent semantic changes with reiser4

[Martin J. Bligh]

--On Thursday, August 26, 2004 11:15:19 -0700 Linus Torvalds <torvalds@osdl.org> wrote:
> On Thu, 26 Aug 2004, Rik van Riel wrote:
>> 
>> It's a relief to know that nobody's taking my humorous
>> suggestion seriously, but now we still have the "standard
>> Unix tools can't manipulate files" problem...
> 
> I disagree. They can manipulate the files a whole lot better than they can 
> manipulate xattr's.
> 
> For example, you _could_ probably (but hey, maybe "tar" tries to strip
> slashes off the end of filenames, so this might not work due to silly
> reasons like that) back up a compound file with
> 
> 	tar cvf file.tar file file/
> 
> although unpacking it would require that tar be taught about the thing. 
> And you definitely could write a script to do the thing, ie even with an 
> unmodified tar you could do
> 
> 	tar cvf file-archive.tar file
> 	cd file
> 	tar cvf ../attribute-archive.tar .
> 
> which is a hell of a lot better than what you can do with the fsattr 
> interfaces and unmodified legacy applications.
> 
> So one of the advantages of "dir-as-file/file-as-dir" is exactly that you
> _can_ manipulate the data with legacy tools. Sure, things that traverse a
> directory tree might need some (likely fairly trivial) modifications if
> they really want to take advantage of the subfiles, but that's still
> likely to be _much_ less of an issue than with fsattr's that have a 
> totally different model entirely.

What would "test -d" and "test -f" return on these magic beasties? I can't
think of any combinations that wouldn't confuse the crap out of userspace.

M.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Martin J. Bligh wrote:
> 
> What would "test -d" and "test -f" return on these magic beasties? I can't
> think of any combinations that wouldn't confuse the crap out of userspace.

"It's a feature".

The S_ISDIR/S_ISREG tests show real information: it shows not only user
intent ("you should consider this a file, even if it has attributes"), but
also whether it is a directory or a container.

And there's a real technical difference there: the streams contained
within a file are bound to that file. The files contained within a
directory are _independent_ of that directory. Big difference. HUGE
difference.

So it's not confusing. If it tests as a file, you think of it as a file.  
It may have attributes aka named streams associated with it, and you may
be able to open those attributes by treating the file as a directory, but
that doesn't really change the fact that it's a file.

The _big_ difference is that when you can make the compound object _look_ 
like a directory, that means that you can now manage the attributes with 
standard tools. They are still attributes, though.

		Linus

Re: silent semantic changes with reiser4

[Martin J. Bligh]

> On Thu, 26 Aug 2004, Martin J. Bligh wrote:
>> 
>> What would "test -d" and "test -f" return on these magic beasties? I can't
>> think of any combinations that wouldn't confuse the crap out of userspace.
> 
> "It's a feature".
> 
> The S_ISDIR/S_ISREG tests show real information: it shows not only user
> intent ("you should consider this a file, even if it has attributes"), but
> also whether it is a directory or a container.
> 
> And there's a real technical difference there: the streams contained
> within a file are bound to that file. The files contained within a
> directory are _independent_ of that directory. Big difference. HUGE
> difference.
> 
> So it's not confusing. If it tests as a file, you think of it as a file.  
> It may have attributes aka named streams associated with it, and you may
> be able to open those attributes by treating the file as a directory, but
> that doesn't really change the fact that it's a file.
> 
> The _big_ difference is that when you can make the compound object _look_ 
> like a directory, that means that you can now manage the attributes with 
> standard tools. They are still attributes, though.

I think what you're saying is that they'd both return positive, right?

Unfortunately I think lots of code in userspace assumes files / directories
are mutually exclusive conditions, and acts accordingly ... eg:

if [ -f file ]
	do file stuff
else
	do directory stuff
fi.

Now I'm not saying they're not broken ... but seems like a dangerous thing 
to change in the midst of a stable kernel series.

M.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1459 bytes --]

Am Donnerstag, den 26.08.2004, 14:00 -0700 schrieb Martin J. Bligh:

> > "It's a feature".
> > 
> > The S_ISDIR/S_ISREG tests show real information: it shows not only user
> > intent ("you should consider this a file, even if it has attributes"), but
> > also whether it is a directory or a container.
> > 
> > And there's a real technical difference there: the streams contained
> > within a file are bound to that file. The files contained within a
> > directory are _independent_ of that directory. Big difference. HUGE
> > difference.
> > 
> > So it's not confusing. If it tests as a file, you think of it as a file.  
> > It may have attributes aka named streams associated with it, and you may
> > be able to open those attributes by treating the file as a directory, but
> > that doesn't really change the fact that it's a file.
> > 
> > The _big_ difference is that when you can make the compound object _look_ 
> > like a directory, that means that you can now manage the attributes with 
> > standard tools. They are still attributes, though.
> 
> I think what you're saying is that they'd both return positive, right?

No. A file is still a file and S_ISREG will thus return false.
The application should just try to open it as a directory if it wants to
access the assiociated embedded attribute files. If the open fails,
well, it knows that the filesystem (or Linux version, whatever) doesn't
support this feature.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Martin J. Bligh wrote:
> 
> I think what you're saying is that they'd both return positive, right?

No. I'd say that a file would look like a file, even if it has attributes.

It wouldn't show as a directory at all - unless you start looking at 
attributes. Because it really _is_ a file, and it's "directory aspect" is 
really nothing but a way to make its named streams visible.

So you really should consider it a perfectly regular file, and so only 
S_ISREG() will return true, and S_ISDIR() will return false.

Think of it this way: when you add a named attribute to current files 
using the xattr interfaces, do you start thinking of the file as a 
directory? No. Even though it actually now is a starting point for finding 
more information.

It's just that thanks to it's directory aspect, people and apps that 
_care_ about the attributes suddenly can trivially access them. You can 
access them in shell scripts, you can access them in programs, you can 
access them in perl. With no special knowledge necessary.

Remember: in the file-as-directory model, we're always talking about just 
_one_ object. That one object just happens to have namd streams or 
attributes or whatever you want to call them associated with it, and you 
can _see_ them through the normal directory interfaces. But that doesn't 
really make the object a "directory". It's still a file.

In other words, the "directory" part is just a _view_ into the file. A
view that potentially exposes a lot _more_ of the file, but we're still
talking about the same file.

In contrast, a S_IFDIR-like _directory_ is something else entirely. When 
you view the things in that, you aren't looking at data "inside" the 
directory. You're looking at somethign totally independent.

		Linus

Re: silent semantic changes with reiser4

[Jamie Lokier]

Linus Torvalds wrote:
> In other words, the "directory" part is just a _view_ into the file. A
> view that potentially exposes a lot _more_ of the file, but we're still
> talking about the same file.
> 
> In contrast, a S_IFDIR-like _directory_ is something else entirely. When 
> you view the things in that, you aren't looking at data "inside" the 
> directory. You're looking at somethign totally independent.

One of the constraints is that you can create any name in an
S_IFDIR-like directory, but you probably can't create any name "in" an
S_IFREG-like file, because the names have meaning -- except in
designated subtrees of the view intended for holding arbitrary names.

-- Jamie

Re: silent semantic changes with reiser4

[David Greaves]

Linus Torvalds wrote:

>On Thu, 26 Aug 2004, Martin J. Bligh wrote:
>  
>
>>I think what you're saying is that they'd both return positive, right?
>>    
>>
>
>No. I'd say that a file would look like a file, even if it has attributes.
>
>It wouldn't show as a directory at all - unless you start looking at 
>attributes. Because it really _is_ a file, and it's "directory aspect" is 
>really nothing but a way to make its named streams visible.
>
>So you really should consider it a perfectly regular file, and so only 
>S_ISREG() will return true, and S_ISDIR() will return false.
>
>  
>
ie, logically:

S_ISREG() no longer precludes O_DIRECTORY from succeeding
conversley
O_DIRECTORY succeeding no longer implies S_ISDIR()

However, apps making those assumptions will not notice new behaviour.

?

David

Re: silent semantic changes with reiser4

[Miquel van Smoorenburg]

In article <57730000.1093554054@flay>,
Martin J. Bligh <mbligh@aracnet.com> wrote:
>> On Thu, 26 Aug 2004, Martin J. Bligh wrote:
>>> 
>>> What would "test -d" and "test -f" return on these magic beasties? I can't
>>> think of any combinations that wouldn't confuse the crap out of userspace.
>> 
>I think what you're saying is that they'd both return positive, right?

test -f file  -> true
test -f file/ -> false
test -d file  -> false
test -d file/ -> true

Mike.
-- 
"In times of universal deceit, telling the truth becomes
 a revolutionary act." -- George Orwell.

Re: silent semantic changes with reiser4

[Chris Wright]

* Linus Torvalds (torvalds@osdl.org) wrote:
> The _big_ difference is that when you can make the compound object _look_ 
> like a directory, that means that you can now manage the attributes with 
> standard tools. They are still attributes, though.

For things that actually use such attributes to determine file access, do the
attributes get attributes?

thanks,
-chris
-- 
Linux Security Modules     http://lsm.immunix.org     http://lsm.bkbits.net

Re: silent semantic changes with reiser4

[Anton Altaparmakov]

On Thu, 2004-08-26 at 21:54, Linus Torvalds wrote:
> The S_ISDIR/S_ISREG tests show real information: it shows not only user
> intent ("you should consider this a file, even if it has attributes"), but
> also whether it is a directory or a container.
> 
> And there's a real technical difference there: the streams contained
> within a file are bound to that file. The files contained within a
> directory are _independent_ of that directory. Big difference. HUGE
> difference.
> 
> So it's not confusing. If it tests as a file, you think of it as a file.  
> It may have attributes aka named streams associated with it, and you may
> be able to open those attributes by treating the file as a directory, but
> that doesn't really change the fact that it's a file.

What about the attributes/named streams of a directory though?  If you
open it as a directory, you would get the files inside the directory. 
So how do you get at the attributes and named streams of the directory
itself using this interface?

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/

Re: silent semantic changes with reiser4

[Hans Reiser]

Why are you guys even considering going to any pain at all to distort 
semantics for the sake of backup?  tar is easy, we'll fix it and send in 
a patch. 

Re: silent semantic changes with reiser4

[Rik van Riel]

On Fri, 27 Aug 2004, Hans Reiser wrote:

> Why are you guys even considering going to any pain at all to distort 
> semantics for the sake of backup?  tar is easy, we'll fix it and send in 
> a patch. 

Because not everybody uses tar.  Quite a few people use a
network backup system, while others use duplicity, RPM uses
cpio internally and big companies tend to use proprietary
network backup suites.

Breaking people's setup is something to worry about.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>On Fri, 27 Aug 2004, Hans Reiser wrote:
>
>  
>
>>Why are you guys even considering going to any pain at all to distort 
>>semantics for the sake of backup?  tar is easy, we'll fix it and send in 
>>a patch. 
>>    
>>
>
>Because not everybody uses tar.  Quite a few people use a
>network backup system, while others use duplicity, RPM uses
>cpio internally and big companies tend to use proprietary
>network backup suites.
>
>Breaking people's setup is something to worry about.
>
>  
>
It is the tail that should not wag the dog.  New semantics are going to 
break backups other than dd.   We need a LOT of new semantics if we 
don't want to be inferior to Apple and MS.

We should implement backup plugin methods though, so that we only have 
to break things once....

Re: silent semantic changes with reiser4

[Lee Revell]

On Fri, 2004-08-27 at 14:27, Hans Reiser wrote:
> Rik van Riel wrote:
> >On Fri, 27 Aug 2004, Hans Reiser wrote:
> >
> >>Why are you guys even considering going to any pain at all to distort 
> >>semantics for the sake of backup?  tar is easy, we'll fix it and send in 
> >>a patch. 
> >
> >Because not everybody uses tar.  Quite a few people use a
> >network backup system, while others use duplicity, RPM uses
> >cpio internally and big companies tend to use proprietary
> >network backup suites.
> >
> >Breaking people's setup is something to worry about.
> >
> It is the tail that should not wag the dog.  New semantics are going to 
> break backups other than dd.   We need a LOT of new semantics if we 
> don't want to be inferior to Apple and MS.
> 
> We should implement backup plugin methods though, so that we only have 
> to break things once....

Every Linux shop I know of uses amanda anyway, which will not be
affected by these changes, as it uses dump/restore.  restore will need
minor changes, I think dump might not have to be changed at all.

If your backup system can't easily handle these changes, that's a pretty
good indication that it's poorly designed.

Lee

Re: silent semantic changes with reiser4

[Rik van Riel]

On Fri, 27 Aug 2004, Hans Reiser wrote:

> It is the tail that should not wag the dog.

That's no reason for dismissing technical questions with
vigorous handwaving.

> New semantics are going to break backups other than dd.  We need a LOT
> of new semantics if we don't want to be inferior to Apple and MS.

Technical objections need to be addressed, otherwise Linux
will end up inferior to Apple and MS.  Whatever choice we
end up making, it should be something that's both workable
and elegant.

It would be really nice if all the technical issues were
worked out on linux-kernel before the files-as-directories
stuff gets merged.  There are quite a few technical questions
in the thread that haven't been answered yet...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>
>It would be really nice if all the technical issues were
>worked out on linux-kernel before the files-as-directories
>stuff gets merged.  There are quite a few technical questions
>in the thread that haven't been answered yet...
>
>  
>
Well, did I answer the technical question with the 
filename/pseudos/backup method answer in your opinion?

Re: silent semantic changes with reiser4

[Christian Mayrhuber]

On Friday 27 August 2004 16:12, Rik van Riel wrote:

> Because not everybody uses tar.  Quite a few people use a
> network backup system, while others use duplicity, RPM uses
> cpio internally and big companies tend to use proprietary
> network backup suites.
Big backup systems support scripts that can be run prior to
backup and post restore. You aready have to use star to
do a metadata backup the acl's/ea into some tgz, which
can be unpacked after restore.

> Breaking people's setup is something to worry about.
Yeah, maybe. But as usual on UN*X  like systems most admins
are able to come up with a quick shell/perl script to solve their
particular problem.

-- 
lg, Chris

Re: silent semantic changes with reiser4

[Rik van Riel]

On Fri, 27 Aug 2004, Hans Reiser wrote:

> Why are you guys even considering going to any pain at all to distort 
> semantics for the sake of backup?  tar is easy, we'll fix it and send in 
> a patch. 

It's not as easy as you make it out, and not just because
there are a few dozen backup programs that need fixing.

The problem is more fundamental than that.  Some of the
file streams proposed need to be backed up, while others
are alternative presentations of the file, which should
not be backed up.

Currently I see no way to distinguish between the stuff
that should be backed up and the stuff that shouldn't.

That problem needs to be resolved before we can even start
thinking about fixing archivers...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Spam]

> On Fri, 27 Aug 2004, Hans Reiser wrote:

>> Why are you guys even considering going to any pain at all to distort
>> semantics for the sake of backup?  tar is easy, we'll fix it and send in
>> a patch. 

> It's not as easy as you make it out, and not just because
> there are a few dozen backup programs that need fixing.

> The problem is more fundamental than that.  Some of the
> file streams proposed need to be backed up, while others
> are alternative presentations of the file, which should
> not be backed up.

  No, not really. This is a user decision and should be options in the
  backup  software.  I don't think it is up to the kernel, filesystem,
  or  the  OS  in  general to decide what information the user want to
  retain or not.

> Currently I see no way to distinguish between the stuff
> that should be backed up and the stuff that shouldn't.

> That problem needs to be resolved before we can even start
> thinking about fixing archivers...

  The  archivers  should,  as  I  said,  allow  the user to choose. It
  shouldn't be automatic. Default, should IMO be to store everything.

  ~S

Re: silent semantic changes with reiser4

[Spam]

  

>>>>>> "Spam" == Spam  <spam@tnonline.net> writes:

>> Rik van Riel wrote:
>>> On Fri, 27 Aug 2004, Hans Reiser wrote:
>>>> Why are you guys even considering going to any pain at all to
>>>> distort semantics for the sake of backup?  tar is easy, we'll fix it
>>>> and send in a patch.

>>> It's not as easy as you make it out, and not just because there are a
>>> few dozen backup programs that need fixing.

>>> The problem is more fundamental than that.  Some of the file streams
>>> proposed need to be backed up, while others are alternative
>>> presentations of the file, which should not be backed up.

Spam>>   No, not really. This is a user decision and should be options in
Spam>> the backup software.  I don't think it is up to the kernel,
Spam>> filesystem, or the OS in general to decide what information the
Spam>> user want to retain or not.

> Why not just define an attribute named something like "do-not-backup"?
> Then whatever program that generates the thumbnail can automatically add
> the do-not-backup bit, and the backup software knows to ignore it.
> (Obviously, that bit should apply recursively down the subtree.)

  This  is  about  the  same  idea  as the archive flag in DOS/Windows
  environments.  It  didn't really work to well IMO. If meta files are
  definable  by  users/application then the backup system could create
  its own meta files with the specific information it needs.

  ~S

  

Re: silent semantic changes with reiser4

[Spam]

  

> Spam wrote:
>> > The problem is more fundamental than that.  Some of the
>> > file streams proposed need to be backed up, while others
>> > are alternative presentations of the file, which should
>> > not be backed up.
>> 
>>   No, not really. This is a user decision and should be options in the
>>   backup  software.  I don't think it is up to the kernel, filesystem,
>>   or  the  OS  in  general to decide what information the user want to
>>   retain or not.

> It is helpful for the OS, or a naming convention, to indicate what
> _is information_ though.

> It makes no sense to backup two or more copies of the _same
> information_, and it makes even less sense to try to restore them as
> it'll either be slow, fail (you can't always write to alternative
> presentations), or cause unwanted side effects.

> Just like when you backup a dynamic web site.  You store the files
> which the server is using.  You don't use "wget" to store the
> generated pages, that's not a useful backup and you can't restore from it.

  I do not agree. Everything can be considered information, even if it
  is derived from an already existing file.

  If  the  user  wants  to  not  backup certain things then that is an
  option to tell the backup program.

>> > Currently I see no way to distinguish between the stuff
>> > that should be backed up and the stuff that shouldn't.
>> 
>> > That problem needs to be resolved before we can even start
>> > thinking about fixing archivers...
>> 
>>   The  archivers  should,  as  I  said,  allow  the user to choose. It
>>   shouldn't be automatic. Default, should IMO be to store everything.

> Don't try to store different views of the same thing.
> When you try to restore, you _won't_ necessarily get back what you stored.

  No, it would be restored as it were before with the meta information
  intact.

> Whereas if you follow the OS's advice, and skip virtual files, then
> backup and restore will recreate the filesystem, which is what you want.

> _That's_ storing everything.  It's what you want from a backup.

  It would be storing everything but the virtual files.


> -- Jamie

Re: silent semantic changes with reiser4

[Rik van Riel]

On Sat, 28 Aug 2004, Spam wrote:

> > Whereas if you follow the OS's advice, and skip virtual files, then
> > backup and restore will recreate the filesystem, which is what you want.
> 
> > _That's_ storing everything.  It's what you want from a backup.
> 
>   It would be storing everything but the virtual files.

Thing is, there is no way to distinguish between what are
virtual files and what are actual streams hidden inside a
file.  You don't know what should and shouldn't be backed
up...

Hans says there must be a way, somehow, but I haven't seen
him tell this mailing list what exactly it is ;)

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Fri, 27 Aug 2004, Rik van Riel wrote:
> 
> Thing is, there is no way to distinguish between what are
> virtual files and what are actual streams hidden inside a
> file.  You don't know what should and shouldn't be backed
> up...

I think that lack of distinguishing poiwer is more serious for 
directories. The more I think I think about it, the more I wonder whether 
Solaris did things right - having a special operation to "cross the 
boundary".

I suspect Solaris did it that way because it's a hell of a lot easier to 
do it like that, but regardless, it would solve the issue of real 
directories having both real children _and_ the "extra streams".

		Linus

Re: silent semantic changes with reiser4

[Hans Reiser]

Just have a special name instead of a special boundary, or, better, have 
a filename/pseudos/backup method that outputs everything needed to 
backup the object "filename".

Hans

Linus Torvalds wrote:

>On Fri, 27 Aug 2004, Rik van Riel wrote:
>  
>
>>Thing is, there is no way to distinguish between what are
>>virtual files and what are actual streams hidden inside a
>>file.  You don't know what should and shouldn't be backed
>>up...
>>    
>>
>
>I think that lack of distinguishing poiwer is more serious for 
>directories. The more I think I think about it, the more I wonder whether 
>Solaris did things right - having a special operation to "cross the 
>boundary".
>
>I suspect Solaris did it that way because it's a hell of a lot easier to 
>do it like that, but regardless, it would solve the issue of real 
>directories having both real children _and_ the "extra streams".
>
>		Linus
>
>
>  
>

Re: silent semantic changes with reiser4

[Spam]

  I  am  thinking  that  is  there  was a proper API for accessing the
  filesystem  then this problem wouldn't arise because things could be
  done  behind  the curtains inside the API, instead of having all the
  tools to be rewritten to know.

  Think of FAT32, for example, the new filenames will be retained even
  if  you use old tools on them. It is only if you mount it as a FAT16
  volume they will be lost.

  Even  if  this  is fixed for Reiser4 now. Next time someone wants to
  make  changes like this we have the exact same problem yet again.
  

> Just have a special name instead of a special boundary, or, better, have
> a filename/pseudos/backup method that outputs everything needed to 
> backup the object "filename".

> Hans

> Linus Torvalds wrote:

>>On Fri, 27 Aug 2004, Rik van Riel wrote:
>>  
>>
>>>Thing is, there is no way to distinguish between what are
>>>virtual files and what are actual streams hidden inside a
>>>file.  You don't know what should and shouldn't be backed
>>>up...
>>>    
>>>
>>
>>I think that lack of distinguishing poiwer is more serious for 
>>directories. The more I think I think about it, the more I wonder whether
>>Solaris did things right - having a special operation to "cross the 
>>boundary".
>>
>>I suspect Solaris did it that way because it's a hell of a lot easier to
>>do it like that, but regardless, it would solve the issue of real 
>>directories having both real children _and_ the "extra streams".
>>
>>		Linus
>>
>>
>>  
>>

Re: silent semantic changes with reiser4

[Helge Hafting]

On Fri, Aug 27, 2004 at 07:05:35PM -0700, Linus Torvalds wrote:
> 
> 
> On Fri, 27 Aug 2004, Rik van Riel wrote:
> > 
> > Thing is, there is no way to distinguish between what are
> > virtual files and what are actual streams hidden inside a
> > file.  You don't know what should and shouldn't be backed
> > up...
> 
> I think that lack of distinguishing poiwer is more serious for 
> directories. The more I think I think about it, the more I wonder whether 
> Solaris did things right - having a special operation to "cross the 
> boundary".
> 
> I suspect Solaris did it that way because it's a hell of a lot easier to 
> do it like that, but regardless, it would solve the issue of real 
> directories having both real children _and_ the "extra streams".

There are many ways of doing this. Several extra streams to a directory
that aren't ordinary files in the directory?

It seems to me that we can get a lot of nice functionality in a simpler way:
Instead of thinking about a number of streams attached to something
that is either an ordinary file or directory, just say that the only
change will be that a directory may have a _single_ file stream in
addition to being a plain directory.

The conceptual change is smaller.  Still you achieve multiple
streams, the "main" stream in such a dir+file is the single stream
attached to the directory.  All the "extra streams" are files inside the 
(normal) directory.

* This way, the extra streams aren't "special" in any way and
  should work fine with existing tools. 
* Serving such a filesystem to clients that use multiple streams
  will allow any number of streams per file for them
* Moving the directory+file thing will move both the directory and
  the special stream, and all the extra streams follow because they're
  �part of the directory as usual�.

If the VFS is to be extended in order to support file-as-directory (or
vice versa) then hopefully it can be done in a simple way.

Helge Hafting
 

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004, Helge Hafting wrote:
>
> > I think that lack of distinguishing power is more serious for 
> > directories. The more I think I think about it, the more I wonder whether 
> > Solaris did things right - having a special operation to "cross the 
> > boundary".
> > 
> > I suspect Solaris did it that way because it's a hell of a lot easier to 
> > do it like that, but regardless, it would solve the issue of real 
> > directories having both real children _and_ the "extra streams".
> 
> There are many ways of doing this. Several extra streams to a directory
> that aren't ordinary files in the directory?

Well.. Yes. We already have "." and "..", which are "special extra
streams" in a sense. However, people expect them, and know to ignore them. 
The same wouldn't be true of new naming.

> It seems to me that we can get a lot of nice functionality in a simpler way:
> Instead of thinking about a number of streams attached to something
> that is either an ordinary file or directory, just say that the only
> change will be that a directory may have a _single_ file stream in
> addition to being a plain directory.

That doesn't really help us. What would the name be, and how could you 
avoid clashes? 

> If the VFS is to be extended in order to support file-as-directory (or
> vice versa) then hopefully it can be done in a simple way.

I'm pretty confident that we can extend the VFS layer to support named
streams (see the technical discussion with Al, rather than the flames in
this thread). I also clearly believe that it is worth it, but I'm starting
to wonder if we should have a special open flag to make people select the
stream.

If you look at the Solaris interface, the _nice_ part about "openat()" is 
that you can do something like

	file = open(filename, O_RDONLY);
	if (file < 0)
		return -ENOENT;
	icon = openat(file, "icon", O_RDONLY | O_XATTR);
	if (icon < 0)
		icon = default_icon_file;
	..

and it will work regardless of whether "filename" is a directory or a 
regular file, if I've understood correctly.

Now, I think that makes sense for several reasons:
 - single case
 - race-free (think "stat()" vs "fstat()" races).
 - I think we want to do "openat()" regardless of whether we ever 
   support extended attributes or not ("openat()" is nice for doing 
   "namei()" in user space even in the absense of any attributes or 
   named streams).

So what we can do is
 - implement openat() regardless, and expect to do the Solaris thing for 
   it if we ever do streams.
 - _also_ support the "implied named attributes" for regular files, so 
   that you don't have to use "openat()" to access them.

Comments? Does anybody hate "openat()" for any reason (regardless of 
attributes)? We can easily support it, we'd just need to pass in the file 
to use as part of the "nameidata" thing or add an argument (it would also 
possibly be cleaner if we made "fs->pwd" be a "struct file").

		Linus

Re: silent semantic changes with reiser4

[Rik van Riel]

On Sat, 28 Aug 2004, Linus Torvalds wrote:
> Well.. Yes. We already have "." and "..", which are "special extra
> streams" in a sense. However, people expect them, and know to ignore them. 
> The same wouldn't be true of new naming.

If the streams that don't contain data weren't real
files, but instead fifos, backup programs would ignore
them fine.

Tar seems to ignore special nodes ...

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[viro]

On Sat, Aug 28, 2004 at 11:09:38AM -0700, Linus Torvalds wrote:
> Comments? Does anybody hate "openat()" for any reason (regardless of 
> attributes)? We can easily support it, we'd just need to pass in the file 
> to use as part of the "nameidata" thing or add an argument (it would also 
> possibly be cleaner if we made "fs->pwd" be a "struct file").

What would your openat() produce?  Normal struct file?  Then what's going
to be its vfsmount/dentry and what will they be attached to?

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> What would your openat() produce?  Normal struct file?  Then what's going
> to be its vfsmount/dentry and what will they be attached to?

Normal file descriptor, exactly like "open()".

And it's going to have all the same vfsmount/dentry thing it would have if 
you looked it up the whole way. I don't understand your question..

Ignore the O_XATTR thing for a while, and assume it's just a convenient
combination of "fchdir + open" (plus "fchdir back" of course, but that's 
beside the point - openat() doesn't ever really change cwd).

Going back to O_XATTR: that would end up doing the "special vfsmount"  
magic at the beginning of the lookup. If the dentry you started with
wasn't marked D_HYBRID, it would just return -ENOTDIR.

So we could do openat() _without_ any of the lookup_mnt() etc special 
cases. This interface is independent of whether we want to expose the 
attributes through a normal lookup - we can do either, both, or neither as 
we choose.

The nice thing about "openat()" is
 - people can definitely find uses for it even without attributes. 
 - it's "portable". Well, at least somebody else does the same thing, 
   which is nice for user-space developers. You don't use a Linux-only 
   interface, you use a Linux/Solaris one, which makes a lot of people a 
   lot more happy.

Remember, portability has always been very important to Linux, and
Linux-only features while nice are certainly not as nice as features you
can also find in other places.

NIH is a disease.

		Linus

Re: silent semantic changes with reiser4

[viro]

On Sat, Aug 28, 2004 at 11:44:52AM -0700, Linus Torvalds wrote:
> Going back to O_XATTR: that would end up doing the "special vfsmount"  
> magic at the beginning of the lookup. If the dentry you started with
> wasn't marked D_HYBRID, it would just return -ENOTDIR.

OK, let me restate the question - what do we get from pwd if we do
fchdir() to such beast?

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
>
> On Sat, Aug 28, 2004 at 11:44:52AM -0700, Linus Torvalds wrote:
> > Going back to O_XATTR: that would end up doing the "special vfsmount"  
> > magic at the beginning of the lookup. If the dentry you started with
> > wasn't marked D_HYBRID, it would just return -ENOTDIR.
> 
> OK, let me restate the question - what do we get from pwd if we do
> fchdir() to such beast?

We'll see the "attribute path". We could (if we want to) mark the point 
where we walked into attribute space somehow (since we can see it by just 
looking at the vfsmount: d_root/d_mountpoint being the same), but even if 
we don't, we'll get a sane-looking path.

(It's not just "getcwd()", it's /proc interfaces too, for opening files,
and we already have the notion of magic markers like "(deleted)" to show
human-readable information).

The question about what to do at the "attribute point" (and there may
actually be several, if the filesystem supports attributes on attribute
files) likely depends on whether we support the previously discussed
"lookup()" magic for attributes.

So if we do support it, I think we should just make the attribute point
look exactly like a normal directory, since that path would work as-is. If
we don't support it (and at directory points), we might want to just
consider it a special root.

NOTE! Anybody who tries to use the "getcwd()" string as a real path is 
already broken - you have pathname permissions that may not make it 
possible to look up the path you see.

So we have multiple options:

 - support "file/attribute" lookup: show the path as-is, so you'd show it 
   as
	"/path/to/file/attribute"

 - alternatively, even if you _do_ support the normal lookup, show it with 
   a double slash (which will still be a valid path), just as a visual 
   clue:
	"/path/to/file//attribute"

 - for directories, or if we do _not_ support the extended lookup format,
   we could show it the same way we show deleted files, something like 
   this:
	"/path/to/file/attribute (attr)"

 - using "http notation" for non-standard-namespaces (we already do this
   for sockets and pipes, for example)

	"attr:attribute@/path/to/file"

pick your poison.

		Linus

Re: silent semantic changes with reiser4

[Andries Brouwer]

On Sat, Aug 28, 2004 at 12:16:47PM -0700, Linus Torvalds wrote:

If I see it correctly, you want to group a file and some
ancillary files together.

The Unix way would be to make a directory and put them all there:
	xterm/xterm
	xterm/xterm.icon

But you are unsatisfied and want
	xterm
	xterm/xterm.icon

As long as we agree that the latter really means the former,
there are no problems in finding out what should happen.

The conclusion is, that a directory carries an additional bit
that says "if I am opened as a regular file then use the file
of the same name inside".

Now there is no attribute space, just a shorthand.

Andries

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004, Andries Brouwer wrote:
> 
> If I see it correctly, you want to group a file and some
> ancillary files together.
> 
> The Unix way would be to make a directory and put them all there:
> 	xterm/xterm
> 	xterm/xterm.icon
> 
> But you are unsatisfied and want
> 	xterm
> 	xterm/xterm.icon
> 
> As long as we agree that the latter really means the former,
> there are no problems in finding out what should happen.
> 
> The conclusion is, that a directory carries an additional bit
> that says "if I am opened as a regular file then use the file
> of the same name inside".
> 
> Now there is no attribute space, just a shorthand.

It's more than a shorthand, though. _Much_ more.

There's the small issue of atomicity and locking.

There's the small issue of hardlinks.

Both of those are why it would need special support.

		Linus

Re: silent semantic changes with reiser4

[Andries Brouwer]

On Sat, Aug 28, 2004 at 12:46:10PM -0700, Linus Torvalds wrote:

> > Now there is no attribute space, just a shorthand.
> 
> It's more than a shorthand, though. _Much_ more.
> ...
> Both of those are why it would need special support.

Yes - support, I do not argue against that.
But I argue against an attribute space.

This xterm.icon was xterm.icons - with both large and small
icons. And these icons have copyrights. Etc. There is just
an arbitrary tree below this xterm filesystem object.
The files below are in no way different from the files above.

Unix says: everything is a file. Now we get: everything is
a directory. And some directories have a file attached.

Andries

Re: silent semantic changes with reiser4

[viro]

On Sat, Aug 28, 2004 at 12:16:47PM -0700, Linus Torvalds wrote:
> So if we do support it, I think we should just make the attribute point
> look exactly like a normal directory, since that path would work as-is. If
> we don't support it (and at directory points), we might want to just
> consider it a special root.
> 
> NOTE! Anybody who tries to use the "getcwd()" string as a real path is 
> already broken - you have pathname permissions that may not make it 
> possible to look up the path you see.

OK, forget getcwd().  What does lookup of .. do from that point?  *Especially*
for stuff you've got from regular files.  That's the decision that needs to
be made.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> OK, forget getcwd().  What does lookup of .. do from that point?  *Especially*
> for stuff you've got from regular files.  That's the decision that needs to
> be made.

I think that will decide on whether we expose attributes through the 
normal namespace or not.

If we do expose them in the normal namespace, then ".." should work the
way the namespace looks: if you do ".." on the "attribute directory" of a
file, you get the directory that the file was in. Ie an old-style 
user-space "getcwd()" would give the right path (well, an old-style 
user-space getcwd() would probably refuse the file on the base that it is 
S_IFREG, but ignoring that..)

If we _don't_ expose it in the normal namespace, we should should either
just error out (logically you'd get the file itself, but I really don't
wan tto have ".." return a non-directory, because _that_ really might
confuse things), or you'd just return the same directory (ie it would be a
"local root" in the namespace you got moved to).

So let's try to be self-consistent with how we expose it in the normal 
namespace.

		Linus

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Sat, Aug 28, 2004 at 10:12:39PM -0700, Linus Torvalds wrote:
> 
> If we do expose them in the normal namespace, then ".." should work the
> way the namespace looks: if you do ".." on the "attribute directory" of a
> file, you get the directory that the file was in. Ie an old-style 
> user-space "getcwd()" would give the right path (well, an old-style 
> user-space getcwd() would probably refuse the file on the base that it is 
> S_IFREG, but ignoring that..)

Errr, why don't we check what Solaris does in these cases and
see if that makes sense. They do seem to have made some progress
along this path and it wouldn't make sense to create new semantics
just for the sake of it.

Jeremy.

Re: silent semantic changes with reiser4

[Paul Jakma]

On Sat, 28 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:

> OK, let me restate the question - what do we get from pwd if we do
> fchdir() to such beast?

On solaris: ENOTDIR from getcwd(), with 'pwd' executed inside a shell 
started with runat.

.. entry exists, but doesnt lead anywhere, cd .. returning ENOTDIR 
too.

regards,
-- 
Paul Jakma	paul@clubi.ie	paul@jakma.org	Key ID: 64A2FF6A
Fortune:
If you didn't get caught, did you really do it?

Re: silent semantic changes with reiser4

[Hans Reiser]

I object to openat().....

My reason is that the things that distinguish between objects should be 
the names, not the choice of system call.  The reason for this is that 
it improves closure and namespace unification to do so, because it 
allows all the objects to be accessed within the same namespace.

Yes, it can be useful to allow a namespace to exclude some objects, but 
that exclusion should not be mandated.  If you want to exclude, you 
should cd or chroot to /proc/nopseudos and find a view of the filesystem 
that excludes metas, or mount with -nopseuodos.

Do you see why I say this?  I can say a lot more about the damage of 
fragmenting namespaces into multiple apis.... Why look at xattrs....;-)

Hans


Linus Torvalds wrote:

>
>
>I'm pretty confident that we can extend the VFS layer to support named
>streams (see the technical discussion with Al, rather than the flames in
>this thread). I also clearly believe that it is worth it, but I'm starting
>to wonder if we should have a special open flag to make people select the
>stream.
>
>If you look at the Solaris interface, the _nice_ part about "openat()" is 
>that you can do something like
>
>	file = open(filename, O_RDONLY);
>	if (file < 0)
>		return -ENOENT;
>	icon = openat(file, "icon", O_RDONLY | O_XATTR);
>	if (icon < 0)
>		icon = default_icon_file;
>	..
>
>and it will work regardless of whether "filename" is a directory or a 
>regular file, if I've understood correctly.
>
>Now, I think that makes sense for several reasons:
> - single case
> - race-free (think "stat()" vs "fstat()" races).
> - I think we want to do "openat()" regardless of whether we ever 
>   support extended attributes or not ("openat()" is nice for doing 
>   "namei()" in user space even in the absense of any attributes or 
>   named streams).
>
>So what we can do is
> - implement openat() regardless, and expect to do the Solaris thing for 
>   it if we ever do streams.
> - _also_ support the "implied named attributes" for regular files, so 
>   that you don't have to use "openat()" to access them.
>
>Comments? Does anybody hate "openat()" for any reason (regardless of 
>attributes)? We can easily support it, we'd just need to pass in the file 
>to use as part of the "nameidata" thing or add an argument (it would also 
>possibly be cleaner if we made "fs->pwd" be a "struct file").
>
>		Linus
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>

The Necessity of File Types (was silent semantic changes with reiser4)

[Alexander G. M. Smith]

Linus Torvalds wrote:
> I'm pretty confident that we can extend the VFS layer to support
> named streams [...]

Hans Reiser wrote on Sat, 28 Aug 2004 15:29:33 -0700:
> I object to openat().....
> 
> My reason is that the things that distinguish between objects should be 
> the names, not the choice of system call.  The reason for this is that 
> it improves closure and namespace unification to do so, because it 
> allows all the objects to be accessed within the same namespace.

I have to agree with Hans there.  Just have one main kind of file system object.  No separate stream system, other than as an optional alternative way of viewing things.  Treat it as a file or a directory or as an attribute as the context implies.  Have just one open call to open it, another to enumerate its children, and another to read and write data to it.  Paths to file system objects, no matter what their purpose, are all just plain paths (you can get fancy later with sub-name-spaces and name space boundaries).

But how do you tell what kind of thing it is?  How do you tell if it is important enough to back up?  How do you tell that a thing is intended to be treated as a directory containing other things rather than as a stand-alone object?  File types are needed for that!

In BeOS we had a MIME type database and all important files were given an attribute of BEOS:TYPE which contained their MIME string.  A system-wide database associated applications that handled each type, and also listed the standard attributes for each MIME type (such as Subject/From/To/... for "text/e-mail" files).  It even included some extra info about which attributes should be shown to the user and which were editable by the user in the desktop directory window GUI.  By the way, the database was just a tree of files (one for each MIME type using the same structure as MIME names) with attributes attached to them containing the relevant meta-meta-data.

Unclassified files (ones with no type, such as those generated by "cp" and other non-attribute aware programs) were treated as application/octet-stream.  There was also a background task that would try to identify them when the computer was idle, or if you tried to open them in a file requestor.  It used the file extension and also looked for characteristic signature bytes to pick the type.

Unfortunately BeOS had fragmented the type system by having a separate system of 4 character codes to identify the type of attributes (since attributes had their own API separate from files).  That would tell you the attribute was an INT32 or  C-String or Date.

I'd like to see one unified type system for the unified file/directory/attribute thing (has anyone named those multipurpose file system objects yet or can I call them Fildirutes?).  That means a global list of types and an attribute attached to all files needing a type.  That standard child file (used as an attribute) would be named something like "..meta.mimetype".  The global list would contain information about whether that type of thing needed to be backed up, what its standard computed attributes would be, and all sorts of common properties that all files of that type need.  Plus I'd add the primitive types, so "primitive/int32" and "primitive/date64" would be possible types for things which are simple attributes.

If you're worried about extra files, those file types could be stored as a unique ID (I recommend 8 character codes, using semi-readable text, for portability reasons) in each file inode.  The type would only appear as a file to users of the file system.

Another need for the type system is the need to index attributes.  Someone has to tell the OS that "MAIL:date-received" is a 64 bit number of microseconds since the start of time, so that the OS knows how to sort the index for that attribute.

Jan Harkes wrote on Sat, 28 Aug 2004 12:11:14 -0400:
> From what I saw in one part of the discussion, it allows for infinite
> depth recursion (file/metas/metas/metas/...). Some applications are
> going to have a lot of fun with that.

Some of the type information can be faked out as appearing as files.  So looking for the MIME type of a MIME type file (thing/..meta.mimetype/..meta.mimetype) would pretend to be a file that contained "text/mime-string", but not actually represent anything on disk.  Instead stock answers for the really primitive data types would be generated by the file system, recursively if needed.

The recursion should be there.  Just like it is for "." and "..", which "ls -R" for some reason doesn't traverse :-)

By the way, I disagree on Hans with using a directory for the meta data, I'd prefer to just have "..meta." as the name prefix of the metadata things.  Otherwise there's an extra directory level in there, making ..meta/ different from other directories (making browsing it a bit awkward since it doesn't have metadata of its own - like window coordinates for displaying the directory).

- Alex

Re: The Necessity of File Types (was silent semantic changes with reiser4)

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander G. M. Smith wrote:
[...]
| By the way, I disagree on Hans with using a directory for the meta
data, I'd prefer to just have "..meta." as the name prefix of the
metadata things.  Otherwise there's an extra directory level in there,
making ..meta/ different from other directories (making browsing it a
bit awkward since it doesn't have metadata of its own - like window
coordinates for displaying the directory).

You'd rather do "cat ..meta.pseudo" than "ls ..metas/"?
And we already have weird directories.  /proc doesn't do things that
/tmp does.  I also doubt proc will ever store that kind of metadata.

I don't like either "metas" or "..metas", but you'll have to look in the
archives for why.  I am NOT starting that again.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTPJW3gHNmZLgCUhAQKuEw/+PjrDfSlZvuTUwiOSa/kU05qjTKkIp05F
nBkrSIUKSUfoYN3gILucIGOjjJ5qBRW3wLIwWyO+raOAzHFTdylvR1rEWP9h/JUG
Psz3X8RE8cubiuNwjyJawI87cPX8HW1xQxgkiiquZN2L5XbvHsFsbwAJV1tJLnrM
2lL6IBUXcxDIh7pdB0/z+4t945ElT3Rl09CLP/MfP4GNvt/UWjhbeHNWdeLq93C+
w0irn6VIsm+1+GKqEhpYLLMj9V4cbBY6m7XMuBzH+gEyu5exAMqGzPLGDiG/I6y/
rXI6IF94i2p4sLMv0LmtEvQsV37LFIKRBrWVScV5oiQ93LPVLG6pYmAi6RLsHN03
QazMmcRwjVvrTAf0KS21KXaE0qYbvgoQpteUEkPZLRkczSmiry2d3e6gBQlCV7xj
9Hzkm9HxR0BjxmRaMVLEWZ86t08OHkl0Ab186DiHUC5RlL2FPIvdyYthXo9wwtI/
XlA03Rvc3kl6DVRO8UM/+fa8v5MDn58ZMI2QgdPrU0V2Q9bGv/woByszXaDA+IVJ
ZiHQ8IgShYlkA279DffOrZnDPnZMOWa7iRYetX5PYPoAyfkrmFKofT8va+dDL7pf
HA5crUR5ws1EZZ8lUfrtXXnsBeEn9PS0lUHvJY7zZ4tJfZ65Fa+IiDpK5p+1tJZi
ylCBNEidK/8=
=CiZ9
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sat, 28 Aug 2004, Hans Reiser wrote:
>
> I object to openat().....

Sound slike you object to O_XATTRS, not openat() itself.

Realize that openat() works independently of any special streams, it's
fundamentally a "look up name starting from this file" (rather than
"starting from root" or "starting from cwd").

		Linus

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Sat, 28 Aug 2004, Hans Reiser wrote:
>  
>
>>I object to openat().....
>>    
>>
>
>Sound slike you object to O_XATTRS, not openat() itself.
>
>Realize that openat() works independently of any special streams, it's
>fundamentally a "look up name starting from this file" (rather than
>"starting from root" or "starting from cwd").
>
>		Linus
>
>
>  
>
well, isn't that namespace fragmentation by definition?  If you can't go

cat filenameA/metas/permissions > filenameB/metas/permissions

find / -exec cat {}/permissions \; | grep 4777 | wc -l

then aren't you missing the whole point of this exercise which is to 
allow the whole OS to be better integrated into a more unified namespace 
so that data can leap from one tool to another, and one container to 
another, with the greatest of ease?  If cat cannot access the metadata 
without changing the code of cat, then all the elegance goes poof.

It completely baffles me what disabling filenameA/.. does for us.  Why 
add asymmetries?  Ease of implementation is no excuse for such asymmetry.

Tomorrow I am going to send a little essay I wrote this evening on these 
metafiles.

Re: silent semantic changes with reiser4

[CaT]

On Sun, Aug 29, 2004 at 02:36:50AM -0700, Hans Reiser wrote:
> Linus Torvalds wrote:
> >On Sat, 28 Aug 2004, Hans Reiser wrote:
> >>I object to openat().....
> >
> >Sound slike you object to O_XATTRS, not openat() itself.
> >
> >Realize that openat() works independently of any special streams, it's
> >fundamentally a "look up name starting from this file" (rather than
> >"starting from root" or "starting from cwd").
>
> well, isn't that namespace fragmentation by definition?  If you can't go
> 
> cat filenameA/metas/permissions > filenameB/metas/permissions
> 
> find / -exec cat {}/permissions \; | grep 4777 | wc -l

My apologies if this question has been raised (and answered) elsewhere.
I tried to keep up with the flamefest/discussion but it wasn't always
possible.

How would you do it for directories without making certain names
illegal? ie taking the above you'd do:

cat dirA/metas/permissions > dirB/metas/permissions

But how does that distinguish an access to the metas for dirA and dirB
from an access to a file called permissions within the metas directories
of dirA and dirB?

I think what Linus is trying to do, as far as naming, is create a
consistant access scheme that works for all types of objects in the
fs without making certain names illegal (and with plugins, those
names are not even a fixed thing I believe).

Feel free to correct me if I'm wrong, ofcourse. :)

-- 
    Red herrings strewn hither and yon.

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Sun, Aug 29, 2004 at 05:05:17PM -0400, Hubert Chan wrote:
>Reiserfs list, and I don't think there was much consensus that came out

It's like Gentoo users and patch sets ;)

>of it.  Currently for Reiser4, AFAIK, the "metas" name is a compile-time
>option that you can change by changing a #define, if you're worried
>about name clashes.

http://mjt.nysv.org/reiser/reiser4.metas.patch

-- 
mjt

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sun, 29 Aug 2004, Hans Reiser wrote:
> >
> >Realize that openat() works independently of any special streams, it's
> >fundamentally a "look up name starting from this file" (rather than
> >"starting from root" or "starting from cwd").
>
> well, isn't that namespace fragmentation by definition?

No.

There's no difference between

	fd = open("/usr/bin/yes", O_RDWR);

and

	dirfd = open("/usr/bin", O_RDONLY | O_DIRECTORY);
	fd = openat(dirfd, "yes", O_RDWR);

apart from error checking and permissions..

No new namespace.

Only O_XATTR implies a "namespace change".

		Linus

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Sun, 29 Aug 2004, Hans Reiser wrote:
>  
>
>>>Realize that openat() works independently of any special streams, it's
>>>fundamentally a "look up name starting from this file" (rather than
>>>"starting from root" or "starting from cwd").
>>>      
>>>
>>well, isn't that namespace fragmentation by definition?
>>    
>>
>
>No.
>
>There's no difference between
>
>	fd = open("/usr/bin/yes", O_RDWR);
>
>and
>
>	dirfd = open("/usr/bin", O_RDONLY | O_DIRECTORY);
>	fd = openat(dirfd, "yes", O_RDWR);
>  
>
The difference is that cat uses open() not openat().

With your model, can I do:

cat filenameA/metas/permissions > filenameB/metas/permissions

find / -exec cat {}/permissions \; | grep 4777 | wc -l

If yes, then we are talking past each other somehow rather than 
disagreeing. If metafiles can be opened with both open and openat() in 
your model, then we are discussing some small detail.

I think the answer is no though, in which case you are missing the point 
of the new design. Is the answer no?

Re: silent semantic changes with reiser4

[Nicholas Miell]

On Sun, 2004-08-29 at 12:53, Hans Reiser wrote:
> With your model, can I do:
> 
> cat filenameA/metas/permissions > filenameB/metas/permissions
> 

runat filenameA "cat permissions" | runat filenameB "cat > permissions"

> find / -exec cat {}/permissions \; | grep 4777 | wc -l

find / -exec runat {} "cat permissions" \; | grep 4777 | wc -l

Although, whether or not the kernel exposes file permissions as an
attribute named permissions is up to the filesystem. (And the wrong
thing to do, in my opinion, but that's irrelevant.)

> If yes, then we are talking past each other somehow rather than 
> disagreeing. If metafiles can be opened with both open and openat() in 
> your model, then we are discussing some small detail.

It can be opened by both, but in order to do it via open(2), you need to
fchdir(2) to the attribute directory as returned by openat(2) and then
use a relative pathname.

> I think the answer is no though, in which case you are missing the point 
> of the new design. Is the answer no?
-- 
Nicholas Miell <nmiell@comcast.net>

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sun, 29 Aug 2004, Hans Reiser wrote:
>
> The difference is that cat uses open() not openat().

Hans. Step back a bit. Stop this idiocy about thinking that this is all 
about you, and all about your reiser4 "metas" etc.

This is about VFS interfaces, and about being generic and _useful_.

Quite frankly, there is absolutely _zero_ point to make attributes just 
because reiser4 can implement them. Zero. Nada. Zilch. It just won't 
matter, because people are NOT GOING TO SWITCH TO REISER4 IN MASS.

So in order to make this useful, it has to be:
 - stable. This means that if Al Viro asks about locking and aliasing
   issues, you don't ignore it, you ask "how high?"
 - portable. Preferably across operating systems, but at the very _least_ 
   across filesystems. See the NFSv4 thing, for example.
 - useful to the people who would actually use it. Right now that's samba, 
   and not a whole lot else.  For everybody else, the pain of not having 
   _all_ major filesystems support it is probably so insurmountable that
   it's just not going to happen.

Do you get it? If you can't get your mind out of your little current 
reiser4 issues, don't even bother talking about it.

Look through the thread on "possible design issues for hybrids" for me
actually talking with Al on how to handle problem #1. That's one important
thing, and in fact, that one is important whether we expose it to a normal
"open()" call or not, since all the same aliasing issues exist
_regardless_ of interfaces.

"openat()" happens to be _one_ solution to #2 and #3. You haven't given 
any constructive input to _any_ of the problems, and you keep on raising 
totally irrelevant issues. Face the music, Hans. If you can't solve those 
three problems, whatever you do simply DOES NOT MATTER.

In particular, in the two last emails to you, I tried to show that
"openat()" is _independent_ of streams/attributes. That it has semantics
that have absolutely _nothing_ to do with metas. And in BOTH CASES you
could not wrap your brain around that fact, and kept on harping about
meta's.

There are bigger issues here. I'm perfectly happy to _also_ expose the
streams as regular filenames, but I've tried to explain that that doesn't
work for directories, for example, so YOU CANNOT -JUST- talk about the
filename-only thing. You do need to have a different namespace.  Full
stop. What's so hard to understand about that?

So stop ignoring the issues. Stop ignoring the fact that not everything 
revolves around how you happened to implement something. Get with the 
program. 

		Linus

Re: silent semantic changes with reiser4

[Alexander Lyamin]

Sat, Aug 28, 2004 at 10:01:23PM -0700, Linus Torvalds wrote:
> 
> 
> On Sat, 28 Aug 2004, Hans Reiser wrote:
> >
> > I object to openat().....
> 
> Sound slike you object to O_XATTRS, not openat() itself.

xattr deserve to be objected. 
quickiehackie design practice.

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: silent semantic changes with reiser4

[viro]

On Sat, Aug 28, 2004 at 10:01:23PM -0700, Linus Torvalds wrote:
> 
> 
> On Sat, 28 Aug 2004, Hans Reiser wrote:
> >
> > I object to openat().....
> 
> Sound slike you object to O_XATTRS, not openat() itself.
> 
> Realize that openat() works independently of any special streams, it's
> fundamentally a "look up name starting from this file" (rather than
> "starting from root" or "starting from cwd").

Linus, openat() is trivial to implement in userland *IF* we are talking
about stuff already in namespace.  Proof:

int openat(int fd, char *path, int flags, mode_t mode)
{
	int len = snprintf(NULL, 0, "/proc/self/%d/%s", fd, path);
	char name[len + 1];
	sprintf(name, "/proc/self/%d/%s", fd, path);
	return open(name, flags, mode);
}

7 lines.  Replace VLA with alloca() or malloc() in pre-C99 environment.

In other words, there's nothing magical about syscall itself - compatibility
with Solaris is not an issue here.  And basic functionality (i.e. what you've
described above) can be trivially achieved with no extra primitives at all.

Now, if you want to use that puppy as explicit indicator of magic steps taken,
fine, but that again says nothing about the primitives we want to express the
magic steps in question - reproducing openat() as one of the libc functions
won't be hard anyway.

So I have to agree with Hans on that one - I believe that as a fundamental
primitive it's wrong choice.  Convenient to have in libc - sure, why not?

Re: silent semantic changes with reiser4

[John Stoffel]

Hans> My reason is that the things that distinguish between objects
Hans> should be the names, not the choice of system call.  The reason
Hans> for this is that it improves closure and namespace unification
Hans> to do so, because it allows all the objects to be accessed
Hans> within the same namespace.

So who decides what the names mean in a specific context?  How do we
get consistency across applications?  In the Apple and MicroSoft
cases, it's a single company which sets the standards for the names of
the streams and what they mean.  I have no knowledge of how Sun is
doing this.  I don't see us setting up a single naming authority and
enforcement mechanism.  

I went and read the Pike paper on namespaces which Hans quotes much
earlier in this thread, and I think it isn't applicable to this
thread.  In the paper Pike compares the MVS, VMS, DOS and Unix name
spaces for disk layouts.  It's an interesting paper, though a bit
dated since it's 20 years old.  The concepts are good, but the
examples aren't any more.  How many Unix systems today have
/n/faces/... to link email addresses to digitized images of people
faces? 

The big point of this paper though was to make the syntax simple, and
the semantics clear and unambiguous.  For example, they point out how
the "Cedar" file system added in VMS style version numbers to
filenames with the syntax "<name>!<version>" and they pointedly ask
what does "/usr/rob!3/bin/cat-v mean?  Which I think just points out
that the syntax of streams (metas) needs be carefully thought out, and
how that syntax implies various semantics.  

Lots of people keep asking for ACLs, but time and time again people
point out that while the Unix permission model isn't as flexible, it
is _simpler_ and easier to keep track of.  

As an example of complexity, just keeping track of our /etc/sudoers
file with all the users, hosts and commands that can be defined is a
huge hassle.  You pretty much need to have a 'lint' style program to
do the consistency checking.  

Hans> Yes, it can be useful to allow a namespace to exclude some
Hans> objects, but that exclusion should not be mandated.  

Conversely, should a namespace *require* that objects always be there?
What happens when I mount an NFS filesystem onto a Resierfs4 tree?
How does the user/apps determine when the semantics of the underlying
namespace have changed?  

Hans> If you want to exclude, you should cd or chroot to
Hans> /proc/nopseudos and find a view of the filesystem that excludes
Hans> metas, or mount with -nopseuodos.

Hans> Do you see why I say this?  I can say a lot more about the
Hans> damage of fragmenting namespaces into multiple apis.... Why look
Hans> at xattrs....;-)

I think one of the advantages of the xattrs is that the core namespace
doesn't change, nor do the semantics.  They are layered on top and can
be explicitly looked at.  

A real life problem of namespaces is Network Appliance Filers.  If you
have a filer with both NFS and CIFS exports, it will nicely manage the
CIFS ACLs in harmony (for some level you determine) with the NFS and
Unix permissions.  But to backup that data, and to store that
information, you need to use either their 'dump' tool or NDMP.  If you
NFS mount the filesystem to your backup host and then use a simple tar
to back it up, you lose the information since that window you use to
look into the namespace is restricted.  

It's a gotcha.  Until we can (or maybe computing in general) come up
with a new namespace paradym which we can all agree to, this is going
to be a big issue.  

John

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Sat, Aug 28, 2004 at 11:09:38AM -0700, Linus Torvalds wrote:
> 
> If you look at the Solaris interface, the _nice_ part about "openat()" is 
> that you can do something like
> 
> 	file = open(filename, O_RDONLY);
> 	if (file < 0)
> 		return -ENOENT;
> 	icon = openat(file, "icon", O_RDONLY | O_XATTR);
> 	if (icon < 0)
> 		icon = default_icon_file;
> 	..
> 
> and it will work regardless of whether "filename" is a directory or a 
> regular file, if I've understood correctly.
> 
> Now, I think that makes sense for several reasons:
>  - single case
>  - race-free (think "stat()" vs "fstat()" races).
>  - I think we want to do "openat()" regardless of whether we ever 
>    support extended attributes or not ("openat()" is nice for doing 
>    "namei()" in user space even in the absense of any attributes or 
>    named streams).
> 
> So what we can do is
>  - implement openat() regardless, and expect to do the Solaris thing for 
>    it if we ever do streams.

Hurrah ! Definately the best way to go as far as Samba is concerned.
Makes our portability easier. Now the NT interface allows an open on
a named data stream without an open on the containing file (ie.
you can do CreateFile("filename::STREAMNAME",....) without doing
CreateFile("filename",..." first, but I personally think those
semantics are a nasty hack. It makes much more sense (IMHO) to
ensure that you already have an open fd on a file before allowing
access to the named streams inside, Windows be damned. If we have
some semantic issues mapping from the Windows semantics to Solaris/Linux
semantics then so be it, it's a Samba problem so to speak :-).

Thanks,

	Jeremy.

Re: silent semantic changes with reiser4

[Helge Hafting]

Linus Torvalds wrote:

>On Sat, 28 Aug 2004, Helge Hafting wrote:
>  
>
>>>I think that lack of distinguishing power is more serious for 
>>>directories. The more I think I think about it, the more I wonder whether 
>>>Solaris did things right - having a special operation to "cross the 
>>>boundary".
>>>
>>>I suspect Solaris did it that way because it's a hell of a lot easier to 
>>>do it like that, but regardless, it would solve the issue of real 
>>>directories having both real children _and_ the "extra streams".
>>>      
>>>
>>There are many ways of doing this. Several extra streams to a directory
>>that aren't ordinary files in the directory?
>>    
>>
>
>Well.. Yes. We already have "." and "..", which are "special extra
>streams" in a sense. However, people expect them, and know to ignore them. 
>The same wouldn't be true of new naming.
>
>  
>
>>It seems to me that we can get a lot of nice functionality in a simpler way:
>>Instead of thinking about a number of streams attached to something
>>that is either an ordinary file or directory, just say that the only
>>change will be that a directory may have a _single_ file stream in
>>addition to being a plain directory.
>>    
>>
>
>That doesn't really help us. What would the name be, and how could you 
>avoid clashes? 
>  
>
The name for the file stream and the directory would be the same,
distinguished by how they're used.  I.e. fopen("filedirname", "rw")
gets the stream, while chdir("filedirname") changes into the directory.
fopen("filedirname/substream", "rw"); opens some stream inside
the directory - as usual.

My idea is that the "extra streams" aren't special at all, they
are simply files in a directory and will therefore work with
any normal tool.  cd'ing into such a directory won't be special
either - because it is a plain directory. A collection of named streams
is usually a bunch of files in a directory - nothing new there.

Someone serving files for clients using
other os'es may attach special meaning to them, but I don't see the need
for a special meaning under linux.

So the only new thing is that the directory have a stream of
its own.  Then there must be a way to turn a ordinary
file into such a directory - people may then implement
thumbnails, summaries and so on as substreams.  Where
the substreams really are plain files in this slightly
special directory.

Perhaps mkdir with an existing filename could succeed,
creating a mixed file/directory, if the underlying fs support it.

>  
>
>>If the VFS is to be extended in order to support file-as-directory (or
>>vice versa) then hopefully it can be done in a simple way.
>>    
>>
>
>I'm pretty confident that we can extend the VFS layer to support named
>streams (see the technical discussion with Al, rather than the flames in
>this thread). I also clearly believe that it is worth it, but I'm starting
>to wonder if we should have a special open flag to make people select the
>stream.
>
>If you look at the Solaris interface, the _nice_ part about "openat()" is 
>that you can do something like
>
>	file = open(filename, O_RDONLY);
>	if (file < 0)
>		return -ENOENT;
>	icon = openat(file, "icon", O_RDONLY | O_XATTR);
>	if (icon < 0)
>		icon = default_icon_file;
>	..
>
>and it will work regardless of whether "filename" is a directory or a 
>regular file, if I've understood correctly.
>  
>
An alternative: support open("filename/icon", O_CREAT);
and have this turn "filename" into file-as-directory if it wasn't so
already.  Then it works both with files and directories.
A new flag might be needed if old programs somehow depend on the stuff
above failing for plain files.

>Now, I think that makes sense for several reasons:
> - single case
> - race-free (think "stat()" vs "fstat()" races).
> - I think we want to do "openat()" regardless of whether we ever 
>   support extended attributes or not ("openat()" is nice for doing 
>   "namei()" in user space even in the absense of any attributes or 
>   named streams).
>
>So what we can do is
> - implement openat() regardless, and expect to do the Solaris thing for 
>   it if we ever do streams.
> - _also_ support the "implied named attributes" for regular files, so 
>   that you don't have to use "openat()" to access them.
>
>Comments? Does anybody hate "openat()" for any reason (regardless of 
>attributes)? We can easily support it, we'd just need to pass in the file 
>to use as part of the "nameidata" thing or add an argument (it would also 
>possibly be cleaner if we made "fs->pwd" be a "struct file").
>
>  
>
No problem with openat()

Helge Hafting

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>
>
>Currently I see no way to distinguish between the stuff
>that should be backed up and the stuff that shouldn't.
>
>  
>
We create filename/pseudos/backup, and that tells the archiver what to 
do.....

Re: silent semantic changes with reiser4

[Paul Jackson]

Hans wrote:
> We create filename/pseudos/backup, and that tells the archiver what to 
> do.....

Instead of exposing the old semantics under a new interface, why not
expose the new semantics under a new interface.

There exist plenty of programs that know the old Unix semantics.  There
don't exist many working programs that use the new semantics that you're
adding.

I raise again the example of how Windows adapted to long filenames.  Old
DOS and FAT programs, including my Unix backups of today, see a 8.3 name
space.  Only code that knows the new magic sees the long names.

If given the choice of breaking much old, existing stuff, or some new,
mostly not yet existing stuff, does not it make more sense to break what
mostly doesn't exist yet?

One possible way to do this, of no doubt many:

 * Stealing a corner of the existing filename space for
   some magic names with the new semantics.

 * A new option on open(2), hence opendir(3), that lights up
   these magic names.

 * Doing any of the classic pathname calls with such a
   new magic name exposes the new semantics - such calls
   as:
	access execve mkdir mknod mount readlink
	rename rmdir stat truncate unlink

This means essentially constructing a map between old and new,
such that changes made in either view are sane and visible
from the other view.

-- 
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <pj@sgi.com> 1.650.933.1373

Re: silent semantic changes with reiser4

[Alexander Lyamin]

Fri, Aug 27, 2004 at 11:08:57PM -0700, Paul Jackson wrote:
> Hans wrote:
> > We create filename/pseudos/backup, and that tells the archiver what to 
> > do.....
> 
> Instead of exposing the old semantics under a new interface, why not
> expose the new semantics under a new interface.

yeah. that could work.

> There exist plenty of programs that know the old Unix semantics.  There
> don't exist many working programs that use the new semantics that you're
> adding.
> 
> I raise again the example of how Windows adapted to long filenames.  Old
> DOS and FAT programs, including my Unix backups of today, see a 8.3 name
> space.  Only code that knows the new magic sees the long names.
> 
> If given the choice of breaking much old, existing stuff, or some new,
> mostly not yet existing stuff, does not it make more sense to break what
> mostly doesn't exist yet?
> 
> One possible way to do this, of no doubt many:
> 
>  * Stealing a corner of the existing filename space for
>    some magic names with the new semantics.
> 
>  * A new option on open(2), hence opendir(3), that lights up
>    these magic names.
> 
>  * Doing any of the classic pathname calls with such a
>    new magic name exposes the new semantics - such calls
>    as:
> 	access execve mkdir mknod mount readlink
> 	rename rmdir stat truncate unlink
> 
> This means essentially constructing a map between old and new,
> such that changes made in either view are sane and visible
> from the other view.

It would be intresting to hear comments from Hans Reiser on proposals stated  above...


-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: silent semantic changes with reiser4

[Hans Reiser]

Alexander Lyamin wrote:

>Fri, Aug 27, 2004 at 11:08:57PM -0700, Paul Jackson wrote:
>  
>
>>
>>One possible way to do this, of no doubt many:
>>
>> * Stealing a corner of the existing filename space for
>>   some magic names with the new semantics.
>>
>> * A new option on open(2), hence opendir(3), that lights up
>>   these magic names.
>>
>> * Doing any of the classic pathname calls with such a
>>   new magic name exposes the new semantics - such calls
>>   as:
>>	access execve mkdir mknod mount readlink
>>	rename rmdir stat truncate unlink
>>
>>This means essentially constructing a map between old and new,
>>such that changes made in either view are sane and visible
>>from the other view.
>>    
>>
>
>It would be intresting to hear comments from Hans Reiser on proposals stated  above...
>
>
>  
>
just use a view, and skip the options on the system calls.  if you cd to 
/nometas/your_home_directory_path you don't see the metafiles.  Why is a 
view better than a syscall flag?  Because it lets the user choose what 
he wants without recompiling to do it.  This kind of a view requires no 
coding because you can just mount the root filesystem two ways, one with 
the -nopseudos mount option, and one without it.

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 11:28:42AM -0700, Hans Reiser wrote:
> just use a view, and skip the options on the system calls.  if you cd to 
> /nometas/your_home_directory_path you don't see the metafiles.  Why is a 
> view better than a syscall flag?  Because it lets the user choose what 
> he wants without recompiling to do it.  This kind of a view requires no 
> coding because you can just mount the root filesystem two ways, one with 
> the -nopseudos mount option, and one without it.

*What*?

OK, now I want detailed explanation of the reasons why that doesn't create
cache coherency problems.

Do you have an analysis of locking in the entire thing?

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 07:36:29PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> > he wants without recompiling to do it.  This kind of a view requires no 
> > coding because you can just mount the root filesystem two ways, one with 
> > the -nopseudos mount option, and one without it.
> 
> *What*?
> 
> OK, now I want detailed explanation of the reasons why that doesn't create
> cache coherency problems.
> 
> Do you have an analysis of locking in the entire thing?

And I am very, very serious about that - we are talking about very nasty
minefield and design choices in that area have fundamental impact on the
entire layer, wherever it is located.

It's *NOT* something that you can leave until later and hope it somehow
falls into place - it can be merged in steps, but you MUST know the goal
on that level.  To rearchitect later might be possible (even though you
will a hell of a time avoiding plugins breakage), but it will be *hard*.

Re: silent semantic changes with reiser4

[Hans Reiser]

viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Sun, Aug 29, 2004 at 07:36:29PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
>  
>
>>>he wants without recompiling to do it.  This kind of a view requires no 
>>>coding because you can just mount the root filesystem two ways, one with 
>>>the -nopseudos mount option, and one without it.
>>>      
>>>
>>*What*?
>>
>>OK, now I want detailed explanation of the reasons why that doesn't create
>>cache coherency problems.
>>
>>Do you have an analysis of locking in the entire thing?
>>    
>>
>
>And I am very, very serious about that - we are talking about very nasty
>minefield and design choices in that area have fundamental impact on the
>entire layer, wherever it is located.
>
>It's *NOT* something that you can leave until later and hope it somehow
>falls into place - it can be merged in steps, but you MUST know the goal
>on that level.  To rearchitect later might be possible (even though you
>will a hell of a time avoiding plugins breakage), but it will be *hard*.
>
>
>  
>
How about if you educate me on the problems you see for a bit before I 
respond? I think it might help us move into a constructive discussion.

You are more experienced with your code than I am.... but I firmly 
believe that having different views of the same filesystem in which some 
files are visible in one view but not in another is technically feasible 
for some VFS, though I'd like to learn about the problems you see for 
doing it in the current VFS.

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 01:06:41PM -0700, Hans Reiser wrote:

> How about if you educate me on the problems you see for a bit before I 
> respond? I think it might help us move into a constructive discussion.

A slew of cache coherency issues (and memory consumption on top of that).
Rigth now we have a signle dentry tree for all fs instances, no matter
how many times it and its subtrees are visible in the tree.  Which,
obviously, avoids all that crap.  As soon as we start trying to have
multiple trees over the same fs, we are in for a *lot* of fun.

The basic question is how do you propagate the changes from one tree
to another and what do you do when change to "invisible" entry happens
via a tree where it is visible.  Unfortunately, all obvious solutions
either hit the pathname resolution and hit it *hard* (both in scalability
and in price of uncontended case) or create a mess for stuff like
NFS silly-rename semantics, VFAT aliases, etc.

Saying that some unspecified data structures might work doesn't help.
obviously - any variant will have tradeoffs of its own and has to be
discussed individually.  fsdevel is there for exactly that sort of
stuff, so if you have any specific proposals you want to discuss, you
are more than welcome.

Again, right now all cache coherency issues are sidestepped by having a single
instance of dentry tree per superblock.  Each mount/binding is represented by
vfsmount and _refers_ to a (sub)tree of dentry tree of that fs - dentries
and inodes are the same (ditto for private data structures owned by filesystem,
obviously).

So we have a forest of dentry trees (one per filesystem) and forest of
vfsmount trees (one tree per namespace).  Each vfsmount corresponds to
a chunk of namespace - think of the full user-visible tree cut into
pieces by mountpoints.  For every piece we know
	a) what fs it's from (->mnt_sb)
	b) what subtree of that fs it corresponds (->mnt_root)
	c) what piece it's attached to (->mnt_parent)
	d) where in that piece it's attached (->mnt_mountpoint)
	e) where to find its siblings and children
	f) which namespace (== tree in vfsmount forest) it's from

Point in a namespace is determined by pair (vfsmount, dentry) - which
piece we are looking at and where in that piece we are.  Pathname lookups
operate on such pairs - as long as we do not cross into another piece
we just step from dentry to dentry, when we cross mountpoint towards root,
we flip to (mnt_parent, mnt_mountpoint) first, when we cross mountpoint
into mounted filesystem, we do hash lookup (lookup_mnt()) by our pair,
find vfsmount with such mnt_parent and mnt_mountpoint and step into
(vfsmount, vfsmount->mnt_root).

Lifetime of vfsmounts is controlled by a simple refcount.  Having vfsmount
attached to tree contributes +1, so does having it pinned down by lookup/
chdir/chroot/opened file.  When the last vfsmount goes away, filesystem is
shut down (basically, that's what happens on umount).  Binding simply creates
a vfsmount pointing to target and attaches it at source.

Cloning a namespace copies the vfsmount tree of parent process' namespace
and flips vfsmounts of our root and cwd into the corresponding nodes in that
copy.  Normally, fork()/clone() just increments namespace refcount instead
of cloning it, so all children share namespace with parent.

When all processes in a namespace are gone its refcount goes to zero.  At
that point we dissolve all attachments in its vfsmount tree (which drops
a reference to each vfsmount and can in turn lead to fs shutdowns).

Lazy umount does similar operation with specified subtree - it detaches
all pieces mounted anywhere in that subtree and drops reference to each.
Ones that are not busy will be shut down immediately, ones that are will
go away as soon as they stop being busy.

vfsmounts can have flags of their own - one of the pending projects is to
allow individual vfsmounts to be read-only.  Noexec, nosuid and nodev are
already per-mountpoint.

Tree of vfsmounts is protected by
	a) vfsmount spinlock (protects hash lookups, mostly)
	b) per-namespace semaphore (that one is obviously blocking)
	c) in cases when we can go from 0 to 1 vfsmounts with given dentry
as a mountpoint, we also hold ->d_inode->i_sem on the mountpoint-to-be
(that closes races between rmdir/mount, rename/mount, etc.)

Dentry tree is messier - these days we have dcache_lock and RCU stuff.
If you can help getting the documentation out of these guys, you've got
a *lot* of thanks from a lot of people.  As it is, it's RTFS country.
I can answer specific questions there, but I won't even try to produce
readable manual covering the entire thing.

Directory operations have exclusion based on ->i_sem - see
Documentation/filesystems/directory-locking for description and proof
of correctness.

For operations that can destroy an object (unlink/rmdir/overwriting rename)
we try to unhash the victim dentry first, so the filesystems that can't
handle unlinked-but-busy stuff can detect such attempt by seeing the
victim still hashed _and_ can be sure that if it's not hashed, nobody will
come and see it (lookup would have to go to filesystem code, since we
have the sucker unhashed and we have exclusion there).  If operation is
unsuccessful, we rehash the victim.

Files involved:
	fs/dcache.c
	fs/super.c
	fs/inode.c
	fs/namespace.c
	include/linux/dcache.h
	include/linux/fs.h
	include/linux/mount.h

If you need more details on any specific area (or gaps in the above) - just
ask.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sun, 29 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> A slew of cache coherency issues (and memory consumption on top of that).
> Rigth now we have a signle dentry tree for all fs instances, no matter
> how many times it and its subtrees are visible in the tree.  Which,
> obviously, avoids all that crap.  As soon as we start trying to have
> multiple trees over the same fs, we are in for a *lot* of fun.

Al, I think you should make the argument a bit more specific, because I 
doubt a lot of people understand just what the problems are with aliased 
names. Just a few examples of the problems involved will illuminate things 
very well, I think. People who haven't been intimate with the name caches 
probably simply don't understand why you worry so much.

I'll start out with some trivial examples, just to let Hans and others get
an idea about what the issues are. I think examples of problems are often 
better ways to explain them than the abstract issues themselves.

Aliases: let's say that you have filename "a" hard-linked to filename "b", 
and you have a directory structure of streams under there. So you have

	a/file1
	a/dir1/file2
	a/dir2/file3

and (through the hard-link with "b") you have aliases of all these same 
names available as "b/file1", "b/dir1/file2" etc).

Now, imagine that you have two processes doing

	mv a/dir1 a/dir2/newdir

and

	mv b/dir2 b/dir1/newdir

at the same time. Both of them MUST NOT SUCCEED, for pretty obvious 
reasons (you'd have moved two directories within each other, and now 
neither would be accessible any more).

How do you handle locking for this situation?

Another interesting case is what happens when you have looked up and cache
the filename "a/file1" and then another process does "rm b/file1". How do
you update the _other_ cached copy, since they had two different names,
but _both_ names went away at the same time. Also again, how do you handle
locking?

The general VFS layer has a lot of rules, and avoids these problems by
simply never having aliases between two directories. If the same directory
shows up multiple times (which can happen with bind mounts), they have the
exact same dentry for the directory, it's just found through two different
vfsmount instances. That's why vfsmounts exist - they allow the same name
cache entry to show up in different places at the same time.

So when we do a bind mount, and the same directory shows up under two
different names "a" and "b", and we do a "rm b/file1", it _automatically_
disappears from "a/file1" too, simply by virtue of "a" and "b" literally
being the same dentry. No aliasing ever happens, and this makes coherency
and locking much easier (which is not to say that they are trivial, but
they are pretty damn clear in comparison to the alternatives).

What Al (and others) worries about is that the reiser4 name handling has
_none_ of these issues figured out and protected against. You can protect 
against them by taking very heavy locks (you can trivially protect against 
all races by taking one large lock around any operation), but the fact is, 
that is just not an option for high-performance name lookup. 

These aliasing/locking rules need to be global and wll-though-out. Not 
just fix the two examples above, but be shown to be safe _in_general_. 

		Linus

Re: silent semantic changes with reiser4

[Linus Torvalds]

[ Linux-kernel cc'd, because I don't think the question is stupid, and I 
  can't even fully answer the kNFSd thing other than point to it as a
  problem. ]

On Mon, 30 Aug 2004, Grzegorz Kulewski wrote:
> 
> Sorry if my qestion is stupid, but why can't we deal with (hard)links to 
> directories in (nearly) same way we deal with bind mounts (= making 
> exactly one object representing target and only referencing to it)?

On a VFS level we could, these days, I think. But realize that bind mounts
and the vfsmounts are pretty recent things.

We don't have any filesystems that support the notion, though, and we 
don't have any interfaces for the filesystem to tell us about it right 
now. The VFS layer could try to figure it out on its own from aliasing 
information, so the latter may be a non-issue, but the former is why 
nobody does it.

And even if Linux _these days_ could handle hardlinked directories, the
fact is that they would cause slightly more memory usage (due to the
vfsmounts), and that nobody else can handle such filesystems - including
older versions of Linux. So nobody would likely use the feature (not to
mention that nobody is even really asking for it ;).

And the lack of filesystem support is not theoretical. It's not easy to 
just retrofit directory hardlinks on a UNIX filesystem. The ".." entry 
actually exists on _disk_ on traditional unix filesystems, and with 
hardlinks on directories, that's a real problem. A hardlinked directory 
has multiple parents.

Also, while the VFS layer no longer cares (to it, ".." is purely virtual,
and it never uses it), the NFS export routines still do actually want to
get the on-disk parent. A filesystem that can't do that may be unable to
be exported with full semantics (ie you might get ESTALE errors after
server reboots, although you'd have to ask somebody with more kNFSd
knowledge than me on exactly why that is the case ;)

			Linus

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 03:37:16PM -0700, Linus Torvalds wrote:
> > Sorry if my qestion is stupid, but why can't we deal with (hard)links to 
> > directories in (nearly) same way we deal with bind mounts (= making 
> > exactly one object representing target and only referencing to it)?
> 
> On a VFS level we could, these days, I think. But realize that bind mounts
> and the vfsmounts are pretty recent things.
 
Bindings won't replace hardlinks.
	a) lifetime rules and keeping stuff busy
	b) who's bound on top?  Note that for real hardlinks to directories
(not just "directory on top of file" hybrids) it's a serious question
	c) for real hardlinks we would want at least rename() working

Re: silent semantic changes with reiser4

[Neil Brown]

On Sunday August 29, torvalds@osdl.org wrote:
> 
> Also, while the VFS layer no longer cares (to it, ".." is purely virtual,
> and it never uses it), the NFS export routines still do actually want to
> get the on-disk parent. A filesystem that can't do that may be unable to
> be exported with full semantics (ie you might get ESTALE errors after
> server reboots, although you'd have to ask somebody with more kNFSd
> knowledge than me on exactly why that is the case ;)
> 

The VFS requires all directories to have full paths from the
filesystem root in the dcache.  This is needed for loop detection when
renaming directories.

suppose and NFS client: (UPPERCASE words refer to file handles)
 Says "lookup ROOT 'a'" and gets A  (file handle for /a)
 Says "lookup A 'b'" and gets B     (file handle for /a/b)
 waits for /a and /a/b to be flushed from the dcache
 Says "RENAME ROOT 'a' to B 'z'"    (i.e. asks for /a to be moved to /a/b/c)

how can the VFS detect that that is not allowed?  It needs to know
that B is a subdirectory of /a.  The complete path cannot fit in
filehandle, so to needs to be computed by repeated lookup of "..",
which must be on disk.


Even without that, storing ".." on disk makes lots of sense for
fsck-like tools.

NeilBrown

Re: silent semantic changes with reiser4

[Stephan von Krawczynski]

On Sun, 29 Aug 2004 15:37:16 -0700 (PDT)
Linus Torvalds <torvalds@osdl.org> wrote:

> [...]
> And even if Linux _these days_ could handle hardlinked directories, the
> fact is that they would cause slightly more memory usage (due to the
> vfsmounts), and that nobody else can handle such filesystems - including
> older versions of Linux. So nobody would likely use the feature (not to
> mention that nobody is even really asking for it ;).

Huh? Me about a year ago ;-)
Been in fact pretty much boo'd for it :-)

I therefore declare as this years hot issue:
How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being
available for years I guess, but with 2.6 supporting lots of GIDs becoming very
actual...
:-)))
Sorry for being a bit off-topic but shouldn't we first solve the obvious
(broken) and simple issues around fs?

Regards,
Stephan

Re: silent semantic changes with reiser4

[Frank van Maarseveen]

On Thu, Sep 02, 2004 at 12:51:56PM +0200, Stephan von Krawczynski wrote:
> 
> I therefore declare as this years hot issue:
> How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being
                       ^^
The limit for NFS is 16.


> available for years I guess, but with 2.6 supporting lots of GIDs becoming very
> actual...

thank you for reminding me that I still need to port it to 2.6. The patch came
into existence around 2.2.17 IIRC.

-- 
Frank

Re: silent semantic changes with reiser4

[Stephan von Krawczynski]

On Thu, 2 Sep 2004 13:26:23 +0200
Frank van Maarseveen <frankvm@xs4all.nl> wrote:

> On Thu, Sep 02, 2004 at 12:51:56PM +0200, Stephan von Krawczynski wrote:
> > 
> > I therefore declare as this years hot issue:
> > How to use more than 32 GIDs on nfs? Frank van Maarseveens' patch being
>                        ^^
> The limit for NFS is 16.

Yes, of course. Sorry for that exaggeration ;-)

> > available for years I guess, but with 2.6 supporting lots of GIDs becoming
> > very actual...
> 
> thank you for reminding me that I still need to port it to 2.6. The patch
> came into existence around 2.2.17 IIRC.

Oh it will be very welcome once your done, the only way to circumvent the
situation currently is to give world-rights, which is of course _bad_.

Thanks in advance for this patch,
Stephan

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 17:50, skreiv Linus Torvalds:
> The general VFS layer has a lot of rules, and avoids these problems by
> simply never having aliases between two directories. If the same directory
> shows up multiple times (which can happen with bind mounts), they have the
> exact same dentry for the directory, it's just found through two different
> vfsmount instances. That's why vfsmounts exist - they allow the same name
> cache entry to show up in different places at the same time.

So could you explain what is stopping us from reducing the whole problem
to the bind mount problem? IOW have "a/" be a directory that acts as if
it is dynamically bind mounted on top of the file "a".


Is it just the fantasy of supporting hard-links across "stream
boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
nobody wants to have to add cyclic graph detection to their filesystems
anyway. 8-)

What other issues would need to be addressed?

Cheers,
  Trond

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 07:17:10PM -0400, Trond Myklebust wrote:
> På su , 29/08/2004 klokka 17:50, skreiv Linus Torvalds:
> > The general VFS layer has a lot of rules, and avoids these problems by
> > simply never having aliases between two directories. If the same directory
> > shows up multiple times (which can happen with bind mounts), they have the
> > exact same dentry for the directory, it's just found through two different
> > vfsmount instances. That's why vfsmounts exist - they allow the same name
> > cache entry to show up in different places at the same time.
> 
> So could you explain what is stopping us from reducing the whole problem
> to the bind mount problem? IOW have "a/" be a directory that acts as if
> it is dynamically bind mounted on top of the file "a".
> 
> 
> Is it just the fantasy of supporting hard-links across "stream
> boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
> nobody wants to have to add cyclic graph detection to their filesystems
> anyway. 8-)
> 
> What other issues would need to be addressed?

Go upthread and read.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sun, 29 Aug 2004, Trond Myklebust wrote:
> 
> So could you explain what is stopping us from reducing the whole problem
> to the bind mount problem? IOW have "a/" be a directory that acts as if
> it is dynamically bind mounted on top of the file "a".

Hey, I suggested people do exactly that. See the other thread between me
and Al Viro on exact implementation issues - even some code snippets ;)

The problem really ends up being directories with attributes (where we
can't just overmount the existing directory). That's where "openat()" 
helps us. 

(The other problem is the purely _practical_ problem of reiser4 going
behind the VFS layer, and thus _not_ getting the aliasing and locking
right, but that's an implementation issue in my book, and nothing really
fundamental).

> Is it just the fantasy of supporting hard-links across "stream
> boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
> nobody wants to have to add cyclic graph detection to their filesystems
> anyway. 8-)

It's easy enough to do the graph detection at the VFS layer, exactly 
because of the density of the dentry graph. 

(Of course, nfs-exporting a filesystem breaks that density, thanks to the
"lookup by fh" stuff, so we might not allow it for an NFS client). 

I suspect most filesystems wouldn't allow hard-links across "stream 
boundaries" in the first place. I _suspect_ that most stream 
implementations would bind the attributes more tightly than that to the 
file that owns them. reiser4 migth be the only one that might ever support 
something like that.

> What other issues would need to be addressed?

 - Whether we want it in the first place
 - whether we need the "separate namespace" thing that O_XATTR and
   openat() brings us (they certainly solve the problem, but maybe it is
   solvable another way too)
 - how to actually test this out in practice (ie getting reiser4 to do the
   proper thing wrt the VFS layer, but preferably _also_ having another
   filesystem like NFSv4 or cifs that actually uses this and shows what
   the problems are).
 - whether it makes any sense at all unless we also make at least a few 
   other filesystems support it, so that people start using it as an 
   "expected feature" rather than a "works only on a couple of machines".

And probably others.

			Linus

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 19:54, skreiv Linus Torvalds:

> The problem really ends up being directories with attributes (where we
> can't just overmount the existing directory). That's where "openat()" 
> helps us. 

Well, yes there has to be a distinction between a true bind mount which
actually covers the file or directory, and something like the stream
"bind mount" which doesn't.

The stream "bind mount" is just there to allow you to root the
attributes in a single tree. It can be made functionally entirely
equivalent to the openat(), but uses pathname semantics (e.g., "//") to
denote the attribute fork instead of an extra function call.

> > Is it just the fantasy of supporting hard-links across "stream
> > boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
> > nobody wants to have to add cyclic graph detection to their filesystems
> > anyway. 8-)
> 
> It's easy enough to do the graph detection at the VFS layer, exactly 
> because of the density of the dentry graph. 

Don't you end up having to lock the entire paths b/c/d and a/e/f in
order to prevent "ln a b/c/d/a; ln b a/e/f/b"?

> > What other issues would need to be addressed?

>  - how to actually test this out in practice (ie getting reiser4 to do the
>    proper thing wrt the VFS layer, but preferably _also_ having another
>    filesystem like NFSv4 or cifs that actually uses this and shows what
>    the problems are).

As I said, NFSv4 can be made ready pretty quickly: Bruce is already
finishing up the xattr implementation.

>  - whether it makes any sense at all unless we also make at least a few 
>    other filesystems support it, so that people start using it as an 
>    "expected feature" rather than a "works only on a couple of machines".

NTFS? ;-)

Cheers,
  Trond

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Sun, 29 Aug 2004, Trond Myklebust wrote:
> 
> Well, yes there has to be a distinction between a true bind mount which
> actually covers the file or directory, and something like the stream
> "bind mount" which doesn't.
> 
> The stream "bind mount" is just there to allow you to root the
> attributes in a single tree. It can be made functionally entirely
> equivalent to the openat(), but uses pathname semantics (e.g., "//") to
> denote the attribute fork instead of an extra function call.

Using '//' would be nice, but would break real apps. If I remember
correctly, POSIX specifies that '//' can be special at the _beginning_ of
a path, but in the middle, it has to act like a single '/'.

And that's not just theory - it's quite common for programs to just 
concatenate a directory name (which may or may not end with a slash) with 
another path-name that starts with a slash. So you _will_ see existing 
scripts and programs using things like "/usr/include//sys/type.h", and 
they'd break if "//" would switch from "regular namespace" to "attribute 
namespace".

So I don't see any way to extend pathname semantics to distinguish between 
"directory contents" and "directory attribute stream". 

> > It's easy enough to do the graph detection at the VFS layer, exactly 
> > because of the density of the dentry graph. 
> 
> Don't you end up having to lock the entire paths b/c/d and a/e/f in
> order to prevent "ln a b/c/d/a; ln b a/e/f/b"?

That's not the problem - since it's in memory, we can just get the dcache 
lock, and do it locked for t least local filesystems.

However, being prodded by Andries, I think I'm wrong _anyway_. Since the
dcache is only "dense" down one path to the root, and doesn't contain all
the alternate ways of getting to a particular directory, I came to the
conclusion that the VFS layer can't actually do cyclic detection after
all...

So together with the fact that nobody really _wants_ hardlinks to 
directories, I think the right answer is "no". It's not a problem as long 
as the attributes streams are always tied to the file/directory they are 
attributes of - then the "directory link" is really just a file link, and 
can't cause any cycles.

> >  - how to actually test this out in practice (ie getting reiser4 to do the
> >    proper thing wrt the VFS layer, but preferably _also_ having another
> >    filesystem like NFSv4 or cifs that actually uses this and shows what
> >    the problems are).
> 
> As I said, NFSv4 can be made ready pretty quickly: Bruce is already
> finishing up the xattr implementation.

Do we have any servers that implement it? I think NFSv4 might be a good 
test-case if so.

> >  - whether it makes any sense at all unless we also make at least a few 
> >    other filesystems support it, so that people start using it as an 
> >    "expected feature" rather than a "works only on a couple of machines".
> 
> NTFS? ;-)

Hey, I see the smiley, but I'd still like to point out that not many 
people use it under Linux, and while I think writing to it might be stable 
these days, I don't believe named streams are necessarily going to 
materialize all that quickly..

		Linus

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 07:31:49PM -0700, Linus Torvalds wrote:
> And that's not just theory - it's quite common for programs to just 
> concatenate a directory name (which may or may not end with a slash) with 
> another path-name that starts with a slash. So you _will_ see existing 
> scripts and programs using things like "/usr/include//sys/type.h", and 
> they'd break if "//" would switch from "regular namespace" to "attribute 
> namespace".
> 
> So I don't see any way to extend pathname semantics to distinguish between 
> "directory contents" and "directory attribute stream". 

I do, actually.  There might be a way and it is kinda-sorta similar to
your openat() variant, but lives in normal namespace.  No, I'm not too
fond of that, but since we are discussing weird variants anyway...

	a) associated directory tree of object is not automounted on top
of it.  Instead of that, we always do detached vfsmount (and do it on demand -
see below)
	b) we have a bunch of pseudo-symlinks in /proc/<pid>/fd/ - same
kind as what we already have there, but instead of (file->f_vfsmount,
file->f_dentry) they lead to associated vfsmount (allocated if needed).
Once we get a reference to such guy, we can
	* do further lookups
	* chdir there and poke around
	* hell, we can even bind it someplace (that will require slight change
in attach_mnt() logics, but it's not hard) and get it permanently mounted

Since it's not attached anywhere, normal GC logics works just fine.  And
yes, they are usable from scripts, etc. -
	exec 42<foo/bar/baz
	cat /proc/self/fd/#42/whatever/crap/you/want
and enjoy.

> However, being prodded by Andries, I think I'm wrong _anyway_. Since the
> dcache is only "dense" down one path to the root, and doesn't contain all
> the alternate ways of getting to a particular directory, I came to the
> conclusion that the VFS layer can't actually do cyclic detection after
> all...

<blinks>
<rereads a bunch of earlier postings>

Oh, _that_'s what you meant...  No, we definitely have no chance in hell
to catch loops, dcache or not.  It costs too much - we need to examine
a *lot* of nodes to do that and all of them would have to be read at some
point.  We either need entire fs tree locked in core or we might have to
reread it on every rename().  The former will kill us on memory use, the
latter - on amount of IO.

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Mon, 30 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> > 
> > So I don't see any way to extend pathname semantics to distinguish between 
> > "directory contents" and "directory attribute stream". 
> 
> I do, actually.  There might be a way and it is kinda-sorta similar to
> your openat() variant, but lives in normal namespace.  No, I'm not too
> fond of that, but since we are discussing weird variants anyway...
> 
> 	a) associated directory tree of object is not automounted on top
> of it.  Instead of that, we always do detached vfsmount (and do it on demand -
> see below)
> 	b) we have a bunch of pseudo-symlinks in /proc/<pid>/fd/ - same
> kind as what we already have there, but instead of (file->f_vfsmount,
> file->f_dentry) they lead to associated vfsmount (allocated if needed).
> Once we get a reference to such guy, we can
> 	* do further lookups
> 	* chdir there and poke around
> 	* hell, we can even bind it someplace (that will require slight change
> in attach_mnt() logics, but it's not hard) and get it permanently mounted

Well, the above _is_ the same as "openat()", really. It's just using a
filesystem starting point to emulate a new system call. Same thing
conceptually. You could do pretty much any system call as a filesystem
action if you wanted to ;)

I don't disagree with doing so - as a way to expose the new system call to
scripts. But I don't think you're being entirely intellectually honest if
you think this suddendly makes it be "one namespace". It's still a
secondary namespace rooted in an entry in the normal ones - exactly like
"openat()". 

For a non-script, a native "openat()" interface would be more efficient
and less confusing, and conceptually no different from yours.  No reason
we couldn't have both, since they are 100% equivalent and would share the
same code anyway...

		Linus

Re: silent semantic changes with reiser4

[viro]

On Sun, Aug 29, 2004 at 08:55:02PM -0700, Linus Torvalds wrote:
> Well, the above _is_ the same as "openat()", really. It's just using a
> filesystem starting point to emulate a new system call. Same thing
> conceptually. You could do pretty much any system call as a filesystem
> action if you wanted to ;)

Umm...  Yes and no - open() is not the only thing you can do there.  You
can emulate some of the other stuff with open() + fchdir() + syscall, but...
Note that it also gives you access to other tasks' files (provided that
tasks are yours, so you can get to their /proc/.../fd).

> I don't disagree with doing so - as a way to expose the new system call to
> scripts. But I don't think you're being entirely intellectually honest if
> you think this suddendly makes it be "one namespace". It's still a
> secondary namespace rooted in an entry in the normal ones - exactly like
> "openat()". 

Well - it *does* expose these objects to all normal syscalls.  E.g. you
can unlink() a component in there without
	fd2 = open(".", ...);
	fd3 = openat(fd, ".", ....)
	fchdir(fd3);
	unlink("blah");
	fchdir(fd2);
and similar horrors (now have fun adding locking for multithreaded process,
etc.).

In that sense it is, indeed, the same namespace.

> For a non-script, a native "openat()" interface would be more efficient
> and less confusing, and conceptually no different from yours.  No reason
> we couldn't have both, since they are 100% equivalent and would share the
> same code anyway...

I'm not sure that openat() is the right interface for e.g. fileservers.
And no, I'm not saying that above is suitable for them - IMO neither variant
is what they need.  The thing is, fileserver almost certainly wants to
create and remove objects.  And you either end up with new syscalls for
doing *that* relative to opened fd or you do tons of fchdir(), which
makes benefits of openat() dubious in the best case.

Arguments about O_NOFOLLOW on the intermediate stages are bullshit, IMNSHO -
if they want to make some parts of tree inaccessible, they should simply
mkdir /tmp/FOAD; chmod 0 /tmp/FOAD; mount --bind /tmp/FOAD <blocked path>
in the namespace their daemon is running in.  And forget all that crap
about filtering pathnames and blocking symlinks on intermediate stages
(the latter is obviously worthless without the former since one can simply
substitute the symlink body in the pathname).

Re: silent semantic changes with reiser4

[viro]

On Mon, Aug 30, 2004 at 05:46:37AM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
> Arguments about O_NOFOLLOW on the intermediate stages are bullshit, IMNSHO -
> if they want to make some parts of tree inaccessible, they should simply
> mkdir /tmp/FOAD; chmod 0 /tmp/FOAD; mount --bind /tmp/FOAD <blocked path>
> in the namespace their daemon is running in.  And forget all that crap
> about filtering pathnames and blocking symlinks on intermediate stages
> (the latter is obviously worthless without the former since one can simply
> substitute the symlink body in the pathname).

Ehh...  After looking at that for a while...  No, it's not that simple
and removing the stuff that way won't do what these guys want, at least
not without something else.  Frankly, what I've seen worries me a lot -
it looks like there is a missing primitive here that would be saner
than this sort of filtering.

It appears that most of this stuff would be covered by a fast way to tell
if the resulting object belongs to given subtree.  That could be arranged
(not without some changes, but doable), but I'm not sure that it's enough
to cover the stuff they are really trying to do.  It does look like an
interesting problem and current solutions certainly suck.  And I very
much doubt that "do a lookup if it doesn't run into anything that could
be too tricky for our pathname-based checks, otherwise let's do it step-by-step
from userland" is the right approach here.

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 22:31, skreiv Linus Torvalds:

> So together with the fact that nobody really _wants_ hardlinks to 
> directories, I think the right answer is "no". It's not a problem as
> long as the attributes streams are always tied to the file/directory
> they are attributes of - then the "directory link" is really just a
> file link, and can't cause any cycles.

Right.

> Do we have any servers that implement it? I think NFSv4 might be a good 
> test-case if so.

I believe Solaris Express already has an implementation, so I should
have at least one server implementation available in the basement.
So probably does Hummingbird, since their NFSv4 server runs on Windows.

CITI is working on the Linux code, but I'm not sure how far they've come
on the server side. I'll ask Bruce in the morning...

Cheers,
  Trond

Re: silent semantic changes with reiser4

[Brad Boyer]

On Sun, Aug 29, 2004 at 07:31:49PM -0700, Linus Torvalds wrote:
> On Sun, 29 Aug 2004, Trond Myklebust wrote:
> > >  - how to actually test this out in practice (ie getting reiser4 to do the
> > >    proper thing wrt the VFS layer, but preferably _also_ having another
> > >    filesystem like NFSv4 or cifs that actually uses this and shows what
> > >    the problems are).
> > 
> > As I said, NFSv4 can be made ready pretty quickly: Bruce is already
> > finishing up the xattr implementation.
> 
> Do we have any servers that implement it? I think NFSv4 might be a good 
> test-case if so.

I know the mention of it in the past has brought up ridicule, but
netatalk requires some level of multi-stream support in the export
to the client, and currently just emulates it on the server side
by hiding the extra files in various directories. It should be
possible to modify it to use named streams/attributes for the
extra data. If that happens, it would be nice if hfs and hfsplus
could export the data so it could be used. The view exported by
netatalk does match the internal structure of hfs, after all.

(For those of you that aren't familiar with the Macintosh, a Mac
style filesystem has two peer streams, data and resource. They
are each a seekable bytestream, and are actually at the same
level inside the filesystem. In addition, there is a large set
of fixed length attributes stored, such as the file type info
and icon position. In addition, hfsplus also supports full
named streams and arbitrary metadata, tho that isn't used much.)

	Brad Boyer
	flar@allandria.com

Re: silent semantic changes with reiser4

[Tom Vier]

On Sun, Aug 29, 2004 at 07:31:49PM -0700, Linus Torvalds wrote:
> Using '//' would be nice, but would break real apps. If I remember
> correctly, POSIX specifies that '//' can be special at the _beginning_ of
> a path, but in the middle, it has to act like a single '/'.

FWIW, QNX uses // for it's network fs (nfs?). //0/ means current node, //1/
first node, //2/ second, etc. I didn't even know posix said anything about
// before.

-- 
Tom Vier <tmv@comcast.net>
DSA Key ID 0x15741ECE

Re: silent semantic changes with reiser4

[Alexander G. M. Smith]

Trond Myklebust wrote on Sun, 29 Aug 2004 19:17:10 -0400:
> Is it just the fantasy of supporting hard-links across "stream
> boundaries" (as in "touch a b; ln b a/b; ln a b/a")? I'm pretty sure
> nobody wants to have to add cyclic graph detection to their filesystems
> anyway. 8-)

Been there, done that.  It works in a rough form in an experimental RAM file system for BeOS.  But deleting files requires a graph traversal to see if it is isolating just one item or a whole bunch of them.  It also implies keeping track of multiple parent directories for everything.  It does make it useful for the user - you can have files and directories in multiple places, so you don't have to decide on one classification for a file (or directory).  One other downside is a lot of locking complexity (or not if you don't need fine grained locking) to guarantee atomicity of the delete.

For user convenience, the low level delete function returns an error code if there is more than one item cut off from the root, but it could just as well delete all of them (sort of like rmdir giving an error if the directory isn't empty rather than just deleting everything).

- Alex

Re: silent semantic changes with reiser4

[Hans Reiser]

You are assuming that the hiding has to be embodied by the dentry cache. 
That sounds hard. Let's not do that; I am too lazy.

We can put it at a layer above that. Let us have a "mask" concept. For 
the purposes of this discussion, the mask can be made totally trivial 
and hardcoded, with none of the expressive power of my new DoD project, 
and can probably be coded in a few hours to exclude the metas. For my 
new DoD project we are creating a configurable, scalable, sophisticated 
"mask" abstraction, and you must successfully pass through the mask 
before you can ask if the filesystem contains X/Y/Z. (For the darpa 
project with the general purpose masks with specifications and all that 
we will take 6 months, not a few hours.)

The rest shall wait until after I get a bite to eat, ok?

Hans

viro@parcelfarce.linux.theplanet.co.uk wrote:

>On Sun, Aug 29, 2004 at 01:06:41PM -0700, Hans Reiser wrote:
>
>  
>
>>How about if you educate me on the problems you see for a bit before I 
>>respond? I think it might help us move into a constructive discussion.
>>    
>>
>
>A slew of cache coherency issues (and memory consumption on top of that).
>Rigth now we have a signle dentry tree for all fs instances, no matter
>how many times it and its subtrees are visible in the tree.  Which,
>obviously, avoids all that crap.  As soon as we start trying to have
>multiple trees over the same fs, we are in for a *lot* of fun.
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

I think we have to either:

1) Link only to the file and not the directory in the file-directory 
duality. This means we change the semantics of hard link so that it only 
links to the data and the standard metadata, and the optional 
attributes/streams/files-in-directory are not seen by the second link. 
You can think of this as meaning that hard links only connect to the 
file and not the directory in the file-directory wave-particle duality

or, 2) we should ask Alexander Smith to help with applying the graph 
traversal cycle detection code that he wrote.

I can go either way contentedly for now. 2) is the right long term 
solution. 1) is probably the right short term solution.

Ok, Linus and Viro, now I see why it was hard. Being able to effectively 
connect to compound documents only with symlinks is a bit distasteful, 
but it is quite livable, and I very much hope you decide it is better 
than fragmenting the namespace.

Hans

Re: silent semantic changes with reiser4

[Alexander G. M. Smith]

Hans Reiser wrote on Mon, 30 Aug 2004 00:42:13 -0700:
> or, 2) we should ask Alexander Smith to help with applying the graph 
> traversal cycle detection code that he wrote.

I put my foot into it, didn't I?  It's not too bad, one trick to simplify
things is to have the concept of a main parent and step parents.  The file
system name space graph is kept structured so that by always going upward
through the main parent, you're guaranteed to reach the root.  It doesn't
save anything on traversal when deleting files (you still have to visit
all the relevant nodes to check for cycles), but allows you to replace
hard links with a new kind of dynamic symbolic link.  The hard link
pretends to be a symbolic link presenting the "true path" to its target,
which changes as the target moves around (so it's not as reliable as real
hard link).  Old software doesn't see the cycles in the graph, just follows
the single true path to any particular object like it currently does, and
sees a bunch of symbolic links to everything else.  New software can see
the full parent list (not just a single "..") and the actual hard links.

> Ok, Linus and Viro, now I see why it was hard. Being able to effectively 
> connect to compound documents only with symlinks is a bit distasteful, 
> but it is quite livable, and I very much hope you decide it is better 
> than fragmenting the namespace.

Still, having hard links to attributes or directories of attributes is
useful.  For example, if several contact manager entries reference the
same person, it would be nice to have that represented as a hard link to
the attribute directory tree describing that person.  That's assuming you
implement contacts (name, address, etc) as a bunch of attributes in a
directory.

- Alex

Re: silent semantic changes with reiser4

[Paul Stewart]

On 2004.08.27 23:08, Paul Jackson wrote:
> Hans wrote:
> > We create filename/pseudos/backup, and that tells the archiver what  
> > to do.....
> 
> Instead of exposing the old semantics under a new interface, why not
> expose the new semantics under a new interface.

Here's another take on the same theme.  To see attrs on files, one can  
either use a newly developed application which can use special new  
syscalls/flags on syscalls a Paul Jackson recommends.  However from an  
old shell or application one can also open the attribute node on
/home/myself/foo.txt by checking out /attr/home/myself/foo.txt/, which  
points to the "as directory" node on the filesystem that foo.txt points  
to.

The strange part of this idea is that the /attr filesystem wouldn't be  
conventionally browsable.  An opendir() on any path would return the  
attributes, not the subdirectory contents, even if the target was a  
conventional directory.  File based operations (open(), unlink(), etc)  
would operate on the "as directory" node of the dirname() of the file  
argument.  The /attr filesystem would be completely virtual, honoring  
the permissions, crossing filesystems, and inheriting the root  
directory of the calling process.

Just a thought from a Linux fs layman -- certainly one of the Reiser  
unwashed masses.

--
Paul

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Mon, 30 Aug 2004, Paul Stewart wrote:
> 
> Here's another take on the same theme.  To see attrs on files, one can  
> either use a newly developed application which can use special new  
> syscalls/flags on syscalls a Paul Jackson recommends.  However from an  
> old shell or application one can also open the attribute node on
> /home/myself/foo.txt by checking out /attr/home/myself/foo.txt/, which  
> points to the "as directory" node on the filesystem that foo.txt points  
> to.

This is the same idea as Al Viro's /proc/self/fd/#42/attr issue, except 
yours has two fundamental problem: races and ambiguities.

If you open a filename in some "secondary" tree (be it /proc or //attr or 
whatever) based on the filename in the primary one, you have two issues 
that you need to work out:

 - how do you handle a name change in the primary tree at the same time as 
   lookup
 - how do you handle the ambiguity of
	//attr/usr/bin/emacs/icon
   (is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the 
   "emacs/icon" attribute on "/usr/bin").

The ambiguity can be handled by saying that attributes only have one
component (ie only the _last_ component of a lookup is the attribute
name). But the race between primary tree and secondary tree cannot be
handled in a normal name-space.

What Al did was to avoid both by "fixing" the attribute lookup point with 
another open - _exactly_ the same way "openat()" handles it. So Al's 
naming convention avoids both the ambiguity and the primary tree name 
change races by first opening the primary tree file, and then explicitly 
using that file as the "anchor" in the secondary tree. He did it in /proc, 
where we obviously already do export an open fd as an anchor-point.

> The strange part of this idea is that the /attr filesystem wouldn't be  
> conventionally browsable.

That may make it non-intuitive to use, but that's not the real problem.  
See above.

		Linus

Re: silent semantic changes with reiser4

[Alexander Lyamin]

Fri, Aug 27, 2004 at 11:28:51AM -0700, Hans Reiser wrote:
> Rik van Riel wrote:
> 
> >
> >
> >Currently I see no way to distinguish between the stuff
> >that should be backed up and the stuff that shouldn't.
> >
> > 
> >
> We create filename/pseudos/backup, and that tells the archiver what to 
> do.....

its ugly idea.

representations should be distinguished from sources. and its reasonably
to put this distinguisher somewhere in namespace. but filename/pseudos/backup
is plain ugly.
-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander Lyamin wrote:
| Fri, Aug 27, 2004 at 11:28:51AM -0700, Hans Reiser wrote:
|
|>Rik van Riel wrote:
|>
|>
|>>
|>>Currently I see no way to distinguish between the stuff
|>>that should be backed up and the stuff that shouldn't.
|>>
|>>
|>>
|>
|>We create filename/pseudos/backup, and that tells the archiver what to
|>do.....
|
|
| its ugly idea.
|
| representations should be distinguished from sources. and its reasonably
| to put this distinguisher somewhere in namespace. but
filename/pseudos/backup
| is plain ugly.

How about filename/metas/serialized, or filename/metas/fschunk?

And I did mention before about how it'd be nice for it to be an fs chunk...

Imagine 5000 tiny little files that, by the time reiser's done with
them, fit inside 5 blocks or so.  It's insanely cheaper to copy the five
blocks than to tar them up.  It also means that we don't have to worry
about supporting tar on other platforms -- if they don't have reiser4,
they need a separate converter.  If apps are going to use reiser4
metadata for anything significant, they probably would link against some
sort of userland implementation of reiser4 as a Windows library, say, to
access the files elsewhere.

That's extreme, but you get the idea.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTPKc3gHNmZLgCUhAQJ5xw//T+kLDCHlfQEaGM9XAeFDZU9f4m4PWjet
n2xODVIxaT7WmnZ4nQ3rRTiJjsXvRBcPdWUTFRZl6NbLqa/K6ESpUJAZsZynRhXd
NLJ9gEpeqcUrXBLrmTD5AbzENOLdqDIo2qQ+hde68H5vo+naa5xtTyrMiHNHQWbf
9P45WKXYt36madg4KjAhZcYB+T843fc/ItjLQBTwxBNOykBcZ82UUNwGeGbiqczm
uXftCD6BNtKsXnJxu2jVc8CK93lOw4SIf7BY09PFXgK2XTpAsKzOwwqpBtdxSzqD
lkz+nGEeYZk6GRA3N1OTbPJnboee7WHZzMyvXDMBsExfR6puV1SYA2NWhj9uGDmS
nYNZwdN+RimiouD1D9DD+oCQe/Gr66THicex5FM37cMBdaDf0jGblvpDdQRfkhcX
2KGLCt1ax620ia2945/+/+6iQqM7+oTQsVNziYjVGQ4gCNRzZh8SBtM0yUZuecNO
H84fWYLICGos+Qk7FyCuQOquTSVRWKadkiDaOSre2hFk+S9VmDWkghJSNGrnqoKi
pFYFrUzL+NXPHs6G8b8EKAaQQt+Z8iHE6xgdc+yS56Pdrq8apRzeQv2zhgZaRq9J
kCRgvYo40xFMX4B+EqexxIiwc6nPUD5dFbt0OnJMEk4pduFq64lMVdoQA8Qp4X8a
0feuVuaWHr0=
=ymeH
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Hans Reiser]

David Masover wrote:

>
>
> And I did mention before about how it'd be nice for it to be an fs 
> chunk...

Please define fs chunk. My mind has an open door.;-)

>
> Imagine 5000 tiny little files that, by the time reiser's done with
> them, fit inside 5 blocks or so. It's insanely cheaper to copy the five
> blocks than to tar them up. It also means that we don't have to worry
> about supporting tar on other platforms -- if they don't have reiser4,
> they need a separate converter. If apps are going to use reiser4
> metadata for anything significant, they probably would link against some
> sort of userland implementation of reiser4 as a Windows library, say, to
> access the files elsewhere.
>
> That's extreme, but you get the idea.

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
| David Masover wrote:
|
|>
|>
|> And I did mention before about how it'd be nice for it to be an fs
|> chunk...
|
|
| Please define fs chunk. My mind has an open door.;-)

I think this is sort of like dump/restore, but for a subset of the tree.
~ So it's faster for all the reasons dump/restore are faster than tar for
a whole filesystem.  But I don't know all that much about either
dump/restore or reiser4 format plugins.

I'm defining an "fs chunk" as a complete, read-only reiser4 filesystem
of the data in question.  As a snapshot.  I'm imagining it being faster
to create and ultimately smaller and more intelligently packed than a
tarball.

What if you could get a blocklist for a directory?  Of course some
things would have to be changed to completely divorce it from the parent
filesystem and have it still be usable.

I'm imagining that literally grabbing a chunk of the filesystem -- using
its own storage layer -- would be faster than tarring.  Think like if
directories had blocklists as well as files.  It would also be smaller
- -- see below.  And it completely eliminates the question of "what may I
never back up" -- the fs already knows that.  Without having to be told
by plugins.  If the fs stores it to disk, the backup stores it, unless
it's part of the "cache" proposal.

This is a rough idea, and has issues.  For instance, how do we know
what's cache and what's not?  If you have foo.gz/ugz cached and someone
writes to foo.gz/ugz, I think that should be written to the cache and
the original foo.gz should be removed -- regenerated if necessary from
foo.gz/ugz.  But the reverse is true -- write to foo.gz and foo.gz/ugz
should go away, to be regenerated if necessary.  Only one should go into
the backup, though.  I think foo.gz goes into the backup and is
regenerated from foo.gz/ugz if necessary.  More generally, when you have
two redundant streams, backup the smaller one.

|>
|> Imagine 5000 tiny little files that, by the time reiser's done with
|> them, fit inside 5 blocks or so. It's insanely cheaper to copy the five
|> blocks than to tar them up. It also means that we don't have to worry
|> about supporting tar on other platforms -- if they don't have reiser4,
|> they need a separate converter. If apps are going to use reiser4
|> metadata for anything significant, they probably would link against some
|> sort of userland implementation of reiser4 as a Windows library, say, to
|> access the files elsewhere.
|>
|> That's extreme, but you get the idea.
|
|

P.S.  Sorry for the dupe, Hans, accidently hit "reply" instead of
"reply-to-all"...

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUuS3gHNmZLgCUhAQKWCQ/+LTNqHvndRAMqzdAJ0rugZl30bqqPmrFs
vNqSwEx09RUxN6caOd+C4F7oVj/as0AJrNqyN5YR6/t6BKGxh+xEDnKdKiIRR5qp
/TJmsMSkh54MSqaZZPL5EB6/DjWscBN7zKCaaCN6f8xB6KU2BCOK5pNSHDvYhISA
uyD8AXrsOLRGC82dNtmh68KdrG+7VcTZMohulXNnosLLRJSuKeYEphB0i5HrvXQn
nZ010yb6LnmKtNKD+w7bFxt3ul+lQo8gti7zBJqb7mV4pa1MnQNhGZRgvey/Pq/l
ahXrrABauUFNwM4PfFWUoY9zjOdCufQpWev/mNGuoo9BS2ZiRg8LXoUpf2bOvi6I
yOTMRb37w6+w37Jv8B0vYPehW3bmuJpb+jDyZ5AYS0GygJmm3bEIFZYupYt2pU/v
7EBIzLszCEnqPbm0glGdWT5jyYiZGucvDK+W4qQ3ymEqaBIdGIti4G56mQ7bTQer
DPVT4dv38OE/PEtKb6xLQUaos7hBg+sWM5+mT9B+7kX/cYDhtdBkZpYqf10jAg34
4ufl2NS48CxqCcml7RayPVAuKT1ZN9sDcfXVYnPw33R94/lziZnfJrIInbnf9MZK
+P9v6WlFi6+9GZjwqFK9ZCJBoTCKRq1lGiy5flTDf1APAgpmU6CS/0Da4s1QiLLI
YRoeX8Nv8Us=
=WFSh
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Denis Vlasenko]

On Thursday 26 August 2004 20:57, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Linus Torvalds wrote:
> > > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> >
> > No no. The stream you get from a directory is totally _independent_ of
> > the contents of the directory. Anything else would be pointless.
>
> It's a relief to know that nobody's taking my humorous
> suggestion seriously, but now we still have the "standard
> Unix tools can't manipulate files" problem...

Is it possible to sufficiently hide "dirs inside files"
so that old tools will be unable to see them?

I just checked:

ls -d /foo  does lstat64("/foo", ...)
ls -d /foo/ does lstat64("/foo", ...)
	but
ls -d /foo/. does lstat64("/foo/.", ...)

Will it work out if "dir inside file" will only be visible when referred as "file/."?
--
vda

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004, Denis Vlasenko wrote:
> 
> Is it possible to sufficiently hide "dirs inside files"
> so that old tools will be unable to see them?

Certainly possible.

> I just checked:
> 
> ls -d /foo  does lstat64("/foo", ...)
> ls -d /foo/ does lstat64("/foo", ...)
> 	but
> ls -d /foo/. does lstat64("/foo/.", ...)
> 
> Will it work out if "dir inside file" will only be visible when referred as "file/."?

That would likely be the _easiest_ approach for the kernel, and does solve 
the problem with some apps knowingly removing the trailing '/'.

Note that we could try this out with existing filesystems with very 
minimal changes:

 - make directory bind mounts work on top of files ("graft_tree()")
 - make open_namei() and friend _not_ do the mount-point following for the 
   last component if it's a non-directory.
 - probably some trivial fixups I haven't thought about. There might be 
   some places that use "S_ISDIR()" to check for whether something can be 
   looked up, but the main path walking already just checks whether there
   is a ".lookup" operation or not.

This would already allow people to "try out" how different applications 
would react to a file that can show up both as a directory and a file. The 
patch might end up being less than 25 lines or so, the difficulty is in 
finding all the right places.

Al, anything I missed?

(And yes, it's a quick hack, but it's a quick hack that would probably
mimic a good part of what we would have to do internally in the VFS layer
to support this notion anyway).

		Linus

Re: silent semantic changes with reiser4

[viro]

[reposted in thread]

On Thu, Aug 26, 2004 at 11:46:33AM -0700, Linus Torvalds wrote:
> Note that we could try this out with existing filesystems with very 
> minimal changes:
> 
>  - make directory bind mounts work on top of files ("graft_tree()")
>  - make open_namei() and friend _not_ do the mount-point following for the 
>    last component if it's a non-directory.
>  - probably some trivial fixups I haven't thought about. There might be 
>    some places that use "S_ISDIR()" to check for whether something can be 
>    looked up, but the main path walking already just checks whether there
>    is a ".lookup" operation or not.
> 
> This would already allow people to "try out" how different applications 
> would react to a file that can show up both as a directory and a file. The 
> patch might end up being less than 25 lines or so, the difficulty is in 
> finding all the right places.

The real issue is what to do with unlink() et.al. on these guys.  Note
that "unlink is OK if all we have there is a bunch of directory mounts"
won't work well - we have no good way to check that condition.

Even funnier one is what we do if we have directory mounted there *and*
have something mounted on stuff in that directory.

Yes, that's one of the probable directions for such stuff, but there's a
lot of fun semantics questions and answers to them will matter a lot.

Hey, if we lose the "can't unlink/rmdir/rename over something that is
a mountpoint in other life" - I'm happy and we can get a lot of much
more interesting stuff to work.  It will take some work (e.g. making
sure we can find all vfsmounts over given mountpoint and sorting out
the locking issues, which won't be trivial), but the main obstacle in
that direction is not in architecture - it's in SuS and tradition; as
the matter of fact, our life would be much easier if we stopped trying
to give -EBUSY here and just dissolved all subtrees mounted on anything
that has that dentry.

Re: silent semantic changes with reiser4

[viro]

On Thu, Aug 26, 2004 at 08:13:23PM +0100, viro@parcelfarce.linux.theplanet.co.uk wrote:
 
> Hey, if we lose the "can't unlink/rmdir/rename over something that is
> a mountpoint in other life" - I'm happy and we can get a lot of much
> more interesting stuff to work.  It will take some work (e.g. making
> sure we can find all vfsmounts over given mountpoint and sorting out
> the locking issues, which won't be trivial), but the main obstacle in
> that direction is not in architecture - it's in SuS and tradition; as
> the matter of fact, our life would be much easier if we stopped trying
> to give -EBUSY here and just dissolved all subtrees mounted on anything
> that has that dentry.

Argh...  OK, now I remember why I went for -EBUSY for unlink() (we obviously
are not bound by SuS on that one).  Consider the following scenario:
	* local file foo got something else bound on it for a while
	* we are tight on space - time to clean up
	* oh, look - contents of foo is junk
	* rm foo
	* ... oh, fuck, there goes the underlying file.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 619 bytes --]

Am Donnerstag, den 26.08.2004, 21:32 +0100 schrieb :

> Argh...  OK, now I remember why I went for -EBUSY for unlink() (we obviously
> are not bound by SuS on that one).  Consider the following scenario:
> 	* local file foo got something else bound on it for a while
> 	* we are tight on space - time to clean up
> 	* oh, look - contents of foo is junk
> 	* rm foo
> 	* ... oh, fuck, there goes the underlying file.

Right. Good thinking. You can't delete a directory while there regular
files in it either.

Another question: /mnt/test is a mountpoint. Why can I rename `mnt' but
can't rename `test'?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> Argh...  OK, now I remember why I went for -EBUSY for unlink() (we obviously
> are not bound by SuS on that one).  Consider the following scenario:
> 	* local file foo got something else bound on it for a while
> 	* we are tight on space - time to clean up
> 	* oh, look - contents of foo is junk
> 	* rm foo
> 	* ... oh, fuck, there goes the underlying file.

Hey, that's a valid reason for doing -EBUSY for normal bind-mounts, but it
actually _is_ what we want for an "implied-by-way-of-container-mount". 
After all, when you do a "rm foo", you do mean "remove the container foo".

I replied to your earlier off-list mail in private, so let's re-iterate
for the list: the easiest way to handle this is to just have a "mount
option", and have "MNT_ALLOWUNLINK" that gets set for containers, and that
users could possibly choose to set for regular mounts too (as a mount
option) if they really want to (and if we want them to).

So there's no reason we'd have to drop existing mount behaviour only 
because we also have special files that look like mountpoints. 

		Linus

[some sanity for a change] possible design issues for hybrids

[viro]

[subject changed and *please* let's keep the wankfest out of that branch;
we are talking about possible ways to handle hybrids, *NOT* their desirability,
effect on DARPA funding, size of KDE developers' genitals experienced by
Aunt Tillie, etc.]

On Thu, Aug 26, 2004 at 01:47:37PM -0700, Linus Torvalds wrote:
> Hey, that's a valid reason for doing -EBUSY for normal bind-mounts, but it
> actually _is_ what we want for an "implied-by-way-of-container-mount". 
> After all, when you do a "rm foo", you do mean "remove the container foo".
> 
> I replied to your earlier off-list mail in private, so let's re-iterate
> for the list: the easiest way to handle this is to just have a "mount
> option", and have "MNT_ALLOWUNLINK" that gets set for containers, and that
> users could possibly choose to set for regular mounts too (as a mount
> option) if they really want to (and if we want them to).
> 
> So there's no reason we'd have to drop existing mount behaviour only 
> because we also have special files that look like mountpoints. 

All right, let's see where that would take us.

1) we would need to find all vfsmounts over given dentry.  Probably a cyclic
list (we want to check if there are normal mounts/bindings among those and
we want to dissolve them if there's none).

2) we would need to do something about locking, since mount trees in other
guys' namespaces are protected by semaphores of their own.

3) what do we do on umount(2)?  We can get a bunch of vfsmounts hanging off
it.  MNT_DETACH will have no problems, but normal umount() is a different
story.  Note that it's not just hybrid-related problem - implementing the
mount traps will cause the same kind of trouble,

4) OK, we have those hybrids and want to create vfsmounts when crossing a
mountpoint.  When do they go away, anyway?  When we don't reference them
anymore?  Right now "attached to mount tree" == "+1 to refcount" and detaching
happens explicitly - outside of the "dropping the final reference" path.
Might become a locking issue.

5) Creation of these vfsmounts: fs should somehow tell us whether it wants
one or not (at the very least, we should stop *somewhere*).  Can we use
the same dentry/inode?  I'm not sure and I really doubt that we'd like that.

6) if it's a method, where should it live, *especially* if we want them on
device nodes.  Note that inode_operations belongs to underlying fs, so it's
not particulary good place for device case.

7) automount folks want partially shared mount trees (well, mirrored,
actually).  The basic idea is that while namespace boundary is a trust
boundary, we might want to be able to say "I trust this guy to handle
that subtree under /home/stuff/foobar/mounts".  It's the same situation
as with shared mappings (I want separate address space, but I'm willing
to share that chunk of memory with other process), except that we want
it to be allowed to become asymmetric (shared r/o mapping with somebody
else having it r/w).   That stuff (and mount traps) is the next pending
major work on the mount trees.  We probably want hybrids work to go with it,
since they affect the same data structures and need more or less the same
changes.
[And yes, this is the open season on design discussions for shared subtrees -
automount folks are welcome to join]

8) what should happen when something is mounted on top of directory-over-file?
How do we treat such beasts?  What are the implications?

9) how do we recognize such mountpoints in the path lookups?  It *is* a
hot path, so we should be careful in that area; the impact will be felt
by everything in the system.

10) how do we deal with directories, anyway?  Mixing "attributes" with
normal directory contents is going to be fun, what with lseek() insanity.
That's not an issue for hybrids, but it is one for anybody who wants
any sort of common metadata exported that way.  Note that it's really
important for fs writers - having two different pieces of code to export
the same information (for directories and non-directories resp.) is going
to become a prime breeding ground for bugs.

11) if we go for your "here's stuff that belongs in device node viewed
as directory", how would that play with fs metadata exporters?  Again,
due to the insanity of lseek() on directories it's *very* hard to deal
with unions, when parts of directory come from different chunks of code.

That's it for starters.  Technical answers/questions/comments are welcome.
Generic masturbation => over there in the parent thread, please.

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

[ This is quite possibly just impossible and buggy, but here's my
  implementation notes. You asked for them. ]

On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> All right, let's see where that would take us.
> 
> 1) we would need to find all vfsmounts over given dentry.  Probably a cyclic
> list (we want to check if there are normal mounts/bindings among those and
> we want to dissolve them if there's none).

Not per-inode? dentries are a bitmore memory-constrained than inodes, and 
we only need this for filesystems that want to support it, so we wouldn't 
need to put this information in each dentry.

Since the vfsmounts have back-pointers to the dentry they are mounted on,
you can still do a "per-dentry" traversal, by just doign the inode list
and checking the dentry pointer. No?

Alternatively, we could just put the list on an external hash-chain 
entirely, and hash off the dentry. It depends on how often we end up 
needing it.

Or we could just put it in the dentry itself. I'd hate to make it any 
bigger than it already is, but maybe it doesn't matter that much.

> 2) we would need to do something about locking, since mount trees in other
> guys' namespaces are protected by semaphores of their own.

Ok, I'll admit that I don't know how to handle namespaces. These things 
should just go into a global namespace, and I was kind of assuming it 
would happen automatically in "lookup_mnt()" or something like that. A 
special case in lookup_mnt which says something like "if you didn't find a 
vfsmount, we create a new one for you".

It should be reasonably easy to create new ones on-the-fly, since we'd
have all the information (the parent vfsmount comes stated, and the
vfsmount we create would point to the same things that the "base" one
would).

> 3) what do we do on umount(2)?  We can get a bunch of vfsmounts hanging off
> it.  MNT_DETACH will have no problems, but normal umount() is a different
> story.  Note that it's not just hybrid-related problem - implementing the
> mount traps will cause the same kind of trouble,

Don't allow umount. It's not something the user can unmount - the mount is 
"implied" in the file. 

> 4) OK, we have those hybrids and want to create vfsmounts when crossing a
> mountpoint.  When do they go away, anyway?  When we don't reference them
> anymore?  Right now "attached to mount tree" == "+1 to refcount" and detaching
> happens explicitly - outside of the "dropping the final reference" path.
> Might become a locking issue.

Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously 
clossed something over when I blathered about the "create the vfsmount on 
the fly" thing above ;)

> 5) Creation of these vfsmounts: fs should somehow tell us whether it wants
> one or not (at the very least, we should stop *somewhere*).  Can we use
> the same dentry/inode?  I'm not sure and I really doubt that we'd like that.

Why not? When doing the ->lookup() operation, the filesystem would create
the vfsmount and bind it to the current vfsmount. That guarantees that it
has a vfsmount, and will mean that it will show up positive with the
"d_mountpoint()" query, which in turn will cause us to do the
"lookup_mnt()".

Which in turn will create the other vfsmounts as needed, if you have 
multiple namespaces.

So I _think_ creation is easy. Getting rid of the dang things migth be 
harder.

> 6) if it's a method, where should it live, *especially* if we want them on
> device nodes.  Note that inode_operations belongs to underlying fs, so it's
> not particulary good place for device case.

Why not just let the existing .lookup method initialize the mount-point 
thing? After that, it's all in the VFS layer (I'd hate to have filesystems 
mess around with vfsmounts - they'll just get it wrong).

> 7) automount folks want partially shared mount trees (well, mirrored,
> actually).

I don't think you can get partial sharing on one of these puppies. You'd 
always have one vfsmount per namespace (well, lazily created, so maybe in 
practice you'd see a lot fewer).

> 8) what should happen when something is mounted on top of directory-over-file?
> How do we treat such beasts?  What are the implications?

Allow file-on-file mounts - it will just totally hide the thing (in that
namespace, at least). But don't allow the dir-on-file thing (that we
already don't allow).

> 9) how do we recognize such mountpoints in the path lookups?  It *is* a
> hot path, so we should be careful in that area; the impact will be felt
> by everything in the system.

I don't think you'll have any special cases. Same d_mountpount(), same 
lookup_mnt().

> 10) how do we deal with directories, anyway?  Mixing "attributes" with
> normal directory contents is going to be fun, what with lseek() insanity.

You couldn't get at the attributes that way anyway, so I think the point 
is moot. The "real" directory always takes over.

Crazy people could try to just use the regular "xattrs" interfaces if they 
really want attributes on directories. You wouldn't ever be able to use 
the "easy" one.

> 11) if we go for your "here's stuff that belongs in device node viewed
> as directory", how would that play with fs metadata exporters?  Again,
> due to the insanity of lseek() on directories it's *very* hard to deal
> with unions, when parts of directory come from different chunks of code.

Don't go there. See above. Directories would be just plain directories, 
you could never see their metadata. Same goes for at least symlinks, and 
possibly other filetypes too (ie at least initially, a block or character 
special device will just take over the whole "file_operations", which 
includes "readdir", so it's actually hard to have the filesystem do 
anything about those).

		Linus

Re: [some sanity for a change] possible design issues for hybrids

[viro]

On Thu, Aug 26, 2004 at 03:04:21PM -0700, Linus Torvalds wrote:
 
> > 2) we would need to do something about locking, since mount trees in other
> > guys' namespaces are protected by semaphores of their own.
> 
> Ok, I'll admit that I don't know how to handle namespaces. These things 
> should just go into a global namespace, and I was kind of assuming it 
> would happen automatically in "lookup_mnt()" or something like that. A 
> special case in lookup_mnt which says something like "if you didn't find a 
> vfsmount, we create a new one for you".
> 
> It should be reasonably easy to create new ones on-the-fly, since we'd
> have all the information (the parent vfsmount comes stated, and the
> vfsmount we create would point to the same things that the "base" one
> would).

Erm...  What do we do upon unlink()?  I'm killing a file, fs it's in is
mounted in a dozen of places (no namespaces, just chroot jails, whatever).
We need to find all vfsmounts to be killed by that.

And BTW that's an argument against anchoring that list in inode - unlink()
on foo should not screw bar/... even if bar and foo are links to the same
file.  So we'll need to check for dentry match anyway.
 
> > 3) what do we do on umount(2)?  We can get a bunch of vfsmounts hanging off
> > it.  MNT_DETACH will have no problems, but normal umount() is a different
> > story.  Note that it's not just hybrid-related problem - implementing the
> > mount traps will cause the same kind of trouble,
> 
> Don't allow umount. It's not something the user can unmount - the mount is 
> "implied" in the file. 

See below.

> > 4) OK, we have those hybrids and want to create vfsmounts when crossing a
> > mountpoint.  When do they go away, anyway?  When we don't reference them
> > anymore?  Right now "attached to mount tree" == "+1 to refcount" and detaching
> > happens explicitly - outside of the "dropping the final reference" path.
> > Might become a locking issue.
> 
> Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously 
> clossed something over when I blathered about the "create the vfsmount on 
> the fly" thing above ;)

> > 5) Creation of these vfsmounts: fs should somehow tell us whether it wants
> > one or not (at the very least, we should stop *somewhere*).  Can we use
> > the same dentry/inode?  I'm not sure and I really doubt that we'd like that.
> 
> Why not? When doing the ->lookup() operation, the filesystem would create
> the vfsmount and bind it to the current vfsmount. That guarantees that it
> has a vfsmount, and will mean that it will show up positive with the
> "d_mountpoint()" query, which in turn will cause us to do the
> "lookup_mnt()".

Several paragraphs below you are saying that you don't like fs messing with
vfsmounts.  Use of ->lookup() would mean that we should not only create
and attach vfsmounts from within fs code, but would actually have to make
->lookup() return vfsmount+dentry, AFAICS.
 
> > 6) if it's a method, where should it live, *especially* if we want them on
> > device nodes.  Note that inode_operations belongs to underlying fs, so it's
> > not particulary good place for device case.
> 
> Why not just let the existing .lookup method initialize the mount-point 
> thing? After that, it's all in the VFS layer (I'd hate to have filesystems 
> mess around with vfsmounts - they'll just get it wrong).

> Allow file-on-file mounts - it will just totally hide the thing (in that
> namespace, at least). But don't allow the dir-on-file thing (that we
> already don't allow).

Err...  What about dir-on-dir-that-is-on-file?  I.e. mount on foo/. when foo
is a file?
 
> > 9) how do we recognize such mountpoints in the path lookups?  It *is* a
> > hot path, so we should be careful in that area; the impact will be felt
> > by everything in the system.

> I don't think you'll have any special cases. Same d_mountpount(), same 
> lookup_mnt().

See above on use ->lookup()

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> > 
> > It should be reasonably easy to create new ones on-the-fly, since we'd
> > have all the information (the parent vfsmount comes stated, and the
> > vfsmount we create would point to the same things that the "base" one
> > would).
> 
> Erm...  What do we do upon unlink()?  I'm killing a file, fs it's in is
> mounted in a dozen of places (no namespaces, just chroot jails, whatever).
> We need to find all vfsmounts to be killed by that.

But that should be trivial: that's what the per-inode vfsmount list was 
(your first question in the last email).

> And BTW that's an argument against anchoring that list in inode - unlink()
> on foo should not screw bar/... even if bar and foo are links to the same
> file.  So we'll need to check for dentry match anyway.

And again - I talked about this in the previous email. Even if you anchor 
the list in "struct inode", or you do it with a totally external 
hash-list, you'll always have the "vfsmount->mnt_mountpoint" pointer to 
point to the dentry. So you can just iterate over the list, and 
cherry-pick the ones that point to the dentry you are removing.

>  
> > > 3) what do we do on umount(2)?  We can get a bunch of vfsmounts hanging off
> > > it.  MNT_DETACH will have no problems, but normal umount() is a different
> > > story.  Note that it's not just hybrid-related problem - implementing the
> > > mount traps will cause the same kind of trouble,
> > 
> > Don't allow umount. It's not something the user can unmount - the mount is 
> > "implied" in the file. 
> 
> See below.
> 
> > > 4) OK, we have those hybrids and want to create vfsmounts when crossing a
> > > mountpoint.  When do they go away, anyway?  When we don't reference them
> > > anymore?  Right now "attached to mount tree" == "+1 to refcount" and detaching
> > > happens explicitly - outside of the "dropping the final reference" path.
> > > Might become a locking issue.
> > 
> > Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously 
> > clossed something over when I blathered about the "create the vfsmount on 
> > the fly" thing above ;)
> 
> > > 5) Creation of these vfsmounts: fs should somehow tell us whether it wants
> > > one or not (at the very least, we should stop *somewhere*).  Can we use
> > > the same dentry/inode?  I'm not sure and I really doubt that we'd like that.
> > 
> > Why not? When doing the ->lookup() operation, the filesystem would create
> > the vfsmount and bind it to the current vfsmount. That guarantees that it
> > has a vfsmount, and will mean that it will show up positive with the
> > "d_mountpoint()" query, which in turn will cause us to do the
> > "lookup_mnt()".
> 
> Several paragraphs below you are saying that you don't like fs messing with
> vfsmounts.  Use of ->lookup() would mean that we should not only create
> and attach vfsmounts from within fs code, but would actually have to make
> ->lookup() return vfsmount+dentry, AFAICS.

No, lookup would just return the dentry, but the dentry would already be 
filled in with the mount-point information.

And you can do that with a simple vfs helper function, ie the filesystem 
itself would just need to do

	pseudo_mount(dentry, inode);

thing - which just fills in dentry->d_mountpoint with a new vfsmount
thing. It would allocate a new root dentry (for the pseudo-mount) and a
new vfsmount, and make dentry->d_mountpoint point to it.

IOW, the filesystem itself would never mess around with d_mountpoint 
itself.

> Err...  What about dir-on-dir-that-is-on-file?  I.e. mount on foo/. when foo
> is a file?

Hmm.. We might as well allow it, I suspect. It's not like it should hurt.  
We'd end up following the mount-chain twice, but we already have that
issue with multi-mount cases..

		Linus

Re: [some sanity for a change] possible design issues for hybrids

[viro]

On Thu, Aug 26, 2004 at 03:45:09PM -0700, Linus Torvalds wrote:
> No, lookup would just return the dentry, but the dentry would already be 
> filled in with the mount-point information.
> 
> And you can do that with a simple vfs helper function, ie the filesystem 
> itself would just need to do
> 
> 	pseudo_mount(dentry, inode);
> 
> thing - which just fills in dentry->d_mountpoint with a new vfsmount
> thing. It would allocate a new root dentry (for the pseudo-mount) and a
> new vfsmount, and make dentry->d_mountpoint point to it.

What dentry->d_mountpoint?  No such thing...

Note that we can't get vfsmount by dentry - that's the point of having these
guys in the first place.  So I'm not sure what you are trying to do here -
dentry + inode is definitely not enough to attach any vfsmounts anywhere.

That's not about namespaces - same fs mounted in several places will give
the same problem - one dentry, many vfsmounts.  And we obviously *can't*
have one vfsmount for all of them - if the same fs is mounted on /foo and
/bar, we will have the same dentry for /foo/splat and /bar/splat.  So
what should we get for /foo/splat/. and /bar/splat/.?  Same dentry *and*
same vfsmount?  I'd expect .. from the former to give /foo and from the
latter - /bar...

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> 
> What dentry->d_mountpoint?  No such thing...

Sorry - set "dentry->d_mounted++" + "add vfsmount/dentry to hashes".

Yes, it's not a direct list off the dentry, but it effectively is the same
thing.

So basically: the "d_mounted++" just makes sure we get into
"lookup_mnt()". That's where we will usually find the actual mount thing.

And that's also where the special case comes in: if we _don't_ find the 
mount thing there, that's where we need to create it. That will only 
happen if somebody looks it up using another namespace, though, so it 
should be rare.

And when it does happen, we can just create a new vfsmount - we have all
the information there (we'll have to walk the per-inode-or-whatever
vfsmount list to find all the information to populate the thing with, of
course. But we need that list _anyway_, so it should be a fairly 
straightforward special case).

			Linus

Re: [some sanity for a change] possible design issues for hybrids

[viro]

On Thu, Aug 26, 2004 at 04:24:51PM -0700, Linus Torvalds wrote:
> So basically: the "d_mounted++" just makes sure we get into
> "lookup_mnt()". That's where we will usually find the actual mount thing.
> 
> And that's also where the special case comes in: if we _don't_ find the 
> mount thing there, that's where we need to create it. That will only 
> happen if somebody looks it up using another namespace, though, so it 
> should be rare.

No.  Trivial example:

mount --bind /foo /bar
mount /dev/sda1 /bar/baz

do lookup for /foo/baz.  No namespaces involved, no vfsmounts found, d_mounted
positive and we certainly do *not* want anything to be created at that point.

Re: [some sanity for a change] possible design issues for hybrids

[Linus Torvalds]

On Fri, 27 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:

> On Thu, Aug 26, 2004 at 04:24:51PM -0700, Linus Torvalds wrote:
> > So basically: the "d_mounted++" just makes sure we get into
> > "lookup_mnt()". That's where we will usually find the actual mount thing.
> > 
> > And that's also where the special case comes in: if we _don't_ find the 
> > mount thing there, that's where we need to create it. That will only 
> > happen if somebody looks it up using another namespace, though, so it 
> > should be rare.
> 
> No.  Trivial example:
> 
> mount --bind /foo /bar
> mount /dev/sda1 /bar/baz
> 
> do lookup for /foo/baz.  No namespaces involved, no vfsmounts found, d_mounted
> positive and we certainly do *not* want anything to be created at that point.

Right. We obviously need to mark the dentry somehow, and only do the 
"create vfsmount" special case in this special case. If we didn't do that, 
then it wouldn't be a special case, now would it?

So clearly lookup_mnt() needs to check the dentry in the failure case. The 
marking could be in any of three places
 - mark the dentry itself by just using a dentry flag ("DCACHE_AUTOVFSMNT")
   or by having a dentry operation for this.
 - mark the inode itself (same logic as dentry)
 - look up the first vfsmount (on the inode list), and look if that one is 
   of the automatic type.

Clearly we should not _always_ create a vfsmount, that would just break 
the existign logic.

		Linus

Re: [some sanity for a change] possible design issues for hybrids

[viro]

On Thu, Aug 26, 2004 at 04:57:01PM -0700, Linus Torvalds wrote:
> Right. We obviously need to mark the dentry somehow, and only do the 
> "create vfsmount" special case in this special case. If we didn't do that, 
> then it wouldn't be a special case, now would it?
> 
> So clearly lookup_mnt() needs to check the dentry in the failure case. The 
> marking could be in any of three places
>  - mark the dentry itself by just using a dentry flag ("DCACHE_AUTOVFSMNT")
>    or by having a dentry operation for this.
>  - mark the inode itself (same logic as dentry)
>  - look up the first vfsmount (on the inode list), and look if that one is 
>    of the automatic type.
> 
> Clearly we should not _always_ create a vfsmount, that would just break 
> the existign logic.

Hmm...  IOW, you are suggesting to use normal trigger for lookup_mnt() and
then have extra check in lookup_mnt() failure exit.  That will probably
work (and I'd prefer to mark dentry here).

Freeing these guys will not be fun.  The final reference could be given
up under very different locking conditions - it's certainly a blocking
operation (it can trigger final umount, among other things), but having
it trigger removal from mount tree(s) can get ugly.  We might be able to
tweak that, but it will probably need require major surgery in several
places.

Right now we assume that vfsmount lock *and* per-namespace semaphore are
held when detaching from the tree (attaching what could be the first at
given dentry --- same + i_sem on its ->i_sem).  We also assume that vfsmount is
detached before refcount reaches zero.  The thing is, that final refcount
can be dropped while holding namespace semaphore.  It's not fatal (we'll
need to hold onto these guys until dropping that semaphore or add an analog
of mntput() for places under namespace sem), but it can get messy.

Situation with unlink(): we need to (a) check that all vfsmounts over
given point are "hybrid" ones (locking is not a problem at that point)
and (b) after successful operation (unlink() and rename() alike, AFAICS)
we want to go and kill everything under the victim dentry.  *That*
can get very ugly, since fs could be mounted in a lot of places and/or
in a lot of namespaces.

Note that we also need to do in VFS what NFS tries to do in revalidation -
if dentry is invalidated, everything under it should be detached and
dropped.  So it's not something new - we do need something that would
be called outside of any superblock/namespace/vfsmount locks (both
unlink() and revalidation qualify) and would kill the vfsmounts that
become unreachable.

Locking rules for attaching might need adjustment (the logics with ->i_sem
is that we want !d_mountpoint() stay true around call of ->unlink(),
->rmdir() and ->rename(); so cloning or replacing doesn't need that
protection.  Now it might become interesting - we'll need to be careful
in pivot_root(2) and possibly some other places).

I'll see what can be done (it'll definitely take careful looking at the
code), but there is a chance of things getting very ugly.

BTW, shared subtrees *are* compatible with that puppy - there's a lot
of corner cases to look at, but AFAICS they are all doable.  Lifetime
rules can get ugly in some of them, but that's on par with the situation
without sharing.

One thing that looks like a bad interface: we get forcible "use same
dentry for file and directory" with that design.  That can lead to
a big can of worms - without a large audit I wouldn't bet a dime on
ability to get that right (and same applies to reiser4 code as-is,
for the same reasons).  No idea how bad it will turn out to be - right
now all I can say is that we might run into trouble and that we almost
certainly will get extra "be careful not to do <list of things>" rules
out of that.

Re: [some sanity for a change] possible design issues for hybrids

[Anton Altaparmakov]

On Thu, 2004-08-26 at 23:04, Linus Torvalds wrote:
> [ This is quite possibly just impossible and buggy, but here's my
>   implementation notes. You asked for them. ]
> On Thu, 26 Aug 2004 viro@parcelfarce.linux.theplanet.co.uk wrote:
> > All right, let's see where that would take us.
> > 3) what do we do on umount(2)?  We can get a bunch of vfsmounts hanging off
> > it.  MNT_DETACH will have no problems, but normal umount() is a different
> > story.  Note that it's not just hybrid-related problem - implementing the
> > mount traps will cause the same kind of trouble,
> 
> Don't allow umount. It's not something the user can unmount - the mount is 
> "implied" in the file. 
> 
> > 4) OK, we have those hybrids and want to create vfsmounts when crossing a
> > mountpoint.  When do they go away, anyway?  When we don't reference them
> > anymore?  Right now "attached to mount tree" == "+1 to refcount" and detaching
> > happens explicitly - outside of the "dropping the final reference" path.
> > Might become a locking issue.
> 
> Ahh. Umm.. Yes. I think this might be the real problem. Unless I seriously 
> clossed something over when I blathered about the "create the vfsmount on 
> the fly" thing above ;)

If it is created on the fly, it should be "easy" to destroy on the fly
using time-based expiry, i.e. a kernel daemon going over all of those
beasts every X seconds (X = 5 perhaps?) and doing something like:

for (each vfsmount) {
	lock_vfsmount(vfsmount);
	if (MOUNT_IS_BUSY(vfsmount)) {
		unlock_vfsmount(vfsmount);
		continue;
	}
	if (current_time() < (vfsmount->last_used_time +
			vfsmount->expire_after)) {
		unlock_vfsmount(vfsmount);
		continue;
	}
	destroy_locked_vfsmount(vfsmount);
}

Wouldn't that work?

> > 10) how do we deal with directories, anyway?  Mixing "attributes" with
> > normal directory contents is going to be fun, what with lseek() insanity.
> 
> You couldn't get at the attributes that way anyway, so I think the point 
> is moot. The "real" directory always takes over.
> 
> Crazy people could try to just use the regular "xattrs" interfaces if they 
> really want attributes on directories. You wouldn't ever be able to use 
> the "easy" one.

But that defeats the whole point of the hybrid objects!  We might as
well just keep the xattrs interface and throw away the new one if we
will have to keep the old one anyway so that we can do named
streams/attributes on directories.  Windows (and other OS?) certainly do
allow them and do use them on directories as well as files...

> > 11) if we go for your "here's stuff that belongs in device node viewed
> > as directory", how would that play with fs metadata exporters?  Again,
> > due to the insanity of lseek() on directories it's *very* hard to deal
> > with unions, when parts of directory come from different chunks of code.
> 
> Don't go there. See above. Directories would be just plain directories, 
> you could never see their metadata. Same goes for at least symlinks, and 
> possibly other filetypes too (ie at least initially, a block or character 
> special device will just take over the whole "file_operations", which 
> includes "readdir", so it's actually hard to have the filesystem do 
> anything about those).

Ah, but at least in other OS (Windows is the one I know about)
directories _also_ have the same semantics as files with respect to
named streams/attributes.

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/

Re: [some sanity for a change] possible design issues for hybrids

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 09:56:39AM +0100, Anton Altaparmakov wrote:
> If it is created on the fly, it should be "easy" to destroy on the fly
> using time-based expiry, i.e. a kernel daemon going over all of those
> beasts every X seconds (X = 5 perhaps?) and doing something like:
> 
> for (each vfsmount) {
> 	lock_vfsmount(vfsmount);
> 	if (MOUNT_IS_BUSY(vfsmount)) {
> 		unlock_vfsmount(vfsmount);
> 		continue;
> 	}
> 	if (current_time() < (vfsmount->last_used_time +
> 			vfsmount->expire_after)) {
> 		unlock_vfsmount(vfsmount);
> 		continue;
> 	}
> 	destroy_locked_vfsmount(vfsmount);
> }
> 
> Wouldn't that work?

That would work for a low number of them.  But with Hans' "visions" we'd
have a damn lot of them at which point this isn't really scalable.

Re: silent semantic changes with reiser4

[Jonathan Abbey]

[-- Attachment #1: Type: text/plain, Size: 930 bytes --]

On Thu, Aug 26, 2004 at 09:28:41PM +0300, Denis Vlasenko wrote:
|
| Is it possible to sufficiently hide "dirs inside files"
| so that old tools will be unable to see them?
| 
| I just checked:
| 
| ls -d /foo  does lstat64("/foo", ...)
| ls -d /foo/ does lstat64("/foo", ...)
| 	but
| ls -d /foo/. does lstat64("/foo/.", ...)
| 
| Will it work out if "dir inside file" will only be visible when referred as "file/."?

I'm used to using ls symlink/. to get ls to show me the directory on
the far side of a symbolic link.  That's a pretty analagous case to
the one we're discussing here, I think?

 Jon

-- 
-------------------------------------------------------------------------------
Jonathan Abbey 				              jonabbey@arlut.utexas.edu
Applied Research Laboratories                 The University of Texas at Austin
GPG Key: 71767586 at keyserver pgp.mit.edu, http://www.ganymeta.org/workkey.gpg

[-- Attachment #2: Type: application/pgp-signature, Size: 185 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Jonathan Abbey wrote:
> | Will it work out if "dir inside file" will only be visible when
> referred as "file/."?
> 
> I'm used to using ls symlink/. to get ls to show me the directory on
> the far side of a symbolic link.  That's a pretty analagous case to
> the one we're discussing here, I think?

By the way, do symlinks have metadata?  Where do you find it? :)

-- Jamie

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 733 bytes --]

Am Donnerstag, den 26.08.2004, 21:16 +0100 schrieb Jamie Lokier:

> > | Will it work out if "dir inside file" will only be visible when
> > referred as "file/."?
> > 
> > I'm used to using ls symlink/. to get ls to show me the directory on
> > the far side of a symbolic link.  That's a pretty analagous case to
> > the one we're discussing here, I think?
> 
> By the way, do symlinks have metadata?  Where do you find it? :)

Oops. :)

At least in reiser4 they don't have, or at least you can't access them.
ln -s foo bar; cd bar/metas shows me the content of foo/metas.

symlinks are special anyway. Their rights are 777. The only thing they
can have is an owner. No chance to do a symlink/metas/uid then. Hmm.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Nikita Danilov]

Christophe Saout writes:
 > Am Donnerstag, den 26.08.2004, 21:16 +0100 schrieb Jamie Lokier:
 > 
 > > > | Will it work out if "dir inside file" will only be visible when
 > > > referred as "file/."?
 > > > 
 > > > I'm used to using ls symlink/. to get ls to show me the directory on
 > > > the far side of a symbolic link.  That's a pretty analagous case to
 > > > the one we're discussing here, I think?
 > > 
 > > By the way, do symlinks have metadata?  Where do you find it? :)
 > 
 > Oops. :)
 > 
 > At least in reiser4 they don't have, or at least you can't access them.

They do.

 > ln -s foo bar; cd bar/metas shows me the content of foo/metas.

That's because lookup for "bar" performs symlink resolution.
 
 > 
 > symlinks are special anyway. Their rights are 777. The only thing they

Symlink is represented by normal inode in UNIX. It has full set of
attributes. It can even be hard-linked.

 > can have is an owner. No chance to do a symlink/metas/uid then. Hmm.
 > 

Nikita.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 453 bytes --]

Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:

>  > At least in reiser4 they don't have, or at least you can't access them.
> 
> They do.
> 
>  > ln -s foo bar; cd bar/metas shows me the content of foo/metas.
> 
> That's because lookup for "bar" performs symlink resolution.

So I can't access them and it is pointless. ;-)

BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
this makes sense. :)


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Emil Larsson]

Christophe Saout wrote:

>Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:
>
>  
>
>> > At least in reiser4 they don't have, or at least you can't access them.
>>
>>They do.
>>
>> > ln -s foo bar; cd bar/metas shows me the content of foo/metas.
>>
>>That's because lookup for "bar" performs symlink resolution.
>>    
>>
>
>So I can't access them and it is pointless. ;-)
>
>BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
>this makes sense. :)
>
>  
>
I prefer this one:

hepburn:/# mount
/dev/sda3 on / type reiser4 (rw)
hepburn:/# touch apa
hepburn:/# chmod +x apa
hepburn:/# cd apa
hepburn:/apa# ls
ls: reading directory .: Not a directory

So there is already file-as-dir with legacy "ls" - and it works just 
like people seem to want it to do with legacy apps. Not at all.

/Emil

Re: silent semantic changes with reiser4

[Nikita Danilov]

Christophe Saout writes:
 > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:
 > 
 > >  > At least in reiser4 they don't have, or at least you can't access them.
 > > 
 > > They do.
 > > 
 > >  > ln -s foo bar; cd bar/metas shows me the content of foo/metas.
 > > 
 > > That's because lookup for "bar" performs symlink resolution.
 > 
 > So I can't access them and it is pointless. ;-)
 > 
 > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
 > this makes sense. :)

Why? foo/metas is a file system object just like foo. It has owner,
permission bits, so access to its meta-data should be provided, and
uniform way to provide access to the file system object meta-data is to
have these little magic files inside metas directory, which is a file
system object just like metas. It has owner^@^@^@^@*** - Lisp stack
overflow. RESET

Nikita.

 > 

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1323 bytes --]

Am Freitag, den 27.08.2004, 02:21 +0400 schrieb Nikita Danilov:

>  > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
>  > this makes sense. :)
> 
> Why? foo/metas is a file system object just like foo. It has owner,
> permission bits, so access to its meta-data should be provided, and
> uniform way to provide access to the file system object meta-data is to
> have these little magic files inside metas directory, which is a file
> system object just like metas. It has owner^@^@^@^@*** - Lisp stack
> overflow. RESET

Yes, of course. But who's going to need that? The meta files are there
to describe regular unix files. They're just exported using a file
system name space. I don't think we need metadata to describe metadata.
Assuming someone adds a directory inside a file where user can add some
attributes/streams/resource forks. As Linus said, they are just
attributes, even if they seem to behave like normal files, they're not,
they belong to the real file. Adding attributes to attributes to
attributes? That would be like adding ACLS to ACLS. Hello the
madness. ;-)

There really should be a point where you can't recurse further. Assume
there is a backup application that always tries to open every dentry it
finds as directory and then recurse. Oops. :-)


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Hans Reiser]

Nikita Danilov wrote:

>Christophe Saout writes:
> > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:
> > 
> > >  > At least in reiser4 they don't have, or at least you can't access them.
> > > 
> > > They do.
> > > 
> > >  > ln -s foo bar; cd bar/metas shows me the content of foo/metas.
> > > 
> > > That's because lookup for "bar" performs symlink resolution.
> > 
> > So I can't access them and it is pointless. ;-)
> > 
> > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I don't think
> > this makes sense. :)
>
>Why? foo/metas is a file system object just like foo. It has owner,
>permission bits, so access to its meta-data should be provided, and
>uniform way to provide access to the file system object meta-data is to
>have these little magic files inside metas directory, which is a file
>system object just like metas. It has owner^@^@^@^@*** - Lisp stack
>overflow. RESET
>
>Nikita.
>
> > 
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>
I think Christophe is a bit right here.  While in general having 
meta-meta objects makes sense, in this particular instance, I don't see 
the functional need for it.  Can you supply an example of where it would 
be useful?

Re: silent semantic changes with reiser4

[Stephen Wille Padnos]

Hans Reiser wrote:

> Nikita Danilov wrote:
>
>> Christophe Saout writes:
>> > Am Freitag, den 27.08.2004, 01:45 +0400 schrieb Nikita Danilov:
>> > > >  > At least in reiser4 they don't have, or at least you can't 
>> access them.
>> > > > > They do.
>> > > > >  > ln -s foo bar; cd bar/metas shows me the content of 
>> foo/metas.
>> > > > > That's because lookup for "bar" performs symlink resolution.
>> > > So I can't access them and it is pointless. ;-)
>> > > BTW, I can do a cd metas/metas/metas/metas/plugin/metas... I 
>> don't think
>> > this makes sense. :)
>>
>> Why? foo/metas is a file system object just like foo. It has owner,
>> permission bits, so access to its meta-data should be provided, and
>> uniform way to provide access to the file system object meta-data is to
>> have these little magic files inside metas directory, which is a file
>> system object just like metas. It has owner^@^@^@^@*** - Lisp stack
>> overflow. RESET
>>
>> Nikita.
>>
>> > -
>> To unsubscribe from this list: send the line "unsubscribe 
>> linux-kernel" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> Please read the FAQ at  http://www.tux.org/lkml/
>>
>>
>>  
>>
> I think Christophe is a bit right here.  While in general having 
> meta-meta objects makes sense, in this particular instance, I don't 
> see the functional need for it.  Can you supply an example of where it 
> would be useful? 

One example is to supply hints about what to do with the attribute if 
the file is copied / moved.
Several types of attributes have been mentioned:

1) Those that are essentially a cached object, which can be re-derived 
from the source file (e.g. thumbnail image)
2) Those that become invalid if the source file changes (e.g. hash value)
3) Those that should be maintained on file copy (e.g. author, copyright, 
etc)
4) Those that may not want to be copied with the file (e.g. security 
keys or the like)
etc.

This could be accomplished without meta-meta information (like 
permissions), but that wouldn't be very extensible, would it :)

- Steve

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>
>  
>
>No no. The stream you get from a directory is totally _independent_ of the 
>contents of the directory.
>
but the user can link filename/metas/body to filename/metas/tar, if we 
implement a tar plugin, if it happens that the user wants it to be 
totally dependent.

> Anything else would be pointless.
>
>		Linus
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>

Re: silent semantic changes with reiser4

[Diego Calleja]

El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió:

> So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

/bin could be separated (like linus said) but cat /bin/.compound could do
it. This is the /etc/passwd Hans' example, I think:

"If /new_syntax_access_path/big_directory_of_small_files/.glued is a plugin that
aggregates every file in big_directory_of_small_files with a delimiter
separating every file within the aggregation, then one can simply type emacs
/new_syntax_access_path/big_directory_of_small_files/.glued, and the filesystem
has done all the work emacs needs to be effective at this. Not a line of emacs
needs to be changed."

Re: silent semantic changes with reiser4

[Christophe Saout]

On Thu, Aug 26, 2004 at 07:40:10PM +0200, Diego Calleja wrote:

> > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> 
> /bin could be separated (like linus said) but cat /bin/.compound could do
> it. This is the /etc/passwd Hans' example, I think:

Yes, but what about locking? If we have compound files with
individually accessible components a lock on the compound files
could lock its components at the same time.

But what with /usr/.compound? Is it read-only? What if you truncate it?
Does it atomically delete the whole /usr directory tree? What about reading?
It has to go through all these independent files, what if they change while
you are reading the compound file? Does it lock everything? Point-in-time
snapshot? If reiser4 can't do this. ;-)

I don't say that it's impossible to do but I think it's a lot harder than
doing it the other way round.

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Diego Calleja wrote:
> El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <riel@redhat.com> escribió:
> 
> > So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
> 
> /bin could be separated (like linus said) but cat /bin/.compound could
> do it. This is the /etc/passwd Hans' example, I think:

Arghhhh.  I wrote it down to ridicule the idea and now people
are taking it seriously ;(

It should be obvious enough that anything depending on the
kernel parsing file contents will lead to problems.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> > /bin could be separated (like linus said) but cat /bin/.compound could
> > do it. This is the /etc/passwd Hans' example, I think:
> 
> Arghhhh.  I wrote it down to ridicule the idea and now people
> are taking it seriously ;(
> 
> It should be obvious enough that anything depending on the
> kernel parsing file contents will lead to problems.

This is one case where the kernel _isn't_ parsing file contents,
but I agree it's ridiculous :)

-- Jamie

Re: silent semantic changes with reiser4

[Hans Reiser]

Jamie Lokier wrote:

>Rik van Riel wrote:
>  
>
>>>/bin could be separated (like linus said) but cat /bin/.compound could
>>>do it. This is the /etc/passwd Hans' example, I think:
>>>      
>>>
>>Arghhhh.  I wrote it down to ridicule the idea and now people
>>are taking it seriously ;(
>>
>>It should be obvious enough that anything depending on the
>>kernel parsing file contents will lead to problems.
>>    
>>
>
>This is one case where the kernel _isn't_ parsing file contents,
>but I agree it's ridiculous :)
>
>-- Jamie
>  
>
Umm, no, when you write the glued /etc/passwd file it parses the 
delimiters in it to determine what subfile to put the parts of 
/etc/passwd into.

Inheritance with delimiters is the best way to make emacs effective for 
editing lots of small files, Stallman was right and deserves credit for it.

Re: silent semantic changes with reiser4

[Denis Vlasenko]

On Thursday 26 August 2004 17:12, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Denis Vlasenko wrote:
> > I think Hans is not planning turning old "file is a stream of bytes"
> > into eight-stream octopus. One stream will remain as a 'main' one,
> > which contains actual data. Others will keep metadata, etc...
>
> This is exactly what the Samba people want, though.
>
> Office suites can store a document with embedded images
> and spread sheets "easily" by putting the text, the
> images and spread sheets all in different file streams.

Kinda far reaching. It's hard to assess whether that is good or bad idea.
Can we start small, like putting metadata (e.g. ACLs) into these streams?
--
vda

Re: silent semantic changes with reiser4

[Alan Cox]

On Iau, 2004-08-26 at 16:19, Denis Vlasenko wrote:
> > Office suites can store a document with embedded images
> > and spread sheets "easily" by putting the text, the
> > images and spread sheets all in different file streams.
> 
> Kinda far reaching. It's hard to assess whether that is good or bad idea.
> Can we start small, like putting metadata (e.g. ACLs) into these streams?

Openoffice does this in user space and the user space vfs code desktops
use can handle zips so this "just works" already including over NFS,
unlike a kernel proposed method.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Sun, Aug 29, 2004 at 03:30:03PM +0100, Alan Cox wrote:
> 
> Openoffice does this in user space and the user space vfs code desktops
> use can handle zips so this "just works" already including over NFS,
> unlike a kernel proposed method.

Hurrah for OpenOffice. Now all you need to do is to
persuade Microsoft to store Word files in the same
format......

Jeremy.

Re: silent semantic changes with reiser4

[Trond Myklebust]

På su , 29/08/2004 klokka 10:30, skreiv Alan Cox:

> Openoffice does this in user space and the user space vfs code desktops
> use can handle zips so this "just works" already including over NFS,
> unlike a kernel proposed method.

NFS is not the roadblock here. Sure v2 and v3 support would be
impossible to do, but since this is all 2.6 work anyway, people have the
option of using NFSv4, which has full support for streams...

NFSv4 has an OPENATTR call which acts on files to return a filehandle
that works fine with both READDIR and LOOKUP, so if a VFS interface for
streams existed, we could code up full support tomorrow.
As it is, we're having to shoehorn this into the xattr interface. 8-(

Cheers,
  Trond

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Sun, Aug 29, 2004 at 02:41:04PM -0400, Trond Myklebust wrote:
> 
> NFSv4 has an OPENATTR call which acts on files to return a filehandle
> that works fine with both READDIR and LOOKUP, so if a VFS interface for
> streams existed, we could code up full support tomorrow.
> As it is, we're having to shoehorn this into the xattr interface. 8-(

Yeah, that's what I'm really trying to avoid for Samba.
We're currently using the xattr interface for DOS attributes,
and soon for NT ACL support, but it doesn't really fit as
a streams interface (no lseek() support) although someone in
HP coded up a test interface to NT streams that map into xattrs
(most of the streams in a Word file for instance are quite
small) - but I really don't want to have to do this :-).

Jeremy.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Jeremy Allison wrote:
> (most of the streams in a Word file for instance are quite small)

Streams in a Word file?

Are you saying that when I copy a .doc file onto my Linux box and off,
I lose part of a Word document?

-- Jamie

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Sep 01, 2004 at 09:19:45PM +0100, Jamie Lokier wrote:
> Jeremy Allison wrote:
> > (most of the streams in a Word file for instance are quite small)
> 
> Streams in a Word file?

Yep.

> Are you saying that when I copy a .doc file onto my Linux box and off,
> I lose part of a Word document?

Right now no, because when Samba refuses the stream open, Word falls
back into a "tar"-like mode where it linearises the streams into the
data (it's a legacy mode for storing data on a FAT drive, not an NTFS
drive). However, the problem is that no currently supported Microsoft
OS doesn't have streams-capable NTFS support. 

This means that in a future MS-Office revision, this backwards support
may be broken by accident or by design (less likely, Microsoft really
don't do that kind of thing without very high level requests :-) and no
testers at Microsoft will notice (because they only test against MS servers).

Jeremy.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Jeremy Allison wrote:
> > Streams in a Word file?
> 
> Yep.
> 
> > Are you saying that when I copy a .doc file onto my Linux box and off,
> > I lose part of a Word document?
> 
> Right now no, because when Samba refuses the stream open, Word falls
> back into a "tar"-like mode where it linearises the streams into the
> data (it's a legacy mode for storing data on a FAT drive, not an NTFS
> drive). However, the problem is that no currently supported Microsoft
> OS doesn't have streams-capable NTFS support. 

I meant when I copy not using Samba.  For example, I copy the .doc
file in Windows NT to an FTP server.

Does the FTP operation magically linearise the .doc streams on demand?

Or does FTP lose part of the Word document?

-- Jamie

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Sep 01, 2004 at 09:31:01PM +0100, Jamie Lokier wrote:
> 
> I meant when I copy not using Samba.  For example, I copy the .doc
> file in Windows NT to an FTP server.
> 
> Does the FTP operation magically linearise the .doc streams on demand?
> 
> Or does FTP lose part of the Word document?

Good question. It depends if the Microsoft ftp client is streams-aware,
and understands the Microsoft OLE structured storage format and will do
the linearisation on demand or not. I must confess I haven't tested this,
as I don't ever run Windows other than on vmware sessions for Samba testing
these days :-).

Probably a non-Microsoft ftp client would lose part of the word doc.

Jeremy.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Jeremy Allison wrote:
> > I meant when I copy not using Samba.  For example, I copy the .doc
> > file in Windows NT to an FTP server.
> > 
> > Does the FTP operation magically linearise the .doc streams on demand?
> > Or does FTP lose part of the Word document?
> 
> Good question. It depends if the Microsoft ftp client is streams-aware,
> and understands the Microsoft OLE structured storage format and will do
> the linearisation on demand or not. I must confess I haven't tested this,
> as I don't ever run Windows other than on vmware sessions for Samba testing
> these days :-).
> 
> Probably a non-Microsoft ftp client would lose part of the word doc.

So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
will _all_ lose part of a Word document when they handle it on a
Window box?

Ouch!

The only sensible implementation I can imagine would be if the OS
linearised multi-stream Word documents into the non-stream format
automatically for all programs which don't know about streams.

Which is of course what I would like to implement for Linux...

- Jamie

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote:
> Jeremy Allison wrote:
> > > I meant when I copy not using Samba.  For example, I copy the .doc
> > > file in Windows NT to an FTP server.
> > > 
> > > Does the FTP operation magically linearise the .doc streams on demand?
> > > Or does FTP lose part of the Word document?
> > 
> > Good question. It depends if the Microsoft ftp client is streams-aware,
> > and understands the Microsoft OLE structured storage format and will do
> > the linearisation on demand or not. I must confess I haven't tested this,
> > as I don't ever run Windows other than on vmware sessions for Samba testing
> > these days :-).
> > 
> > Probably a non-Microsoft ftp client would lose part of the word doc.
> 
> So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> will _all_ lose part of a Word document when they handle it on a
> Window box?
> 
> Ouch!

Yep. It's the meta data that Word stores in streams that will get lost.

Jeremy.

Re: silent semantic changes with reiser4

[Lee Revell]

On Wed, 2004-09-01 at 16:51, Jeremy Allison wrote:
> On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote:
> > Jeremy Allison wrote:
> > > > I meant when I copy not using Samba.  For example, I copy the .doc
> > > > file in Windows NT to an FTP server.
> > > > 
> > > > Does the FTP operation magically linearise the .doc streams on demand?
> > > > Or does FTP lose part of the Word document?
> > > 
> > > Good question. It depends if the Microsoft ftp client is streams-aware,
> > > and understands the Microsoft OLE structured storage format and will do
> > > the linearisation on demand or not. I must confess I haven't tested this,
> > > as I don't ever run Windows other than on vmware sessions for Samba testing
> > > these days :-).
> > > 
> > > Probably a non-Microsoft ftp client would lose part of the word doc.
> > 
> > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> > will _all_ lose part of a Word document when they handle it on a
> > Window box?
> > 
> > Ouch!
> 
> Yep. It's the meta data that Word stores in streams that will get lost.
> 

This is shocking.  When was this behavior introduced?

Lee

Re: silent semantic changes with reiser4

[Oliver Hunt]

The loss of forks in the file is exxactly the problem you used to have
when transferring native Mac files to a PC...

This meant in order to transfer files to different filesystem you
often needed to tar/zip/whatever them first.

Bare in mind this would let us do the whole MacOS thing of putting an
entire application(plus plugins, etc) inside one "file"...

--Oliver

On Wed, 1 Sep 2004 13:51:40 -0700, Jeremy Allison <jra@samba.org> wrote:
> On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote:
> > Jeremy Allison wrote:
> > > > I meant when I copy not using Samba.  For example, I copy the .doc
> > > > file in Windows NT to an FTP server.
> > > >
> > > > Does the FTP operation magically linearise the .doc streams on demand?
> > > > Or does FTP lose part of the Word document?
> > >
> > > Good question. It depends if the Microsoft ftp client is streams-aware,
> > > and understands the Microsoft OLE structured storage format and will do
> > > the linearisation on demand or not. I must confess I haven't tested this,
> > > as I don't ever run Windows other than on vmware sessions for Samba testing
> > > these days :-).
> > >
> > > Probably a non-Microsoft ftp client would lose part of the word doc.
> > 
> > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> > will _all_ lose part of a Word document when they handle it on a
> > Window box?
> >
> > Ouch!
> 
> Yep. It's the meta data that Word stores in streams that will get lost.
> 
> 
> 
> Jeremy.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: silent semantic changes with reiser4

[Herbert Poetzl]

On Thu, Sep 02, 2004 at 11:22:41AM +1200, Oliver Hunt wrote:
> The loss of forks in the file is exxactly the problem you used to have
> when transferring native Mac files to a PC...
> 
> This meant in order to transfer files to different filesystem you
> often needed to tar/zip/whatever them first.
> 
> Bare in mind this would let us do the whole MacOS thing of putting an
> entire application(plus plugins, etc) inside one "file"...

hum, well aside from the fact that recent MacOS (X) does
not put applications in one "file", it does simply store
related stuff inside a directory which is presented as
one unit by the GUI ... something I consider useful and
which allows to 'move' the application around without
rendering them useless ...

best,
Herbert

> --Oliver
> 
> On Wed, 1 Sep 2004 13:51:40 -0700, Jeremy Allison <jra@samba.org> wrote:
> > On Wed, Sep 01, 2004 at 09:47:46PM +0100, Jamie Lokier wrote:
> > > Jeremy Allison wrote:
> > > > > I meant when I copy not using Samba.  For example, I copy the .doc
> > > > > file in Windows NT to an FTP server.
> > > > >
> > > > > Does the FTP operation magically linearise the .doc streams on demand?
> > > > > Or does FTP lose part of the Word document?
> > > >
> > > > Good question. It depends if the Microsoft ftp client is streams-aware,
> > > > and understands the Microsoft OLE structured storage format and will do
> > > > the linearisation on demand or not. I must confess I haven't tested this,
> > > > as I don't ever run Windows other than on vmware sessions for Samba testing
> > > > these days :-).
> > > >
> > > > Probably a non-Microsoft ftp client would lose part of the word doc.
> > > 
> > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> > > will _all_ lose part of a Word document when they handle it on a
> > > Window box?
> > >
> > > Ouch!
> > 
> > Yep. It's the meta data that Word stores in streams that will get lost.
> > 
> > 
> > 
> > Jeremy.
> > -
> > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> > Please read the FAQ at  http://www.tux.org/lkml/
> >
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: silent semantic changes with reiser4

[Theodore Ts'o]

On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote:
> > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> > will _all_ lose part of a Word document when they handle it on a
> > Window box?
> > 
> > Ouch!
> 
> Yep. It's the meta data that Word stores in streams that will get lost.

And this is why I believe that using streams in application is well,
ill-advised.  Indeed, one of my concerns with providing streams
support is that application authors may make the mistake of using it,
and we will be back to the bad old days (when MacOS made this mistake)
where you will need to binhex files before you ftp them (and unbinhex
them on the otherside) --- and if you forget, the resulting file will
be useless.

I understand why the Samba folks want this feature very badly;
however, hopefully other projects will know enough *not* to use
streams once they become available in Linux....

						- Ted

Re: silent semantic changes with reiser4

[Stuart Young]

On Thu, 2 Sep 2004 22:54, Theodore Ts'o wrote:
> On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote:
> > > So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> > > will _all_ lose part of a Word document when they handle it on a
> > > Window box?
> > >
> > > Ouch!
> >
> > Yep. It's the meta data that Word stores in streams that will get lost.
>
> And this is why I believe that using streams in application is well,
> ill-advised.  Indeed, one of my concerns with providing streams
> support is that application authors may make the mistake of using it,
> and we will be back to the bad old days (when MacOS made this mistake)
> where you will need to binhex files before you ftp them (and unbinhex
> them on the otherside) --- and if you forget, the resulting file will
> be useless.

At least currently (to my knowledge anyway) all stream support in Windows is 
data that is not important, and that can be either regenerated from 
filesystem metadata or (more usually) the main file stream itself.

This sort of data is really where streams excel, by providing a way to access 
data that would otherwise take time/cpu to regenerate over and over, but that 
in itself is not indispensable. Good examples of this are indexes of data 
within a document, details of who owns/created/modified the document, common 
views or reformatting of the data, etc. With audio/video/graphics, you could 
store lower quality transforms of data (eg: stereo to mono, resolution 
reduction, thumbnails, etc) in the streams for a file. With a word document, 
it could be things like an index (assuming it's auto-generated from section 
headings). With a database, it could be the indexes, and a few views that are 
expensive time-wise to generate. All of these are easily regenerated from the 
original data stream, but takes a while. And if you've got the disk, why not 
use it?

If streams were always to be considered volatile, then you could do all sorts 
of interesting things with them. Any disk cleanup mechanism you have could 
also reap old streams specifically if the disk gets below a certain amount 
free. This means that old streams that are hanging about don't end up wasting 
all your disk space. Of course, you'd want a way to disable this (for servers 
mainly), and streams would have to be considered volatile on more than just 
Linux as a platform for this to be useful.

Note that I'm not particularly advocating streams here. I'm just pointing out 
'how' it could be useful. It could be very easy to misuse streams and cause 
huge problems (as per Ted's comments), but it's always good to know the other 
side of the argument.

-- 
 Stuart Young (aka Cef)
 cef-lkml@optusnet.com.au is for LKML and related email only

Re: silent semantic changes with reiser4

[Helge Hafting]

Stuart Young wrote:

>On Thu, 2 Sep 2004 22:54, Theodore Ts'o wrote:
>  
>
>>On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote:
>>    
>>
>>>>So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
>>>>will _all_ lose part of a Word document when they handle it on a
>>>>Window box?
>>>>
>>>>Ouch!
>>>>        
>>>>
>>>Yep. It's the meta data that Word stores in streams that will get lost.
>>>      
>>>
>>And this is why I believe that using streams in application is well,
>>ill-advised.  Indeed, one of my concerns with providing streams
>>support is that application authors may make the mistake of using it,
>>and we will be back to the bad old days (when MacOS made this mistake)
>>where you will need to binhex files before you ftp them (and unbinhex
>>them on the otherside) --- and if you forget, the resulting file will
>>be useless.
>>    
>>
This is not a problem with multiple streams implemented right - as a 
directory.
You don't stay away from directories just because you have to tar
them in order to put them on ftp sites? ;-)

Still, you're right that apps using streams jsut for the hell of it is bad.
That sort of thing happens all the time when something new shows up,
some people will use it without thinking.

>
>At least currently (to my knowledge anyway) all stream support in Windows is 
>data that is not important, and that can be either regenerated from 
>filesystem metadata or (more usually) the main file stream itself.
>
>This sort of data is really where streams excel, by providing a way to access 
>data that would otherwise take time/cpu to regenerate over and over, but that 
>in itself is not indispensable. 
>
Streams as a cache.

>Good examples of this are indexes of data 
>within a document, details of who owns/created/modified the document, common 
>views or reformatting of the data, etc. With audio/video/graphics, you could 
>store lower quality transforms of data (eg: stereo to mono, resolution 
>reduction, thumbnails, etc) in the streams for a file. With a word document, 
>it could be things like an index (assuming it's auto-generated from section 
>headings). With a database, it could be the indexes, and a few views that are 
>expensive time-wise to generate. All of these are easily regenerated from the 
>original data stream, but takes a while. And if you've got the disk, why not 
>use it?
>  
>
Actually, some if this is bad examples of using streams.
Why can't that index be stored _in_ the document?
After all, the word processor is the one who knows the document's
internal format, how to generate the index, and how to use it.  Well,
this example is bad anyway as an index can be created so fast from
a well structured document that there is little need for storing one.
(Example - see how lyx keeps a live index in the "navigate" menu. . .)

A document format may also contain fields specifying who made it, or
even a log of who modified it and when. 

It seems to me that streams are more useful for stuff that the file's
main application don't deal with itself.  Such as attaching icons that
follow the file around.

>If streams were always to be considered volatile, then you could do all sorts 
>of interesting things with them. Any disk cleanup mechanism you have could 
>also reap old streams specifically if the disk gets below a certain amount 
>free. 
>
That would limit streams to caching use _only_, disks fill occationally
and we can't have _useful_ stuff disappearing at random.

Helge Hafting

Re: silent semantic changes with reiser4

[Stuart Young]

On Fri, 3 Sep 2004 17:53, Helge Hafting wrote:
> Stuart Young wrote:
> >On Thu, 2 Sep 2004 22:54, Theodore Ts'o wrote:
> >>On Wed, Sep 01, 2004 at 01:51:40PM -0700, Jeremy Allison wrote:
> >>>>So you're saying SCP, CVS, Subversion, Bitkeeper, Apache and rsyncd
> >>>>will _all_ lose part of a Word document when they handle it on a
> >>>>Window box?
> >>>>
> >>>>Ouch!
> >>>
> >>>Yep. It's the meta data that Word stores in streams that will get lost.
> >>
> >>And this is why I believe that using streams in application is well,
> >>ill-advised.  Indeed, one of my concerns with providing streams
> >>support is that application authors may make the mistake of using it,
> >>and we will be back to the bad old days (when MacOS made this mistake)
> >>where you will need to binhex files before you ftp them (and unbinhex
> >>them on the otherside) --- and if you forget, the resulting file will
> >>be useless.
>
> This is not a problem with multiple streams implemented right - as a
> directory.

This works if you want (possibly editable) views of a single document. It 
doesn't really work that well for other things. Even then, something has to 
handle all the data translations and decoding, and that means you need to 
store all the reference information somewhere so that the app knows what to 
pass the original file to, so that it can be decoded for the user as they've 
requested.

> You don't stay away from directories just because you have to tar
> them in order to put them on ftp sites? ;-)

No, but I know a lot of people who don't understand that with a lot of ftp 
servers that if you append .tar.gz to the directory name, you can fetch the 
directory. Sure it's all user education, but you have to remember that users 
don't generally want to be educated. They just want to get on with doing 
whatever they are doing.

> Still, you're right that apps using streams jsut for the hell of it is bad.
> That sort of thing happens all the time when something new shows up,
> some people will use it without thinking.

Hence why I was suggesting the idea of disposable data in streams. As long as 
people KNOW it's disposable, but useful to keep around as it cuts down the 
time needed to do stuff, then apps will start to pick up transporting streams 
properly. Least then (hopefully) no real information will get lost that is 
important. Once transporting streams becomes commonplace, then perhaps 
streams can be used for more useful things.

> >At least currently (to my knowledge anyway) all stream support in Windows
> > is data that is not important, and that can be either regenerated from
> > filesystem metadata or (more usually) the main file stream itself.
> >
> >This sort of data is really where streams excel, by providing a way to
> > access data that would otherwise take time/cpu to regenerate over and
> > over, but that in itself is not indispensable.
>
> Streams as a cache.

Almost exactly.

> >Good examples of this are indexes of data
> >within a document, details of who owns/created/modified the document,
> > common views or reformatting of the data, etc. With audio/video/graphics,
> > you could store lower quality transforms of data (eg: stereo to mono,
> > resolution reduction, thumbnails, etc) in the streams for a file. With a
> > word document, it could be things like an index (assuming it's
> > auto-generated from section headings). With a database, it could be the
> > indexes, and a few views that are expensive time-wise to generate. All of
> > these are easily regenerated from the original data stream, but takes a
> > while. And if you've got the disk, why not use it?
>
> Actually, some if this is bad examples of using streams.
> Why can't that index be stored _in_ the document?
> After all, the word processor is the one who knows the document's
> internal format, how to generate the index, and how to use it.  Well,
> this example is bad anyway as an index can be created so fast from
> a well structured document that there is little need for storing one.
> (Example - see how lyx keeps a live index in the "navigate" menu. . .)

The point of such information in my examples is that a stream can store 
information in a particular format (ie: an index) that is common to one 
indexing app/library. Such an index can be used by ANY app that knows the 
index format to search the document. This is almost exactly what MS will do 
(if they haven't done it already) with the File Indexing Service. As it's ONE 
library, then any new user app that creates data can add index creation by 
adding one library. And any app that wants to search these indexes would need 
only to add one library, not every library for every format that it wants to 
search. It's essentially an n^2 vs 2n problem.

> A document format may also contain fields specifying who made it, or
> even a log of who modified it and when.

Exactly, but then you need to:
 1. Read in the Document, and in some formats that are either compressed or 
that do not have a fixed format, this may mean you end up reading most of or 
even an entire file to get this data. This is expensive on CPU, disk cache, 
disk access and transfer times with large files, particularly if it's a slow 
data bus to the storage device.
 2. Parse out the information you want, which means that you need to 
understand the format. With an index stream, in a fixed format, you only need 
one library that can handle the format for you. Without streams, you use lots 
of libraries. By using lots of libraries, you end up using lots of memory to 
hold them, and disk access time/transfer speed to load them.

> It seems to me that streams are more useful for stuff that the file's
> main application don't deal with itself.  Such as attaching icons that
> follow the file around.

Icons themselves wouldn't be that good for multi-user environments. Attaching 
an icon type designator that then allows the app that wants to display an 
icon for the file to pull the appropriate icon from somewhere else (and 
therefore can be matched to whatever theme or specific icon the user wants) 
is probably a better idea.

> >If streams were always to be considered volatile, then you could do all
> > sorts of interesting things with them. Any disk cleanup mechanism you
> > have could also reap old streams specifically if the disk gets below a
> > certain amount free.
>
> That would limit streams to caching use _only_, disks fill occationally
> and we can't have _useful_ stuff disappearing at random.

If they transfer a file with streams on Windows using standard (Posix based) 
tools at the moment then they lose this information anyway. And application 
developers aren't all suddenly going to drop whatever it is they're doing to 
implement stream support as soon as it appears. Things take time, and till 
the majority of applications support it, it's useless to assume that the data 
is not going to dissapear because of a user using a non-updated tool. By 
limiting it to caching, at least in the short term, at least no one is going 
to (hopefully!) get bitten by data loss. Later (and I'm guessing a LOT 
later), who knows, streams could actually become useful for more than just 
caching.

-- 
 Stuart Young (aka Cef)
 cef-lkml@optusnet.com.au is for LKML and related email only

Re: silent semantic changes with reiser4

[Horst von Brand]

Stuart Young <cef-lkml@optusnet.com.au> said:

[...]

> Hence why I was suggesting the idea of disposable data in streams. As
> long as people KNOW it's disposable, but useful to keep around as it cuts
> down the time needed to do stuff, then apps will start to pick up
> transporting streams properly. Least then (hopefully) no real information
> will get lost that is important. Once transporting streams becomes
> commonplace, then perhaps streams can be used for more useful things.

How will you prevent people putting the "real" data under some random
stream, "just because it is a prettier name"? (Yes, I've seen Windows users
exporting everything because they found the folder + hand icon
prettier...).  Short answer: You can't. And if you did, then it would be
(another) hell to go through when you start using streams for "useful
data".

[...]

> The point of such information in my examples is that a stream can store
> information in a particular format (ie: an index) that is common to one
> indexing app/library.

Great. Now you just need to convince everybody and Aunt Tillie to use that
same format.

>                       Such an index can be used by ANY app that knows the
> index format to search the document. This is almost exactly what MS will
> do (if they haven't done it already) with the File Indexing Service. As
> it's ONE library, then any new user app that creates data can add index
> creation by adding one library. And any app that wants to search these
> indexes would need only to add one library, not every library for every
> format that it wants to search. It's essentially an n^2 vs 2n problem.

Doable if you can just go and force a format/stream layout/application
suite on each and every user. Won't happen in Linux (and I'm happy for
that).
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Stuart Young]

On Sat, 4 Sep 2004 05:17, Horst von Brand wrote:
> Stuart Young <cef-lkml@optusnet.com.au> said:
> > Hence why I was suggesting the idea of disposable data in streams. As
> > long as people KNOW it's disposable, but useful to keep around as it cuts
> > down the time needed to do stuff, then apps will start to pick up
> > transporting streams properly. Least then (hopefully) no real information
> > will get lost that is important. Once transporting streams becomes
> > commonplace, then perhaps streams can be used for more useful things.
>
> How will you prevent people putting the "real" data under some random
> stream, "just because it is a prettier name"?

This is a function of exporting something complex like file streams through a 
namespace like 'file-as-directory'. As soon as you start putting complex 
things like file streams through an 'easy to use' interface, you will get 
people using it in ways you don't expect. Sure, it has to be useful, and 
relatively easy, but you don't need to make it insanely so.

And if it's all considered volatile, the only ones that NEED to know about it 
are system admins, power users and programmers, all of which have a much 
higher chance of understanding a complex object like a file stream.

> (Yes, I've seen Windows users 
> exporting everything because they found the folder + hand icon
> prettier...).  Short answer: You can't. And if you did, then it would be
> (another) hell to go through when you start using streams for "useful
> data".

Interesting you should say that. Pretty much every stream in Windows at the 
moment is considered to be volatile, as the information could be lost in a 
transfer to a networked machine or filesystem that is not stream enabled. 
Once Windows NT and Win98/ME are 'out of the picture', and FAT* filesystems 
vanish, this sort of behaviour could possibly be expected to change. Notice 
that it's a timeframe over about 5 yrs that they've been (almost silently) 
doing this already. They'll have much the same problem. Since I seriously 
doubt that streams are going to pop up in Linux overnight, with all the app 
support that this entails, we'll have the benefit of seeing just how the 
behemoth handles it, and be able to learn from their mistakes.

Of note on this, the ability to create/manipulate/read/etc streams should 
almost certainly require it's own set of ACL's beyond blindly assuming that 
the standard permissions/ACL's can cover it as is. Document creators should 
be able to create/manipulate a stream, but a searching service does not 
necessarily need to do more than read it. You could even have the ability to 
access or write to a stream other than the default covered by a capability.

> > The point of such information in my examples is that a stream can store
> > information in a particular format (ie: an index) that is common to one
> > indexing app/library.
>
> Great. Now you just need to convince everybody and Aunt Tillie to use that
> same format.

Welcome to standardisation. And in fact it's not so much necessary that 
everyone uses the same index format, as long as the same index format is used 
on one machine. In fact, it's not even that necessary, as each index format 
could use it's own stream to store it's version of the information. Yes it's 
duplication, but who isn't already used to that? (KDE/Gnome, vi/emacs, 
ext3/XFS/reiser/etc). You choose the format for the job, and by personal 
preference. If the apps don't support the format you want, you just lose 
whatever abilities the format gains you, but at least you don't lose the raw 
data.

> >                       Such an index can be used by ANY app that knows the
> > index format to search the document. This is almost exactly what MS will
> > do (if they haven't done it already) with the File Indexing Service. As
> > it's ONE library, then any new user app that creates data can add index
> > creation by adding one library. And any app that wants to search these
> > indexes would need only to add one library, not every library for every
> > format that it wants to search. It's essentially an n^2 vs 2n problem.
>
> Doable if you can just go and force a format/stream layout/application
> suite on each and every user. Won't happen in Linux (and I'm happy for
> that).

As above.

My personal view of streams (if we MUST have it under Linux, and I'm in no way 
at all convinced of that fact) is that it should in many ways be like 
journaling or hash trees in ext3. They're useful functions, but they are not 
'necessary' to the actual storage of data. Sure, they make it easy (and in 
some cases significantly easier) to recover or access data, but they are 
features that do not ultimately stop you using the core of the filesystem, 
it's main objective, which is storing data.

-- 
 Stuart Young (aka Cef)
 cef-lkml@optusnet.com.au is for LKML and related email only

Re: silent semantic changes with reiser4

[David Lang]

what happens when you ue one of the many software packages around that 
lets windows access a NFS server. when you copy the file to a NFS accessed 
drive does it loose part of the data?

David Lang

  On Wed, 1 Sep 2004, Jamie Lokier 
wrote:

> Date: Wed, 1 Sep 2004 21:31:01 +0100
> From: Jamie Lokier <jamie@shareable.org>
> To: Jeremy Allison <jra@samba.org>
> Cc: Trond Myklebust <trond.myklebust@fys.uio.no>,
>     Alan Cox <alan@lxorguk.ukuu.org.uk>,
>     Denis Vlasenko <vda@port.imtp.ilyichevsk.odessa.ua>,
>     Rik van Riel <riel@redhat.com>, Christer Weinigel <christer@weinigel.se>,
>     Spam <spam@tnonline.net>, Andrew Morton <akpm@osdl.org>,
>     wichert@wiggy.net, Linus Torvalds <torvalds@osdl.org>, reiser@namesys.com,
>     hch@lst.de, Linux Filesystem Development <linux-fsdevel@vger.kernel.org>,
>     Linux Kernel Mailing List <linux-kernel@vger.kernel.org>, flx@namesys.com,
>     reiserfs-list@namesys.com
> Subject: Re: silent semantic changes with reiser4
> 
> Jeremy Allison wrote:
>>> Streams in a Word file?
>>
>> Yep.
>>
>>> Are you saying that when I copy a .doc file onto my Linux box and off,
>>> I lose part of a Word document?
>>
>> Right now no, because when Samba refuses the stream open, Word falls
>> back into a "tar"-like mode where it linearises the streams into the
>> data (it's a legacy mode for storing data on a FAT drive, not an NTFS
>> drive). However, the problem is that no currently supported Microsoft
>> OS doesn't have streams-capable NTFS support.
>
> I meant when I copy not using Samba.  For example, I copy the .doc
> file in Windows NT to an FTP server.
>
> Does the FTP operation magically linearise the .doc streams on demand?
>
> Or does FTP lose part of the Word document?
>
> -- Jamie
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Alan Cox]

On Mer, 2004-09-01 at 21:26, Jeremy Allison wrote:
> Right now no, because when Samba refuses the stream open, Word falls
> back into a "tar"-like mode where it linearises the streams into the
> data (it's a legacy mode for storing data on a FAT drive, not an NTFS
> drive). However, the problem is that no currently supported Microsoft
> OS doesn't have streams-capable NTFS support. 

That implies that samba can do the same transform set without kernel
help and you'd even get the advantages of being able to transfer the
stuff around sanely afterwards.

What I don't understand is the tie between Linux having such streams and
Windows doing it for Samba to work. Netatalk has always handle this for
Macintosh and portably. Presumably any Samba support would need to
handle OS's without wacky files for portability too ?

Re: silent semantic changes with reiser4

[Oliver Neukum]

Am Donnerstag, 2. September 2004 11:48 schrieb Alan Cox:
> What I don't understand is the tie between Linux having such streams and
> Windows doing it for Samba to work. Netatalk has always handle this for
> Macintosh and portably. Presumably any Samba support would need to
> handle OS's without wacky files for portability too ?

Can you do an atomic rename of all streams without kernel support?

	Regards
		Oliver

Re: silent semantic changes with reiser4

[Alan Cox]

On Iau, 2004-09-02 at 12:13, Oliver Neukum wrote:
> Am Donnerstag, 2. September 2004 11:48 schrieb Alan Cox:
> > What I don't understand is the tie between Linux having such streams and
> > Windows doing it for Samba to work. Netatalk has always handle this for
> > Macintosh and portably. Presumably any Samba support would need to
> > handle OS's without wacky files for portability too ?
> 
> Can you do an atomic rename of all streams without kernel support?

That depends how SAMBA chooses to handle the problem internally and how
it chooses to store the data. The netatalk people have atomicity from
the view of clients but not from the unix fs internal view.

Alan

Re: silent semantic changes with reiser4

[Brad Boyer]

On Thu, Sep 02, 2004 at 10:48:46AM +0100, Alan Cox wrote:
> What I don't understand is the tie between Linux having such streams and
> Windows doing it for Samba to work. Netatalk has always handle this for
> Macintosh and portably. Presumably any Samba support would need to
> handle OS's without wacky files for portability too ?

I'm not 100% sure on the samba side, but I think there is a pretty
significant difference. On the Mac, the problem of copying forks and
metadata onto non-Mac systems was recognized early on. There are several
standard formats for serialized versions of this data. If you take the
files that netatalk writes and copy them directly to a Mac separately,
there are tools that can convert them back to the original format with
all the data intact. I've never seen such a thing for NTFS named streams.

	Brad Boyer
	flar@allandria.com

Re: silent semantic changes with reiser4

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brad Boyer wrote:
| On Thu, Sep 02, 2004 at 10:48:46AM +0100, Alan Cox wrote:
|
|>What I don't understand is the tie between Linux having such streams and
|>Windows doing it for Samba to work. Netatalk has always handle this for
|>Macintosh and portably. Presumably any Samba support would need to
|>handle OS's without wacky files for portability too ?
|
|
| I'm not 100% sure on the samba side, but I think there is a pretty
| significant difference. On the Mac, the problem of copying forks and
| metadata onto non-Mac systems was recognized early on. There are several
| standard formats for serialized versions of this data. If you take the
| files that netatalk writes and copy them directly to a Mac separately,
| there are tools that can convert them back to the original format with
| all the data intact. I've never seen such a thing for NTFS named streams.

I had an idea for how to solve this.  Search the archives for
"serialization" or "serialize".

Basically, it involves creating a better interface to a better "dump".
You do
	cat foo/serialize > foo.img
and send foo.img over the network.  Or you just tell mutt/scp/whatever
to grab foo/serialize instead of foo.  Can't be worse than a mac
resource fork, and it looks more user-friendly to me.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTlauHgHNmZLgCUhAQK7Iw//SAi7I84OIz4xHh5S7i1xc+JugPvELyh2
dvrVkETvQlvRcABZ0GYlCQQPPzL+QcTVnruMRAUZGrGCPJdTkxfae41tX5KDS8/4
OpPWEDki+FSJrv+Mn9Pbm3I5Bxlhu0+nuNOIS0HSULo+/IBV4f/ldv8TUKSbXeXJ
YjAXJJECpWTlYwF8sTvM/ALHpo+6xEtJq5gQxoUFnw4Pio8eycalS9m2cDs/N6rq
PYcOjb7pWjGEk+9qimmwIcX5LnBXl8L9OhXqMQoR3Of+blniIrEOtg/0WLrwzu62
rW+rBxDAfoDxIZZvquf/gyJ6stO8QzGeQqoxxTUXhbI5PUtD+qGO+tWT6+/EljHF
qNri65JwB8PetbqdjWmsTO3V+FVSRy3hu4/vTNpnmwnR8H+yilWWDc7Tnsjql+dh
9j4ycfT5xDAA9dC6APOSV5AEgI9z8FntQZzOCPe1lyLD7Qwjdak+3Nw2I3yNS+Pl
Xa5ijSm3IKJ+JqKpZKRzCRyXRQdeAj2kGaWQP4Ui3D/RwBEHtn7shwdJ9Hku6KkN
9ZyiTPqY5XGiyLrJWqRc9MraepGhuxxtYcq65A+vdJq2fxA5+XRF8UGbYc9KW5lF
1fYSXA2DBNJjeNEvZw8LWI54OrIMSlhj3dWLX8WkkXog2gWO7ly0E6OiA0odZV6M
i9sDeRClqlM=
=CFb4
-----END PGP SIGNATURE-----

Re: silent semantic changes with reiser4

[Jamie Lokier]

Alan Cox wrote:
> On Iau, 2004-08-26 at 16:19, Denis Vlasenko wrote:
> > > Office suites can store a document with embedded images
> > > and spread sheets "easily" by putting the text, the
> > > images and spread sheets all in different file streams.
> > 
> > Kinda far reaching. It's hard to assess whether that is good or bad idea.
> > Can we start small, like putting metadata (e.g. ACLs) into these streams?
> 
> Openoffice does this in user space and the user space vfs code desktops
> use can handle zips so this "just works" already including over NFS,
> unlike a kernel proposed method.

OpenOffice is very sensible, and I agree that a method of handling
document parts that _requires_ kernel support would be foolish.

However, please see my message "The argument for fs assistance in
handling archives" for how kernel support can improve performance of
programs like OpenOffice on large compound documents, while preserving
the sensible behaviour and remaining portable.

-- Jamie

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 16:00, Denis Vlasenko wrote:
>
> I think Hans is not planning turning old "file is a stream of bytes"
> into eight-stream octopus. One stream will remain as a 'main' one,
> which contains actual data. Others will keep metadata, etc...
>
> Note that even today's files aren't simple "streams of bytes"
> because they also have names,mode bits etc.

This is just exactly how NTFS does work: there's an unnamed stream (which is 
the one that the majority of userspace programs do manipulate) which contains 
the user data (the pixels from a PMG image, or text from a text file, etc.), 
and there are named streams which not all kind of applications do know how to 
handle.

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Andrew Morton wrote:

> All of which can be handled in userspace library code.
> 
> What compelling reason is there for doing this in the kernel?

There's a compelling reason to do it in userspace.  If an
unaware program copies or moves such a file with streams
inside, it doesn't break the streams and aware programs will
continue to see them.

OTOH, if we had the streams in the kernel, unaware applications
would continuously break the metadata and streams that the
streams aware programs expect !

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Jamie Lokier]

Rik van Riel wrote:
> > All of which can be handled in userspace library code.
> > 
> > What compelling reason is there for doing this in the kernel?
> 
> There's a compelling reason to do it in userspace.  If an
> unaware program copies or moves such a file with streams
> inside, it doesn't break the streams and aware programs will
> continue to see them.
> 
> OTOH, if we had the streams in the kernel, unaware applications
> would continuously break the metadata and streams that the
> streams aware programs expect !

You appear not to have read any of my mails on this topic.

Properly implemented metadata can:

  (1) operate in both modes simultaneously;
  (2) work with unaware applications;
  (3) provide performance enhancements to aware applications;
  (4) provide storage enhancements to both;
  (5) provide useful features that work with standard unmodified unix tools,

all at once.  That includes program copies.

-- Jamie

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Jamie Lokier wrote:

> Properly implemented metadata can:
[snip world peace]
> all at once.  That includes program copies.

Show me the design for such a properly implemented thing.

None of the designs I've seen proposed so far are able
to do what you claim.

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Hans Reiser]

Rik van Riel wrote:

>On Thu, 26 Aug 2004, Andrew Morton wrote:
>
>  
>
>>All of which can be handled in userspace library code.
>>
>>What compelling reason is there for doing this in the kernel?
>>    
>>
>
>There's a compelling reason to do it in userspace.  If an
>unaware program copies or moves such a file with streams
>inside, it doesn't break the streams and aware programs will
>continue to see them.
>
>OTOH, if we had the streams in the kernel, unaware applications
>would continuously break the metadata and streams that the
>streams aware programs expect !
>
>  
>
Well, first off, you don't want streams in the kernel, you want all the 
little pieces that can be composed together into a stream if you so 
choose.  Streams are ugly, the pieces are all cool.

Re: silent semantic changes with reiser4

[Jeremy Allison]

On Thu, Aug 26, 2004 at 02:49:56AM -0700, Andrew Morton wrote:
> Spam <spam@tnonline.net> wrote:
> >
> >    Yes,  for  example  documents,  image  files  etc. The multiple data
> >    streams  can  contain thumbnails, info about who is editing the file
> >    (useful for networked files) etc. Could be used for version handling
> >    and much more.
> 
> All of which can be handled in userspace library code.
> 
> What compelling reason is there for doing this in the kernel?

Because without kernel support there is no way someone can
publish a new metadata type and have it automatically supported
by all application data files (ie. most apps ignore it, and only
apps that are aware of it can see it). Without kernel support
you have to have all apps agree on a data format. And
that's harder to do than getting linux kernel VFS engineers
to agree on things :-).

Jeremy.

Re: silent semantic changes with reiser4

[Rik van Riel]

On Thu, 26 Aug 2004, Jeremy Allison wrote:

> Because without kernel support there is no way someone can
> publish a new metadata type and have it automatically supported
> by all application data files (ie. most apps ignore it, and only
> apps that are aware of it can see it).

So your backup software ignores it, and after a restore you've
lost your new metadata ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: silent semantic changes with reiser4

[Felipe Alfaro Solana]

On Thursday 26 August 2004 20:17, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Jeremy Allison wrote:
> > Because without kernel support there is no way someone can
> > publish a new metadata type and have it automatically supported
> > by all application data files (ie. most apps ignore it, and only
> > apps that are aware of it can see it).
>
> So your backup software ignores it, and after a restore you've
> lost your new metadata ?

I think there's no easy way to fix this, except fixing the backup software to 
support xattrs's/streams/forks/whatever. Even on Windows, it has happened 
this way with older apps that don't support named streams.

Re: silent semantic changes with reiser4

[Hans Reiser]

Well said.

Jeremy Allison wrote:

> Yes, but then you fix your backup software and everyone lives
>
>happily ever after :-)......
>
>It's a better solution than changing *every single application
>in existance* (the "userspace library" approach).
>
>IMHO of course.
>
>Jeremy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Hans Reiser]

I think Jeremy is accurate in saying the below.

Jeremy Allison wrote:

>>What compelling reason is there for doing this in the kernel?
>>    
>>
>
>
>Because without kernel support there is no way someone can
>publish a new metadata type and have it automatically supported
>by all application data files (ie. most apps ignore it, and only
>apps that are aware of it can see it). Without kernel support
>you have to have all apps agree on a data format. And
>that's harder to do than getting linux kernel VFS engineers
>to agree on things :-).
>
>Jeremy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Anton Altaparmakov]

On Thu, 2004-08-26 at 00:32, Andrew Morton wrote:
> Spam <spam@tnonline.net> wrote:
> >
> > > In other words, if reiserfs does something special, we should make 
> > > standard interfaces for doing that special thing, so that everybody can
> > > do it without stepping on other peoples toes.
> > 
> >   Agreed  that would be the best. But how much time and effort will it
> >   be
> 
> Zero.
> 
> We can add these new features tomorrow, as reiser4-only features, with a
> plan in hand to generalise them later.
> 
> -->>__if__<<-- we think these are features which Linux should offer.

Please don't forget that if the reiser4 features are merged as they are
now, then we will likely be stuck with the API reiser4 chooses.  There
will be tools that will rely on it springing up no doubt.

Moving the reiser4 features to VFS later is fine and good, but what if
the VFS doesn't want the same API for those features?  Either we would
have to allow reiser4 to continue providing the old API even though the
VFS now provides a new, shiny API or we would have to break all existing
API users on reiser4.  Things like "I rebooted into the latest kernel
and my computer failed to boot because essential app FOO failed to
access the reiser4 API - Help!" spring to mind.

Yes, I know I am painting a rather black picture here and I know you
might well say "screw apps", its been done plenty of times in Linux
kernel development before...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net
WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/

Re: silent semantic changes with reiser4

[Christian Mayrhuber]

On Thursday 26 August 2004 11:03, Anton Altaparmakov wrote:

> Please don't forget that if the reiser4 features are merged as they are
> now, then we will likely be stuck with the API reiser4 chooses.  There
> will be tools that will rely on it springing up no doubt.
>
> Moving the reiser4 features to VFS later is fine and good, but what if
> the VFS doesn't want the same API for those features?  Either we would
> have to allow reiser4 to continue providing the old API even though the
> VFS now provides a new, shiny API or we would have to break all existing
> API users on reiser4.  Things like "I rebooted into the latest kernel
> and my computer failed to boot because essential app FOO failed to
> access the reiser4 API - Help!" spring to mind.

Andrew Morton wrote:
>b) accept the reiser4-only extensions with a view to turning them into
>   kernel-wide extensions at some time in the future, so all filesystems
>   will offer the extensions (as much as poss) or

If option b) is chosen Reiser4 can become a playground.

There is the reiser4() syscall which you surely don't want to implement for 
other filesystems.
Once there is some experience with this new fancy stuff the dust what
is useful/insecure, etc. and what is not will settle and can be condensed
into a vfs api.
Apps like samba and user scripts will have to be adapted once this is
the case, but this should not be to big a problem if this stuff is marked 
experimental.

People which want something stable can continue to use xattrs and a
magnitude of filesystems for now.

-- 
lg, Chris

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 12:45:47PM +0200, Christian Mayrhuber wrote:
> There is the reiser4() syscall which you surely don't want to implement for 
> other filesystems.

Actually a syscall that goes into filesystem code is the last thing we want.
But it's not in the submission, so let's keep that flame^H^H^H^Hdiscussion
for later.

> Once there is some experience with this new fancy stuff the dust what
> is useful/insecure, etc. and what is not will settle and can be condensed
> into a vfs api.
> Apps like samba and user scripts will have to be adapted once this is
> the case, but this should not be to big a problem if this stuff is marked 
> experimental.
> 
> People which want something stable can continue to use xattrs and a
> magnitude of filesystems for now.

Sure, no one stops you from playing around with new semantics.  But please
don't add them to the linux kernel stable series until we have semantics we
a) want to stick to for a while and b) actually work.

Re: silent semantic changes with reiser4

[Olivier Galibert]

On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote:
> Sure, no one stops you from playing around with new semantics.  But please
> don't add them to the linux kernel stable series until we have semantics we
> a) want to stick to for a while and b) actually work.

He's not proposing to add it to 2.4, is he?

  OG.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote:
> On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote:
> > Sure, no one stops you from playing around with new semantics.  But please
> > don't add them to the linux kernel stable series until we have semantics we
> > a) want to stick to for a while and b) actually work.
> 
> He's not proposing to add it to 2.4, is he?

We're talking about 2.6 here.  2.4 is the obsolete kernel series these days.

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 01:44:15PM +0100, Christoph Hellwig wrote:
>On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote:
>> He's not proposing to add it to 2.4, is he?
>We're talking about 2.6 here.  2.4 is the obsolete kernel series these days.

So run it in -mm or fork it to 2.7.

2.7 might not be a bad idea for trying such bleeding edge revolution
stuff as this.

-- 
mjt

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 03:49:29PM +0300, Markus Törnqvist wrote:
> 2.7 might not be a bad idea for trying such bleeding edge revolution
> stuff as this.

I already suggested to Linus that 2.7 is probably the right place to
experiment with that.

Re: silent semantic changes with reiser4

[Adrian Bunk]

On Thu, Aug 26, 2004 at 02:43:34PM +0200, Olivier Galibert wrote:
> On Thu, Aug 26, 2004 at 11:52:29AM +0100, Christoph Hellwig wrote:
> > Sure, no one stops you from playing around with new semantics.  But please
> > don't add them to the linux kernel stable series until we have semantics we
> > a) want to stick to for a while and b) actually work.
> 
> He's not proposing to add it to 2.4, is he?

2.6 is a stable series...

>   OG.

cu
Adrian

-- 

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Re: silent semantic changes with reiser4

[Hans Reiser]

Spam wrote:

>
>>  The fact that one filesystem will
>>offer features which other filesystems do not and cannot offer makes me
>>queasy for some reason.
>>    
>>
>
>  This last sentence makes me wonder. Where is Linux heading? The idea
>  that   a  FS  cannot  contain  features that no other FS has is very
>  scary.
>
>  I  am  all  for  uniformity, but not at the expense of shutting down
>  advanced progress that Linux is so badly needing.
>
>  This talk about old UNIX seems like people want to still live in the
>  70'ies and not look forward. Please wake up!
>
>  ~S
>
>
>
>
>  
>
well said.

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrew Morton wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>>I had not intended to respond to this because I have nothing positive to 
>>say, but Andrew said I needed to respond and suggested I should copy 
>>Linus.
>>    
>>
>
>Yes, but I didn't say "flame Christoph and ignore the issues" ;)
>  
>
Oh....;-)

>There are lots of little things to do with implementation, coding style,
>module exports, deadlocks, what code goes where, etc.  These are all normal
>daily kernel business and we should set them aside for now and concentrate
>on the bigger issues.
>  
>
Yes, you are right, but I am not sure Viro will go along with that..... ;-)

>And as I see it, there are two big issues:
>
>a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not
>   anticipate and 
>
>b) it does this within the context of just a single filesystem.
>
>I see three possible responses:
>
>a) accept the reiser4-only extensions as-is (possibly with post-review
>   modifications, of course) or
>
>b) accept the reiser4-only extensions with a view to turning them into
>   kernel-wide extensions at some time in the future, so all filesystems
>   will offer the extensions (as much as poss) or
>
>c) reject the extensions.
>
>
>My own order of preference is b) c) a). 
>
I don't object to b), though I think b) should wait for 2.7 and reiser4 
should not.

> The fact that one filesystem will
>offer features which other filesystems do not and cannot offer makes me
>queasy for some reason.
>  
>
Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
for Apple, and that means we need to put search engine and database 
functionality into the filesystem.  It takes 11 years of serious 
research to build a clean storage layer able to handle doing that.  
Reiser4 has done that, finally.  None of the other Linux filesystems 
have.  The next major release of ReiserFS is going to be bursting with 
semantic enhancements, because the prerequisites for them are in place 
now.  None of the other Linux filesystems have those prerequisites.  
They won't be able to keep up with the semantic enhancements.  This 
metafiles and file-directories stuff is actually fairly trivial stuff.

Look guys, in 1993 I anticipated the battle would be here, and I build 
the foundation for a defensive tower right at the spot MS and Apple are 
now maneuvering towards.  Help me get the next level on the tower before 
they get here.  It is one hell of a foundation, they won't be able to 
shake it, their trees are not as powerful.  Don't move reiser4 into vfs, 
use reiser4 as the vfs.  Don't write filesystems, write file plugins and 
disk format plugins and all the other kinds of plugins, and you won't be 
missing any expressive power that you really want....

Give Saveliev and I some credit.  10 years of hard work at an ivory 
tower nobody thought mattered.  Now the battle leaves the browser and 
swings our way.  Don't duplicate that infrastructure, use it. 

There is so much we could use help with if talented people like you 
chose to contribute.

>b) means that at some time in the future we need to hoist the reiser4
>extensions (at a conceptual level) (and probably with restrictions) up into
>the VFS.  This will involve much thought, argument and work.
>
>
>To get us started on this route it would really help me (and, probably,
>others) if you could describe what these API extensions are in a very
>simple way, without referring to incomprehehsible web pages,
>
what is not comprehensible....?

> and without
>using terms which non-reiser4 people don't understand.
>  
>
Well, I agree that there is value in defining things in more detail than 
we have.

>It would be best if each extension was addressed in a separate email
>thread.
>
>We also need to discuss what a reiser4 "module" is, what its capabilities
>are, and what licensing implications they have.
>
>Then, we can look at each one and say "yup, that makes sense - we want
>Linux to do that" and we can also think about how we would implement it at
>the VFS level.
>
>If we follow the above route I believe we can make progress in a technical
>direction and not get deadlocked on personal/political stuff.
>
>
>Now, an alternative to all the above is to just merge reiser4 as-is, after
>addressing all the lower-level coding issues.  And see what happens.  That
>may be a thing which Linus wishes to do.  I'm easy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Andrew Morton]

Hans Reiser <reiser@namesys.com> wrote:
>
>  >To get us started on this route it would really help me (and, probably,
>  >others) if you could describe what these API extensions are in a very
>  >simple way, without referring to incomprehehsible web pages,
>  >
>  what is not comprehensible....?

Pretty much anything at www.namesys.com.  The amount of time which is
needed to extract the technical info which one is looking for vastly
exceeds a gnat-like attention span.

As a starting point, please prepare a bullet-point list of
userspace-visible changes which the filesystem introduces, or is planned to
introduce.

And describe the "plugin" system.  Why does the filesystem need such a
thing (other filesystems get their features via `patch -p1')?

And what are the licensing implications of plugins?  Are they derived
works?  Must they be GPL'ed?

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrew Morton wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>> >To get us started on this route it would really help me (and, probably,
>> >others) if you could describe what these API extensions are in a very
>> >simple way, without referring to incomprehehsible web pages,
>> >
>> what is not comprehensible....?
>>    
>>
>
>Pretty much anything at www.namesys.com.  The amount of time which is
>needed to extract the technical info which one is looking for vastly
>exceeds a gnat-like attention span.
>
>As a starting point, please prepare a bullet-point list of
>userspace-visible changes which the filesystem introduces, or is planned to
>introduce.
>
>And describe the "plugin" system.  Why does the filesystem need such a
>thing (other filesystems get their features via `patch -p1')?
>  
>
It takes 6 months or more to become competent to change a usual 
filesystem.  Creating a new reiser4 plugin is a weekend programmer fun 
hack to do.  Weekend programmers matter, because they tend to have 
clever ideas based on understanding a need they have.   How many people 
can easily add new features to ext3 or reiserfs V3?  Very few. 

What happens if you need a disk format change?

Well, in V4, you can easily compose a plugin from plugin methods of 
other plugins, write a little piece of code with the one thing you want 
different, and add it in.  Disk format changes, no big deal, add a new 
disk format plugin, or a new item plugin, or a new node plugin, etc., 
and you got your new format.

There is a huge difference between code that is designed for reuse, and 
code that is not.  That is the difference between V3 and V4.  We were 
looking at our V3 balancing code, and it special cased each different 
kind of item (an item is a piece of something, which the balancing code 
chops objects into as it squeezes them into nodes).  It looked like the 
complexity of the balancing code was going to be N squared, where N was 
the number of different kinds of items.

So, we created item handlers, and wrote balancing code that could slice 
and dice and merge any item that implemented all of the operations 
required of an item handler.  From there it grew, and we made everything 
pluggable.

Adding features to the new code is far less than the time cost of adding 
features to the old code.  I've seen Nikita complain that something 
would take 6 weeks to do (changing key assignment algorithms), and then 
it takes him 3 afternoons, and it was because of the plugins, because 
when we did something similar in V3 it took 3 man months.

>And what are the licensing implications of plugins?  Are they derived
>works?  
>
Yes.

Hans

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 05:24, Hans Reiser wrote:
> Andrew Morton wrote:
> >
> >And describe the "plugin" system.  Why does the filesystem need such a
> >thing (other filesystems get their features via `patch -p1')?
> >  
> >
> It takes 6 months or more to become competent to change a usual 
> filesystem.  Creating a new reiser4 plugin is a weekend programmer fun 
> hack to do.  Weekend programmers matter, because they tend to have 
> clever ideas based on understanding a need they have.   How many people 
> can easily add new features to ext3 or reiserfs V3?  Very few. 
> 
> What happens if you need a disk format change?
> 
> Well, in V4, you can easily compose a plugin from plugin methods of 
> other plugins, write a little piece of code with the one thing you want 
> different, and add it in.  Disk format changes, no big deal, add a new 
> disk format plugin, or a new item plugin, or a new node plugin, etc., 
> and you got your new format.
> 

OK, real world example.  My roommate has an AKAI MPC-2000, a very
popular hardware sampler from the 90's.  The disk format is known,there
are a few utilities to edit the disks on a PC and extract the PCM
samples, but there are no tools to mount it on a modern PC.  Are you
saying that, since I know the MPC disk format, I could write a reiser4
plugin to mount an MPC drive?

If so, then Hans has an excellent point.  Users do want this kind of
thing, and it is worth having to fix tar et al.

Lee 

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote:
> OK, real world example.  My roommate has an AKAI MPC-2000, a very
> popular hardware sampler from the 90's.  The disk format is known,there
> are a few utilities to edit the disks on a PC and extract the PCM
> samples, but there are no tools to mount it on a modern PC.  Are you
> saying that, since I know the MPC disk format, I could write a reiser4
> plugin to mount an MPC drive?
> 
> If so, then Hans has an excellent point.  Users do want this kind of
> thing, and it is worth having to fix tar et al.

You don't need reiser4 for that, writing read-only linux filesystems is
trivial as soon as you have a specification of the ondisk format.

Re: silent semantic changes with reiser4

[Lee Revell]

On Thu, 2004-08-26 at 15:34, Christoph Hellwig wrote:
> On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote:
> > OK, real world example.  My roommate has an AKAI MPC-2000, a very
> > popular hardware sampler from the 90's.  The disk format is known,there
> > are a few utilities to edit the disks on a PC and extract the PCM
> > samples, but there are no tools to mount it on a modern PC.  Are you
> > saying that, since I know the MPC disk format, I could write a reiser4
> > plugin to mount an MPC drive?
> > 
> > If so, then Hans has an excellent point.  Users do want this kind of
> > thing, and it is worth having to fix tar et al.
> 
> You don't need reiser4 for that, writing read-only linux filesystems is
> trivial as soon as you have a specification of the ondisk format.
> 

Of course I could just write an MPC filesystem driver.  And, like Andrew
said, a filesystem normally gets new features with 'patch -p1'.  My
question, which was answered by a previous post, was whether the same
could be done in a reiser4 plugins.

Lee

Re: silent semantic changes with reiser4

[Dmitry Baryshkov]

Hello,
On Thu, Aug 26, 2004 at 03:26:55PM -0400, Lee Revell wrote:
> On Thu, 2004-08-26 at 05:24, Hans Reiser wrote:
> > 
> > Well, in V4, you can easily compose a plugin from plugin methods of 
> > other plugins, write a little piece of code with the one thing you want 
> > different, and add it in.  Disk format changes, no big deal, add a new 
> > disk format plugin, or a new item plugin, or a new node plugin, etc., 
> > and you got your new format.
> > 
> 
> OK, real world example.  My roommate has an AKAI MPC-2000, a very
> popular hardware sampler from the 90's.  The disk format is known,there
> are a few utilities to edit the disks on a PC and extract the PCM
> samples, but there are no tools to mount it on a modern PC.  Are you
> saying that, since I know the MPC disk format, I could write a reiser4
> plugin to mount an MPC drive?
> 

Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4
plugins? From Hans' words it seems so. If this is correct, then maybe
reiser4 core should be updated to completely replace current VFS layer?
Then it's a good point to create a branch (in old development model it
would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and
rewrite all (or at least most used) FS as reiser4 plugins. Then
everybody will be happy.

But this looks too good to be true. Perhaps I misunderstood Hans' words
aboud 'new disk format', did I?

-- 
With best wishes
Dmitry Baryshkov

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 821 bytes --]

Am Freitag, den 27.08.2004, 00:30 +0400 schrieb Dmitry Baryshkov:

> Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4
> plugins? From Hans' words it seems so. If this is correct, then maybe
> reiser4 core should be updated to completely replace current VFS layer?
> Then it's a good point to create a branch (in old development model it
> would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and
> rewrite all (or at least most used) FS as reiser4 plugins. Then
> everybody will be happy.
> 
> But this looks too good to be true. Perhaps I misunderstood Hans' words
> aboud 'new disk format', did I?

No. You can change the format the reiser4 storage tree is stored in. As
long as other filesystems don't use the same underlying storage tree
this is not possible.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Dmitry Baryshkov]

Hello,
On Thu, Aug 26, 2004 at 10:38:12PM +0200, Christophe Saout wrote:
> 
> No. You can change the format the reiser4 storage tree is stored in. As
> long as other filesystems don't use the same underlying storage tree
> this is not possible.
> 

Thank you for clarification. As I wrote, it's too good to be true.

-- 
With best wishes
Dmitry Baryshkov

Re: silent semantic changes with reiser4

[Nikita Danilov]

Christophe Saout writes:
 > Am Freitag, den 27.08.2004, 00:30 +0400 schrieb Dmitry Baryshkov:
 > 
 > > Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4
 > > plugins? From Hans' words it seems so. If this is correct, then maybe
 > > reiser4 core should be updated to completely replace current VFS layer?
 > > Then it's a good point to create a branch (in old development model it
 > > would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and
 > > rewrite all (or at least most used) FS as reiser4 plugins. Then
 > > everybody will be happy.
 > > 
 > > But this looks too good to be true. Perhaps I misunderstood Hans' words
 > > aboud 'new disk format', did I?
 > 
 > No. You can change the format the reiser4 storage tree is stored in. As
 > long as other filesystems don't use the same underlying storage tree
 > this is not possible.

Wrong, plugin is called just below entry point from the VFS to the
file-system back-end. It can use reiser4 tree, or any storage layer it
wants. Or none at all: think about pseudo-files like
foo/metas/uid---they are also implemnted by plugins.

Programmer dedicated enough can (besides programming Fortran in any
language), write reiser4 plugin for any file-system. I will leave
description of this activity to people with better command over English
language.

Nikita.

 > 

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 594 bytes --]

Am Freitag, den 27.08.2004, 01:33 +0400 schrieb Nikita Danilov:

> Wrong, plugin is called just below entry point from the VFS to the
> file-system back-end. It can use reiser4 tree, or any storage layer it
> wants. Or none at all: think about pseudo-files like
> foo/metas/uid---they are also implemnted by plugins.

Yes. But which plugins are we talking about?

What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/?

Of course you can implement another filesystem inside the plugins but
they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Nikita Danilov]

Christophe Saout writes:
 > Am Freitag, den 27.08.2004, 01:33 +0400 schrieb Nikita Danilov:
 > 
 > > Wrong, plugin is called just below entry point from the VFS to the
 > > file-system back-end. It can use reiser4 tree, or any storage layer it
 > > wants. Or none at all: think about pseudo-files like
 > > foo/metas/uid---they are also implemnted by plugins.
 > 
 > Yes. But which plugins are we talking about?
 > 
 > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/?
 > 
 > Of course you can implement another filesystem inside the plugins but
 > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right?
 > 

That was the message of my message.

Nikita.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 02:03:36AM +0400, Nikita Danilov wrote:
>  > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/?
>  > 
>  > Of course you can implement another filesystem inside the plugins but
>  > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right?
>  > 
> 
> That was the message of my message.

And I think Christophe and me already agreed in this thread that these
upper level plugin facility should go away before merge anyway.  We made
the mistake of not requesting removel of that one for XFS and now SGI
blocks it's removal.

Re: silent semantic changes with reiser4

[Hans Reiser]

Christoph Hellwig wrote:

>On Fri, Aug 27, 2004 at 02:03:36AM +0400, Nikita Danilov wrote:
>  
>
>> > What about fs/reiser4/plugin/node/ and fs/reiser4/plugin/disk_format/?
>> > 
>> > Of course you can implement another filesystem inside the plugins but
>> > they wouldn't use fs/reiser4/*.c, so this would be rather stupid. Right?
>> > 
>>
>>That was the message of my message.
>>    
>>
>
>And I think Christophe and me already agreed in this thread that these
>upper level plugin facility should go away before merge anyway.  We made
>the mistake of not requesting removel of that one for XFS and now SGI
>blocks it's removal.
>
>-
>To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
>the body of a message to majordomo@vger.kernel.org
>More majordomo info at  http://vger.kernel.org/majordomo-info.html
>Please read the FAQ at  http://www.tux.org/lkml/
>
>
>  
>
Your arrogance is misplaced for a man who has never done a significant 
piece of research.

Christoph, we are doing work, you are in the way, get out of the way.

Hans

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 01:12:44AM -0700, Hans Reiser wrote:
> Your arrogance is misplaced for a man who has never done a significant 
> piece of research.
> 
> Christoph, we are doing work, you are in the way, get out of the way.

Could you please stop the namecalling.  And if you want to check who has
done significant work on Linux check the commit logs for yoour and for
my shortname.

Re: silent semantic changes with reiser4

[Hans Reiser]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 15:26 -0400 schrieb Lee Revell:
>
>  
>
>>>Well, in V4, you can easily compose a plugin from plugin methods of 
>>>other plugins, write a little piece of code with the one thing you want 
>>>different, and add it in.  Disk format changes, no big deal, add a new 
>>>disk format plugin, or a new item plugin, or a new node plugin, etc., 
>>>and you got your new format.
>>>      
>>>
>>OK, real world example.  My roommate has an AKAI MPC-2000, a very
>>popular hardware sampler from the 90's.  The disk format is known,there
>>are a few utilities to edit the disks on a PC and extract the PCM
>>samples, but there are no tools to mount it on a modern PC.  Are you
>>saying that, since I know the MPC disk format, I could write a reiser4
>>plugin to mount an MPC drive?
>>    
>>
>
>No, the underlying storage must be a reiser4-like tree. 
>
Not necessarily.  We just encourage it....  Reiser4 is a body of code 
that can be sliced and diced as you choose, and it is designed for easy 
slicing.

>If you want to
>mount an MPC drive, write an MPC filesystem.
>
>  
>
However, this last sentence is probably sensible advice.  A mount point 
is probably the right interface from reiser4 for what you want.

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Fri, Aug 27, 2004 at 01:05:45AM -0700, Hans Reiser wrote:
> Not necessarily.  We just encourage it....  Reiser4 is a body of code 
> that can be sliced and diced as you choose, and it is designed for easy 
> slicing.

So what about slicing it into a simple posix fs as a start so we can review
that while still discussing semantic issues.  Now that would be a refreshing
thing after all that talking, no?

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2289 bytes --]

Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton:

> And describe the "plugin" system.  Why does the filesystem need such a
> thing (other filesystems get their features via `patch -p1')?

Ok, let me try. (Hans, please don't shoot me if I don't get every detail
right, I'm trying to simplify and translate)

The reiser4 core is just a big and fast storage layer using a single
tree. It is bound to a mount point and uses the page cache to manage its
memory. The only thing it can do on its own is to flush dirty data to
the disk when the VM wants to (memory pressure) or the VFS wants to
(unmount, sync) or some "plugin" wants to.
Additionally it's completely atomic (with respect to crashes, no
isolation like in databases), comparable to data=journal but without the
overhead of using a journal for everything.

Up to this point there are no users of this "database" and it does not
implement any of the VFS methods for a filesystem (except mount and
unmount perhaps).

All the functionality is provided to the plugins. You can insert,
remove, lookup or modify key:object pairs (with an index into the
object). The object is a sequence of units. For files a unit would be 1
byte and the index would be the byte offset. For directories the unit
would be 1 entry and the index would be the filename.

Now there are some plugins that define how the storage layout on the
disk is (some kind of "backend" plugins).

*And* there are plugins which are users of the "reiser4 client API" and
implement the actual VFS methods.

There's a UNIX directory plugin and a UNIX file plugin.

Directories, inodes and file content are just key:object pairs and the
plugins know how to operate on these.

There's a new plugin in work that also implements UNIX file semantics
but stores the data for that file encrypted and/or compressed.

These plugins live between the VFS and the storage layer.

Just like the filesystems live between the VFS and the block layer (Hans
would say that filesystems are VFS plugins ;-)).

> And what are the licensing implications of plugins?  Are they derived
> works?  Must they be GPL'ed?

I suppose yes, at least currently, since they can only be linked with
reiser4, there's no module infrastructure.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 02:18:49PM +0200, Christophe Saout wrote:
> The reiser4 core is just a big and fast storage layer using a single
> tree. It is bound to a mount point and uses the page cache to manage its
> memory. The only thing it can do on its own is to flush dirty data to
> the disk when the VM wants to (memory pressure) or the VFS wants to
> (unmount, sync) or some "plugin" wants to.
> Additionally it's completely atomic (with respect to crashes, no
> isolation like in databases), comparable to data=journal but without the
> overhead of using a journal for everything.
> 
> Up to this point there are no users of this "database" and it does not
> implement any of the VFS methods for a filesystem (except mount and
> unmount perhaps).

Now that part is interesting from the how to sell it to non-Linux users
POV, but not for the linux kernel.

> All the functionality is provided to the plugins. You can insert,
> remove, lookup or modify key:object pairs (with an index into the
> object). The object is a sequence of units. For files a unit would be 1
> byte and the index would be the byte offset. For directories the unit
> would be 1 entry and the index would be the filename.
> 
> Now there are some plugins that define how the storage layout on the
> disk is (some kind of "backend" plugins).
> 
> *And* there are plugins which are users of the "reiser4 client API" and
> implement the actual VFS methods.

Here comes the problem.  All the access checking/race avoidance/loop
creation avoiced, in short the posix+extension semantics are implemented
in the Linux VFS layer.  If you want to allow a second access method
(e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
inside reiser4.  But this is absolutely not what we want, because we
moved to this common as all previous version in the different
filesystems were buggy, and in fact there are problems that are simply
not solveable at the level at all (rename on directories comes to mind).

So if you want to provide an additional API you'll have to go through
the VFS to get it right - ergo a plugin architecture for upper plugins
is worthless.

Plugins for the disk-format level are an interesting and workable idea
on the other hand.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1732 bytes --]

Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig:

> Now that part is interesting from the how to sell it to non-Linux users
> POV, but not for the linux kernel.

Why? The question was what these plugins are exactly. This is the
answer.

> > *And* there are plugins which are users of the "reiser4 client API" and
> > implement the actual VFS methods.
> 
> Here comes the problem.  All the access checking/race avoidance/loop
> creation avoiced, in short the posix+extension semantics are implemented
> in the Linux VFS layer.  If you want to allow a second access method
> (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
> inside reiser4.

Are you actually listening? If you implement a filesystem there's a
place where you have to implement the Linux VFS methods. I'm talking
about inode_operations and these things. This has nothing to do with
doing anything outside the Linux VFS. And I'm not talking about these
metas either. These really don't belong inside the filesystem.

> So if you want to provide an additional API you'll have to go through
> the VFS to get it right - ergo a plugin architecture for upper plugins
> is worthless.

See, you didn't listen. ;-)

The reiser4 core doesn't know about inodes, directories or files. It's
the glue code between the VFS and the storage layer that does. It's
implemented as a plugin. This has nothing to do with semantic
enhancements yet. These should be removed for now and made a 2.7 topic.

Maybe Namesys could provide them as an additional patch from their
webpage for those who need it. The plugins that would benefit from this
(encrypted and compressed file storage) aren't ready yet anyway.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 03:00:34PM +0200, Christophe Saout wrote:
> Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig:
> 
> > Now that part is interesting from the how to sell it to non-Linux users
> > POV, but not for the linux kernel.
> 
> Why? The question was what these plugins are exactly. This is the
> answer.

I explained it below..

> > > *And* there are plugins which are users of the "reiser4 client API" and
> > > implement the actual VFS methods.
> > 
> > Here comes the problem.  All the access checking/race avoidance/loop
> > creation avoiced, in short the posix+extension semantics are implemented
> > in the Linux VFS layer.  If you want to allow a second access method
> > (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
> > inside reiser4.
> 
> Are you actually listening? If you implement a filesystem there's a
> place where you have to implement the Linux VFS methods. I'm talking
> about inode_operations and these things. This has nothing to do with
> doing anything outside the Linux VFS. And I'm not talking about these
> metas either. These really don't belong inside the filesystem.

as I wrote in this mail this absolutely _does_ belong in the filesystem,
it's a major part of it and isn't easily separatable.

> > So if you want to provide an additional API you'll have to go through
> > the VFS to get it right - ergo a plugin architecture for upper plugins
> > is worthless.
> 
> See, you didn't listen. ;-)
> 
> The reiser4 core doesn't know about inodes, directories or files. It's
> the glue code between the VFS and the storage layer that does. It's
> implemented as a plugin. This has nothing to do with semantic
> enhancements yet. These should be removed for now and made a 2.7 topic.

Oh yes, it is.  As soon as you use different access methods on an
overlapping set of objects you see exactly the problems I described.

If they don't overlap there's no point for the plugins to start with,
you'll better turn the core into a library that can be used by different
projects then.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]

Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig:

> > Why? The question was what these plugins are exactly. This is the
> > answer.
> 
> I explained it below..

It's hard to talk to you if you're not even trying to understand what
I'm trying to say. Andrew wanted to know what these plugins are. I tried
to give an answer and then you come along to tell me that it's not
relevant. Sure, Linux doesn't care about what the filesystem does
internally but Andrew wanted to know that.

> > Are you actually listening? If you implement a filesystem there's a
> > place where you have to implement the Linux VFS methods. I'm talking
> > about inode_operations and these things. This has nothing to do with
> > doing anything outside the Linux VFS. And I'm not talking about these
> > metas either. These really don't belong inside the filesystem.
> 
> as I wrote in this mail this absolutely _does_ belong in the filesystem,
> it's a major part of it and isn't easily separatable.

Huh?

Now you're completely confusing me.

First you say that that file-as-a-directory is crap then you say that it
does belong into the filesystem?

I'll try again:

inode_operations, etc... are implemented as a "reiser4 plugin". This
means that reiser4 is usable as filesystem, you can store files in a
directory hierarchy. There's absolute nothing special here. Absolutely
nothing. And yes, reiser4 plugins are invisible from the VFS. It looks
like a normal filesystem because it behaves like one.

The "metas" I was referring in the second part are this file-as-a-
directory thing. These don't belong in the filesystem and should
therefore be removed.

The latter doesn't necessarily have anything to do with plugins.
The plugins are plugins for the storage layer, not for some semantic
enhancement.

Note that I changed the topic to avoid further confusion.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 2520 bytes --]

On Thu, Aug 26, 2004 at 03:17:53PM +0200, Christophe Saout wrote:
> It's hard to talk to you if you're not even trying to understand what
> I'm trying to say. Andrew wanted to know what these plugins are. I tried
> to give an answer and then you come along to tell me that it's not
> relevant. Sure, Linux doesn't care about what the filesystem does
> internally but Andrew wanted to know that.

Huh?  I don understand what reiser4 plugins are, and I tell you that a
subset of these are _wrong_ for use in the kernel. 

> > > Are you actually listening? If you implement a filesystem there's a
> > > place where you have to implement the Linux VFS methods. I'm talking
> > > about inode_operations and these things. This has nothing to do with
> > > doing anything outside the Linux VFS. And I'm not talking about these
> > > metas either. These really don't belong inside the filesystem.
> > 
> > as I wrote in this mail this absolutely _does_ belong in the filesystem,
> > it's a major part of it and isn't easily separatable.
> 
> Huh?
> 
> Now you're completely confusing me.
> 
> First you say that that file-as-a-directory is crap then you say that it
> does belong into the filesystem?

I think you're talking about something different then me, I'm not
talking about the magic meta files but the VFS interface in general.

This VFS interface is an integral part of ævery filesystem, and it
doesn't make a whole lot to put it into a plugin.  If you want your
filesystem core portable it does to a certain extent make sense to
abstract them out, but as someone who's worked on a few such 'portable'
filesystems I can tell you that it doesn't work out as expected.

> I'll try again:
> 
> inode_operations, etc... are implemented as a "reiser4 plugin". This
> means that reiser4 is usable as filesystem, you can store files in a
> directory hierarchy. There's absolute nothing special here. Absolutely
> nothing. And yes, reiser4 plugins are invisible from the VFS. It looks
> like a normal filesystem because it behaves like one.

Now kill the whole plugin mess and let them talk directly to another.
My first mail explained to you why it doesn't make sense to have
multiple such plugins to work on a common subset of data.

> The latter doesn't necessarily have anything to do with plugins.
> The plugins are plugins for the storage layer, not for some semantic
> enhancement.

Again plugins below the pagecache (if that's what you call the "storage
layer") _do_ make sense.  plugins at the semantic layer don't.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

Am Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig:

> > First you say that that file-as-a-directory is crap then you say that it
> > does belong into the filesystem?
> 
> I think you're talking about something different then me, I'm not
> talking about the magic meta files but the VFS interface in general.
> 
> This VFS interface is an integral part of very filesystem, and it
> doesn't make a whole lot to put it into a plugin.

Right. That's why these plugins are linked in uncoditionally. It doesn't
work without them. Hence "plugins" is not a very good name.

> If you want your
> filesystem core portable it does to a certain extent make sense to
> abstract them out, but as someone who's worked on a few such 'portable'
> filesystems I can tell you that it doesn't work out as expected.

That's your opinion. reiser4 seems to work very well.

> Now kill the whole plugin mess and let them talk directly to another.

This is how reiserfs v3 did it. And Hans says that it actually
complicated things.

> My first mail explained to you why it doesn't make sense to have
> multiple such plugins to work on a common subset of data.

What I understood is that you can select exactly one plugin that e.g.
handles the file data. The default plugin is optimized for normal files,
another one could implement transparent compression or encryption. Some
of these plugins also give the storage layer hints how to group files
together to optimized performance. Neither of these things mess with the
VFS.

> > The latter doesn't necessarily have anything to do with plugins.
> > The plugins are plugins for the storage layer, not for some semantic
> > enhancement.
> 
> Again plugins below the pagecache (if that's what you call the "storage
> layer") _do_ make sense.  plugins at the semantic layer don't.

Yes, that's what I was trying to say.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 03:35:07PM +0200, Christophe Saout wrote:
> > If you want your
> > filesystem core portable it does to a certain extent make sense to
> > abstract them out, but as someone who's worked on a few such 'portable'
> > filesystems I can tell you that it doesn't work out as expected.
> 
> That's your opinion. reiser4 seems to work very well.

So how many OSes is it ported to currently?  The problem is not that it
doesn't work but that it's really hard to maintain.  And remember that
this maintaince overhead is not just for Hans but for everyone that does
VFS-level work.  Unless of course we get a blanko permission to break it
as soon as fixing the mess becomes non-trivial.

> What I understood is that you can select exactly one plugin that e.g.
> handles the file data. The default plugin is optimized for normal files,
> another one could implement transparent compression or encryption. Some
> of these plugins also give the storage layer hints how to group files
> together to optimized performance. Neither of these things mess with the
> VFS.

compression or encryption must sit below the pagecache to work nicely,
and this hint things that usually sit at the pagecache level.  But let's
assume you have a valid use for different file_operations, why don't you
simply add in different file_operations instead of adding another
internal dispatch layer?  

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2493 bytes --]

Am Donnerstag, den 26.08.2004, 15:40 +0200 schrieb Christoph Hellwig:

> > That's your opinion. reiser4 seems to work very well.
> 
> So how many OSes is it ported to currently?  The problem is not that it
> doesn't work but that it's really hard to maintain.  And remember that
> this maintaince overhead is not just for Hans but for everyone that does
> VFS-level work.

Where does it add overhead? It's just the interface between the reiser4
storage layer and the rest abstracted out.

Ok, there's the code that dispatches the VFS method calls to the plugin
call but I would say that's about it. I didn't develop reiser4 so I
don't know.

/* ->mkdir() VFS method in reiser4 inode_operations */
static int
reiser4_mkdir(struct inode *parent      /* inode of parent
                                         * directory */ ,
              struct dentry *dentry     /* dentry of new object to
                                         * create */ ,
              int mode /* new object's mode */ )
{
        reiser4_object_create_data data;

        reiser4_stat_inc_at(parent->i_sb, vfs_calls.mkdir);

        data.mode = S_IFDIR | mode;
        data.id = DIRECTORY_FILE_PLUGIN_ID;
        return invoke_create_method(parent, dentry, &data);
}

The invoke_create_method then just calls the create method of the plugin
after setting up a context and lookup up the plugin.

xfs_iops.c seems to do something similar.

>   Unless of course we get a blanko permission to break it
> as soon as fixing the mess becomes non-trivial.

Well, I think so. It's done relatively often.

> > What I understood is that you can select exactly one plugin that e.g.
> > handles the file data. The default plugin is optimized for normal files,
> > another one could implement transparent compression or encryption. Some
> > of these plugins also give the storage layer hints how to group files
> > together to optimized performance. Neither of these things mess with the
> > VFS.
> 
> compression or encryption must sit below the pagecache to work nicely,
> and this hint things that usually sit at the pagecache level.  But let's
> assume you have a valid use for different file_operations, why don't you
> simply add in different file_operations instead of adding another
> internal dispatch layer?  

I don't know, ask Hans. How could the VFS know it a filesystem wants to
do something specific with a file that is completely transparent to the
VFS?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

> > compression or encryption must sit below the pagecache to work nicely,
> > and this hint things that usually sit at the pagecache level.  But let's
> > assume you have a valid use for different file_operations, why don't you
> > simply add in different file_operations instead of adding another
> > internal dispatch layer?  
> 
> I don't know, ask Hans. How could the VFS know it a filesystem wants to
> do something specific with a file that is completely transparent to the
> VFS?

The VFS shouldn't, that the whole point.  That's why it allows the
filesystem to register different method tables for each object.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1191 bytes --]

Am Donnerstag, den 26.08.2004, 17:37 +0200 schrieb Christoph Hellwig:

> > > compression or encryption must sit below the pagecache to work nicely,
> > > and this hint things that usually sit at the pagecache level.  But let's
> > > assume you have a valid use for different file_operations, why don't you
> > > simply add in different file_operations instead of adding another
> > > internal dispatch layer?  
> > 
> > I don't know, ask Hans. How could the VFS know it a filesystem wants to
> > do something specific with a file that is completely transparent to the
> > VFS?
> 
> The VFS shouldn't, that the whole point.  That's why it allows the
> filesystem to register different method tables for each object.

Only the objects it can distinguish.

>         ops->file    = reiser4_file_operations;
>         ops->symlink = reiser4_symlink_inode_operations;
>         ops->special = reiser4_special_inode_operations;
>         ops->dentry  = reiser4_dentry_operations;
>         ops->as      = reiser4_as_operations;

How could reiser4 register other operations for files that should be
stored encrypted or compressed? It's all under reiser4_file_operations.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 05:48:54PM +0200, Christophe Saout wrote:
> > > I don't know, ask Hans. How could the VFS know it a filesystem wants to
> > > do something specific with a file that is completely transparent to the
> > > VFS?
> > 
> > The VFS shouldn't, that the whole point.  That's why it allows the
> > filesystem to register different method tables for each object.
> 
> Only the objects it can distinguish.

Yes, every inode can have different operation vectors.  Which is the
smallest possible object the VFS knows about.

> >         ops->file    = reiser4_file_operations;
> >         ops->symlink = reiser4_symlink_inode_operations;
> >         ops->special = reiser4_special_inode_operations;
> >         ops->dentry  = reiser4_dentry_operations;
> >         ops->as      = reiser4_as_operations;
> 
> How could reiser4 register other operations for files that should be
> stored encrypted or compressed? It's all under reiser4_file_operations.

Again, your confusing upper and lower plugins.  For things happening
below the pagecache you could register different address_space
operations which sometimes makes sense.  But you want e.g. different
inode_operations for directories vs symlinks vs files.

Please read through some linux filesystem code, okay :)

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 986 bytes --]

Am Donnerstag, den 26.08.2004, 18:06 +0200 schrieb Christoph Hellwig:

> Again, your confusing upper and lower plugins.  For things happening
> below the pagecache you could register different address_space
> operations which sometimes makes sense.  But you want e.g. different
> inode_operations for directories vs symlinks vs files.

Ok, I see your point. aops. Sorry.
Looking at the code this could be done. The wrappers that dispatch the
operations are really small and call the plugin that is registered with
the inode of the mapping. Instead it could have directly set the
corresponding operations. Right. The wrappers are doing a few things
before calling the plugin. That could be done the other way round too.
But that's more of an implementation issue and could still be changed.

> Please read through some linux filesystem code, okay :)

I'm partly familiar with that, I just either misunderstood you or wasn't
really thinking (probably the latter...).


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 06:14:19PM +0200, Christophe Saout wrote:
> Ok, I see your point. aops. Sorry.
> Looking at the code this could be done. The wrappers that dispatch the
> operations are really small and call the plugin that is registered with
> the inode of the mapping. Instead it could have directly set the
> corresponding operations. Right. The wrappers are doing a few things
> before calling the plugin. That could be done the other way round too.
> But that's more of an implementation issue and could still be changed.

I agree that it's an implementation issue.  But it's also a good proof
for how Hans tries to ignore all the existing infrastructure for various
reasons.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 964 bytes --]

Am Donnerstag, den 26.08.2004, 17:25 +0100 schrieb Christoph Hellwig:

> > Looking at the code this could be done. The wrappers that dispatch the
> > operations are really small and call the plugin that is registered with
> > the inode of the mapping. Instead it could have directly set the
> > corresponding operations. Right. The wrappers are doing a few things
> > before calling the plugin. That could be done the other way round too.
> > But that's more of an implementation issue and could still be changed.
> 
> I agree that it's an implementation issue.  But it's also a good proof
> for how Hans tries to ignore all the existing infrastructure for various
> reasons.

I definitely agree with you on that one.

It bugged me from the beginning because things like these are very
likely to suffer from unnecessary problems that have been solved in the
common code or might be solved when correctly done in the common code in
the future.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Hans Reiser]

Christophe Saout wrote:

>
>
>I don't know, ask Hans. How could the VFS know it a filesystem wants to
>do something specific with a file that is completely transparent to the
>VFS?
>
>  
>
To know what method to use, you must determine the pluginid, and then 
find the method within that plugin for that vfs operation.

As for overhead, well, who eats whose dust in the benchmarks....?

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:
 > Christophe Saout wrote:
 > 
 > >
 > >
 > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
 > >do something specific with a file that is completely transparent to the
 > >VFS?
 > >
 > >  
 > >
 > To know what method to use, you must determine the pluginid, and then 
 > find the method within that plugin for that vfs operation.
 > 
 > As for overhead, well, who eats whose dust in the benchmarks....?

Whoever sponsors the benchmark usually wins. Had you forgotten that
mongo setup used by http://www.namesys.com/benchmarks.html was specially
`tuned' to reach peak reiser4 performance? Remember why you decided to
turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
_second_ stalls with reiser4 under high io loads (large atom is
obviously being flushed and everyone waits on it...). In my opinion, it
is such things that are of utmost importance for real reiser4
acceptance, not how to name `metas' sub-directory.

Nikita.

Re: reiser4 plugins

[Hans Reiser]

Nikita Danilov wrote:

>Hans Reiser writes:
> > Christophe Saout wrote:
> > 
> > >
> > >
> > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
> > >do something specific with a file that is completely transparent to the
> > >VFS?
> > >
> > >  
> > >
> > To know what method to use, you must determine the pluginid, and then 
> > find the method within that plugin for that vfs operation.
> > 
> > As for overhead, well, who eats whose dust in the benchmarks....?
>
>Whoever sponsors the benchmark usually wins. Had you forgotten that
>mongo setup used by http://www.namesys.com/benchmarks.html was specially
>`tuned' to reach peak reiser4 performance? Remember why you decided to
>turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
>_second_ stalls with reiser4 under high io loads (large atom is
>obviously being flushed and everyone waits on it...). In my opinion, it
>is such things that are of utmost importance for real reiser4
>acceptance, not how to name `metas' sub-directory.
>
>Nikita.
>
>
>  
>
If you ask real users, they say that reiser4 is fast, and their 
experience matches our benchmark.  You can criticize the benchmark if 
you want, but then you should run your own and publish it.

There are still plenty of things needing fixing though, and you are 
right that they are more important than the pretext being attempted for 
keeping us out of the kernel.  Probably you could supply them with many 
superior pretexts.;-)

The 90 second stalls, those should be fixed by debugging copy on capture 
and turning it on, yes?  You can hardly claim I failed to push for the 
completion of copy on capture....  and of course the reason it is not 
done is simply that there is too much to do....  I (sincerely) hope vs 
enjoys his vacation, the whole team is working too hard.  I am doing 7 
day weeks now to try to make payroll for us and keep darpa happy at the 
same time.

fsync performance needs fixing badly, and Chris is right in his 
diagnosis of what we can do to fix it.

Hans

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:
 > Nikita Danilov wrote:
 > 
 > >Hans Reiser writes:
 > > > Christophe Saout wrote:
 > > > 
 > > > >
 > > > >
 > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
 > > > >do something specific with a file that is completely transparent to the
 > > > >VFS?
 > > > >
 > > > >  
 > > > >
 > > > To know what method to use, you must determine the pluginid, and then 
 > > > find the method within that plugin for that vfs operation.
 > > > 
 > > > As for overhead, well, who eats whose dust in the benchmarks....?
 > >
 > >Whoever sponsors the benchmark usually wins. Had you forgotten that
 > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
 > >`tuned' to reach peak reiser4 performance? Remember why you decided to
 > >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
 > >_second_ stalls with reiser4 under high io loads (large atom is
 > >obviously being flushed and everyone waits on it...). In my opinion, it
 > >is such things that are of utmost importance for real reiser4
 > >acceptance, not how to name `metas' sub-directory.
 > >
 > >Nikita.
 > >
 > >
 > >  
 > >
 > If you ask real users, they say that reiser4 is fast, and their 
 > experience matches our benchmark.  You can criticize the benchmark if 

They experience 90 second stalls. And please, do not tell me how fast
reiser4 is, I spent a lot of time working with it, and I know very well
when it's fast, and when it's deadly slow.

 > you want, but then you should run your own and publish it.

I did, after which you told me to turn OVERWRITE and MODIFY phases off,
beause performance was horrible.

I not criticizing mongo benchmark per se. I think that it is
fundamentally wrong to use results that were deliberatly manipulated to
get best appearance to reiser4 (by omitting work-loads where it performs
poorly) as an argument. It's not clear who will, according to your
colorful expression, `eat dust' as a result of this. Or do you think
that users never overwrite of modify files in real life?

 > The 90 second stalls, those should be fixed by debugging copy on capture 
 > and turning it on, yes?  You can hardly claim I failed to push for the 

No. I described so many times to you, why COC is ineffectual here.

 > Hans

Nikita.

Re: reiser4 plugins

[Hans Reiser]

Nikita Danilov wrote:

> > >Whoever sponsors the benchmark usually wins. Had you forgotten that
> > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
> > >`tuned' to reach peak reiser4 performance? Remember why you decided to
> > >turn OVERWRITE and MODIFY phases off?
>
What I should have done was what I did with fsync performance.  With 
fsync performance I told people that we had not yet tuned for it, please 
wait a bit and we will tune for it, for now it sucks.

Instead what I did was discuss with Zam at the time how it could be 
fixed, leave it off the website until Zam was given a chance to fix it, 
and then I managed to forget about it.   After release one remembers 
what all the things that should have been fixed before release were, sigh.

Zam and I both know what is needed to fix performance in these phases, 
and I just spoke with him on the phone.  I imagine it is 2 weeks of work 
for him, but it could be up to 6 weeks.  He will comment later.

The fix should consist of the following:

1) tweak the relocation policies (zam will say more about this, as he is 
the maintainer of them)

2) optimize overwrite set so that in the modify phase we behave like a 
write twice journaling filesystem, which means implement a reserved for 
the journal only area that exists as long as plenty of disk space is free.

3) finish the repacker (but we won't do that this month....)

The modify phase, which picks a random block to modify, is a worst case 
for a wandering log without a repacker.  We could actually do very very 
well at that kind of activity with a repacker, probably better than a 
fixed journal, but for now we should try acting like a write twice 
journal for atoms that small.

So, having realized my error thanks to the gracious kindness of how you 
pointed it out, we will modify the web site to say that we are not yet 
tuned for and currently suck at modifying random blocks in the middle of 
files, and appending small amounts to the end of them, and overwriting 
small files, but that we think we know what is needed to fix those things.

I think your characterization of my reasons was unkind and also unfair.  
I will pass on saying similarly unkind things about you as you are 
mostly a good person.  You never did seem to understand me.  You didn't 
understand the reasons for my technical designs (when they worked it 
made no sense to you that they did), and you didn't understand me as a 
person. 

I wish you well in your new job, and thank you for the hard work you did 
for me.

Hans

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:
 > Nikita Danilov wrote:
 > 
 > > > >Whoever sponsors the benchmark usually wins. Had you forgotten that
 > > > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
 > > > >`tuned' to reach peak reiser4 performance? Remember why you decided to
 > > > >turn OVERWRITE and MODIFY phases off?
 > >
 > What I should have done was what I did with fsync performance.  With 
 > fsync performance I told people that we had not yet tuned for it, please 
 > wait a bit and we will tune for it, for now it sucks.

Right, and instead of this you are now claiming that these `benchmarks'
show superior reiser4 performance. Moreover, you are doing this
aggressively and proposing other people to `eat the dust' over what
happened to only exist due to your failing to remember something. As
some other notable LKML poster put it `inform yourself before
posting'. :-)

 > 
 > Instead what I did was discuss with Zam at the time how it could be 
 > fixed, leave it off the website until Zam was given a chance to fix it, 

It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were
turned off after switching to large keys.

 > and then I managed to forget about it.   After release one remembers 
 > what all the things that should have been fixed before release were, sigh.
 > 

[...]

 > 
 > I think your characterization of my reasons was unkind and also unfair.  

What fairness and kindness one expects after styling others as `puppies'
in public, and commenting on their hair style in purportedly technical
argument?

[...]

 > 
 > Hans

Nikita.

Re: reiser4 plugins

[Hans Reiser]

Nikita Danilov wrote:

> > 
> > Instead what I did was discuss with Zam at the time how it could be 
> > fixed, leave it off the website until Zam was given a chance to fix it, 
>
>It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were
>turned off after switching to large keys.
>
> 
>
The online repacker is the best technical fix for these problems.  Tight 
packing is generally more damaged in its layout by small changes than 
loose packing, and V4 is the ultimate tight packer.  It is not clear to 
me that making V4 pack more loosely is necessarily a good idea.  These 
phases disturb the original packing at flush time, and the other phases 
don't.  One solution to try is packing less tightly.  Maybe once every 
megabyte skip forward 1 megabyte, and once every 64 megabytes skip to a 
random disk location, when allocating large new atoms.  I think I prefer 
using a repacker, but we should try it and see.  The other solutions of 
the previous email should also help a lot.  The issues of layout in this 
are similar to the ones improved by the fibration plugin.

I remember talking with not just you but zam about how we could fix it, 
and there was too much in the queue of work, and everyone was 
complaining to me that we should be debugging not optimizing, and you 
were the only one who thought it was a big deal.  I guess you still 
think it is a big issue and I still think things are good enough to use 
without it.  I still think Zam understood the issues better than you did 
(allocation is his code not yours).

There are some layout optimizations that a repacker can do best.  Still, 
there are some rough spots in the current allocation policy, and we 
should look at it.

Probably you will have reason to howl if we setup a benchmark which 
disturbs things with these phases, runs the repacker (I hope to have one 
in 6 months), and then measures our read performance compared to other 
filesystems without a repacker....;-)

Hans

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:

[...]

 > 
 > I remember talking with not just you but zam about how we could fix it, 
 > and there was too much in the queue of work, and everyone was 
 > complaining to me that we should be debugging not optimizing, and you 
 > were the only one who thought it was a big deal.  I guess you still 

You are trying to evade the point. And the point is not how performance
in these phases (and their impact on other phases) can be improved, or
what excuses did you have for not `doing' it, but the fact that after it
was found that large keys behave badly, these phases were turned off,
others were switched to lexicographical operation, _and_ resulting
benchmark is used as a basis to call other people names.

 > think it is a big issue and I still think things are good enough to use 
 > without it.  I still think Zam understood the issues better than you did 
 > (allocation is his code not yours).
 > 
 > There are some layout optimizations that a repacker can do best.  Still, 
 > there are some rough spots in the current allocation policy, and we 
 > should look at it.
 > 
 > Probably you will have reason to howl if we setup a benchmark which 

Surely, as a `big dog' what other sounds can I produce? :)

 > disturbs things with these phases, runs the repacker (I hope to have one 
 > in 6 months), and then measures our read performance compared to other 
 > filesystems without a repacker....;-)

That's ok with me, but it would make sense to remove unfair benchmarks
from the site until then.

 > 
 > Hans

Nikita.

Re: reiser4 plugins

[Alex Zarochentsev]

On Fri, Aug 27, 2004 at 10:55:50PM +0400, Nikita Danilov wrote:
> Hans Reiser writes:
>  > Nikita Danilov wrote:
>  > 
>  > >Hans Reiser writes:
>  > > > Christophe Saout wrote:
>  > > > 
>  > > > >
>  > > > >
>  > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
>  > > > >do something specific with a file that is completely transparent to the
>  > > > >VFS?
>  > > > >
>  > > > >  
>  > > > >
>  > > > To know what method to use, you must determine the pluginid, and then 
>  > > > find the method within that plugin for that vfs operation.
>  > > > 
>  > > > As for overhead, well, who eats whose dust in the benchmarks....?
>  > >
>  > >Whoever sponsors the benchmark usually wins. Had you forgotten that
>  > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
>  > >`tuned' to reach peak reiser4 performance? Remember why you decided to
>  > >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
>  > >_second_ stalls with reiser4 under high io loads (large atom is
>  > >obviously being flushed and everyone waits on it...). In my opinion, it
>  > >is such things that are of utmost importance for real reiser4
>  > >acceptance, not how to name `metas' sub-directory.
>  > >
>  > >Nikita.
>  > >
>  > >
>  > >  
>  > >
>  > If you ask real users, they say that reiser4 is fast, and their 
>  > experience matches our benchmark.  You can criticize the benchmark if 
> 
> They experience 90 second stalls. And please, do not tell me how fast
> reiser4 is, I spent a lot of time working with it, and I know very well
> when it's fast, and when it's deadly slow.
> 
>  > you want, but then you should run your own and publish it.
> 
> I did, after which you told me to turn OVERWRITE and MODIFY phases off,
> beause performance was horrible.

Hmm, I think not the modify/overwrite phases performance were the problem.  The
modify and overwrite mongo phases fragment the filesystem too much, it had a
bad effect visible in the read phase and even in delete phase.  

We measured/optimized read performance  in assumption that fs is not
fragmented, why we should measure bad effect of overwrite instead?  

other fs do not relocate already allocated blocks, negative effect of 
file set overwrite is zero for them.  We could do the same for reiser4
by tuning flush algorithm and journal blocks allocation.

If users want to see read and delete performance as they are, overwrite phase
should be excluded, at least until we teach flush algorithm to not break good
block allocation.

If one wants to measure overwite performance, it is easy to enable overwrite
phase in mongo, but read and delete speed will be affected (it was so when I
tried to optimize delete).

> I not criticizing mongo benchmark per se. I think that it is
> fundamentally wrong to use results that were deliberatly manipulated to
> get best appearance to reiser4 (by omitting work-loads where it performs
> poorly) as an argument. It's not clear who will, according to your
> colorful expression, `eat dust' as a result of this. 
> Or do you think
> that users never overwrite of modify files in real life?

not whole file set.  I mean it is unusual to overwrite all files in the fs.

>  > The 90 second stalls, those should be fixed by debugging copy on capture 
>  > and turning it on, yes?  You can hardly claim I failed to push for the 
> 
> No. I described so many times to you, why COC is ineffectual here.

There are other ideas :) but, it seems to me, if we will focus on benchmarks,
we never complete plugins, metafiles and other useful things.  What I want
from reiser4 is a possiblity to encrypt and compress ~/Mail/ :).

> 
>  > Hans
> 
> Nikita.

-- 
Alex.

Re: reiser4 plugins

[Steve Bergman]

On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote:
> >
> If you ask real users, they say that reiser4 is fast, and their 
> experience matches our benchmark.  You can criticize the benchmark if 
> you want, but then you should run your own and publish it.
> 


As a "real" desktop user who just converted all his partitions from ext3
to reiser4, I have not, as yet, noticed any startling performance
increase.  Being slightly slightly irked to hear that the benchmark
numbers that have been paraded around on Slashdot and the internet in
general, at ext3's expense, have had reiser4's "bad" results surgically
extracted, I am running my own benchmarks to get the real story on
reiser4/ext3 mongo performance on my, rather average, desktop hardware.

I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel.

Unfortunately, I get an error from mongo.pl that "Done" is not a numeric
argument at line 439.

I have done this to fix it:


--- mongo.pl    2004-08-27 17:07:01.681723313 -0500
+++ mongo_fixed.pl      2004-08-27 17:06:51.369306735 -0500
@@ -429,8 +429,8 @@
        if ( -e ${ERR_FILE}) {
            &DIE ("\nEXITED WITH FAIL\n");
        }
-       my $real = (split ' ', $time_output[0])[1];
-       my $cpu  = (split ' ', $time_output[2])[1];
+       my $real = (split ' ', $time_output[1])[1];
+       my $cpu  = (split ' ', $time_output[3])[1];
  
        unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) {
          LOG "@time_output";


What it gets me is the "real" line of the "time" output for "STAT
REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME".
i.e. only system time is counted. I believe this was the intent of the
original code, but want to verify before continuing.

Thanks,
Steve Bergman

Re: reiser4 plugins

[Hans Reiser]

Steve Bergman wrote:

>On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote:
>  
>
>>If you ask real users, they say that reiser4 is fast, and their 
>>experience matches our benchmark.  You can criticize the benchmark if 
>>you want, but then you should run your own and publish it.
>>
>>    
>>
>
>
>As a "real" desktop user who just converted all his partitions from ext3
>to reiser4, I have not, as yet, noticed any startling performance
>increase.  Being slightly slightly irked to hear that the benchmark
>numbers that have been paraded around on Slashdot and the internet in
>general, at ext3's expense, have had reiser4's "bad" results surgically
>extracted, I am running my own benchmarks to get the real story on
>reiser4/ext3 mongo performance on my, rather average, desktop hardware.
>
>I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel.
>
>Unfortunately, I get an error from mongo.pl that "Done" is not a numeric
>argument at line 439.
>
>I have done this to fix it:
>
>
>--- mongo.pl    2004-08-27 17:07:01.681723313 -0500
>+++ mongo_fixed.pl      2004-08-27 17:06:51.369306735 -0500
>@@ -429,8 +429,8 @@
>        if ( -e ${ERR_FILE}) {
>            &DIE ("\nEXITED WITH FAIL\n");
>        }
>-       my $real = (split ' ', $time_output[0])[1];
>-       my $cpu  = (split ' ', $time_output[2])[1];
>+       my $real = (split ' ', $time_output[1])[1];
>+       my $cpu  = (split ' ', $time_output[3])[1];
>  
>        unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) {
>          LOG "@time_output";
>
>
>What it gets me is the "real" line of the "time" output for "STAT
>REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME".
>i.e. only system time is counted. I believe this was the intent of the
>original code, but want to verify before continuing.
>
>Thanks,
>Steve Bergman
>
>
>
>  
>
I didn't write this (more precisely, it only vaguely resembles what I 
wrote in 1996).  Are you saying that it reports system time as real 
time?  If yes, then it is an error, we need to go remove a bunch of 
numbers from our benchmarks, and thanks for finding it.

Zam, please comment.

Hans

Re: reiser4 plugins

[Alex Zarochentsev]

On Sat, Aug 28, 2004 at 05:09:28AM -0500, Steve Bergman wrote:
> On Fri, 2004-08-27 at 23:54 -0700, Hans Reiser wrote:
> > >
> > I didn't write this (more precisely, it only vaguely resembles what I 
> > wrote in 1996).  Are you saying that it reports system time as real 
> > time?  If yes, then it is an error, we need to go remove a bunch of 
> > numbers from our benchmarks, and thanks for finding it.
> > 
> > Zam, please comment.
> > 
> 
> No.  I'm saying that on my setup (kernel 2.6.8.1-mm4/perl 5.8.5/bash
> 3.00) the first line returned from running the test is the:
> 
> [1]+  Done ...

> line which throws the array indexes off by 1 and I always get 0 for the
> real time and, yes, the real time gets reported as the system time, I
> think.  Plus I get a warning about the fact that "Done" is not numeric.
> This is obviously a problem with my particular setup.

Yes. but that real/sys time parsing isn't reliable.   

> After bumping the indexes up by 1, I get the correct real time reported
> as "STAT REAL_TIME".  And the system time is reported as "STAT
> CPU_TIME".
> 
> "STAT CPU_UTIL", however is computed in a completely different way based
> on numbers collected from /proc/stat.  If I'm, reading
> linux/Documentation/filesystems/proc.txt correctly, it is trying to
> return the (system time) / (total time).  The total time agrees with
> "STAT REAL_TIME".  However, the (system time) that it comes up with here
> is always considerably higher than "STAT CPU_TIME".

CPU_UTIL counts other background processes too.   The foreground process may just wait
when all work is done by pdflush.  I think CPU_UTIL is more useful than CPU_TIME.

> 
> User error is quite possible, as I am:
> 
> 1. Just getting to know mongo.
> 2. Not a perl guy.

i don't like perl too much, but seems Perl is perfect for the things you just did (the fixes).

> 3. There is obviously something mongo.pl doesn't like about my setup.
> 
> -Steve Bergman
> 
> P.S. In the 2 tests I've run, reiser4 is not doing all that badly
> against ext3 in OVERWRITE and MODIFY, though ext3 does come in faster.
> Reiser4 trails badly on APPEND however, ext3 (data=ordered, without
> htree) being some 2.5 - 4 times faster. 

-- 
Alex.

Re: reiser4 plugins

[Alex Zarochentsev]

On Fri, Aug 27, 2004 at 05:29:07PM -0500, Steve Bergman wrote:
> On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote:
> > >
> > If you ask real users, they say that reiser4 is fast, and their 
> > experience matches our benchmark.  You can criticize the benchmark if 
> > you want, but then you should run your own and publish it.
> > 
> 
> 
> As a "real" desktop user who just converted all his partitions from ext3
> to reiser4, I have not, as yet, noticed any startling performance
> increase.  Being slightly slightly irked to hear that the benchmark
> numbers that have been paraded around on Slashdot and the internet in
> general, at ext3's expense, have had reiser4's "bad" results surgically
> extracted, I am running my own benchmarks to get the real story on
> reiser4/ext3 mongo performance on my, rather average, desktop hardware.
> 
> I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel.
> 
> Unfortunately, I get an error from mongo.pl that "Done" is not a numeric
> argument at line 439.
> 
> I have done this to fix it:
> 
> 
> --- mongo.pl    2004-08-27 17:07:01.681723313 -0500
> +++ mongo_fixed.pl      2004-08-27 17:06:51.369306735 -0500
> @@ -429,8 +429,8 @@
>         if ( -e ${ERR_FILE}) {
>             &DIE ("\nEXITED WITH FAIL\n");
>         }
> -       my $real = (split ' ', $time_output[0])[1];
> -       my $cpu  = (split ' ', $time_output[2])[1];
> +       my $real = (split ' ', $time_output[1])[1];
> +       my $cpu  = (split ' ', $time_output[3])[1];
>   
>         unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) {
>           LOG "@time_output";
> 
> 
> What it gets me is the "real" line of the "time" output for "STAT
> REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME".
> i.e. only system time is counted. I believe this was the intent of the
> original code, but want to verify before continuing.

it works on our test servers which are SuSE9.0/9.1.

I think there is a dependency on system utilities,  "Done" is not printed by
mongo.pl or any other program from the mongo package.

It would be nice to tell us what linux distro you are using and what mongo
phase fails. 

> Thanks,
> Steve Bergman

Thanks.

-- 
Alex.

Re: reiser4 plugins

[Hans Reiser]

Christoph Hellwig wrote:

>
>  The problem is not that it
>doesn't work but that it's really hard to maintain.  
>
You really like to talk about what you know nothing of, yes?

Reiser4 is FAR easier to maintain than V3.  Ask anyone on reiserfs-dev.  
It was designed to reduce our software maintenance costs. 

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Rik van Riel]

On Thu, 26 Aug 2004, Christophe Saout wrote:

> That's your opinion. reiser4 seems to work very well.

Have you tried /bin/cp, or a backup+restore ?

What happened to your file streams ?

-- 
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Chris Mason]

On Thu, 2004-08-26 at 09:52, Rik van Riel wrote:
> On Thu, 26 Aug 2004, Christophe Saout wrote:
> 
> > That's your opinion. reiser4 seems to work very well.
> 
> Have you tried /bin/cp, or a backup+restore ?
> 
> What happened to your file streams ?

Shrug, backup programs can be fixed.  The fact that we couldn't backup
16GB files before O_LARGEFILE wasn't a reason not to do it.

IMHO it's more important to ask the question 'is this important for
linux' and then we can figure out how to make it work.

-chris

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 560 bytes --]

Am Donnerstag, den 26.08.2004, 09:52 -0400 schrieb Rik van Riel:

> > That's your opinion. reiser4 seems to work very well.
> 
> Have you tried /bin/cp, or a backup+restore ?
> 
> What happened to your file streams ?

1. I wasn't talking about reiser4 plugins, not about the
   file-as-a-directory thing or file streams
2. reiser4 doesn't currently have those
3. I don't really like file streams

:-)

But I'm using ACLs on a server and had to forward-port an rsync patch to
make full backups using rsync possible. That's not really better.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Thu, Aug 26, 2004 at 03:35:07PM +0200, Christophe Saout wrote:
> Am Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig:
> 
> > > First you say that that file-as-a-directory is crap then you say that it
> > > does belong into the filesystem?
> > 
> > I think you're talking about something different then me, I'm not
> > talking about the magic meta files but the VFS interface in general.
> > 
> > This VFS interface is an integral part of very filesystem, and it
> > doesn't make a whole lot to put it into a plugin.
> 
> Right. That's why these plugins are linked in uncoditionally. It doesn't
> work without them. Hence "plugins" is not a very good name.

its still plugins no matter what. they just emulate "conventional filesystem"
behavior for VFS, but at some point you might want just to scrap VFS..
if you'd like to.

And I honestly dont understand whats the other Christoph's worries are about.

Its got perfomance. Its there. It can emulate "conventional filesystem" behaviour,
 for legacy apps. Thouse two that are currently crippled by metas, we 
are so happy arguing about, will get fixed fast. Thats a point where you could
happily STOP and live with your happy "conventional filesystem emulation", yet
enjoying perfomance aspects (if perfomance hurts you, do not compile reiserfs,
just like one big-red distro does, nobody would not complain. except your hard-drive
mechanics sick of seeks).
 But most people would not stop, and its good.

Yes, I think it would be nice to have this infrastructure in VFS. Technically.
But its not possible, cause of "committee clusterfuck". Socially. Stupidly.




-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote:
> Yes, I think it would be nice to have this infrastructure in VFS. Technically.
> But its not possible, cause of "committee clusterfuck". Socially. Stupidly.

Please explain your problems.  I've not seen a single actually working
proposal yet.  Current reiser4 implementation apparently oopses and
breaks guaranteed fs semantics, it's not going in in either reiser4 or
the vfs.

Al has started a thread to hash out working semantics, but there's not been
a single namesys person involved.  Similar all of you have absolutely ignore
every single technical question or comment but started a flamefest
consisting mostly of personal attacs and "mine is longer" politics. 

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 01:22:55PM +0200, Christoph Hellwig wrote:
> On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote:
> > Yes, I think it would be nice to have this infrastructure in VFS. Technically.
> > But its not possible, cause of "committee clusterfuck". Socially. Stupidly.
> 
> Please explain your problems.  I've not seen a single actually working
> proposal yet.  Current reiser4 implementation apparently oopses and

Not true. If true - send bugreports.

Its work for me couple of months. there were few hiccups, but they got fixed
quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS
limitations. Yes, i use it with LVM2 and stuff...

de facto its even more stable then reiserfs 3.6, which got issues with
CONFIG_PREEMPT.

> breaks guaranteed fs semantics, it's not going in in either reiser4 or
> the vfs.
A
> 
> Al has started a thread to hash out working semantics, but there's not been
> a single namesys person involved.  Similar all of you have absolutely ignore
Al message had a reply.
"namesys persons" is Hans Reiser.  Sufficient ?

> every single technical question or comment but started a flamefest
> consisting mostly of personal attacs and "mine is longer" politics. 
no need to comment on this.

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote:
> Not true. If true - send bugreports.
> 
> Its work for me couple of months. there were few hiccups, but they got fixed
> quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS
> limitations. Yes, i use it with LVM2 and stuff...

See the mails from Christophe in this thread.

> > breaks guaranteed fs semantics, it's not going in in either reiser4 or
> > the vfs.
> A
> > 
> > Al has started a thread to hash out working semantics, but there's not been
> > a single namesys person involved.  Similar all of you have absolutely ignore
> Al message had a reply.
> "namesys persons" is Hans Reiser.  Sufficient ?

http://marc.theaimsgroup.com/?t=109355582600002&r=1&w=2

I can't see Hans anywhere.  And honestly Hans has been so out of touch
with VFS internals that some person actually understanding the issue
might be helpfull.  That would probably whoever has taken over Nikita's
position.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 01:51:04PM +0200, Christoph Hellwig wrote:
> On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote:
> > Its work for me couple of months. there were few hiccups, but they got fixed
> > quickly by zam@namesys.com. only ext2 partition is /boot cause of BIOS
> > limitations. Yes, i use it with LVM2 and stuff...
> 
> See the mails from Christophe in this thread.

seen. noted. 

> > > breaks guaranteed fs semantics, it's not going in in either reiser4 or
> > > the vfs.
> > A
> > > 
> > > Al has started a thread to hash out working semantics, but there's not been
> > > a single namesys person involved.  Similar all of you have absolutely ignore
> > Al message had a reply.
> > "namesys persons" is Hans Reiser.  Sufficient ?
> 
> http://marc.theaimsgroup.com/?t=109355582600002&r=1&w=2
> 
> I can't see Hans anywhere.  And honestly Hans has been so out of touch
> with VFS internals that some person actually understanding the issue
> might be helpfull.  That would probably whoever has taken over Nikita's
> position.

I assume Vladimir Saveliev could play this out...


-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Markus Törnqvist]

On Sat, Aug 28, 2004 at 03:46:28PM +0400, Alexander Lyamin wrote:
>Not true. If true - send bugreports.

Hate to say it, but apparently symlinking out of metas oopses, but that
should be trivial to fix?

>> Al has started a thread to hash out working semantics, but there's not been
>> a single namesys person involved.  Similar all of you have absolutely ignore
>Al message had a reply.
>"namesys persons" is Hans Reiser.  Sufficient ?

Not to be disrespectful, but Hans has sometimes taken personal shots on the
lists and that's not a productive approach. Not that he'd be the only one
around to do so.

Seems to me, as Christoph said, that some other Namesys guy than Hans should
participate in the Al/Linus branch of the thread with ideas.

Certainly the VFS may have to be extended right about now, so someone who
knows the exact details of Reiser4 should offer ideas and I'm not so sure
it's Hans. Looks like he had the ideas but not the implementation.

There's also the risk that Reiser4's implementation's don't really bend
into even a modified/extended VFS because of the plugins?
That's definitely something that has to be approached properly...

-- 
mjt

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Jan Harkes]

On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote:
> And I honestly dont understand whats the other Christoph's worries are about.

I honestly don't understand what all this 'grassroot' campaining is all
about. Very valid technical points have been made and mostly ignored.

The whole 'it breaks backups' noise is just that, noise.

- It breaks VFS locking rules.
- It pretends to provide a uniform way to store streams or metadata.
  However it only does so files and not for directories, symlinks,
  fifos, unix domain sockets, or device nodes, which makes it less than
  uniform.
- From what I saw in one part of the discussion, it allows for infinite
  depth recursion (file/metas/metas/metas/...). Some applications are
  going to have a lot of fun with that.

And finally,

- When reiserfs3 got merged, it introduced iget3 and read_inode2 in the
  VFS layer. Later on when I started to use them for Coda I almost
  immediately found serious consistency problems, resulting in the
  iget4_locked implementation in the 2.5 kernels.
  
  I don't think anyone ever fixed that race in reiser3. It should hit
  occasionally on concurrent lookups on SMP or preempt kernels. In 2.4,
  Coda needed a semaphore to prevent concurrency during the iget3 lookup
  until the new inode is actually initialized.

> Its got perfomance. Its there. It can emulate "conventional
> filesystem" behaviour, for legacy apps.

So does ramfs.

> Yes, I think it would be nice to have this infrastructure in VFS.
> Technically. But its not possible, cause of "committee clusterfuck".
> Socially. Stupidly.

It will get there, just consider it constructive criticism. If we can
help resolve or refute the technical issues, all the better. We might
even end up with improvements or extensions to the VFS or MM making life
easier for everyone.

Jan

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Nikita Danilov]

Jan Harkes writes:
 > On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote:
 > > And I honestly dont understand whats the other Christoph's worries are about.
 > 

[...]

 > - When reiserfs3 got merged, it introduced iget3 and read_inode2 in the
 >   VFS layer. Later on when I started to use them for Coda I almost
 >   immediately found serious consistency problems, resulting in the
 >   iget4_locked implementation in the 2.5 kernels.
 >   
 >   I don't think anyone ever fixed that race in reiser3. It should hit

Err... it was fixed.


 > Jan
 > 

Nikita.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 12:11:14PM -0400, Jan Harkes wrote:
> On Sat, Aug 28, 2004 at 03:18:07PM +0400, Alexander Lyamin wrote:
> 
> It will get there, just consider it constructive criticism. If we can
> help resolve or refute the technical issues, all the better. We might
> even end up with improvements or extensions to the VFS or MM making life
> easier for everyone.

That would be nice. Some nice points too (metas/metas in just noice tough).

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote:
> This VFS interface is an integral part of Фvery filesystem, and it

VFS never was "an integral part" of ANY filesystem. my dog knows it.
its just unified INTERFACE TO any filesystem (including reiser4).


P.S. I imagine, how much flamed it would be if reiser4 made any intensive
changes in linux VFS code...



-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote:
> Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote:
> > This VFS interface is an integral part of ??very filesystem, and it
> 
> VFS never was "an integral part" of ANY filesystem. my dog knows it.
> its just unified INTERFACE TO any filesystem (including reiser4).

You's misquoting me.  IF you quoted the whole context it'd be pretty
sure that the part of the filesystem that intefaces with the VFS is
meant.

But one could even say VFS is integral part of a linux filesystem as
it does most of the work a filesystem driver does in other operating
systems.

> P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> changes in linux VFS code...

It really depends on how you sent them.  If you had a big patch without
explanations - sure.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 01:12:33PM +0200, Christoph Hellwig wrote:
> On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote:
> > Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote:
> > > This VFS interface is an integral part of ??very filesystem, and it
> > 
> > VFS never was "an integral part" of ANY filesystem. my dog knows it.
> > its just unified INTERFACE TO any filesystem (including reiser4).
> 
> You's misquoting me.  IF you quoted the whole context it'd be pretty
> sure that the part of the filesystem that intefaces with the VFS is
> meant.

No. Its not me "misquoting", its just someone sound plain ?incoherrent?.
Even if I overquoted reply with whole message, its still sound ?incohherent?.

> 
> But one could even say VFS is integral part of a linux filesystem as
> it does most of the work a filesystem driver does in other operating
> systems.

theres no "linux filesystem". there are "linux filesystems".
thanks god.

But I it would be really grate if you'll elaborate your sentence with
example of VFS functionality (lack of it) on said "other operating systems"
and if you'll define "most of work".


> 
> > P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> > changes in linux VFS code...
> 
> It really depends on how you sent them.  If you had a big patch without
> explanations - sure.
It would work with small tweaks, but you just can take a look at reiser4
code and you'll understand that it just could not be chopped in
"set of small patches" altough it could be documented better ofcourse,
but its really well commented already.

some times, some approaches to  some problems  just would not work.

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 04:05:02PM +0400, Alexander Lyamin wrote:
> > But one could even say VFS is integral part of a linux filesystem as
> > it does most of the work a filesystem driver does in other operating
> > systems.
> 
> theres no "linux filesystem". there are "linux filesystems".
> thanks god.

a linux filesystem, not the linux filesystem, please read again.

> But I it would be really grate if you'll elaborate your sentence with
> example of VFS functionality (lack of it) on said "other operating systems"
> and if you'll define "most of work".

most trivial example is namespace locking, in *BSD, Windows, SVR4 and
derivates it's done in the lowlevel filesystem.  In plan9, Linux and
soon DragonlyBSD it's done in the VFS. 

> > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> > > changes in linux VFS code...
> > 
> > It really depends on how you sent them.  If you had a big patch without
> > explanations - sure.
> It would work with small tweaks, but you just can take a look at reiser4
> code and you'll understand that it just could not be chopped in
> "set of small patches" altough it could be documented better ofcourse,
> but its really well commented already.
> 
> some times, some approaches to  some problems  just would not work.

You still haven't even bother explaining what you want to do.  It's hard
to argue against vague uncertainity.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 03:56:55PM +0200, Christoph Hellwig wrote:
> On Sat, Aug 28, 2004 at 04:05:02PM +0400, Alexander Lyamin wrote:
> > > But one could even say VFS is integral part of a linux filesystem as
> > > it does most of the work a filesystem driver does in other operating
> > > systems.
> > 
> > theres no "linux filesystem". there are "linux filesystems".
> > thanks god.
> 
> a linux filesystem, not the linux filesystem, please read again.
> 
> > But I it would be really grate if you'll elaborate your sentence with
> > example of VFS functionality (lack of it) on said "other operating systems"
> > and if you'll define "most of work".
> 
> most trivial example is namespace locking, in *BSD, Windows, SVR4 and
> derivates it's done in the lowlevel filesystem.  In plan9, Linux and
> soon DragonlyBSD it's done in the VFS. 

ok. good examples.

> 
> > > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> > > > changes in linux VFS code...
> > > 
> > > It really depends on how you sent them.  If you had a big patch without
> > > explanations - sure.
> > It would work with small tweaks, but you just can take a look at reiser4
> > code and you'll understand that it just could not be chopped in
> > "set of small patches" altough it could be documented better ofcourse,
> > but its really well commented already.
> > 
> > some times, some approaches to  some problems  just would not work.
> 
> You still haven't even bother explaining what you want to do.  It's hard
> to argue against vague uncertainity.

 o files as directories -  no oppinion on that.
                            
 o metafiles -   AFAIK it was product of Nikita Danilov just playing and fooling.

Probably solution that will satisfy you is to have "legacy (dumb?) mode" which
is leaving all this fancy stuff out of sight.

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 11:23:50PM +0400, Alexander Lyamin wrote:
> Sat, Aug 28, 2004 at 03:56:55PM +0200, Christoph Hellwig wrote:
> > > 
> > > some times, some approaches to  some problems  just would not work.
> > 
> > You still haven't even bother explaining what you want to do.  It's hard
> > to argue against vague uncertainity.
> 
>  o files as directories -  no oppinion on that.
>                             
>  o metafiles -   AFAIK it was product of Nikita Danilov just playing and fooling.
> 
> Probably solution that will satisfy you is to have "legacy (dumb?) mode" which
> is leaving all this fancy stuff out of sight.

and enabled compile time or mount option...

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins

[Hans Reiser]

Alexander Lyamin wrote:

>
>                            
> o metafiles -   AFAIK it was product of Nikita Danilov just playing and fooling.
>  
>
No, it was not, it was an idea I had for so long I cannot remember when, 
maybe 1984, and I proposed it to darpa, and then nikita got told about 
it probably from being told to read the web page on our darpa project, 
and he took on the task of coding it.

Hans

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Markus Törnqvist]

On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote:
>P.S. I imagine, how much flamed it would be if reiser4 made any intensive
>changes in linux VFS code...

Surely it would be flamefest galore, but thanks to Reiser4 there may be
some VFS changes to that direction, maybe completely from Reiser4.
Should keep some people at bay if the chances weren't so much replacing
VFS than extending it.

-- 
mjt

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Sat, 28 Aug 2004, Alexander Lyamin wrote:
> 
> VFS never was "an integral part" of ANY filesystem. my dog knows it.

That's not really true.

Name handling (dentry layer, mounting) is very much an integral part of
the filesystem. Almost everything else in the VFS is "helper functions",
ie a filesystem can choose to ignore buffer heads, page cache etc, but a 
filesystem really cannot ignore or override the VFS naming stuff.

(Arguably the page cache isn't part of the VFS layer at all, it's really a 
memory management thing, although it's so intertwined with the VFS helper 
functions that you can't really draw the line).

> its just unified INTERFACE TO any filesystem (including reiser4).

True, but that's not the whole truth. It's way more than just an
interface. It's a set of rules, and it's in many way the controlling
party.

		Linus

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 10:18:44AM -0700, Linus Torvalds wrote:
>
>
> On Sat, 28 Aug 2004, Alexander Lyamin wrote:
> >
> > VFS never was "an integral part" of ANY filesystem. my dog knows it.
>
> That's not really true.

Theres some truth. And its really arguably, but I think, I see your point.

And taking on the situation with reiser4 with "smartmost VFS" approach,
time should pass and with time and expirience on hands it will be evident what
should go in VFS. Now its may be too early.

Considering "amazing PR skills" of Hans Reiser it was the only viable way
to get this changes in VFS. Cause, ironically, mr. Hellwig that currently
demand it to be scrapped out or go in VFS would instakill Hans Reiser
(i know many people would:) if he only touched holy cow of VFS for any
reiser4 purpose.

As I told before, some technically right things just do not work in this
imperfect world. You have to count with social issues, and Hans Reiser
wont, he just avoid them whenever its possible. 

> Name handling (dentry layer, mounting) is very much an integral part of
> the filesystem. Almost everything else in the VFS is "helper functions",
> ie a filesystem can choose to ignore buffer heads, page cache etc, but a 
> filesystem really cannot ignore or override the VFS naming stuff.
> 
> (Arguably the page cache isn't part of the VFS layer at all, it's really a 
> memory management thing, although it's so intertwined with the VFS helper 
> functions that you can't really draw the line).
> 
> > its just unified INTERFACE TO any filesystem (including reiser4).
> 
> True, but that's not the whole truth. It's way more than just an
> interface. It's a set of rules, and it's in many way the controlling
> party.
> 
> 		Linus

"True, but thats not the whole truth."

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 11:03:50PM +0400, Alexander Lyamin wrote:
> And taking on the situation with reiser4 with "smartmost VFS" approach,
> time should pass and with time and expirience on hands it will be evident what
> should go in VFS. Now its may be too early.
> 
> Considering "amazing PR skills" of Hans Reiser it was the only viable way
> to get this changes in VFS. Cause, ironically, mr. Hellwig that currently
> demand it to be scrapped out or go in VFS would instakill Hans Reiser
> (i know many people would:) if he only touched holy cow of VFS for any
> reiser4 purpose.

Could you please stop talking such inproven crap?

Re: reiser4 plugins

[Hans Reiser]

I think it is reasonable to make the -nopseudos (turns off the metafiles 
) mount option mandatory, until the bugs are resolved.

Our testing did not find these metafile/VFS bugs because of the reason 
for all our bugs, we screwed up. 

There is a distinct difference between some persons and I, which is that 
some think all of reiser4 should be excluded until metafiles are 
implemented by VFS some long time from now, and I, in that I merely 
think buggy optional features should be turned off until they are 
fixed.  I, being renowned for my paranoia and asininity as I am, think 
these persons find it convenient as an excuse to keep us from competing, 
and I think that if we were slower there would be less hassle every time 
we try to get into the kernel. 

While reiser4 has some significant roughnesses remaining in its 
performance, I think the average user would find it performs better than 
other filesystems, and is stable enough for, say, a laptop, and I 
predict that by the time we have it stable enough for mission critical 
servers, all the roughness in various important corner cases will be 
gone.  

Persons benchmarking it with tarballs, please be sure to use tarballs 
created on reiser4, not ext2 tarballs, readdir order matters a lot for 
sorted directory filesystems.

Hans

Re: reiser4 plugins

[Herbert Poetzl]

On Sat, Aug 28, 2004 at 02:41:12PM -0700, Hans Reiser wrote:
> I think it is reasonable to make the -nopseudos (turns off the metafiles 
> ) mount option mandatory, until the bugs are resolved.
> 
> Our testing did not find these metafile/VFS bugs because of the reason 
> for all our bugs, we screwed up. 
> 
> There is a distinct difference between some persons and I, which is that 
> some think all of reiser4 should be excluded until metafiles are 
> implemented by VFS some long time from now, and I, in that I merely 
> think buggy optional features should be turned off until they are 
> fixed.  I, being renowned for my paranoia and asininity as I am, think 
> these persons find it convenient as an excuse to keep us from competing, 
> and I think that if we were slower there would be less hassle every time 
> we try to get into the kernel. 
> 
> While reiser4 has some significant roughnesses remaining in its 
> performance, I think the average user would find it performs better than 
> other filesystems, and is stable enough for, say, a laptop, and I 
> predict that by the time we have it stable enough for mission critical 
> servers, all the roughness in various important corner cases will be 
> gone.  
> 
> Persons benchmarking it with tarballs, please be sure to use tarballs 
> created on reiser4, not ext2 tarballs, readdir order matters a lot for 
> sorted directory filesystems.

hmm, so probably we have to wait until all
tar packagers moved to reiser4, so that the
available tar files are 'sorted properly' ...

best,
Herbert

> Hans
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: reiser4 plugins

[Hans Reiser]

Herbert Poetzl wrote:

>
>hmm, so probably we have to wait until all
>tar packagers moved to reiser4, so that the
>available tar files are 'sorted properly' ...
>
>best,
>Herbert
>
>  
>
Or wait for a repacker, which will probably happen sooner. Distros that 
use reiser4 will probably tar in reiser4 order.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Sat, 28 Aug 2004, Alexander Lyamin wrote:
> 
> Considering "amazing PR skills" of Hans Reiser it was the only viable way
> to get this changes in VFS. Cause, ironically, mr. Hellwig that currently
> demand it to be scrapped out or go in VFS would instakill Hans Reiser
> (i know many people would:) if he only touched holy cow of VFS for any
> reiser4 purpose.

Hey, you don't have to try to convince me - I haven't been arguing against 
the reiser4 approach. I think it's a prototyping scheme, and quite frankly 
I don't think you _can_ get the name lookup right without real VFS support 
(races, namespaces etc are just not something you can solve in the 
filesystem). 

But the fact that name lookup does seriously need VFS support means that I 
don't think we want to merge reiser4 as-is. No way. That doesn't mean that 
I think reiser4 was _wrong_.

And I certainly agree that there are personality issues, and they are not 
all just Hans' ;)

			Linus

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1218 bytes --]

Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig:

> > The reiser4 core doesn't know about inodes, directories or files. It's
> > the glue code between the VFS and the storage layer that does. It's
> > implemented as a plugin. This has nothing to do with semantic
> > enhancements yet. These should be removed for now and made a 2.7 topic.
> 
> Oh yes, it is.  As soon as you use different access methods on an
> overlapping set of objects you see exactly the problems I described.

Right. That shouldn't happen. Since an object is usally tied to exactly
one plugin this can't happen I think. At least as long the programmer
doesn't mess anything up.

> If they don't overlap there's no point for the plugins to start with,
> you'll better turn the core into a library that can be used by different
> projects then.

I think that's what Hans is trying to suggest. The core is theoretically
usable for anything. He made a filesystem of it.

You can see reiser4 as a library and the UNIX directory and file plugins
as the filesystem. That's why they are linked in uncoditionally because
it doesn't make sense otherwise. It's mainly the naming that seems to
confuse people.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig and Christophe Saout, I think you two are
miscommunicating a little.

I've never looked at reiser4 code, but I think I understand what you
are both saying, so I will explain.

Christoph Hellwig wrote:
> Here comes the problem.  All the access checking/race avoidance/loop
> creation avoiced, in short the posix+extension semantics are implemented
> in the Linux VFS layer.  If you want to allow a second access method
> (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
> inside reiser4.

That would be good.  Even better would be if VFS implemented the best
semantics that we agree is nice, after some experimenting with reiser4
lets us discover what works well.

> > The reiser4 core doesn't know about inodes, directories or files. It's
> > the glue code between the VFS and the storage layer that does. It's
> > implemented as a plugin. This has nothing to do with semantic
> > enhancements yet. These should be removed for now and made a 2.7 topic.
> 
> Oh yes, it is.  As soon as you use different access methods on an
> overlapping set of objects you see exactly the problems I described.
> 
> If they don't overlap there's no point for the plugins to start with,
> you'll better turn the core into a library that can be used by different
> projects then.

Firstly, the core library _is_ used by different projects.  That's the
commercial aim of reiser4: to be used by projects other than just as
Linux filesystem.  For example it might be used by a database, or a
portable application which needs to store structured data in a flat
file (with random access).  A good abstraction makes it more useful
than "just" a filesystem.

Secondly, the reiser4 design makes the "high-level plugin" interface a
good clean abstraction.  It's conceptually simple and versatile, so
simplifies the code and makes it easier to try new ideas as well.

I think the disagreement between you two comes down to the idea that a
generally useful API abstraction to file-like objects really should be
somewhere in the VFS, and not specific to a single filesystem.

Unfortunately, the problem is that reiser4 is the only filesystem
which is _technically capable_ of implementing that abstraction in a
practical way, apparently.  (I'm not sure if this is really true.
reiser4's object model is not the same as paths and inodes, but the
impedance mismatch doesn't seem huge.)

Perhaps in future some other filesystems could do it.  Then it would
make sense for the API to be common between them.

At the same time, some of the features which Hans Reiser would like to
offer users -- especially things like metadata indexing -- don't seem
to be very practical on top of the VFS abstraction.

These are conflicting goals.  Hence the argument.

VFS is very POSIX, which rules out some interesting things like
multiple views of an ordinary file (i.e. viewing a flat file and it's
structured form together), coherent files (i.e. metadata which is
calculated from the file contents and always guaranteed to match it),
and real time indexing (i.e. "grep", "locate" and "local-google" but
fancier, always up to date, and still fast).

Those features can be implemented poorly on top of POSIX, and hence
poorly on top of VFS.  We do that today.  It works, it's simple, but
it sucks.  Sometimes it goes wrong (index doesn't match files), it's
slow (locate isn't real-time, and if grep kept content indexes it
would still be too slow to verify them), and it's not really simple as
you have to use special tools for every file format these days, each
with their own unique interface.

I think we can tidy that up a lot, still using the VFS, but with some
extra hooks.  But it's not clear what hooks those would be, and how
the interesting features would be implemented by other POSIX-only
filesystems.  It'd be more useful: I want the fancy stuff to work over
NFS to a bog standard server, or over an old ext2 filesystem that
mustn't be upgraded, not just reiser4, even if I don't have the same
performance and it has to create lots of articial "metadata" files to
simulate the fancy stuff.

-- Jamie

Re: silent semantic changes with reiser4

[Christoph Hellwig]

> Firstly, the core library _is_ used by different projects.  That's the
> commercial aim of reiser4: to be used by projects other than just as
> Linux filesystem.  For example it might be used by a database, or a
> portable application which needs to store structured data in a flat
> file (with random access).  A good abstraction makes it more useful
> than "just" a filesystem.

Which is okay as long as it still fits in the kernel.  Remember that
lots of projects had to adjust so far despite the commerical aims of
their creators.

> I think the disagreement between you two comes down to the idea that a
> generally useful API abstraction to file-like objects really should be
> somewhere in the VFS, and not specific to a single filesystem.

IT should, and in fact it actually is most of the part.  We can have
differnt operation vectors per object, but reiser4 is the only
filesystem in the tree not making use of that but provinding only a
single one and then dispatching internally.  Even XFS that carries
around and horrible internal dispatching layer as IRIX legacy gets
this right.

> Unfortunately, the problem is that reiser4 is the only filesystem
> which is _technically capable_ of implementing that abstraction in a
> practical way, apparently.  (I'm not sure if this is really true.
> reiser4's object model is not the same as paths and inodes, but the
> impedance mismatch doesn't seem huge.)

Umm, no.  In fact every filesystem does this.  There's not too many
objects with different namespace semantic: regular files and special
files vs directories vs symlinks basically, but as soon as you go to the
data patch you can have hundrets of different file_operations on a
single filesystems (for special files).

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> > Unfortunately, the problem is that reiser4 is the only filesystem
> > which is _technically capable_ of implementing that abstraction in a
> > practical way, apparently.  (I'm not sure if this is really true.
> > reiser4's object model is not the same as paths and inodes, but the
> > impedance mismatch doesn't seem huge.)
> 
> Umm, no.  In fact every filesystem does this.  There's not too many
> objects with different namespace semantic: regular files and special
> files vs directories vs symlinks basically, but as soon as you go to the
> data patch you can have hundrets of different file_operations on a
> single filesystems (for special files).

Remember that reiser4 allows you to operate on little pieces of data,
glueing and rearranging them inside files (something like that).

No other filesystem has that capability, and it's a data model which
the fancy features (when they exist) will use.

You can map those pieces to underlying directories and files and
renames and unlinks, so that the fancy stuff works on other
filesystems, but it would be a useless model because those other
filesystems wouldn't be recognisably "ordinary" files any more.

For reiser4 to expose that model through a VFS interface, and the
fancy stuff to use it through the VFS interface, and for the fancy
stuff to work (even imperfectly) on other filesystems which don't
offer those operations, some kind of fall-back "store metadata and
fragment rearrangements in auxiliary files with special names" layer
would be requied.  That's a big job.

I think it's a good job to do (funding, anyone? :), and the right
place for that (imperfect but useful) fall-back layer is userspace
with perhaps minimal VFS support.  Another reason to put the fall-back
layer in userspace is that applications which depend on the fancy
stuff can still be portable to other OSes, working, but a bit
smoother, faster and more "integrated" on reiser4.

-- Jamie

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 05:19:11PM +0100, Jamie Lokier wrote:
> Remember that reiser4 allows you to operate on little pieces of data,
> glueing and rearranging them inside files (something like that).
> 
> No other filesystem has that capability, and it's a data model which
> the fancy features (when they exist) will use.
> 
> You can map those pieces to underlying directories and files and
> renames and unlinks, so that the fancy stuff works on other
> filesystems, but it would be a useless model because those other
> filesystems wouldn't be recognisably "ordinary" files any more.
> 
> For reiser4 to expose that model through a VFS interface, and the
> fancy stuff to use it through the VFS interface, and for the fancy
> stuff to work (even imperfectly) on other filesystems which don't
> offer those operations, some kind of fall-back "store metadata and
> fragment rearrangements in auxiliary files with special names" layer
> would be requied.  That's a big job.

I seem to miss those files in the current implementation.  But again the
big question is how will the interface look?  If the smaller objects
interact with normal files (link/rename, contained in normal
directories), they will have to obey the same locking protocols.  If not
a filesystem could experiment with it of course.

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christoph Hellwig wrote:
> If the smaller objects interact with normal files (link/rename,
> contained in normal directories), they will have to obey the same
> locking protocols.  If not a filesystem could experiment with it of
> course.

I think the smaller objects will consist of special files, which are
magic attributes like the ones you currently see in reiser4, and
ordinary files (which might themselves be containers).

Link/rename/locking etc. should work as usual on the ordinary smaller
files, but probably aren't allowed at all on the special attribute
files, except where there's an appropriate meaning.  (For example
locking certain attributes might have an appropriate meaning).

It would make sense for the special attributes file to not identify as
"regular files" (S_IFREG), but there aren't many mode bits to play
with.  They should also be harder to find than ordinary smaller files.

-- Jamie

Re: silent semantic changes with reiser4

[Hans Reiser]

Christophe Saout wrote:

>Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton:
>
>  
>
>>And describe the "plugin" system.  Why does the filesystem need such a
>>thing (other filesystems get their features via `patch -p1')?
>>    
>>
>
>Ok, let me try. (Hans, please don't shoot me if I don't get every detail
>right, I'm trying to simplify and translate)
>
>The reiser4 core is just a big and fast storage layer using a single
>tree. It is bound to a mount point and uses the page cache to manage its
>memory. The only thing it can do on its own is to flush dirty data to
>the disk when the VM wants to (memory pressure) or the VFS wants to
>(unmount, sync) or some "plugin" wants to.
>Additionally it's completely atomic (with respect to crashes, no
>isolation like in databases), comparable to data=journal but without the
>overhead of using a journal for everything.
>
>Up to this point there are no users of this "database" and it does not
>implement any of the VFS methods for a filesystem (except mount and
>unmount perhaps).
>
>All the functionality is provided to
>
^to^via
apologies for the correction but it might confuse some

> the plugins. You can insert,
>remove, lookup or modify key:object pairs (with an index into the
>object). The object
>
^object^item
perform that substitution everywhere in this email

> is a sequence of units. For files a unit would be 1
>byte and the index would be the byte offset. For directories the unit
>would be 1 entry and the index would be the filename.
>
>Now there are some plugins that define how the storage layout on the
>disk is (some kind of "backend" plugins).
>
>*And* there are plugins which are users of the "reiser4 client API" and
>implement the actual VFS methods.
>
>There's a UNIX directory plugin and a UNIX file plugin.
>
>Directories, inodes and file content are just key:object pairs and the
>plugins know how to operate on these.
>
>There's a new plugin in work that also implements UNIX file semantics
>but stores the data for that file encrypted and/or compressed.
>
>These plugins live between the VFS and the storage layer.
>
>Just like the filesystems live between the VFS and the block layer (Hans
>would say that filesystems are VFS plugins ;-)).
>
>  
>
>>And what are the licensing implications of plugins?  Are they derived
>>works?  Must they be GPL'ed?
>>    
>>
>
>I suppose yes, at least currently, since they can only be linked with
>reiser4, there's no module infrastructure.
>
>  
>

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 01:31:34AM -0700, Hans Reiser wrote:
> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> for Apple, and that means we need to put search engine and database 

Dou you know a nice thing?  We (as in the Linux Community) don't have to
compete with anyone.  Sure, we're usually trying to be better than
anyone else, but unlike companies under maret pressure we can wait until
something is ready.

Again, if you think you want your work in Linux cooperate with the Linux
Developers, and as you have seen in this thread there's a pretty broad
consensus that we want things working at the VFS (and actually working
of course)

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 11:34:07AM +0200, Christoph Hellwig wrote:
>
>Dou you know a nice thing?  We (as in the Linux Community) don't have to
>compete with anyone.  Sure, we're usually trying to be better than
>anyone else, but unlike companies under maret pressure we can wait until
>something is ready.

It won't get ready unless it's made ready, yes?

And what about the time when people will move away from Linux because
Linux has nothing but crappy file systems to offer?
Sure, it's not that we'd have to compete, but...

>Again, if you think you want your work in Linux cooperate with the Linux
>Developers, and as you have seen in this thread there's a pretty broad
>consensus that we want things working at the VFS (and actually working
>of course)

So move it into VFS. Neext! ;)

-- 
mjt

Re: silent semantic changes with reiser4

[Thomas Cataldo]

On Thu, 2004-08-26 at 11:34 +0200, Christoph Hellwig wrote:
> On Thu, Aug 26, 2004 at 01:31:34AM -0700, Hans Reiser wrote:
> > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> > for Apple, and that means we need to put search engine and database 
> 
> Dou you know a nice thing?  We (as in the Linux Community) don't have to
> compete with anyone.  Sure, we're usually trying to be better than
> anyone else, but unlike companies under maret pressure we can wait until
> something is ready.

By the way, this kind of features can be achieved easily in userspace
with, for example, ext3, beagle (a userspace fulltext indexing daemon)
and some (d|i)notify magic.

Re: silent semantic changes with reiser4

[Will Dyson]

Hans Reiser wrote:

> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> for Apple, and that means we need to put search engine and database 
> functionality into the filesystem.  It takes 11 years of serious 

Hans,

I'm very curious about your ideas on how to put search engine and 
database functionality into the filesystem. Although I've let the Befs 
filesystem driver largely stagnate over the past two years, from time to 
time I think about the problem of exporting to userspace the attribute 
indexes that BeFS keeps on disk.

In the original BeOS, they solved the problem by having the filesystem 
driver itself take a text query string and parse it, returning a list of 
inodes that match. The whole business of parsing a query string in the 
kernel (let alone in the filesystem driver) has always seemed ugly to 
me. However, the best alternative I've come up with is to simply export 
the index data as a special file (perhaps in sysfs?) and have userspace 
responsible for searching the index. That would probably work, but it 
wouldn't help other filesystems that implement even a different index 
format, much less a different form of extra searchability.

-- 
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1007 bytes --]

Am Donnerstag, den 26.08.2004, 12:32 -0400 schrieb Will Dyson:

> > Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> > for Apple, and that means we need to put search engine and database 
> > functionality into the filesystem.  It takes 11 years of serious 
> 
> I'm very curious about your ideas on how to put search engine and 
> database functionality into the filesystem.

I could imagine something like this.

There are some special folders or files or whatever under every file
where some userspace can put some metadata. A keyword perhaps.

echo linux > file.bla/keywords/topic

The filesystem might then automatically put these keywords into an index
and then provide a search mechanism elsewhere where it could ask "find
me all dentries with the keyword 'linux'." and it would return a list
like locate does. Only that it's in realtime and also works when moving
the file around (but not when copying with an unaware program for
obvious reasons).


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Jamie Lokier]

Christophe Saout wrote:
> echo linux > file.bla/keywords/topic
> 
> The filesystem might then automatically put these keywords into an index
> and then provide a search mechanism elsewhere where it could ask "find
> me all dentries with the keyword 'linux'." and it would return a list
> like locate does. Only that it's in realtime and also works when moving
> the file around (but not when copying with an unaware program for
> obvious reasons).

It could work even when copying with an unaware program - provided
the're a tool which knows to extract metadata from files where that
hasn't been done already, or where the file's been modified.

-- Jamie

Re: silent semantic changes with reiser4

[Hans Reiser]

Will Dyson wrote:

>
>
> In the original BeOS, they solved the problem by having the filesystem 
> driver itself take a text query string and parse it, returning a list 
> of inodes that match. The whole business of parsing a query string in 
> the kernel (let alone in the filesystem driver) has always seemed ugly 
> to me. 

Why?

> However, the best alternative I've come up with is to simply export 
> the index data as a special file (perhaps in sysfs?) and have 
> userspace responsible for searching the index.

That is the best implementation suggestion I've heard for splitting the 
filesystem into two parts, one in user space and one in kernel, but I 
still don't trust it to work well.

> That would probably work, but it wouldn't help other filesystems that 
> implement even a different index format, much less a different form of 
> extra searchability.
>
After reiser6 is implemented, we'll have a better idea of what parts 
need what from what other parts.  Until we get there, it needs to be 
kept together.  Frankly, I suspect that naming is not going to split so 
easily, but when it is done we'll be able to know.  The problems of 
putting too much functionality into the kernel are less than those of 
partitioning the filesystem code, and defining an inflexible API between 
the two parts.  Kernel interfaces are fairly inflexible because there 
will always be some moron who sidesteps the fs library to program 
directly to the kernel interface, gets screwed when the interface 
changes, has the nerve to complain about it, and works for some powerful 
distro.

Hans

Re: silent semantic changes with reiser4

[Will Dyson]

Hans Reiser wrote:
> Will Dyson wrote:
>>
>> In the original BeOS, they solved the problem by having the filesystem 
>> driver itself take a text query string and parse it, returning a list 
>> of inodes that match. The whole business of parsing a query string in 
>> the kernel (let alone in the filesystem driver) has always seemed ugly 
>> to me. 
> 
> Why?

Hmm. Trying to explain aesthetic judgments is always fun, but I'll try.

String parsing bloats the kernel with code that will be called rarely, 
and doing it inside the filesystem module makes for duplicate code if 
more than one filesystem does it. But a good common parser routine (or a 
kernel api that takes a pre-compiled parse tree) would reduce the bad taste.

The real objection I have is that it puts the kernel in the position of 
doing more work than it has to for the system to operate correctly and 
efficiently. The user wants to know what files match a certain set of 
criteria. The filesystem provides special features which can greatly 
accelerate some searches. Does it make more sense to move the 
functionality of /usr/bin/find into the kernel, or to export the index 
information so that an enhanced version of find can make use of it?

I don't think having find in the kernel would be worth the cost or 
complexity.

However, now that I've had to think about it while writing the above, it 
occurs to me that if we wanted to offer something like the beos's live 
queries, doing it in the kernel might be the way to go.

For the 99.9% of people who were not beos users back in the day, the 
"live query" was a way to register a query string with the kernel, so 
that you got notifications when the list of files matching the query 
changed. It was pretty cool, but this is really just blue-skying since I 
am not stepping up to write such a beast any time soon.

>> However, the best alternative I've come up with is to simply export 
>> the index data as a special file (perhaps in sysfs?) and have 
>> userspace responsible for searching the index.
> 
> That is the best implementation suggestion I've heard for splitting the 
> filesystem into two parts, one in user space and one in kernel, but I 
> still don't trust it to work well.

Why not? And if it really splits the filesystem into two parts depends 
on which of the two following statements you agree with more:

"Fast searching is a feature of the filesystem"
"Maintaining indexes is a feature of the filesystem"


PS.
The file-as-directory attribute/stream/whatever model is a much better 
fit for befs's internals than the xattr api. If the kernel supported it, 
I would enable support for it in befs.

-- 
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman

Re: silent semantic changes with reiser4

[David Ford]

[-- Attachment #1: Type: text/plain, Size: 2800 bytes --]

Will Dyson wrote:

> Hans Reiser wrote:
>
>> Will Dyson wrote:
>>
>>>
>>> In the original BeOS, they solved the problem by having the 
>>> filesystem driver itself take a text query string and parse it, 
>>> returning a list of inodes that match. The whole business of parsing 
>>> a query string in the kernel (let alone in the filesystem driver) 
>>> has always seemed ugly to me. 
>>
>>
>> Why?
>
>
> Hmm. Trying to explain aesthetic judgments is always fun, but I'll try.
>
> String parsing bloats the kernel with code that will be called rarely, 
> and doing it inside the filesystem module makes for duplicate code if 
> more than one filesystem does it. But a good common parser routine (or 
> a kernel api that takes a pre-compiled parse tree) would reduce the 
> bad taste.


Like say.. printk() ?  :)

> The real objection I have is that it puts the kernel in the position 
> of doing more work than it has to for the system to operate correctly 
> and efficiently. The user wants to know what files match a certain set 
> of criteria. The filesystem provides special features which can 
> greatly accelerate some searches. Does it make more sense to move the 
> functionality of /usr/bin/find into the kernel, or to export the index 
> information so that an enhanced version of find can make use of it?


I think it depends largely on where you see file systems going 5, 10, 20 
years from now.  Are they going to be decadent designs or will they be 
flush with meta data and other tidbits?  Where does the heuristic line 
sit when it becomes better to put such a function into the kernel v.s. 
user land?

> I don't think having find in the kernel would be worth the cost or 
> complexity.


Probably not.  But as I view it, file systems are slowly evolving into 
relational objects and the margin between what are files and directories 
versus content within said objects is shrinking.  More and more, people 
are using a relational database to store content that once resided in 
singular files.

Sure, it makes for a much fatter *FS system, but the user doesn't care 
too much, he simply enjoys the ability to manage, search, and cross 
reference his content much more easily.

And then comes the "it's a user land thing" comment again.  And again 
the response.  When does moving this from user land into kernel land 
become a better choice.

With the file systems we have today, it's really not much of a gain 
compared to the cost of the move.

Consider Reiser4 now.  More and more meta data is being used in regards 
to files.  That margin is shrinking.

BeOS broached it, Microsoft is building it.  Just because they did it 
doesn't mean we have to do it nor that it is right.

But it's food for thought.  Especially as more and more tidbits get 
associated with files.

-david


[-- Attachment #2: david+challenge-response.vcf --]
[-- Type: text/x-vcard, Size: 183 bytes --]

begin:vcard
fn:David Ford
n:Ford;David
email;internet:david@blue-labs.org
title:Industrial Geek
tel;home:Ask please
tel;cell:(203) 650-3611
x-mozilla-html:TRUE
version:2.1
end:vcard

Re: silent semantic changes with reiser4

[Hans Reiser]

David Ford wrote:

> Will Dyson wrote:
>
>> Hans Reiser wrote:
>>
>>> Will Dyson wrote:
>>>
>>
>>
>> String parsing bloats the kernel with code that will be called 
>> rarely, and doing it inside the filesystem module makes for duplicate 
>> code if more than one filesystem does it. But a good common parser 
>> routine (or a kernel api that takes a pre-compiled parse tree) would 
>> reduce the bad taste.
>
>
>
> Like say.. printk() ?  :)
>
or namei()....;-)

Seriously, what we are doing is making namei() more powerful.  That is 
all.  Currently namei() is very simple.  Evolution happens.

Hans

Re: silent semantic changes with reiser4

[Hans Reiser]

I think there are two ways to analyze the code boundary issue.  One is 
"does it belong in the kernel?"  Another is, "does it belong in the 
filesystem. and if so should name resolution in a filesystem be split 
into two parts, one in kernel, and one in user space."  In ten years I 
might have the knowledge needed to make such a split, but I know for 
sure that I don't know how to do it today without regretting it 
tomorrow, and I don't really have confidence that I will ever be able to 
do it without losing performance.

Glad that BeFS finds the new model better.:)

Hans

Will Dyson wrote:

> Hans Reiser wrote:
>
>> Will Dyson wrote:
>>
>>>
>>> In the original BeOS, they solved the problem by having the 
>>> filesystem driver itself take a text query string and parse it, 
>>> returning a list of inodes that match. The whole business of parsing 
>>> a query string in the kernel (let alone in the filesystem driver) 
>>> has always seemed ugly to me. 
>>
>>
>> Why?
>
>
> Hmm. Trying to explain aesthetic judgments is always fun, but I'll try.
>
> String parsing bloats the kernel with code that will be called rarely, 
> and doing it inside the filesystem module makes for duplicate code if 
> more than one filesystem does it. But a good common parser routine (or 
> a kernel api that takes a pre-compiled parse tree) would reduce the 
> bad taste.
>
> The real objection I have is that it puts the kernel in the position 
> of doing more work than it has to for the system to operate correctly 
> and efficiently. The user wants to know what files match a certain set 
> of criteria. The filesystem provides special features which can 
> greatly accelerate some searches. Does it make more sense to move the 
> functionality of /usr/bin/find into the kernel, or to export the index 
> information so that an enhanced version of find can make use of it?
>
> I don't think having find in the kernel would be worth the cost or 
> complexity.
>
> However, now that I've had to think about it while writing the above, 
> it occurs to me that if we wanted to offer something like the beos's 
> live queries, doing it in the kernel might be the way to go.
>
> For the 99.9% of people who were not beos users back in the day, the 
> "live query" was a way to register a query string with the kernel, so 
> that you got notifications when the list of files matching the query 
> changed. It was pretty cool, but this is really just blue-skying since 
> I am not stepping up to write such a beast any time soon.
>
>>> However, the best alternative I've come up with is to simply export 
>>> the index data as a special file (perhaps in sysfs?) and have 
>>> userspace responsible for searching the index.
>>
>>
>> That is the best implementation suggestion I've heard for splitting 
>> the filesystem into two parts, one in user space and one in kernel, 
>> but I still don't trust it to work well.
>
>
> Why not? And if it really splits the filesystem into two parts depends 
> on which of the two following statements you agree with more:
>
> "Fast searching is a feature of the filesystem"
> "Maintaining indexes is a feature of the filesystem"
>
>
> PS.
> The file-as-directory attribute/stream/whatever model is a much better 
> fit for befs's internals than the xattr api. If the kernel supported 
> it, I would enable support for it in befs.
>

Re: silent semantic changes with reiser4

[Will Dyson]

Hans Reiser wrote:
> I think there are two ways to analyze the code boundary issue.  One is 
> "does it belong in the kernel?"  Another is, "does it belong in the 
> filesystem. and if so should name resolution in a filesystem be split 
> into two parts, one in kernel, and one in user space."  In ten years I 
> might have the knowledge needed to make such a split, but I know for 
> sure that I don't know how to do it today without regretting it 
> tomorrow, and I don't really have confidence that I will ever be able to 
> do it without losing performance.

I don't see how exporting a set of indices on file attributes to 
userspace constitutes putting part of the name resolution into 
userspace. A file's name (or names, in the case of hardlinks) would 
still be determined entirely within the filesystem.

The more I think about it, the more I am convinced that "the index" is 
the correct primative to use for exposing any filesystem's fast 
searching features.

-- 
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman

Separating Indexing and Searching (was silent semantic changes with reiser4)

[Alexander G. M. Smith]

Will Dyson wrote on Fri, 27 Aug 2004 14:20:29 -0400:
> Hans Reiser wrote:
> > Will Dyson wrote:
> >> In the original BeOS, they solved the problem by having the filesystem 
> >> driver itself take a text query string and parse it, returning a list 
> >> of inodes that match. The whole business of parsing a query string in 
> >> the kernel (let alone in the filesystem driver) has always seemed ugly 
> >> to me. 
> > 
> > Why?
> 
> Hmm. Trying to explain aesthetic judgments is always fun, but I'll try.
> 
> String parsing bloats the kernel with code that will be called rarely, 
> and doing it inside the filesystem module makes for duplicate code if 
> more than one filesystem does it. But a good common parser routine (or a 
> kernel api that takes a pre-compiled parse tree) would reduce the bad taste.

Not only bloat, but time wasted reinventing the wheel.  For my experimental file system (http://members.rogers.com/agmsbeos/AGMSRAMFileSystem.readme - see the Nifty Features chapter), I had to figure out the BFS query language (it was documented but didn't have every little detail explained - some experimentation was needed), then write a tokenizer and parser for it, to convert it down to a parse tree that could be used during the query's life.  Good exercise for the programmer, but duplication of effort.

I also added a few new things to the query language, but because the parsing is done by each file system separately, my "improvements" won't show up in other file systems.

> For the 99.9% of people who were not beos users back in the day, the 
> "live query" was a way to register a query string with the kernel, so 
> that you got notifications when the list of files matching the query 
> changed. It was pretty cool, but this is really just blue-skying since I 
> am not stepping up to write such a beast any time soon.

Kind of like that.  For performance, the query string is converted into a parse tree and then all file operations that touch any attribute mentioned in the query get evaluated against the parse tree to see if the result (in or out of the query result set) has changed.  If it changes, then a kernel facility is used to broadcast a message describing the change and file involved to all parties monitoring that query.  Of course, before becoming live, the existing files that match the query are found by iterating over an index (choosing the appropriate index is an optimization trick).

> >> However, the best alternative I've come up with is to simply export 
> >> the index data as a special file (perhaps in sysfs?) and have 
> >> userspace responsible for searching the index.
> > 
> > That is the best implementation suggestion I've heard for splitting the 
> > filesystem into two parts, one in user space and one in kernel, but I 
> > still don't trust it to work well.
> 
> Why not? And if it really splits the filesystem into two parts depends 
> on which of the two following statements you agree with more:
> 
> "Fast searching is a feature of the filesystem"
> "Maintaining indexes is a feature of the filesystem"

I like that.  Put the indexing and a related attribute change update notification mechanism into the kernel.  User land libraries can build on that to implement whatever query language you want, thus saving work and having a common language for all file system varieties that support indices.

However, one of my (unfinished) experiments was to have magic directories that show query results as their contents.  One attribute of the directory (or even its name) would be the query string.  That way even old software (like "ls") could use queries!  Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level.

- Alex

Re: Separating Indexing and Searching (was silent semantic changes with reiser4)

[Hans Reiser]

Alexander G. M. Smith wrote:

>
>
>However, one of my (unfinished) experiments was to have magic directories that show query results as their contents.  One attribute of the directory (or even its name) would be the query string.  That way even old software (like "ls") could use queries!  Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level.
>  
>
Beautiful example of the sort of thing that I don't want to be locked 
out from doing by having the parser in user space.

Re: Separating Indexing and Searching (was silent semantic changes with reiser4)

[Hans Reiser]

Alexander G. M. Smith wrote:

>
>
>However, one of my (unfinished) experiments was to have magic directories that show query results as their contents.  One attribute of the directory (or even its name) would be the query string.  That way even old software (like "ls") could use queries!  Implementing queries-as-directories might require moving some things back into the kernel, or at least into some plug-in level.
>
>- Alex
>
>
>  
>
Symlinks also.  Symlinks with powerful queries in them would require a 
parser in the kernel.  Thanks for helping me to distill my incoherent 
reasons for the parser being in the kernel.

Re: Separating Indexing and Searching (was silent semantic changes with reiser4)

[Jamie Lokier]

Hans Reiser wrote:
> Symlinks also.  Symlinks with powerful queries in them would require a 
> parser in the kernel.

No they wouldn't.  You can do symlinks with powerful queries in
userspace _today_ (as well as directories which list queries).

Making queries be up to date in real-time is still a problem, but that
has nothing to do with symlinks or directories, or where the parsers live.

> Thanks for helping me to distill my incoherent reasons for the
> parser being in the kernel.

This is one occasion where you're mistaken :)

There may be other reasons for parsing query strings in the kernel
(though I'm not convinced there are any), but this isn't one.

-- Jamie

Re: Separating Indexing and Searching (was silent semantic changes with reiser4)

[Will Dyson]

Alexander G. M. Smith wrote:

> Not only bloat, but time wasted reinventing the wheel.  For my 
> experimental file system 
> (http://members.rogers.com/agmsbeos/AGMSRAMFileSystem.readme - see 
> the Nifty Features chapter), I had to figure out the BFS query

I had a look. Cool stuff!

> I like that.  Put the indexing and a related attribute change update 
> notification mechanism into the kernel.  User land libraries can 
> build on that to implement whatever query language you want, thus 
> saving work and having a common language for all file system 
> varieties that support indices.

Yeah, thats basicly what I was thinking. Right now I'm trying to think
of any kind of filesystem that could support the general concept of 
"Fast Searching", but be somehow unable to export this capability as a 
set of indices on file attributes....

Ok. I give up. It seems like the right abstraction to me.

> However, one of my (unfinished) experiments was to have magic 
> directories that show query results as their contents.  One attribute
>  of the directory (or even its name) would be the query string.  That
>  way even old software (like "ls") could use queries!  Implementing 
> queries-as-directories might require moving some things back into the
>  kernel, or at least into some plug-in level.

That is a pretty awesome idea. It should be done in its own specialized
filesystem though. QueryFS. If the change-notification mechanism in the
kernel was robust enough (a big if), the rest could be done with a
userspace filesystem (via FUSE). Mount a query, cd into it, run grep,
write a little bash script... Very unixy.

-- 
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman

Re: silent semantic changes with reiser4

[Horst von Brand]

Hans Reiser <reiser@namesys.com> said:
> Andrew Morton wrote:

[...]

> > The fact that one filesystem will offer features which other
> > filesystems do not and cannot offer makes me queasy for some reason.

Not me. As long as it is clearly experimental stuff that can be removed at
the head hacker's whim, that is. 2.7 material.

> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
> for Apple,

Says who?

>            and that means we need to put search engine and database 
> functionality into the filesystem.

Please don't. Unix works and is extremely popular because it _doesn't_ try
to be smart (policy vs mechanism distinction, simple abstractions that can
be combined endlessly). If you add this to the filesystems, you either redo
_all_ userland to understand _one_ particular way of doing "smart things"
(which will turn out to be almost exactly the dumbest possible for some
uses, and then you are stuck), or get lots of shards from broken apps (and
users, and sysadmins).

>                                     It takes 11 years of serious 
> research to build a clean storage layer able to handle doing that.  

Great! Build an experimental OS showing how to use it, and through that see
if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

> Reiser4 has done that, finally.  None of the other Linux filesystems 
> have.

How come nobody before you in the almost 30 years of Unix has ever seen the
need for this indispensable feature?

>        The next major release of ReiserFS is going to be bursting with 
> semantic enhancements, because the prerequisites for them are in place 
> now.  None of the other Linux filesystems have those prerequisites.  

To me that would mean that ReiserFS has no place in Linux.

> They won't be able to keep up with the semantic enhancements.  This 
> metafiles and file-directories stuff is actually fairly trivial stuff.

Maybe. But the question of it being of any use aren't trivially answered
"yes". My gut feeling is that the answer is a resounding "no", but that's
just me. Direcories are directories, if you want to ship directory-like
stuff around, use directories (or tarfiles, or whatever). Just look what
happened to the much, much easier stuff of splitting SUID/SGID into
capabilities: Nothing at all whatsoever, because they have no decent place
to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
able to _define_ APIs to follow, but even so it isn't at all easy to do so.

> Look guys, in 1993 I anticipated the battle would be here, and I build 
> the foundation for a defensive tower right at the spot MS and Apple are 
> now maneuvering towards.

Why place a protective tower around the pit into which they are trying
desperately to throw themselves? ;-)

>                           Help me get the next level on the tower before 
> they get here.  It is one hell of a foundation, they won't be able to 
> shake it, their trees are not as powerful.  Don't move reiser4 into vfs, 
> use reiser4 as the vfs.  Don't write filesystems, write file plugins and 
> disk format plugins and all the other kinds of plugins, and you won't be 
> missing any expressive power that you really want....

Better write libraries handling whatever weird format you have in mind for
each use. Even applications. I don't expect to be able to use emacs to edit
a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
a poem. The current trend (which I heartily agree with) is to take stuff
_out_ of the kernel (for flexibility, access, development ease, even
performance).
-- 
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Thu, Aug 26, 2004 at 02:12:32PM -0400, Horst von Brand wrote:

>> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
>> for Apple,
>Says who?

People will say it when people stop using Linux on servers because
they can integrate metadata easier in other operating systems ;)

>Please don't. Unix works and is extremely popular because it _doesn't_ try
>to be smart (policy vs mechanism distinction, simple abstractions that can
>be combined endlessly). If you add this to the filesystems, you either redo

Just more moving little pieces to be combined endlessly.

I think it looks like Reiser4 semantics should go into VFS and not break
legacy posix compatibility.

If you don't want to use a filesystem like that, don't.
Maybe the reason Linux hasn't developed this yet is the fact that Linux
supports n+1 file systems and above and beyond that many userspace ones.
So there has been enough work in maintaining and competing the old
ones and not coming up with new ones.

>_all_ userland to understand _one_ particular way of doing "smart things"
>(which will turn out to be almost exactly the dumbest possible for some
>uses, and then you are stuck), or get lots of shards from broken apps (and
>users, and sysadmins).

This is a question if implementation.

What if the files-as-dirs have a ..metas/mainstream[1] file and are handled
by legacy applications as directories? Nothing breaks, but may look a bit
ugly. Nothing one line of shell scripting would not cure.

OTOH, the user space progs are quickly patched to support all this.

>Great! Build an experimental OS showing how to use it, and through that see
>if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

Why should he?

Parts of Linux are already here and susceptible to experiments in
evolution. (Break out 2.7 for this one, the people want code! :)

>How come nobody before you in the almost 30 years of Unix has ever seen the
>need for this indispensable feature?

Lack of imagination?
Lack of need?
Lack of drive space?

And maybe someone did think among these lines back in the days, it's
not like the current dogma of file systems is the only one that has
ever been used.

>To me that would mean that ReiserFS has no place in Linux.

This is the type of argumentation I can not understand.
Do you not have the freedom not to use it?

And this is not personal, but please say Reiser4, as ReiserFS is coupled
with versions 3.x. They are quite different.

>Maybe. But the question of it being of any use aren't trivially answered
>"yes". My gut feeling is that the answer is a resounding "no", but that's

I think it's a pretty big yes, even if the technical details do need to
be clear before anything is set in stone. That's the reason for the
on-going discussion.

>just me. Direcories are directories, if you want to ship directory-like
>stuff around, use directories (or tarfiles, or whatever). Just look what

Without a common user-space API to deal in them. Oh, joy.
Make the common user-space API? Will never happen.

>Better write libraries handling whatever weird format you have in mind for
>each use. Even applications. I don't expect to be able to use emacs to edit
>a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>a poem. The current trend (which I heartily agree with) is to take stuff
>_out_ of the kernel (for flexibility, access, development ease, even
>performance).

I don't think anyone's talking about using emacs to edit a picture, but
emacs could be used to edit the picture's description.[2]
Maybe with some userspace-hook that exports the exif data via ..metas/
so the kernel won't have to parse it.

Linus wrote about why TCP is not in userspace and I think this is a similar
situation. Technically. But not everything has to be in kernel space
in Reiser4. Or that's what they say and I'm not sure where I stand on
an exif parser in the kernel...

On the social side, I may be wrong in this, but not many userspace 
filesystems have the possibility of becoming a de-facto standard. One 
might say that's because they suck, or they suck because they're 
implemented in userspace or they are just met with prejudice
for the same reason.

In the kernel something like this would have to be done right and people
would put effort into doing it right.

I for one would welcome that.

[1] Sorry, guys, just had to use that name..
[2] You can edit a picture in emacs already, it's just prone to break ;)

-- 
mjt

Re: silent semantic changes with reiser4

[Linus Torvalds]

On Fri, 27 Aug 2004, Markus Törnqvist wrote:
> 
> People will say it when people stop using Linux on servers because
> they can integrate metadata easier in other operating systems ;)

Heh. Considering that WinFS seems to be delayed yet more, I don't think 
that's a very strong argument.

Hell will freeze over before Microsoft does a filesystem right. Besides,
WinFS is likely almost in user mode anyway, ie mostly a library, rather
like the gnome people are already doing with nome storage.

So there's really no point in trying to push your agenda by trying to 
scare people with MS activities. Linux kernel developers do what's right 
because it is _right_, not because somebody else does it.

		Linus

Re[2]: silent semantic changes with reiser4

[Spam]

> On Fri, 27 Aug 2004, Markus Törnqvist wrote:
>> 
>> People will say it when people stop using Linux on servers because
>> they can integrate metadata easier in other operating systems ;)

> Heh. Considering that WinFS seems to be delayed yet more, I don't think
> that's a very strong argument.

> Hell will freeze over before Microsoft does a filesystem right. Besides,
> WinFS is likely almost in user mode anyway, ie mostly a library, rather
> like the gnome people are already doing with nome storage.

  Yes, it is mostly in user land. But, It is still integrated with the
  Windows  API.  This  means that normal programs that use the Windows
  API  will  be  able to use most of the new features (like displaying
  thumbnails, filtered views, etc when you do file operations)

  Besides,  NTFS  is  not  that  bad. It does have both encryption and
  compression  and a big set of ACL features that aren't in most Linux
  filesystems today.

> So there's really no point in trying to push your agenda by trying to
> scare people with MS activities. Linux kernel developers do what's right
> because it is _right_, not because somebody else does it.

  I  agree.  Things  should be done because they are good, not because
  you are threatened to.

  It  is  a shame really that so many people hate Microsoft (or Apple)
  so much they fail to see things that actually are good.

>         Linus

Re: silent semantic changes with reiser4

[Markus Törnqvist]

On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote:
>
>Heh. Considering that WinFS seems to be delayed yet more, I don't think 
>that's a very strong argument.

But there's Mac OS and isn't Solaris doing something along these lines?

Microsoft isn't alone there ;)

>scare people with MS activities. Linux kernel developers do what's right 
>because it is _right_, not because somebody else does it.

Never really doubted that, mind you.
I just think this is both right and a competition situation.

-- 
mjt

Re: silent semantic changes with reiser4

[Bill Huey]

On Fri, Aug 27, 2004 at 01:38:28PM -0700, Linus Torvalds wrote:
> Heh. Considering that WinFS seems to be delayed yet more, I don't think 
> that's a very strong argument.

Which is another argument, and bragging rights, to integrate something
like Reiser FS4. :)

> Hell will freeze over before Microsoft does a filesystem right. Besides,
> WinFS is likely almost in user mode anyway, ie mostly a library, rather
> like the gnome people are already doing with nome storage.

They're trying to push their low level storage model into the kernel
after they saw how slow their userspace version of that is. It makes
sense to have this kind of stuff directly apart of a lower level FS
later
 
> So there's really no point in trying to push your agenda by trying to 
> scare people with MS activities. Linux kernel developers do what's right 
> because it is _right_, not because somebody else does it.

I tried to stay out of this thread, but I couldn't resist after this
post. :) MS has other problems, namely, they're a large company trying
to basically replicated Apple's (NeXT) OS X stuff, but onl 10x larger
and slower than it is. It's really silly what's going on with MS
development as well as the greater open source X community. The sooner
folks realize that all rendering models are moving in the direction of
Apple's OS X, the sooner they can just flat out rip it off.

It's going to require both kernel and userspace changes for this to
happen.

bill

Re: silent semantic changes with reiser4

[Hans Reiser]

Linus Torvalds wrote:

>On Fri, 27 Aug 2004, Markus Törnqvist wrote:
>  
>
>>People will say it when people stop using Linux on servers because
>>they can integrate metadata easier in other operating systems ;)
>>    
>>
>
>Heh. Considering that WinFS seems to be delayed yet more, I don't think 
>that's a very strong argument.
>
>Hell will freeze over before Microsoft does a filesystem right. Besides,
>WinFS is likely almost in user mode anyway, ie mostly a library, rather
>like the gnome people are already doing with nome storage.
>
>So there's really no point in trying to push your agenda by trying to 
>scare people with MS activities. Linux kernel developers do what's right 
>because it is _right_, not because somebody else does it.
>
>		Linus
>
>
>  
>
Apple will get it right.  I promise it.  I have met Dominic, and he is 
very very sharp.  Look at the Tiger demos on their website.  Simple 
interface, looks nice to me.... 

The one area he might screw up is performance, but I don't care to count 
on that.

WinFS first tried to put it all in the FS, and then it became a user 
mode library almost certainly because they are making the standard 
mistakes the database guys make when they try to emulate file systems 
without changing the core balanced tree algorithms, and their 
performance sucked and they had to back off.  It took 11 years for me to 
get it right, and they aren't as crazy-err-persistent as I am.;-)

We might get lucky and have them produce another NTFS, but then again, 
when Microsoft focuses on a task, they do much better at it than they do 
most of the time, and they are focused on WinFS.  They have hired very 
sharp people.  We can hope that they don't know how to use them, but 
when they hire people like Gerard Salton for $1 million a year, there is 
just possibly a chance that they might try to get their money's worth 
out of him.

You should not be complacent about WinFS being delayed to 2007, because 
even if I get funding for enhanced ReiserFS semantics tomorrow we also 
can't get the job done before 2007.  This is big science, not writing a 
device driver.

Finally, how much harm will it be if we do it right and it is important 
and they fail?  Suppose I am wrong about them, and we create a powerful 
unifying namespace for Linux before any other OS does?  Is that so bad?

Creating a powerful namespace at the heart of Linux is the most 
important enhancement you can make to the OS. 

Finally the storage layer is good enough to support putting the 
relationship between keywords (actually keyobjects in my scheme....) and 
their documents directly into the FS without losing performance for 
traditional file system usage patterns, and I get to stop tweaking 
performance and go have fun with semantics in the next major release.

Hans

Re: silent semantic changes with reiser4

[Bill Huey]

On Sat, Aug 28, 2004 at 02:55:00AM -0700, Hans Reiser wrote:
> mistakes the database guys make when they try to emulate file systems 
> without changing the core balanced tree algorithms, and their 
> performance sucked and they had to back off.  It took 11 years for me to 
> get it right, and they aren't as crazy-err-persistent as I am.;-)
> 
> We might get lucky and have them produce another NTFS, but then again, 
> when Microsoft focuses on a task, they do much better at it than they do 
> most of the time, and they are focused on WinFS.  They have hired very 
> sharp people.  We can hope that they don't know how to use them, but 
> when they hire people like Gerard Salton for $1 million a year, there is 
> just possibly a chance that they might try to get their money's worth 
> out of him.

I realize this is a relatively late reply, but...

MS's problem is that they hire very smart folks, but the compartimentalize
them so they don't influence the overall architecture of the entire system.
I guarantee you that even if they did get all of this in place and working,
their arrogance in this matter will screw up how this will be properly used
and never expressed the power of a system like this.

Apple is another story. They're the only other folks that might get this right.

Your system is a dream of mine since I was dorking around with persistent
objects in Smalltalk systems. But getting other folks to understand
something like this and then use it in this way is very difficult and I
don't expect MS to be able to do this nor get .NET to compete with Java
for server side applications. I don't expect traditional Unix folks to
understand why something like this is important also.

It's not about a single technology as much as the system as a whole,
larger picture... That's were a system starts to show it's superiority.

That's my two cents on the matter.

bill

Re: silent semantic changes with reiser4

[Spam]

> Hans Reiser <reiser@namesys.com> said:
>> Andrew Morton wrote:

> [...]

>> > The fact that one filesystem will offer features which other
>> > filesystems do not and cannot offer makes me queasy for some reason.

> Not me. As long as it is clearly experimental stuff that can be removed at
> the head hacker's whim, that is. 2.7 material.

>> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> for Apple,

> Says who?

  At  least  I  as  a user see the benefits largely. The workstation /
  desktop  user has much more need for such features, than perhaps the
  normal Linux sysadmin.

>>            and that means we need to put search engine and database
>> functionality into the filesystem.

> Please don't. Unix works and is extremely popular because it _doesn't_ try
> to be smart (policy vs mechanism distinction, simple abstractions that can
> be combined endlessly). If you add this to the filesystems, you either redo
> _all_ userland to understand _one_ particular way of doing "smart things"
> (which will turn out to be almost exactly the dumbest possible for some
> uses, and then you are stuck), or get lots of shards from broken apps (and
> users, and sysadmins).

  Indeed.  But  you forget one thing. Combining lots of small userland
  tools can actually make things more complex - at least for the user.

  That said, I do not see that anyone has suggested to remove this way
  of  working with applications.

  Are the new and old features mutually exclusive?

>>                                     It takes 11 years of serious 
>> research to build a clean storage layer able to handle doing that.

> Great! Build an experimental OS showing how to use it, and through that see
> if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

  There  are  many  uses already. It could be encryption, compression,
  meta-data,  file streams, sorting, searching and a multitude of uses
  and ideas that haven't been discovered yet.

>> Reiser4 has done that, finally.  None of the other Linux filesystems
>> have.

> How come nobody before you in the almost 30 years of Unix has ever seen the
> need for this indispensable feature?

  I,  as  a  user, have for a long time. Do not drag everyone over the
  same  edge.  Users  have  different  needs.  We  should try to be as
  forthcoming as possible to provide the tools and options for them.

>>        The next major release of ReiserFS is going to be bursting with
>> semantic enhancements, because the prerequisites for them are in place
>> now.  None of the other Linux filesystems have those prerequisites.

> To me that would mean that ReiserFS has no place in Linux.

  Why  do  you say this? I thought Linux embraced advancements and new
  technologies.

>> They won't be able to keep up with the semantic enhancements.  This
>> metafiles and file-directories stuff is actually fairly trivial stuff.

> Maybe. But the question of it being of any use aren't trivially answered
> "yes". My gut feeling is that the answer is a resounding "no", but that's
> just me. Direcories are directories, if you want to ship directory-like
> stuff around, use directories (or tarfiles, or whatever). Just look what
> happened to the much, much easier stuff of splitting SUID/SGID into
> capabilities: Nothing at all whatsoever, because they have no decent place
> to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
> able to _define_ APIs to follow, but even so it isn't at all easy to do so.

  Exactly.  The  problem  I see in this discussion is the lack of just
  this - APIs. Because the lack of good APIs we cannot make any larger
  changes or bring new features because old applications will break.

  I think that Linux should make new API that will completely separate
  applications  from the storage media. This way we will not as easily
  break  applications  when new features are developed. This should be
  the main priority.

  Perhaps  over a transition period the old ways will be allowed while
  applications  are  ported  to  the new APIs. The change is certainly
  going  to be needed at one point or Linux won't be able to evolve as
  quickly as its users are demanding it.

  Application  breakage  has  happened  before between virtually every
  kernel change. It isn't new. Just take devfs and sysfs.


>> Look guys, in 1993 I anticipated the battle would be here, and I build
>> the foundation for a defensive tower right at the spot MS and Apple are
>> now maneuvering towards.

> Why place a protective tower around the pit into which they are trying
> desperately to throw themselves? ;-)

  It  is easy to say this. But the fact is that users can actually use
  these new features for good things.

>>                           Help me get the next level on the tower before
>> they get here.  It is one hell of a foundation, they won't be able to
>> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> disk format plugins and all the other kinds of plugins, and you won't be
>> missing any expressive power that you really want....

> Better write libraries handling whatever weird format you have in mind for
> each use. Even applications. I don't expect to be able to use emacs to edit
> a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
> a poem. The current trend (which I heartily agree with) is to take stuff
> _out_ of the kernel (for flexibility, access, development ease, even
> performance).

  If  I  want  to  be able to quickly find files by a certain author I
  could   search  the  meta-data fields (if they are used) with normal
  tools such as grep.

  Compression or encryption is another example.

  Would  you  rather  patch  every  application  there is for Linux to
  enable support for reading these files and directories?

  

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Horst von Brand <vonbrand@inf.utfsm.cl> said:
>> > Hans Reiser <reiser@namesys.com> said:
>> >> Andrew Morton wrote:

> [...]

>> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> >> for Apple,
>> 
>> > Says who?
>> 
>>   At  least  I  as  a user see the benefits largely. The workstation /
>>   desktop  user has much more need for such features, than perhaps the
>>   normal Linux sysadmin.

> Show benefits, show there are no downsides. Show that the same can't
> possibly be done in userland. Then I'll consider buying it.

  I am not saying it couldn't be done in userland. Most things can.

  The  problem with userland is that it doesn't provide generic access
  across  most  applications. For example GnomeVFS is limited to Gnome
  applications  only.  The  same  for  KDE.  to  be able to coordinate
  application developer on anything is just almost impossible.

  Still.  Why do you oppose plugins, streams and meta files? The could
  be   valuable   and  easy  to use tools for many purposes. One could
  be  would be advanced ACLs defined as a meta-file using XML format.
  

> AFAICS: Smallish benefits for some narrow application niche. Mayor problems
> (no decent multiuser way of using it, extremely complex stuff placed in the
> kernel). Userland solutions are all around us right now, and work just
> fine. No, placing the same gunk in the kernel won't make it magically use
> no time or memory, more like exactly the opposite. Besides, all the visions
> waved around here have very little in common (except the idea that they
> make all restrictions magically dissapear).

> Get a list of required features, show concrete ways of using it, outline
> possible implementations, discuss resource usage. I.e., do "engineering".
> Then we have something to discuss.

  I  am not sure what you refer to as "required features". Do you mean
  example of usage and contents for streams, plugins and meta files?

>> >>            and that means we need to put search engine and database
>> >> functionality into the filesystem.

>> > Please don't. Unix works and is extremely popular because it _doesn't_ try
>> > to be smart (policy vs mechanism distinction, simple abstractions that can
>> > be combined endlessly). If you add this to the filesystems, you either redo
>> > _all_ userland to understand _one_ particular way of doing "smart things"
>> > (which will turn out to be almost exactly the dumbest possible for some
>> > uses, and then you are stuck), or get lots of shards from broken apps (and
>> > users, and sysadmins).

>>   Indeed.  But  you forget one thing. Combining lots of small userland
>>   tools can actually make things more complex - at least for the user.

> So? Placing lots of little things in kernel does the same, plus screws over
> sysadmins, and makes kernel hacker's job harder. I prefer the former alone.

  Difference being that it could be transparent in a way otherwise not
  possible.

>>   That said, I do not see that anyone has suggested to remove this way
>>   of  working with applications.

> Excuse me, if files suddenly become directories, and you can't have some
> "reserved" file/directory names, and... you are breaking lots of stuff.

>>   Are the new and old features mutually exclusive?

> Yep.

>> >>                                     It takes 11 years of serious
>> >> research to build a clean storage layer able to handle doing that.
>> 
>> > Great! Build an experimental OS showing how to use it, and through that see
>> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

>>   There  are  many  uses already.

> Where?

>>                                   It could be encryption, compression,
>>   meta-data,  file streams, sorting, searching and a multitude of uses
>>   and ideas that haven't been discovered yet.

> "Could", "multitude ... haven't been discovered yet" is _not_ "many current
> uses".

>> >> Reiser4 has done that, finally.  None of the other Linux filesystems
>> >> have.

>> > How come nobody before you in the almost 30 years of Unix has ever seen the
>> > need for this indispensable feature?

>>   I,  as  a  user, have for a long time. Do not drag everyone over the
>>   same  edge.  Users  have  different  needs.  We  should try to be as
>>   forthcoming as possible to provide the tools and options for them.

> Why? Breaking the basic paradigms for some user whim is not good OS design
> in my book, sorry. Many "bright ideas" have been thrown out over the years
> for just the same reasons.

>> >>        The next major release of ReiserFS is going to be bursting with
>> >> semantic enhancements, because the prerequisites for them are in place
>> >> now.  None of the other Linux filesystems have those prerequisites.

>> > To me that would mean that ReiserFS has no place in Linux.

>>   Why  do  you say this? I thought Linux embraced advancements and new
>>   technologies.

> If they are right. I think breaking the base of the OS is not a good
> idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where
> this is supposedly so so badly needed?), but not here.

  You  mean  in  Windows  and  MacOS  where much of this functionality
  already exists?

>> >> They won't be able to keep up with the semantic enhancements.  This
>> >> metafiles and file-directories stuff is actually fairly trivial stuff.

>> > Maybe. But the question of it being of any use aren't trivially answered
>> > "yes". My gut feeling is that the answer is a resounding "no", but that's
>> > just me. Direcories are directories, if you want to ship directory-like
>> > stuff around, use directories (or tarfiles, or whatever). Just look what
>> > happened to the much, much easier stuff of splitting SUID/SGID into
>> > capabilities: Nothing at all whatsoever, because they have no decent place
>> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
>> > able to _define_ APIs to follow, but even so it isn't at all easy to do so.

>>   Exactly.  The  problem  I see in this discussion is the lack of just
>>   this - APIs. Because the lack of good APIs we cannot make any larger
>>   changes or bring new features because old applications will break.


  I'll chop this up...
  
> No. There is a complete lack of answers to why it should be done ("I think
> it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from
> enough)

  No   perhaps  not  by itself. But it certainly should be considered.
  These  two  OS'es  have  most  of  the market share. They have great
  resources  trying to make things that work for their users.

  Users  are  more  important  than you think. Linux and any OS can do
  what  it  wants, but if it wants to keep users then it has to listen
  to them.

> , on what the alternatives are ("Sorry, it must be in the kernel
> because it is impossible to get application writers to agree on how to do
> it"

  This  is  a  perfectly  valid  reason.  If  it  is necessary to move
  something  into  the  core of Linux because it couldn't otherwise be
  done then it should seriously be considered.

>  will just add another incompatible way of "doing almost the same", this
> time in the kernel),

   People  on  this  list  are  actually trying to make it to be fully
   compatible with old legacy applications.

   If  you  hadn't noticed, Reiser4 does work. The meta files do work.
   The  discussion  is  wether parts of Reiser4 should be made generic
   and fit into the VFS.

>  and last but not least what the impact is (performance
> wise, application breakage, user/sysadmin expectation breakage,
> incompatibility with other systems, kernel development impact, to list a
> few). Hash this out, and we can talk.

  You  paint up a doomsday picture here. Nothing of this magnitude has
  happened  and  won't. besides, what is wrong if sysadmins would have
  to learn something new?

>>   I think that Linux should make new API that will completely separate
>>   applications  from the storage media. This way we will not as easily
>>   break  applications  when new features are developed. This should be
>>   the main priority.

> The "files + directories" stuff is a nice abstraction, and works today on
> all random-access storage media. You don't care much if it is on a CD, a
> floppy, a disk, a RAID array, or a flash card, do you? It won't work on
> tapes, or printers; but that is to be expected.

  you  didn't  actually  comment  on  what  I  said. I mentioned API's
  nothing about directories and files. API being the keyword here.

  If  there  are  good API's that applications can use, for example to
  access filesystems, then it will be easier and much more possible to
  introduce  new  things into the filesystem without breaking existing
  applications.  One  example of preventing application breakage could
  be   to   have  a  versioning system. Applications could query which
  version is supported, or vice versa.

  From  what  I  understand  it  is  much  harder to do anything today
  because of the fixed ways. This is why we have this discussion.

> [...]

>>   Application  breakage  has  happened  before between virtually every
>>   kernel change. It isn't new. Just take devfs and sysfs.

> Oops, now emacs stopped working because devfs was added? gcc needed a mayor
> overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_
> breakage, not all-around application breakage.

  Emacs  seem  to  work  fine  with  the  file-as-dir  on  my  Reiser4
  partition. And so does actually any other application I have used on
  it. Be it in Gnome, KDE or in plain shell's.

>> >> Look guys, in 1993 I anticipated the battle would be here, and I build
>> >> the foundation for a defensive tower right at the spot MS and Apple are
>> >> now maneuvering towards.
>> 
>> > Why place a protective tower around the pit into which they are trying
>> > desperately to throw themselves? ;-)
>> 
>>   It  is easy to say this. But the fact is that users can actually use
>>   these new features for good things.

> And can abuse them for bad things, and the features aren't free. Balance
> benefits (very minor) against costs (quite large).

  Abuse depends on permissions and security policies which is your job
  as admin to look after.

  It  would  be  possible to limit, for example), the types of plugins
  that a user can either use or load.

  Do you give your users access to modprobe or tools like that?

>> >>                           Help me get the next level on the tower before
>> >> they get here.  It is one hell of a foundation, they won't be able to
>> >> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> >> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> >> disk format plugins and all the other kinds of plugins, and you won't be
>> >> missing any expressive power that you really want....
>> 
>> > Better write libraries handling whatever weird format you have in mind for
>> > each use. Even applications. I don't expect to be able to use emacs to edit
>> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>> > a poem. The current trend (which I heartily agree with) is to take stuff
>> > _out_ of the kernel (for flexibility, access, development ease, even
>> > performance).

>>   If  I  want  to  be able to quickly find files by a certain author I
>>   could   search  the  meta-data fields (if they are used) with normal
>>   tools such as grep.

> Iff the metadata is there. It won't (everybody would have to agree on
> placing that on the files for it to work), so it is useless.

  No  it  is  not  useless.  It  just  means  the  file  has  no  meta
  information. Just as it is today. No change.

>>   Compression or encryption is another example.
>> 
>>   Would  you  rather  patch  every  application  there is for Linux to
>>   enable support for reading these files and directories?

> No. But an encrypted or compressed filessytem doesn't need weird files.
> Easiest on applications is just to serve the decrypted/uncompressed files
> for use via the traditional APIs. Next easiest is provide programs that
> handle cryptography and compression and can be combined with whatever
> unaware program you care to use. Funny, that is exactly the way things work
> now...

  So you can setup a directory that has some files compressed and some
  other  encrypted and yet they will be available to use with tar, cp,
  ls, mc, Nautilus, Emacs etc etc?

Re: silent semantic changes with reiser4

[Nikita Danilov]

Spam writes:
 > 
 > > Spam <spam@tnonline.net> said:
 > >> Horst von Brand <vonbrand@inf.utfsm.cl> said:

[...]

 > 
 > > Show benefits, show there are no downsides. Show that the same can't
 > > possibly be done in userland. Then I'll consider buying it.
 > 
 >   I am not saying it couldn't be done in userland. Most things can.
 > 
 >   The  problem with userland is that it doesn't provide generic access
 >   across  most  applications. For example GnomeVFS is limited to Gnome
 >   applications  only.  The  same  for  KDE.  to  be able to coordinate
 >   application developer on anything is just almost impossible.

Hmm... drag-and-drop also doesn't work constently over all
applications. Let's put it into kernel.

Seriously, kernel prformance is critical to the system, and to achieve
high performance all kernel code runs in single address space (actually,
in portion of address spaces of user processes shared by all
processes). All shared system state is located there. This means that
bug in a kernel affects whole system. This means that kernel should be
kept as simple as possible (and a bit more simple).

This is why only things that cannot be done efficiently in the user
level are put into kernel. And political agendas of various camps of
user-level developers change nothing here.

 > 
 >   Still.  Why do you oppose plugins, streams and meta files? The could
 >   be   valuable   and  easy  to use tools for many purposes. One could
 >   be  would be advanced ACLs defined as a meta-file using XML format.

POSIX has standard ACL API in C (well, "eternal draft" only), one can
use it to extract ACLs from kernel and convert it to any format to one's
heart content. Advantage of this is that when XML goes out of fashion (I
hope this wouldn't yet happen when you will read this message), kernel
API will remain intact.

 >   

Nikita.

Re: silent semantic changes with reiser4

[Markus Törnqvist]

[ This will probably go down in history as the most pointless message
  ever, I just told Nik on IRC I'd take it up on the list :) ]

On Sun, Aug 29, 2004 at 09:22:08PM +0400, Nikita Danilov wrote:
>Spam writes:

>Hmm... drag-and-drop also doesn't work constently over all
>applications. Let's put it into kernel.

What I'm after is that the file system should be able to store
any arbitrary metadata. No matter what the format is.

Maybe I was a bit harsh on XML in my previous email, but whatever.

>Seriously, kernel prformance is critical to the system, and to achieve
>high performance all kernel code runs in single address space (actually,
>in portion of address spaces of user processes shared by all
>processes). All shared system state is located there. This means that
>bug in a kernel affects whole system. This means that kernel should be
>kept as simple as possible (and a bit more simple).

I do agree here. But is it seriously such a big issue to implement
the files-as-dirs kind of thing?

Be it overloading MAY_READ so it works with chdir or implementing openat
or whatever it takes.

I don't think this should be much more, if any more, than the capability
to store this data. Which Reiser4 does. But moved from the Reiser4
level to the VFS level. Right?

>This is why only things that cannot be done efficiently in the user
>level are put into kernel. And political agendas of various camps of
>user-level developers change nothing here.

What I'd want to do is to be able to write a Reiser4 plugin that suits
my needs for something. Not that I'd even know how to, but that's a
matter of time.

Are there other means of doing this than Reiser? I mean Reiser4 plugin
functionality by other means.

>>   Still.  Why do you oppose plugins, streams and meta files? The could
>>   be   valuable   and  easy  to use tools for many purposes. One could
>>   be  would be advanced ACLs defined as a meta-file using XML format.
>POSIX has standard ACL API in C (well, "eternal draft" only), one can
>use it to extract ACLs from kernel and convert it to any format to one's
>heart content. Advantage of this is that when XML goes out of fashion (I
>hope this wouldn't yet happen when you will read this message), kernel
>API will remain intact.

Aa, this apparently is an argument against XML parsing in the kernel,
not the idea that if someone wanted ACLs, that are in XML format, and
could not use them (because the kernel doesn't parse), he should be
able to write such a plugin.

Besides, let's say the Reiser4 XML-ACL plugin has a parser that actually
DOES something. For argument's sake. Would it not be up to the individual
user to disable this plugin? I'd assume it's possible to write plugins
so that they do not affect anything but themselves, meaning this parser.

Of course the problem of copying a file with XML-ACLs to another Reiser4
fs without support for them should be thought of. But I guess this is
where you take the shortest route; the XML-ACL files are intact in the
file's metadata, but totally unusable?

-- 
mjt

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Horst von Brand <vonbrand@inf.utfsm.cl> said:
>> > Hans Reiser <reiser@namesys.com> said:
>> >> Andrew Morton wrote:

> [...]

>> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> >> for Apple,
>> 
>> > Says who?
>> 
>>   At  least  I  as  a user see the benefits largely. The workstation /
>>   desktop  user has much more need for such features, than perhaps the
>>   normal Linux sysadmin.

> Show benefits, show there are no downsides. Show that the same can't
> possibly be done in userland. Then I'll consider buying it.

  I am not saying it couldn't be done in userland. Most things can.

  The  problem with userland is that it doesn't provide generic access
  across  most  applications. For example GnomeVFS is limited to Gnome
  applications  only.  The  same  for  KDE.  to  be able to coordinate
  application developer on anything is just almost impossible.

  Still.  Why do you oppose plugins, streams and meta files? The could
  be   valuable   and  easy  to use tools for many purposes. One could
  be  would be advanced ACLs defined as a meta-file using XML format.
  

> AFAICS: Smallish benefits for some narrow application niche. Mayor problems
> (no decent multiuser way of using it, extremely complex stuff placed in the
> kernel). Userland solutions are all around us right now, and work just
> fine. No, placing the same gunk in the kernel won't make it magically use
> no time or memory, more like exactly the opposite. Besides, all the visions
> waved around here have very little in common (except the idea that they
> make all restrictions magically dissapear).

> Get a list of required features, show concrete ways of using it, outline
> possible implementations, discuss resource usage. I.e., do "engineering".
> Then we have something to discuss.

  I  am not sure what you refer to as "required features". Do you mean
  example of usage and contents for streams, plugins and meta files?

>> >>            and that means we need to put search engine and database
>> >> functionality into the filesystem.

>> > Please don't. Unix works and is extremely popular because it _doesn't_ try
>> > to be smart (policy vs mechanism distinction, simple abstractions that can
>> > be combined endlessly). If you add this to the filesystems, you either redo
>> > _all_ userland to understand _one_ particular way of doing "smart things"
>> > (which will turn out to be almost exactly the dumbest possible for some
>> > uses, and then you are stuck), or get lots of shards from broken apps (and
>> > users, and sysadmins).

>>   Indeed.  But  you forget one thing. Combining lots of small userland
>>   tools can actually make things more complex - at least for the user.

> So? Placing lots of little things in kernel does the same, plus screws over
> sysadmins, and makes kernel hacker's job harder. I prefer the former alone.

  Difference being that it could be transparent in a way otherwise not
  possible.

>>   That said, I do not see that anyone has suggested to remove this way
>>   of  working with applications.

> Excuse me, if files suddenly become directories, and you can't have some
> "reserved" file/directory names, and... you are breaking lots of stuff.

>>   Are the new and old features mutually exclusive?

> Yep.

>> >>                                     It takes 11 years of serious
>> >> research to build a clean storage layer able to handle doing that.
>> 
>> > Great! Build an experimental OS showing how to use it, and through that see
>> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

>>   There  are  many  uses already.

> Where?

>>                                   It could be encryption, compression,
>>   meta-data,  file streams, sorting, searching and a multitude of uses
>>   and ideas that haven't been discovered yet.

> "Could", "multitude ... haven't been discovered yet" is _not_ "many current
> uses".

>> >> Reiser4 has done that, finally.  None of the other Linux filesystems
>> >> have.

>> > How come nobody before you in the almost 30 years of Unix has ever seen the
>> > need for this indispensable feature?

>>   I,  as  a  user, have for a long time. Do not drag everyone over the
>>   same  edge.  Users  have  different  needs.  We  should try to be as
>>   forthcoming as possible to provide the tools and options for them.

> Why? Breaking the basic paradigms for some user whim is not good OS design
> in my book, sorry. Many "bright ideas" have been thrown out over the years
> for just the same reasons.

>> >>        The next major release of ReiserFS is going to be bursting with
>> >> semantic enhancements, because the prerequisites for them are in place
>> >> now.  None of the other Linux filesystems have those prerequisites.

>> > To me that would mean that ReiserFS has no place in Linux.

>>   Why  do  you say this? I thought Linux embraced advancements and new
>>   technologies.

> If they are right. I think breaking the base of the OS is not a good
> idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where
> this is supposedly so so badly needed?), but not here.

  You  mean  in  Windows  and  MacOS  where much of this functionality
  already exists?

>> >> They won't be able to keep up with the semantic enhancements.  This
>> >> metafiles and file-directories stuff is actually fairly trivial stuff.

>> > Maybe. But the question of it being of any use aren't trivially answered
>> > "yes". My gut feeling is that the answer is a resounding "no", but that's
>> > just me. Direcories are directories, if you want to ship directory-like
>> > stuff around, use directories (or tarfiles, or whatever). Just look what
>> > happened to the much, much easier stuff of splitting SUID/SGID into
>> > capabilities: Nothing at all whatsoever, because they have no decent place
>> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
>> > able to _define_ APIs to follow, but even so it isn't at all easy to do so.

>>   Exactly.  The  problem  I see in this discussion is the lack of just
>>   this - APIs. Because the lack of good APIs we cannot make any larger
>>   changes or bring new features because old applications will break.


  I'll chop this up...
  
> No. There is a complete lack of answers to why it should be done ("I think
> it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from
> enough)

  No   perhaps  not  by itself. But it certainly should be considered.
  These  two  OS'es  have  most  of  the market share. They have great
  resources  trying to make things that work for their users.

  Users  are  more  important  than you think. Linux and any OS can do
  what  it  wants, but if it wants to keep users then it has to listen
  to them.

> , on what the alternatives are ("Sorry, it must be in the kernel
> because it is impossible to get application writers to agree on how to do
> it"

  This  is  a  perfectly  valid  reason.  If  it  is necessary to move
  something  into  the  core of Linux because it couldn't otherwise be
  done then it should seriously be considered.

>  will just add another incompatible way of "doing almost the same", this
> time in the kernel),

   People  on  this  list  are  actually trying to make it to be fully
   compatible with old legacy applications.

   If  you  hadn't noticed, Reiser4 does work. The meta files do work.
   The  discussion  is  wether parts of Reiser4 should be made generic
   and fit into the VFS.

>  and last but not least what the impact is (performance
> wise, application breakage, user/sysadmin expectation breakage,
> incompatibility with other systems, kernel development impact, to list a
> few). Hash this out, and we can talk.

  You  paint up a doomsday picture here. Nothing of this magnitude has
  happened  and  won't. besides, what is wrong if sysadmins would have
  to learn something new?

>>   I think that Linux should make new API that will completely separate
>>   applications  from the storage media. This way we will not as easily
>>   break  applications  when new features are developed. This should be
>>   the main priority.

> The "files + directories" stuff is a nice abstraction, and works today on
> all random-access storage media. You don't care much if it is on a CD, a
> floppy, a disk, a RAID array, or a flash card, do you? It won't work on
> tapes, or printers; but that is to be expected.

  you  didn't  actually  comment  on  what  I  said. I mentioned API's
  nothing about directories and files. API being the keyword here.

  If  there  are  good API's that applications can use, for example to
  access filesystems, then it will be easier and much more possible to
  introduce  new  things into the filesystem without breaking existing
  applications.  One  example of preventing application breakage could
  be   to   have  a  versioning system. Applications could query which
  version is supported, or vice versa.

  From  what  I  understand  it  is  much  harder to do anything today
  because of the fixed ways. This is why we have this discussion.

> [...]

>>   Application  breakage  has  happened  before between virtually every
>>   kernel change. It isn't new. Just take devfs and sysfs.

> Oops, now emacs stopped working because devfs was added? gcc needed a mayor
> overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_
> breakage, not all-around application breakage.

  Emacs  seem  to  work  fine  with  the  file-as-dir  on  my  Reiser4
  partition. And so does actually any other application I have used on
  it. Be it in Gnome, KDE or in plain shell's.

>> >> Look guys, in 1993 I anticipated the battle would be here, and I build
>> >> the foundation for a defensive tower right at the spot MS and Apple are
>> >> now maneuvering towards.
>> 
>> > Why place a protective tower around the pit into which they are trying
>> > desperately to throw themselves? ;-)
>> 
>>   It  is easy to say this. But the fact is that users can actually use
>>   these new features for good things.

> And can abuse them for bad things, and the features aren't free. Balance
> benefits (very minor) against costs (quite large).

  Abuse depends on permissions and security policies which is your job
  as admin to look after.

  It  would  be  possible to limit, for example), the types of plugins
  that a user can either use or load.

  Do you give your users access to modprobe or tools like that?

>> >>                           Help me get the next level on the tower before
>> >> they get here.  It is one hell of a foundation, they won't be able to
>> >> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> >> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> >> disk format plugins and all the other kinds of plugins, and you won't be
>> >> missing any expressive power that you really want....
>> 
>> > Better write libraries handling whatever weird format you have in mind for
>> > each use. Even applications. I don't expect to be able to use emacs to edit
>> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>> > a poem. The current trend (which I heartily agree with) is to take stuff
>> > _out_ of the kernel (for flexibility, access, development ease, even
>> > performance).

>>   If  I  want  to  be able to quickly find files by a certain author I
>>   could   search  the  meta-data fields (if they are used) with normal
>>   tools such as grep.

> Iff the metadata is there. It won't (everybody would have to agree on
> placing that on the files for it to work), so it is useless.

  No  it  is  not  useless.  It  just  means  the  file  has  no  meta
  information. Just as it is today. No change.

>>   Compression or encryption is another example.
>> 
>>   Would  you  rather  patch  every  application  there is for Linux to
>>   enable support for reading these files and directories?

> No. But an encrypted or compressed filessytem doesn't need weird files.
> Easiest on applications is just to serve the decrypted/uncompressed files
> for use via the traditional APIs. Next easiest is provide programs that
> handle cryptography and compression and can be combined with whatever
> unaware program you care to use. Funny, that is exactly the way things work
> now...

  So you can setup a directory that has some files compressed and some
  other  encrypted and yet they will be available to use with tar, cp,
  ls, mc, Nautilus, Emacs etc etc?

Re: silent semantic changes with reiser4

[Spam]

  

> Spam <spam@tnonline.net> said:
>> Horst von Brand <vonbrand@inf.utfsm.cl> said:
>> > Hans Reiser <reiser@namesys.com> said:
>> >> Andrew Morton wrote:

> [...]

>> >> Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem
>> >> for Apple,
>> 
>> > Says who?
>> 
>>   At  least  I  as  a user see the benefits largely. The workstation /
>>   desktop  user has much more need for such features, than perhaps the
>>   normal Linux sysadmin.

> Show benefits, show there are no downsides. Show that the same can't
> possibly be done in userland. Then I'll consider buying it.

  I am not saying it couldn't be done in userland. Most things can.

  The  problem with userland is that it doesn't provide generic access
  across  most  applications. For example GnomeVFS is limited to Gnome
  applications  only.  The  same  for  KDE.  to  be able to coordinate
  application developer on anything is just almost impossible.

  Still.  Why do you oppose plugins, streams and meta files? The could
  be   valuable   and  easy  to use tools for many purposes. One could
  be  would be advanced ACLs defined as a meta-file using XML format.
  

> AFAICS: Smallish benefits for some narrow application niche. Mayor problems
> (no decent multiuser way of using it, extremely complex stuff placed in the
> kernel). Userland solutions are all around us right now, and work just
> fine. No, placing the same gunk in the kernel won't make it magically use
> no time or memory, more like exactly the opposite. Besides, all the visions
> waved around here have very little in common (except the idea that they
> make all restrictions magically dissapear).

> Get a list of required features, show concrete ways of using it, outline
> possible implementations, discuss resource usage. I.e., do "engineering".
> Then we have something to discuss.

  I  am not sure what you refer to as "required features". Do you mean
  example of usage and contents for streams, plugins and meta files?

>> >>            and that means we need to put search engine and database
>> >> functionality into the filesystem.

>> > Please don't. Unix works and is extremely popular because it _doesn't_ try
>> > to be smart (policy vs mechanism distinction, simple abstractions that can
>> > be combined endlessly). If you add this to the filesystems, you either redo
>> > _all_ userland to understand _one_ particular way of doing "smart things"
>> > (which will turn out to be almost exactly the dumbest possible for some
>> > uses, and then you are stuck), or get lots of shards from broken apps (and
>> > users, and sysadmins).

>>   Indeed.  But  you forget one thing. Combining lots of small userland
>>   tools can actually make things more complex - at least for the user.

> So? Placing lots of little things in kernel does the same, plus screws over
> sysadmins, and makes kernel hacker's job harder. I prefer the former alone.

  Difference being that it could be transparent in a way otherwise not
  possible.

>>   That said, I do not see that anyone has suggested to remove this way
>>   of  working with applications.

> Excuse me, if files suddenly become directories, and you can't have some
> "reserved" file/directory names, and... you are breaking lots of stuff.

>>   Are the new and old features mutually exclusive?

> Yep.

>> >>                                     It takes 11 years of serious
>> >> research to build a clean storage layer able to handle doing that.
>> 
>> > Great! Build an experimental OS showing how to use it, and through that see
>> > if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

>>   There  are  many  uses already.

> Where?

>>                                   It could be encryption, compression,
>>   meta-data,  file streams, sorting, searching and a multitude of uses
>>   and ideas that haven't been discovered yet.

> "Could", "multitude ... haven't been discovered yet" is _not_ "many current
> uses".

>> >> Reiser4 has done that, finally.  None of the other Linux filesystems
>> >> have.

>> > How come nobody before you in the almost 30 years of Unix has ever seen the
>> > need for this indispensable feature?

>>   I,  as  a  user, have for a long time. Do not drag everyone over the
>>   same  edge.  Users  have  different  needs.  We  should try to be as
>>   forthcoming as possible to provide the tools and options for them.

> Why? Breaking the basic paradigms for some user whim is not good OS design
> in my book, sorry. Many "bright ideas" have been thrown out over the years
> for just the same reasons.

>> >>        The next major release of ReiserFS is going to be bursting with
>> >> semantic enhancements, because the prerequisites for them are in place
>> >> now.  None of the other Linux filesystems have those prerequisites.

>> > To me that would mean that ReiserFS has no place in Linux.

>>   Why  do  you say this? I thought Linux embraced advancements and new
>>   technologies.

> If they are right. I think breaking the base of the OS is not a good
> idea. Perhaps ReiserFS has a place elsewhere (in MacOS, or Windows, where
> this is supposedly so so badly needed?), but not here.

  You  mean  in  Windows  and  MacOS  where much of this functionality
  already exists?

>> >> They won't be able to keep up with the semantic enhancements.  This
>> >> metafiles and file-directories stuff is actually fairly trivial stuff.

>> > Maybe. But the question of it being of any use aren't trivially answered
>> > "yes". My gut feeling is that the answer is a resounding "no", but that's
>> > just me. Direcories are directories, if you want to ship directory-like
>> > stuff around, use directories (or tarfiles, or whatever). Just look what
>> > happened to the much, much easier stuff of splitting SUID/SGID into
>> > capabilities: Nothing at all whatsoever, because they have no decent place
>> > to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be
>> > able to _define_ APIs to follow, but even so it isn't at all easy to do so.

>>   Exactly.  The  problem  I see in this discussion is the lack of just
>>   this - APIs. Because the lack of good APIs we cannot make any larger
>>   changes or bring new features because old applications will break.


  I'll chop this up...
  
> No. There is a complete lack of answers to why it should be done ("I think
> it would be nice", "MSFT is trying to do it", "Look at MacOS" is far from
> enough)

  No   perhaps  not  by itself. But it certainly should be considered.
  These  two  OS'es  have  most  of  the market share. They have great
  resources  trying to make things that work for their users.

  Users  are  more  important  than you think. Linux and any OS can do
  what  it  wants, but if it wants to keep users then it has to listen
  to them.

> , on what the alternatives are ("Sorry, it must be in the kernel
> because it is impossible to get application writers to agree on how to do
> it"

  This  is  a  perfectly  valid  reason.  If  it  is necessary to move
  something  into  the  core of Linux because it couldn't otherwise be
  done then it should seriously be considered.

>  will just add another incompatible way of "doing almost the same", this
> time in the kernel),

   People  on  this  list  are  actually trying to make it to be fully
   compatible with old legacy applications.

   If  you  hadn't noticed, Reiser4 does work. The meta files do work.
   The  discussion  is  wether parts of Reiser4 should be made generic
   and fit into the VFS.

>  and last but not least what the impact is (performance
> wise, application breakage, user/sysadmin expectation breakage,
> incompatibility with other systems, kernel development impact, to list a
> few). Hash this out, and we can talk.

  You  paint up a doomsday picture here. Nothing of this magnitude has
  happened  and  won't. besides, what is wrong if sysadmins would have
  to learn something new?

>>   I think that Linux should make new API that will completely separate
>>   applications  from the storage media. This way we will not as easily
>>   break  applications  when new features are developed. This should be
>>   the main priority.

> The "files + directories" stuff is a nice abstraction, and works today on
> all random-access storage media. You don't care much if it is on a CD, a
> floppy, a disk, a RAID array, or a flash card, do you? It won't work on
> tapes, or printers; but that is to be expected.

  you  didn't  actually  comment  on  what  I  said. I mentioned API's
  nothing about directories and files. API being the keyword here.

  If  there  are  good API's that applications can use, for example to
  access filesystems, then it will be easier and much more possible to
  introduce  new  things into the filesystem without breaking existing
  applications.  One  example of preventing application breakage could
  be   to   have  a  versioning system. Applications could query which
  version is supported, or vice versa.

  From  what  I  understand  it  is  much  harder to do anything today
  because of the fixed ways. This is why we have this discussion.

> [...]

>>   Application  breakage  has  happened  before between virtually every
>>   kernel change. It isn't new. Just take devfs and sysfs.

> Oops, now emacs stopped working because devfs was added? gcc needed a mayor
> overhaul because of /sys vs /proc? Gimme a break. That was _basic system tool_
> breakage, not all-around application breakage.

  Emacs  seem  to  work  fine  with  the  file-as-dir  on  my  Reiser4
  partition. And so does actually any other application I have used on
  it. Be it in Gnome, KDE or in plain shell's.

>> >> Look guys, in 1993 I anticipated the battle would be here, and I build
>> >> the foundation for a defensive tower right at the spot MS and Apple are
>> >> now maneuvering towards.
>> 
>> > Why place a protective tower around the pit into which they are trying
>> > desperately to throw themselves? ;-)
>> 
>>   It  is easy to say this. But the fact is that users can actually use
>>   these new features for good things.

> And can abuse them for bad things, and the features aren't free. Balance
> benefits (very minor) against costs (quite large).

  Abuse depends on permissions and security policies which is your job
  as admin to look after.

  It  would  be  possible to limit, for example), the types of plugins
  that a user can either use or load.

  Do you give your users access to modprobe or tools like that?

>> >>                           Help me get the next level on the tower before
>> >> they get here.  It is one hell of a foundation, they won't be able to
>> >> shake it, their trees are not as powerful.  Don't move reiser4 into vfs,
>> >> use reiser4 as the vfs.  Don't write filesystems, write file plugins and
>> >> disk format plugins and all the other kinds of plugins, and you won't be
>> >> missing any expressive power that you really want....
>> 
>> > Better write libraries handling whatever weird format you have in mind for
>> > each use. Even applications. I don't expect to be able to use emacs to edit
>> > a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit
>> > a poem. The current trend (which I heartily agree with) is to take stuff
>> > _out_ of the kernel (for flexibility, access, development ease, even
>> > performance).

>>   If  I  want  to  be able to quickly find files by a certain author I
>>   could   search  the  meta-data fields (if they are used) with normal
>>   tools such as grep.

> Iff the metadata is there. It won't (everybody would have to agree on
> placing that on the files for it to work), so it is useless.

  No  it  is  not  useless.  It  just  means  the  file  has  no  meta
  information. Just as it is today. No change.

>>   Compression or encryption is another example.
>> 
>>   Would  you  rather  patch  every  application  there is for Linux to
>>   enable support for reading these files and directories?

> No. But an encrypted or compressed filessytem doesn't need weird files.
> Easiest on applications is just to serve the decrypted/uncompressed files
> for use via the traditional APIs. Next easiest is provide programs that
> handle cryptography and compression and can be combined with whatever
> unaware program you care to use. Funny, that is exactly the way things work
> now...

  So you can setup a directory that has some files compressed and some
  other  encrypted and yet they will be available to use with tar, cp,
  ls, mc, Nautilus, Emacs etc etc?

Re: silent semantic changes with reiser4

[Jamie Lokier]

Horst von Brand wrote:
> Build an experimental OS showing how to use it, and through that see
> if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.

Hey, if you can think of a way to build an experimental OS and get it
used in real, day-to-day, down-to-earth, practice - i.e. used by a
diverse audience of thousands of vocal groups doing different things
with it, please let me know.

Until you do we have to lobby for some things to be tried in Linux
before we know if they really work.

-- Jamie

Re: silent semantic changes with reiser4

[Martin J. Bligh]

> Don't move reiser4 into vfs, use reiser4 as the vfs. 

Ah ... that seems to be a good insight into what you're driving for.
That would seem to be good for you ... and bad for other filesystems.
So whilst I can see why you'd want it - but I don't think it's a good idea.
Competition is good.

Andrew wrote:

> And what are the licensing implications of plugins?  Are they derived
> works?  Must they be GPL'ed?

You replied:

>>And what are the licensing implications of plugins?  Are they derived
>>works?  
>
> Yes.

I find it odd that you cut out the last part of his question. So must
they be GPL'ed or not?

M.

vfs2 (was Re: silent semantic changes with reiser4)

[Pavel Machek]

Hi!

> >Yes, but I didn't say "flame Christoph and ignore the issues" ;)
> > 
> >
> Oh....;-)
...
> enhancements.  This metafiles and file-directories stuff is actually 
> fairly trivial stuff.

Now I'm scared.

not as powerful.  Don't 
> move reiser4 into vfs, use reiser4 as the vfs.  Don't write 
> filesystems, write file plugins and disk format plugins and all the 
> other kinds of plugins, and you won't be missing any expressive power 
> that you really want....

So you want to replace vfs. Fine, but you should have told us before.

Also vfs replacement probably should not be hidden in fs/reiserfs4...


-- 
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         

file-as-dir vs. dir? (was Re: silent semantic changes with reiser4)

[Andy Lutomirski]

(Apologies for beating on a dead horse here...)

I seems like one confusion here is that everyone has a different idea of 
what the semantics we're talking about are.  I see two main ones:

Hans's: A file, a directory, and an attribute are functionally 
equivalent (except for S_ISxxx and hardlinks).  That is, /usr/bin/metas 
makes sense, and it's not talking about a program called metas.  This 
also means that /foo/metas/metas might exist and needs dealing with.

Linus's (I think):  A directory is just a directory (no attributes and 
no read()able data).  A file can contain attributes, where attributes 
can be "file" attributes or "directory" attributes.  That is, a file is 
also a subtree with posix-like semantics (except for hardlink stuff). 
So doing "touch /tmp/foo; cat /tmp/foo/metas" fails, rather than doing 
something that's probably useless.  "touch /tmp/foo; touch /tmp/foo/bar; 
touch /tmp/foo/bar/baz" fails on the last touch because bar is a file 
attribute and recursive magic is disallowed.

Which one are we talking about?

FWIW, I like the latter version a lot better, as it removes a lot of 
ambiguity.  If I see a path like /tmp/foo/metas/uid, it is either a uid 
attribute (i.e. writing it has security implications) or it is a 
standard file (i.e. writing it  just writes it).  But _I can tell which 
one_ by fstat()ing /tmp/foo!  If it's a directory, than I have either a 
named stream called uid or a genuine file called uid (and I can tell 
which by fstat()ing /tmp/foo/metas), but if it's a file then I have the 
magic uid.  And I'm guaranteed that there's no other funny business, 
because /tmp/foo is a "file with a subtree," which means that it's not 
an attribute.  This way I know exactly what I'm dealing with.

As an added bonus, we could have O_NOMETAS which means that "files" may 
not be traversed.  Then someone who wants to make sure they get a real 
file can do it.  If recursive files-as-dirs were allowed, that might not 
do quite what the caller expected.  If you really wanted a directory 
with some data attached, but it in dir/.mystuff, because that's probably 
what you meant, and any existing tool will do exactly the right thing 
with it.

In the former version, AFAIK, I have a mess.  In UNIX, a path is a path. 
  It's trivial to understand and there's no funny business parsing it. 
But now a path has a slippery meaning where the token "metas" could be a 
plain old token or it could be magic or it could be magic some other 
way.  The security implications and the possbility for something to 
break horribly are scary.

--Andy

RE: silent semantic changes with reiser4

[David Dabbs]

> 
> I'm probably not the first to suggest this idea, and it's probably not a
> very good idea, but here's my idea anyhow:
> 
> You have a file "/usr/bin/emacs"
> with a metadata property in the overlaid namespace
> "/usr/bin/emacs/[[..]metas/]icon"
> 
> According to some, this could cause some confusion.  Howabout instead:
> 
> You have a file "/usr/bin/emacs"
> with a metadata property in a slightly separated namespace
> "/metas/usr/bin/emacs/icon"
>

Timothy, see my similar proposal at
http://marc.theaimsgroup.com/?l=reiserfs&m=109511131923831&w=2 
which viro vetoed. I suspect this horse is quite dead.


David

Re: silent semantic changes with reiser4

[Gunnar Ritter]

Chris Siebenmann <cks@utcc.utoronto.ca> wrote:

>  I believe that this approach clashes violently with Hans Reiser's
> goals for reiser4, which I believe plans to have every regular file
> also have a stream component (for through-filesystem access to the
> stat data, if nothing else). Many of the other visions on exhibit in
> the thread seem to be almost as expansive, envisioning such streamed
> files used widely for various sorts of content annotation in places
> such as GNOME/KDE.

Although some individuals did tell about their 'visions' in this thread,
there were serious arguments against their use for anything else except
specialities like CIFS support. Many people do just not care about most
of the 'visions'.

>  I believe Linus Torvald's 'openat()' approach would maintain POSIX
> compatability, because it doesn't change the POSIX-visible namespace.

But it has the serious problem that handling a S_IFREG file in just the
usual manner will lead to silent data loss. Even with all POSIX issues
left aside, this seems hardly acceptable to me.

> [This is a separate issue to whether it would work better this way,
>  or whether it can be POSIX compliant without this. Hans Reiser at least
>  seems to not care very much about POSIX compliance for things like this,
>  so this sort of argument is unlikely to make him change his activities.]

As long as POSIX is only violated by some patch, people can ignore this
problem by just not applying it. Such will be possible even if Reiser4 is
integrated in the future; lack of POSIX conformance for FAT filesystems is
also not much of an issue.

The interesting point is that future VFS changes do not lead to POSIX
violations.

	Gunnar

Re: silent semantic changes with reiser4

[Gunnar Ritter]

Chris Siebenmann <cks@utcc.utoronto.ca> wrote:

>  Note that 'cp' is already not POSIX compliant on most Linux systems,
> thanks to GNU libc: 'cp foo -X' ought to work under the POSIX rules I've
> seen, but most Linux systems will have cp helpfully interpret the '-X'
> as a switch (and because it's a bad switch, explode). Evidently strict
> POSIX compatability is not very high on people's priority lists.

This issue is currently discussed on the Austin Group list, but in
contrast to your assumption, there are efforts to get both sides
closer together here, cf.
<http://www.opengroup.org/austin/mailarchives/ag/msg07261.html>.

>  Thus I would expect that the GNU fileutils people would be reasonably
> happy to make cp copy additional file streams and the like by default if
> they actually caught on in Linux.

It is clearly preferred to have such things as extensions that do not
violate the standard.

	Gunnar

Re: silent semantic changes with reiser4

[michael mattie]

In reading this thread I have not found one example *app* that cannot be 
implemented in XML. From
my understanding the big win of hybrid files is that they expose the 
structure of files to the Unix toolset.
Assume that this is possible and you can cd through the structure, you have 
acheived zero code sharing, only a never ending job of intergrating a 
infinite list of format interpreters. With XML you have a
concesus on syntax (structure) that is independant of the semantics and a 
significant and error
prone chunk of code can be shoved into a userspace library. I don't see 
anything like this in
hybrid files.

So far the vision I see in this thread for hybrid files is icons, mime 
types, character encoding, thumb nails, and meta data ? This is hetrogenous 
content ! I see no win in a VFS interface to a ad-hoc poor re-implementation 
of w3.org. I do admit it can be exciting to cd through the internal 
structure of a file, I do it all the time with Xpath and XSH [ see 
http://xsh.sourceforge.net/ ] from CPAN. The big win
of hybrid files can be had with a install. Transparent traversals of archive 
files should be shoved
into the shell with all of the other ugly hacks where it can be caged and 
easily killed with a signal.
Even windows programmer's reached this conclusion for some of their cruft.

My conclusion from reading this thread is that a proposal for hybrid files 
should demonstrate a significant win over the work of w3.org before such a 
radical semantic alteration is made. I hope
the comments of a humble and grateful user can be of some use and please 
accept my apology
if I have mis-understood this discussion or squandered your time.

_________________________________________________________________
Express yourself instantly with MSN Messenger! Download today - it's FREE! 
hthttp://messenger.msn.click-url.com/go/onm00200471ave/direct/01/

Re: silent semantic changes with reiser4

[SUBODH SHRIVASTAVA]

Hi,

I am using Reiser4. just want to report some thing, i
am not sure whether it is a normal behavior.

sshrivas@subbu-gentoo /opt/HelixPlayer/Bin $ ls -al
total 42
drwxr-sr-x   2 sshrivas 1000     3 Aug 28 18:21 .
drwxr-xr-x  11 root     root    15 Aug 28 18:21 ..
-rwxr-xr-x   1 sshrivas 1000 39174 Aug 28 18:21 setup

sshrivas@subbu-gentoo /opt/HelixPlayer/Bin/setup $ ls
-al
ls: reading directory .: Not a directory
total 0

Best Regards
Subodh Shrivastava

Re: silent semantic changes with reiser4

[Samium Gromoff]

Alexander Viro wrote:
> Arguments about O_NOFOLLOW on the intermediate stages are bullshit, IMNSHO -
> if they want to make some parts of tree inaccessible, they should simply
> mkdir /tmp/FOAD; chmod 0 /tmp/FOAD; mount --bind /tmp/FOAD <blocked path>
> in the namespace their daemon is running in.  And forget all that crap
> about filtering pathnames and blocking symlinks on intermediate stages
> the latter is obviously worthless without the former since one can simply
> substitute the symlink body in the pathname).

This made me wonder -- why not have a dedicated /dev/noaccess special node
for exactly such patterns of usage?

Yes that`ll made it Linux-specific and such... but hey -- if it saves
someone for for literally (?) no cost why not?

regards, Samium Gromoff