RE: reiser4 plugins

RE: reiser4 plugins

[Al Boldi]

Hans Reiser wrote:
> Steve, there is a remark about XFS below which you are going to be 
> more expert on.
> 
> Theodore Ts'o wrote:
> 
>>
>>XFS has similar issues where it assumes that hardware has powerfail 
>>interrupts, and that the OS can use said powerfail interrupt to stop 
>>DMA's in its tracks on an power failure, so that you don't have 
>>garbage written to key filesystem data structures when the memory 
>>starts suffering from the dropping voltage on the power bus faster 
>>than the DMA engine or the disk drives.  So XFS is a great filesystem
>>--- but you'd better be running it on a UPS, or on a system which has 
>>power fail interrupts and an OS that knows what to do.  Ext3, because 
>>it does physical block journalling, does not suffer from this problem.
>>(Yes, Resierfs uses logical journalling as well, so it suffers from 
>>the same problem.)
>>

True now, not so around 2.4.20 when XFS was rock-solid. I think they tried
to improve on performance and broke something. I wish they would fix that
because it forced me back to ext3, as in consistency over performance any
time.

Re: silent semantic changes with reiser4

[Andrew Morton]

Hans Reiser <reiser@namesys.com> wrote:
>
> I had not intended to respond to this because I have nothing positive to 
> say, but Andrew said I needed to respond and suggested I should copy 
> Linus.

Yes, but I didn't say "flame Christoph and ignore the issues" ;)

There are lots of little things to do with implementation, coding style,
module exports, deadlocks, what code goes where, etc.  These are all normal
daily kernel business and we should set them aside for now and concentrate
on the bigger issues.

And as I see it, there are two big issues:

a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not
   anticipate and 

b) it does this within the context of just a single filesystem.

I see three possible responses:

a) accept the reiser4-only extensions as-is (possibly with post-review
   modifications, of course) or

b) accept the reiser4-only extensions with a view to turning them into
   kernel-wide extensions at some time in the future, so all filesystems
   will offer the extensions (as much as poss) or

c) reject the extensions.


My own order of preference is b) c) a).  The fact that one filesystem will
offer features which other filesystems do not and cannot offer makes me
queasy for some reason.

b) means that at some time in the future we need to hoist the reiser4
extensions (at a conceptual level) (and probably with restrictions) up into
the VFS.  This will involve much thought, argument and work.


To get us started on this route it would really help me (and, probably,
others) if you could describe what these API extensions are in a very
simple way, without referring to incomprehehsible web pages, and without
using terms which non-reiser4 people don't understand.

It would be best if each extension was addressed in a separate email
thread.

We also need to discuss what a reiser4 "module" is, what its capabilities
are, and what licensing implications they have.

Then, we can look at each one and say "yup, that makes sense - we want
Linux to do that" and we can also think about how we would implement it at
the VFS level.

If we follow the above route I believe we can make progress in a technical
direction and not get deadlocked on personal/political stuff.


Now, an alternative to all the above is to just merge reiser4 as-is, after
addressing all the lower-level coding issues.  And see what happens.  That
may be a thing which Linus wishes to do.  I'm easy.

Re: silent semantic changes with reiser4

[Hans Reiser]

Andrew Morton wrote:

>Hans Reiser <reiser@namesys.com> wrote:
>  
>
>>I had not intended to respond to this because I have nothing positive to 
>>say, but Andrew said I needed to respond and suggested I should copy 
>>Linus.
>>    
>>
>
>Yes, but I didn't say "flame Christoph and ignore the issues" ;)
>  
>
Oh....;-)

>There are lots of little things to do with implementation, coding style,
>module exports, deadlocks, what code goes where, etc.  These are all normal
>daily kernel business and we should set them aside for now and concentrate
>on the bigger issues.
>  
>
Yes, you are right, but I am not sure Viro will go along with that..... ;-)

>And as I see it, there are two big issues:
>
>a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not
>   anticipate and 
>
>b) it does this within the context of just a single filesystem.
>
>I see three possible responses:
>
>a) accept the reiser4-only extensions as-is (possibly with post-review
>   modifications, of course) or
>
>b) accept the reiser4-only extensions with a view to turning them into
>   kernel-wide extensions at some time in the future, so all filesystems
>   will offer the extensions (as much as poss) or
>
>c) reject the extensions.
>
>
>My own order of preference is b) c) a). 
>
I don't object to b), though I think b) should wait for 2.7 and reiser4 
should not.

> The fact that one filesystem will
>offer features which other filesystems do not and cannot offer makes me
>queasy for some reason.
>  
>
Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem 
for Apple, and that means we need to put search engine and database 
functionality into the filesystem.  It takes 11 years of serious 
research to build a clean storage layer able to handle doing that.  
Reiser4 has done that, finally.  None of the other Linux filesystems 
have.  The next major release of ReiserFS is going to be bursting with 
semantic enhancements, because the prerequisites for them are in place 
now.  None of the other Linux filesystems have those prerequisites.  
They won't be able to keep up with the semantic enhancements.  This 
metafiles and file-directories stuff is actually fairly trivial stuff.

Look guys, in 1993 I anticipated the battle would be here, and I build 
the foundation for a defensive tower right at the spot MS and Apple are 
now maneuvering towards.  Help me get the next level on the tower before 
they get here.  It is one hell of a foundation, they won't be able to 
shake it, their trees are not as powerful.  Don't move reiser4 into vfs, 
use reiser4 as the vfs.  Don't write filesystems, write file plugins and 
disk format plugins and all the other kinds of plugins, and you won't be 
missing any expressive power that you really want....

Give Saveliev and I some credit.  10 years of hard work at an ivory 
tower nobody thought mattered.  Now the battle leaves the browser and 
swings our way.  Don't duplicate that infrastructure, use it. 

There is so much we could use help with if talented people like you 
chose to contribute.

>b) means that at some time in the future we need to hoist the reiser4
>extensions (at a conceptual level) (and probably with restrictions) up into
>the VFS.  This will involve much thought, argument and work.
>
>
>To get us started on this route it would really help me (and, probably,
>others) if you could describe what these API extensions are in a very
>simple way, without referring to incomprehehsible web pages,
>
what is not comprehensible....?

> and without
>using terms which non-reiser4 people don't understand.
>  
>
Well, I agree that there is value in defining things in more detail than 
we have.

>It would be best if each extension was addressed in a separate email
>thread.
>
>We also need to discuss what a reiser4 "module" is, what its capabilities
>are, and what licensing implications they have.
>
>Then, we can look at each one and say "yup, that makes sense - we want
>Linux to do that" and we can also think about how we would implement it at
>the VFS level.
>
>If we follow the above route I believe we can make progress in a technical
>direction and not get deadlocked on personal/political stuff.
>
>
>Now, an alternative to all the above is to just merge reiser4 as-is, after
>addressing all the lower-level coding issues.  And see what happens.  That
>may be a thing which Linus wishes to do.  I'm easy.
>
>
>  
>

Re: silent semantic changes with reiser4

[Andrew Morton]

Hans Reiser <reiser@namesys.com> wrote:
>
>  >To get us started on this route it would really help me (and, probably,
>  >others) if you could describe what these API extensions are in a very
>  >simple way, without referring to incomprehehsible web pages,
>  >
>  what is not comprehensible....?

Pretty much anything at www.namesys.com.  The amount of time which is
needed to extract the technical info which one is looking for vastly
exceeds a gnat-like attention span.

As a starting point, please prepare a bullet-point list of
userspace-visible changes which the filesystem introduces, or is planned to
introduce.

And describe the "plugin" system.  Why does the filesystem need such a
thing (other filesystems get their features via `patch -p1')?

And what are the licensing implications of plugins?  Are they derived
works?  Must they be GPL'ed?

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2289 bytes --]

Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton:

> And describe the "plugin" system.  Why does the filesystem need such a
> thing (other filesystems get their features via `patch -p1')?

Ok, let me try. (Hans, please don't shoot me if I don't get every detail
right, I'm trying to simplify and translate)

The reiser4 core is just a big and fast storage layer using a single
tree. It is bound to a mount point and uses the page cache to manage its
memory. The only thing it can do on its own is to flush dirty data to
the disk when the VM wants to (memory pressure) or the VFS wants to
(unmount, sync) or some "plugin" wants to.
Additionally it's completely atomic (with respect to crashes, no
isolation like in databases), comparable to data=journal but without the
overhead of using a journal for everything.

Up to this point there are no users of this "database" and it does not
implement any of the VFS methods for a filesystem (except mount and
unmount perhaps).

All the functionality is provided to the plugins. You can insert,
remove, lookup or modify key:object pairs (with an index into the
object). The object is a sequence of units. For files a unit would be 1
byte and the index would be the byte offset. For directories the unit
would be 1 entry and the index would be the filename.

Now there are some plugins that define how the storage layout on the
disk is (some kind of "backend" plugins).

*And* there are plugins which are users of the "reiser4 client API" and
implement the actual VFS methods.

There's a UNIX directory plugin and a UNIX file plugin.

Directories, inodes and file content are just key:object pairs and the
plugins know how to operate on these.

There's a new plugin in work that also implements UNIX file semantics
but stores the data for that file encrypted and/or compressed.

These plugins live between the VFS and the storage layer.

Just like the filesystems live between the VFS and the block layer (Hans
would say that filesystems are VFS plugins ;-)).

> And what are the licensing implications of plugins?  Are they derived
> works?  Must they be GPL'ed?

I suppose yes, at least currently, since they can only be linked with
reiser4, there's no module infrastructure.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 02:18:49PM +0200, Christophe Saout wrote:
> The reiser4 core is just a big and fast storage layer using a single
> tree. It is bound to a mount point and uses the page cache to manage its
> memory. The only thing it can do on its own is to flush dirty data to
> the disk when the VM wants to (memory pressure) or the VFS wants to
> (unmount, sync) or some "plugin" wants to.
> Additionally it's completely atomic (with respect to crashes, no
> isolation like in databases), comparable to data=journal but without the
> overhead of using a journal for everything.
> 
> Up to this point there are no users of this "database" and it does not
> implement any of the VFS methods for a filesystem (except mount and
> unmount perhaps).

Now that part is interesting from the how to sell it to non-Linux users
POV, but not for the linux kernel.

> All the functionality is provided to the plugins. You can insert,
> remove, lookup or modify key:object pairs (with an index into the
> object). The object is a sequence of units. For files a unit would be 1
> byte and the index would be the byte offset. For directories the unit
> would be 1 entry and the index would be the filename.
> 
> Now there are some plugins that define how the storage layout on the
> disk is (some kind of "backend" plugins).
> 
> *And* there are plugins which are users of the "reiser4 client API" and
> implement the actual VFS methods.

Here comes the problem.  All the access checking/race avoidance/loop
creation avoiced, in short the posix+extension semantics are implemented
in the Linux VFS layer.  If you want to allow a second access method
(e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
inside reiser4.  But this is absolutely not what we want, because we
moved to this common as all previous version in the different
filesystems were buggy, and in fact there are problems that are simply
not solveable at the level at all (rename on directories comes to mind).

So if you want to provide an additional API you'll have to go through
the VFS to get it right - ergo a plugin architecture for upper plugins
is worthless.

Plugins for the disk-format level are an interesting and workable idea
on the other hand.

Re: silent semantic changes with reiser4

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1732 bytes --]

Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig:

> Now that part is interesting from the how to sell it to non-Linux users
> POV, but not for the linux kernel.

Why? The question was what these plugins are exactly. This is the
answer.

> > *And* there are plugins which are users of the "reiser4 client API" and
> > implement the actual VFS methods.
> 
> Here comes the problem.  All the access checking/race avoidance/loop
> creation avoiced, in short the posix+extension semantics are implemented
> in the Linux VFS layer.  If you want to allow a second access method
> (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
> inside reiser4.

Are you actually listening? If you implement a filesystem there's a
place where you have to implement the Linux VFS methods. I'm talking
about inode_operations and these things. This has nothing to do with
doing anything outside the Linux VFS. And I'm not talking about these
metas either. These really don't belong inside the filesystem.

> So if you want to provide an additional API you'll have to go through
> the VFS to get it right - ergo a plugin architecture for upper plugins
> is worthless.

See, you didn't listen. ;-)

The reiser4 core doesn't know about inodes, directories or files. It's
the glue code between the VFS and the storage layer that does. It's
implemented as a plugin. This has nothing to do with semantic
enhancements yet. These should be removed for now and made a 2.7 topic.

Maybe Namesys could provide them as an additional patch from their
webpage for those who need it. The plugins that would benefit from this
(encrypted and compressed file storage) aren't ready yet anyway.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: silent semantic changes with reiser4

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 03:00:34PM +0200, Christophe Saout wrote:
> Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig:
> 
> > Now that part is interesting from the how to sell it to non-Linux users
> > POV, but not for the linux kernel.
> 
> Why? The question was what these plugins are exactly. This is the
> answer.

I explained it below..

> > > *And* there are plugins which are users of the "reiser4 client API" and
> > > implement the actual VFS methods.
> > 
> > Here comes the problem.  All the access checking/race avoidance/loop
> > creation avoiced, in short the posix+extension semantics are implemented
> > in the Linux VFS layer.  If you want to allow a second access method
> > (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity
> > inside reiser4.
> 
> Are you actually listening? If you implement a filesystem there's a
> place where you have to implement the Linux VFS methods. I'm talking
> about inode_operations and these things. This has nothing to do with
> doing anything outside the Linux VFS. And I'm not talking about these
> metas either. These really don't belong inside the filesystem.

as I wrote in this mail this absolutely _does_ belong in the filesystem,
it's a major part of it and isn't easily separatable.

> > So if you want to provide an additional API you'll have to go through
> > the VFS to get it right - ergo a plugin architecture for upper plugins
> > is worthless.
> 
> See, you didn't listen. ;-)
> 
> The reiser4 core doesn't know about inodes, directories or files. It's
> the glue code between the VFS and the storage layer that does. It's
> implemented as a plugin. This has nothing to do with semantic
> enhancements yet. These should be removed for now and made a 2.7 topic.

Oh yes, it is.  As soon as you use different access methods on an
overlapping set of objects you see exactly the problems I described.

If they don't overlap there's no point for the plugins to start with,
you'll better turn the core into a library that can be used by different
projects then.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1907 bytes --]

Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig:

> > Why? The question was what these plugins are exactly. This is the
> > answer.
> 
> I explained it below..

It's hard to talk to you if you're not even trying to understand what
I'm trying to say. Andrew wanted to know what these plugins are. I tried
to give an answer and then you come along to tell me that it's not
relevant. Sure, Linux doesn't care about what the filesystem does
internally but Andrew wanted to know that.

> > Are you actually listening? If you implement a filesystem there's a
> > place where you have to implement the Linux VFS methods. I'm talking
> > about inode_operations and these things. This has nothing to do with
> > doing anything outside the Linux VFS. And I'm not talking about these
> > metas either. These really don't belong inside the filesystem.
> 
> as I wrote in this mail this absolutely _does_ belong in the filesystem,
> it's a major part of it and isn't easily separatable.

Huh?

Now you're completely confusing me.

First you say that that file-as-a-directory is crap then you say that it
does belong into the filesystem?

I'll try again:

inode_operations, etc... are implemented as a "reiser4 plugin". This
means that reiser4 is usable as filesystem, you can store files in a
directory hierarchy. There's absolute nothing special here. Absolutely
nothing. And yes, reiser4 plugins are invisible from the VFS. It looks
like a normal filesystem because it behaves like one.

The "metas" I was referring in the second part are this file-as-a-
directory thing. These don't belong in the filesystem and should
therefore be removed.

The latter doesn't necessarily have anything to do with plugins.
The plugins are plugins for the storage layer, not for some semantic
enhancement.

Note that I changed the topic to avoid further confusion.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 2774 bytes --]

On Thu, Aug 26, 2004 at 03:17:53PM +0200, Christophe Saout wrote:
> It's hard to talk to you if you're not even trying to understand what
> I'm trying to say. Andrew wanted to know what these plugins are. I tried
> to give an answer and then you come along to tell me that it's not
> relevant. Sure, Linux doesn't care about what the filesystem does
> internally but Andrew wanted to know that.

Huh?  I don understand what reiser4 plugins are, and I tell you that a
subset of these are _wrong_ for use in the kernel. 

> > > Are you actually listening? If you implement a filesystem there's a
> > > place where you have to implement the Linux VFS methods. I'm talking
> > > about inode_operations and these things. This has nothing to do with
> > > doing anything outside the Linux VFS. And I'm not talking about these
> > > metas either. These really don't belong inside the filesystem.
> > 
> > as I wrote in this mail this absolutely _does_ belong in the filesystem,
> > it's a major part of it and isn't easily separatable.
> 
> Huh?
> 
> Now you're completely confusing me.
> 
> First you say that that file-as-a-directory is crap then you say that it
> does belong into the filesystem?

I think you're talking about something different then me, I'm not
talking about the magic meta files but the VFS interface in general.

This VFS interface is an integral part of ævery filesystem, and it
doesn't make a whole lot to put it into a plugin.  If you want your
filesystem core portable it does to a certain extent make sense to
abstract them out, but as someone who's worked on a few such 'portable'
filesystems I can tell you that it doesn't work out as expected.

> I'll try again:
> 
> inode_operations, etc... are implemented as a "reiser4 plugin". This
> means that reiser4 is usable as filesystem, you can store files in a
> directory hierarchy. There's absolute nothing special here. Absolutely
> nothing. And yes, reiser4 plugins are invisible from the VFS. It looks
> like a normal filesystem because it behaves like one.

Now kill the whole plugin mess and let them talk directly to another.
My first mail explained to you why it doesn't make sense to have
multiple such plugins to work on a common subset of data.

> The latter doesn't necessarily have anything to do with plugins.
> The plugins are plugins for the storage layer, not for some semantic
> enhancement.

Again plugins below the pagecache (if that's what you call the "storage
layer") _do_ make sense.  plugins at the semantic layer don't.

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1905 bytes --]

Am Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig:

> > First you say that that file-as-a-directory is crap then you say that it
> > does belong into the filesystem?
> 
> I think you're talking about something different then me, I'm not
> talking about the magic meta files but the VFS interface in general.
> 
> This VFS interface is an integral part of very filesystem, and it
> doesn't make a whole lot to put it into a plugin.

Right. That's why these plugins are linked in uncoditionally. It doesn't
work without them. Hence "plugins" is not a very good name.

> If you want your
> filesystem core portable it does to a certain extent make sense to
> abstract them out, but as someone who's worked on a few such 'portable'
> filesystems I can tell you that it doesn't work out as expected.

That's your opinion. reiser4 seems to work very well.

> Now kill the whole plugin mess and let them talk directly to another.

This is how reiserfs v3 did it. And Hans says that it actually
complicated things.

> My first mail explained to you why it doesn't make sense to have
> multiple such plugins to work on a common subset of data.

What I understood is that you can select exactly one plugin that e.g.
handles the file data. The default plugin is optimized for normal files,
another one could implement transparent compression or encryption. Some
of these plugins also give the storage layer hints how to group files
together to optimized performance. Neither of these things mess with the
VFS.

> > The latter doesn't necessarily have anything to do with plugins.
> > The plugins are plugins for the storage layer, not for some semantic
> > enhancement.
> 
> Again plugins below the pagecache (if that's what you call the "storage
> layer") _do_ make sense.  plugins at the semantic layer don't.

Yes, that's what I was trying to say.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Thu, Aug 26, 2004 at 03:35:07PM +0200, Christophe Saout wrote:
> > If you want your
> > filesystem core portable it does to a certain extent make sense to
> > abstract them out, but as someone who's worked on a few such 'portable'
> > filesystems I can tell you that it doesn't work out as expected.
> 
> That's your opinion. reiser4 seems to work very well.

So how many OSes is it ported to currently?  The problem is not that it
doesn't work but that it's really hard to maintain.  And remember that
this maintaince overhead is not just for Hans but for everyone that does
VFS-level work.  Unless of course we get a blanko permission to break it
as soon as fixing the mess becomes non-trivial.

> What I understood is that you can select exactly one plugin that e.g.
> handles the file data. The default plugin is optimized for normal files,
> another one could implement transparent compression or encryption. Some
> of these plugins also give the storage layer hints how to group files
> together to optimized performance. Neither of these things mess with the
> VFS.

compression or encryption must sit below the pagecache to work nicely,
and this hint things that usually sit at the pagecache level.  But let's
assume you have a valid use for different file_operations, why don't you
simply add in different file_operations instead of adding another
internal dispatch layer?  

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 2493 bytes --]

Am Donnerstag, den 26.08.2004, 15:40 +0200 schrieb Christoph Hellwig:

> > That's your opinion. reiser4 seems to work very well.
> 
> So how many OSes is it ported to currently?  The problem is not that it
> doesn't work but that it's really hard to maintain.  And remember that
> this maintaince overhead is not just for Hans but for everyone that does
> VFS-level work.

Where does it add overhead? It's just the interface between the reiser4
storage layer and the rest abstracted out.

Ok, there's the code that dispatches the VFS method calls to the plugin
call but I would say that's about it. I didn't develop reiser4 so I
don't know.

/* ->mkdir() VFS method in reiser4 inode_operations */
static int
reiser4_mkdir(struct inode *parent      /* inode of parent
                                         * directory */ ,
              struct dentry *dentry     /* dentry of new object to
                                         * create */ ,
              int mode /* new object's mode */ )
{
        reiser4_object_create_data data;

        reiser4_stat_inc_at(parent->i_sb, vfs_calls.mkdir);

        data.mode = S_IFDIR | mode;
        data.id = DIRECTORY_FILE_PLUGIN_ID;
        return invoke_create_method(parent, dentry, &data);
}

The invoke_create_method then just calls the create method of the plugin
after setting up a context and lookup up the plugin.

xfs_iops.c seems to do something similar.

>   Unless of course we get a blanko permission to break it
> as soon as fixing the mess becomes non-trivial.

Well, I think so. It's done relatively often.

> > What I understood is that you can select exactly one plugin that e.g.
> > handles the file data. The default plugin is optimized for normal files,
> > another one could implement transparent compression or encryption. Some
> > of these plugins also give the storage layer hints how to group files
> > together to optimized performance. Neither of these things mess with the
> > VFS.
> 
> compression or encryption must sit below the pagecache to work nicely,
> and this hint things that usually sit at the pagecache level.  But let's
> assume you have a valid use for different file_operations, why don't you
> simply add in different file_operations instead of adding another
> internal dispatch layer?  

I don't know, ask Hans. How could the VFS know it a filesystem wants to
do something specific with a file that is completely transparent to the
VFS?


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: reiser4 plugins

[Hans Reiser]

Christophe Saout wrote:

>
>
>I don't know, ask Hans. How could the VFS know it a filesystem wants to
>do something specific with a file that is completely transparent to the
>VFS?
>
>  
>
To know what method to use, you must determine the pluginid, and then 
find the method within that plugin for that vfs operation.

As for overhead, well, who eats whose dust in the benchmarks....?

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:
 > Christophe Saout wrote:
 > 
 > >
 > >
 > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
 > >do something specific with a file that is completely transparent to the
 > >VFS?
 > >
 > >  
 > >
 > To know what method to use, you must determine the pluginid, and then 
 > find the method within that plugin for that vfs operation.
 > 
 > As for overhead, well, who eats whose dust in the benchmarks....?

Whoever sponsors the benchmark usually wins. Had you forgotten that
mongo setup used by http://www.namesys.com/benchmarks.html was specially
`tuned' to reach peak reiser4 performance? Remember why you decided to
turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
_second_ stalls with reiser4 under high io loads (large atom is
obviously being flushed and everyone waits on it...). In my opinion, it
is such things that are of utmost importance for real reiser4
acceptance, not how to name `metas' sub-directory.

Nikita.

Re: reiser4 plugins

[Hans Reiser]

Nikita Danilov wrote:

>Hans Reiser writes:
> > Christophe Saout wrote:
> > 
> > >
> > >
> > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
> > >do something specific with a file that is completely transparent to the
> > >VFS?
> > >
> > >  
> > >
> > To know what method to use, you must determine the pluginid, and then 
> > find the method within that plugin for that vfs operation.
> > 
> > As for overhead, well, who eats whose dust in the benchmarks....?
>
>Whoever sponsors the benchmark usually wins. Had you forgotten that
>mongo setup used by http://www.namesys.com/benchmarks.html was specially
>`tuned' to reach peak reiser4 performance? Remember why you decided to
>turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
>_second_ stalls with reiser4 under high io loads (large atom is
>obviously being flushed and everyone waits on it...). In my opinion, it
>is such things that are of utmost importance for real reiser4
>acceptance, not how to name `metas' sub-directory.
>
>Nikita.
>
>
>  
>
If you ask real users, they say that reiser4 is fast, and their 
experience matches our benchmark.  You can criticize the benchmark if 
you want, but then you should run your own and publish it.

There are still plenty of things needing fixing though, and you are 
right that they are more important than the pretext being attempted for 
keeping us out of the kernel.  Probably you could supply them with many 
superior pretexts.;-)

The 90 second stalls, those should be fixed by debugging copy on capture 
and turning it on, yes?  You can hardly claim I failed to push for the 
completion of copy on capture....  and of course the reason it is not 
done is simply that there is too much to do....  I (sincerely) hope vs 
enjoys his vacation, the whole team is working too hard.  I am doing 7 
day weeks now to try to make payroll for us and keep darpa happy at the 
same time.

fsync performance needs fixing badly, and Chris is right in his 
diagnosis of what we can do to fix it.

Hans

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:
 > Nikita Danilov wrote:
 > 
 > >Hans Reiser writes:
 > > > Christophe Saout wrote:
 > > > 
 > > > >
 > > > >
 > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
 > > > >do something specific with a file that is completely transparent to the
 > > > >VFS?
 > > > >
 > > > >  
 > > > >
 > > > To know what method to use, you must determine the pluginid, and then 
 > > > find the method within that plugin for that vfs operation.
 > > > 
 > > > As for overhead, well, who eats whose dust in the benchmarks....?
 > >
 > >Whoever sponsors the benchmark usually wins. Had you forgotten that
 > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
 > >`tuned' to reach peak reiser4 performance? Remember why you decided to
 > >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
 > >_second_ stalls with reiser4 under high io loads (large atom is
 > >obviously being flushed and everyone waits on it...). In my opinion, it
 > >is such things that are of utmost importance for real reiser4
 > >acceptance, not how to name `metas' sub-directory.
 > >
 > >Nikita.
 > >
 > >
 > >  
 > >
 > If you ask real users, they say that reiser4 is fast, and their 
 > experience matches our benchmark.  You can criticize the benchmark if 

They experience 90 second stalls. And please, do not tell me how fast
reiser4 is, I spent a lot of time working with it, and I know very well
when it's fast, and when it's deadly slow.

 > you want, but then you should run your own and publish it.

I did, after which you told me to turn OVERWRITE and MODIFY phases off,
beause performance was horrible.

I not criticizing mongo benchmark per se. I think that it is
fundamentally wrong to use results that were deliberatly manipulated to
get best appearance to reiser4 (by omitting work-loads where it performs
poorly) as an argument. It's not clear who will, according to your
colorful expression, `eat dust' as a result of this. Or do you think
that users never overwrite of modify files in real life?

 > The 90 second stalls, those should be fixed by debugging copy on capture 
 > and turning it on, yes?  You can hardly claim I failed to push for the 

No. I described so many times to you, why COC is ineffectual here.

 > Hans

Nikita.

Re: reiser4 plugins

[Hans Reiser]

Nikita Danilov wrote:

> > >Whoever sponsors the benchmark usually wins. Had you forgotten that
> > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
> > >`tuned' to reach peak reiser4 performance? Remember why you decided to
> > >turn OVERWRITE and MODIFY phases off?
>
What I should have done was what I did with fsync performance.  With 
fsync performance I told people that we had not yet tuned for it, please 
wait a bit and we will tune for it, for now it sucks.

Instead what I did was discuss with Zam at the time how it could be 
fixed, leave it off the website until Zam was given a chance to fix it, 
and then I managed to forget about it.   After release one remembers 
what all the things that should have been fixed before release were, sigh.

Zam and I both know what is needed to fix performance in these phases, 
and I just spoke with him on the phone.  I imagine it is 2 weeks of work 
for him, but it could be up to 6 weeks.  He will comment later.

The fix should consist of the following:

1) tweak the relocation policies (zam will say more about this, as he is 
the maintainer of them)

2) optimize overwrite set so that in the modify phase we behave like a 
write twice journaling filesystem, which means implement a reserved for 
the journal only area that exists as long as plenty of disk space is free.

3) finish the repacker (but we won't do that this month....)

The modify phase, which picks a random block to modify, is a worst case 
for a wandering log without a repacker.  We could actually do very very 
well at that kind of activity with a repacker, probably better than a 
fixed journal, but for now we should try acting like a write twice 
journal for atoms that small.

So, having realized my error thanks to the gracious kindness of how you 
pointed it out, we will modify the web site to say that we are not yet 
tuned for and currently suck at modifying random blocks in the middle of 
files, and appending small amounts to the end of them, and overwriting 
small files, but that we think we know what is needed to fix those things.

I think your characterization of my reasons was unkind and also unfair.  
I will pass on saying similarly unkind things about you as you are 
mostly a good person.  You never did seem to understand me.  You didn't 
understand the reasons for my technical designs (when they worked it 
made no sense to you that they did), and you didn't understand me as a 
person. 

I wish you well in your new job, and thank you for the hard work you did 
for me.

Hans

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:
 > Nikita Danilov wrote:
 > 
 > > > >Whoever sponsors the benchmark usually wins. Had you forgotten that
 > > > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
 > > > >`tuned' to reach peak reiser4 performance? Remember why you decided to
 > > > >turn OVERWRITE and MODIFY phases off?
 > >
 > What I should have done was what I did with fsync performance.  With 
 > fsync performance I told people that we had not yet tuned for it, please 
 > wait a bit and we will tune for it, for now it sucks.

Right, and instead of this you are now claiming that these `benchmarks'
show superior reiser4 performance. Moreover, you are doing this
aggressively and proposing other people to `eat the dust' over what
happened to only exist due to your failing to remember something. As
some other notable LKML poster put it `inform yourself before
posting'. :-)

 > 
 > Instead what I did was discuss with Zam at the time how it could be 
 > fixed, leave it off the website until Zam was given a chance to fix it, 

It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were
turned off after switching to large keys.

 > and then I managed to forget about it.   After release one remembers 
 > what all the things that should have been fixed before release were, sigh.
 > 

[...]

 > 
 > I think your characterization of my reasons was unkind and also unfair.  

What fairness and kindness one expects after styling others as `puppies'
in public, and commenting on their hair style in purportedly technical
argument?

[...]

 > 
 > Hans

Nikita.

Re: reiser4 plugins

[Hans Reiser]

Nikita Danilov wrote:

> > 
> > Instead what I did was discuss with Zam at the time how it could be 
> > fixed, leave it off the website until Zam was given a chance to fix it, 
>
>It wasn't Zam. It was me to begin with. OVERWRITE and MODIFY phases were
>turned off after switching to large keys.
>
> 
>
The online repacker is the best technical fix for these problems.  Tight 
packing is generally more damaged in its layout by small changes than 
loose packing, and V4 is the ultimate tight packer.  It is not clear to 
me that making V4 pack more loosely is necessarily a good idea.  These 
phases disturb the original packing at flush time, and the other phases 
don't.  One solution to try is packing less tightly.  Maybe once every 
megabyte skip forward 1 megabyte, and once every 64 megabytes skip to a 
random disk location, when allocating large new atoms.  I think I prefer 
using a repacker, but we should try it and see.  The other solutions of 
the previous email should also help a lot.  The issues of layout in this 
are similar to the ones improved by the fibration plugin.

I remember talking with not just you but zam about how we could fix it, 
and there was too much in the queue of work, and everyone was 
complaining to me that we should be debugging not optimizing, and you 
were the only one who thought it was a big deal.  I guess you still 
think it is a big issue and I still think things are good enough to use 
without it.  I still think Zam understood the issues better than you did 
(allocation is his code not yours).

There are some layout optimizations that a repacker can do best.  Still, 
there are some rough spots in the current allocation policy, and we 
should look at it.

Probably you will have reason to howl if we setup a benchmark which 
disturbs things with these phases, runs the repacker (I hope to have one 
in 6 months), and then measures our read performance compared to other 
filesystems without a repacker....;-)

Hans

Re: reiser4 plugins

[Nikita Danilov]

Hans Reiser writes:

[...]

 > 
 > I remember talking with not just you but zam about how we could fix it, 
 > and there was too much in the queue of work, and everyone was 
 > complaining to me that we should be debugging not optimizing, and you 
 > were the only one who thought it was a big deal.  I guess you still 

You are trying to evade the point. And the point is not how performance
in these phases (and their impact on other phases) can be improved, or
what excuses did you have for not `doing' it, but the fact that after it
was found that large keys behave badly, these phases were turned off,
others were switched to lexicographical operation, _and_ resulting
benchmark is used as a basis to call other people names.

 > think it is a big issue and I still think things are good enough to use 
 > without it.  I still think Zam understood the issues better than you did 
 > (allocation is his code not yours).
 > 
 > There are some layout optimizations that a repacker can do best.  Still, 
 > there are some rough spots in the current allocation policy, and we 
 > should look at it.
 > 
 > Probably you will have reason to howl if we setup a benchmark which 

Surely, as a `big dog' what other sounds can I produce? :)

 > disturbs things with these phases, runs the repacker (I hope to have one 
 > in 6 months), and then measures our read performance compared to other 
 > filesystems without a repacker....;-)

That's ok with me, but it would make sense to remove unfair benchmarks
from the site until then.

 > 
 > Hans

Nikita.

Re: reiser4 plugins

[Alex Zarochentsev]

On Fri, Aug 27, 2004 at 10:55:50PM +0400, Nikita Danilov wrote:
> Hans Reiser writes:
>  > Nikita Danilov wrote:
>  > 
>  > >Hans Reiser writes:
>  > > > Christophe Saout wrote:
>  > > > 
>  > > > >
>  > > > >
>  > > > >I don't know, ask Hans. How could the VFS know it a filesystem wants to
>  > > > >do something specific with a file that is completely transparent to the
>  > > > >VFS?
>  > > > >
>  > > > >  
>  > > > >
>  > > > To know what method to use, you must determine the pluginid, and then 
>  > > > find the method within that plugin for that vfs operation.
>  > > > 
>  > > > As for overhead, well, who eats whose dust in the benchmarks....?
>  > >
>  > >Whoever sponsors the benchmark usually wins. Had you forgotten that
>  > >mongo setup used by http://www.namesys.com/benchmarks.html was specially
>  > >`tuned' to reach peak reiser4 performance? Remember why you decided to
>  > >turn OVERWRITE and MODIFY phases off? People on #reiser4 report 90
>  > >_second_ stalls with reiser4 under high io loads (large atom is
>  > >obviously being flushed and everyone waits on it...). In my opinion, it
>  > >is such things that are of utmost importance for real reiser4
>  > >acceptance, not how to name `metas' sub-directory.
>  > >
>  > >Nikita.
>  > >
>  > >
>  > >  
>  > >
>  > If you ask real users, they say that reiser4 is fast, and their 
>  > experience matches our benchmark.  You can criticize the benchmark if 
> 
> They experience 90 second stalls. And please, do not tell me how fast
> reiser4 is, I spent a lot of time working with it, and I know very well
> when it's fast, and when it's deadly slow.
> 
>  > you want, but then you should run your own and publish it.
> 
> I did, after which you told me to turn OVERWRITE and MODIFY phases off,
> beause performance was horrible.

Hmm, I think not the modify/overwrite phases performance were the problem.  The
modify and overwrite mongo phases fragment the filesystem too much, it had a
bad effect visible in the read phase and even in delete phase.  

We measured/optimized read performance  in assumption that fs is not
fragmented, why we should measure bad effect of overwrite instead?  

other fs do not relocate already allocated blocks, negative effect of 
file set overwrite is zero for them.  We could do the same for reiser4
by tuning flush algorithm and journal blocks allocation.

If users want to see read and delete performance as they are, overwrite phase
should be excluded, at least until we teach flush algorithm to not break good
block allocation.

If one wants to measure overwite performance, it is easy to enable overwrite
phase in mongo, but read and delete speed will be affected (it was so when I
tried to optimize delete).

> I not criticizing mongo benchmark per se. I think that it is
> fundamentally wrong to use results that were deliberatly manipulated to
> get best appearance to reiser4 (by omitting work-loads where it performs
> poorly) as an argument. It's not clear who will, according to your
> colorful expression, `eat dust' as a result of this. 
> Or do you think
> that users never overwrite of modify files in real life?

not whole file set.  I mean it is unusual to overwrite all files in the fs.

>  > The 90 second stalls, those should be fixed by debugging copy on capture 
>  > and turning it on, yes?  You can hardly claim I failed to push for the 
> 
> No. I described so many times to you, why COC is ineffectual here.

There are other ideas :) but, it seems to me, if we will focus on benchmarks,
we never complete plugins, metafiles and other useful things.  What I want
from reiser4 is a possiblity to encrypt and compress ~/Mail/ :).

> 
>  > Hans
> 
> Nikita.

-- 
Alex.

Re: reiser4 plugins

[Steve Bergman]

On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote:
> >
> If you ask real users, they say that reiser4 is fast, and their 
> experience matches our benchmark.  You can criticize the benchmark if 
> you want, but then you should run your own and publish it.
> 


As a "real" desktop user who just converted all his partitions from ext3
to reiser4, I have not, as yet, noticed any startling performance
increase.  Being slightly slightly irked to hear that the benchmark
numbers that have been paraded around on Slashdot and the internet in
general, at ext3's expense, have had reiser4's "bad" results surgically
extracted, I am running my own benchmarks to get the real story on
reiser4/ext3 mongo performance on my, rather average, desktop hardware.

I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel.

Unfortunately, I get an error from mongo.pl that "Done" is not a numeric
argument at line 439.

I have done this to fix it:


--- mongo.pl    2004-08-27 17:07:01.681723313 -0500
+++ mongo_fixed.pl      2004-08-27 17:06:51.369306735 -0500
@@ -429,8 +429,8 @@
        if ( -e ${ERR_FILE}) {
            &DIE ("\nEXITED WITH FAIL\n");
        }
-       my $real = (split ' ', $time_output[0])[1];
-       my $cpu  = (split ' ', $time_output[2])[1];
+       my $real = (split ' ', $time_output[1])[1];
+       my $cpu  = (split ' ', $time_output[3])[1];
  
        unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) {
          LOG "@time_output";


What it gets me is the "real" line of the "time" output for "STAT
REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME".
i.e. only system time is counted. I believe this was the intent of the
original code, but want to verify before continuing.

Thanks,
Steve Bergman

Re: reiser4 plugins

[Hans Reiser]

Steve Bergman wrote:

>On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote:
>  
>
>>If you ask real users, they say that reiser4 is fast, and their 
>>experience matches our benchmark.  You can criticize the benchmark if 
>>you want, but then you should run your own and publish it.
>>
>>    
>>
>
>
>As a "real" desktop user who just converted all his partitions from ext3
>to reiser4, I have not, as yet, noticed any startling performance
>increase.  Being slightly slightly irked to hear that the benchmark
>numbers that have been paraded around on Slashdot and the internet in
>general, at ext3's expense, have had reiser4's "bad" results surgically
>extracted, I am running my own benchmarks to get the real story on
>reiser4/ext3 mongo performance on my, rather average, desktop hardware.
>
>I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel.
>
>Unfortunately, I get an error from mongo.pl that "Done" is not a numeric
>argument at line 439.
>
>I have done this to fix it:
>
>
>--- mongo.pl    2004-08-27 17:07:01.681723313 -0500
>+++ mongo_fixed.pl      2004-08-27 17:06:51.369306735 -0500
>@@ -429,8 +429,8 @@
>        if ( -e ${ERR_FILE}) {
>            &DIE ("\nEXITED WITH FAIL\n");
>        }
>-       my $real = (split ' ', $time_output[0])[1];
>-       my $cpu  = (split ' ', $time_output[2])[1];
>+       my $real = (split ' ', $time_output[1])[1];
>+       my $cpu  = (split ' ', $time_output[3])[1];
>  
>        unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) {
>          LOG "@time_output";
>
>
>What it gets me is the "real" line of the "time" output for "STAT
>REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME".
>i.e. only system time is counted. I believe this was the intent of the
>original code, but want to verify before continuing.
>
>Thanks,
>Steve Bergman
>
>
>
>  
>
I didn't write this (more precisely, it only vaguely resembles what I 
wrote in 1996).  Are you saying that it reports system time as real 
time?  If yes, then it is an error, we need to go remove a bunch of 
numbers from our benchmarks, and thanks for finding it.

Zam, please comment.

Hans

Re: reiser4 plugins

[Alex Zarochentsev]

On Fri, Aug 27, 2004 at 05:29:07PM -0500, Steve Bergman wrote:
> On Fri, 2004-08-27 at 11:15 -0700, Hans Reiser wrote:
> > >
> > If you ask real users, they say that reiser4 is fast, and their 
> > experience matches our benchmark.  You can criticize the benchmark if 
> > you want, but then you should run your own and publish it.
> > 
> 
> 
> As a "real" desktop user who just converted all his partitions from ext3
> to reiser4, I have not, as yet, noticed any startling performance
> increase.  Being slightly slightly irked to hear that the benchmark
> numbers that have been paraded around on Slashdot and the internet in
> general, at ext3's expense, have had reiser4's "bad" results surgically
> extracted, I am running my own benchmarks to get the real story on
> reiser4/ext3 mongo performance on my, rather average, desktop hardware.
> 
> I am using the latest Mongo on FC/rawhide and the 2.6.8.1-mm4 kernel.
> 
> Unfortunately, I get an error from mongo.pl that "Done" is not a numeric
> argument at line 439.
> 
> I have done this to fix it:
> 
> 
> --- mongo.pl    2004-08-27 17:07:01.681723313 -0500
> +++ mongo_fixed.pl      2004-08-27 17:06:51.369306735 -0500
> @@ -429,8 +429,8 @@
>         if ( -e ${ERR_FILE}) {
>             &DIE ("\nEXITED WITH FAIL\n");
>         }
> -       my $real = (split ' ', $time_output[0])[1];
> -       my $cpu  = (split ' ', $time_output[2])[1];
> +       my $real = (split ' ', $time_output[1])[1];
> +       my $cpu  = (split ' ', $time_output[3])[1];
>   
>         unless ( $real =~ /\s*\d+/ && $cpu =~ /\s*\d+/) {
>           LOG "@time_output";
> 
> 
> What it gets me is the "real" line of the "time" output for "STAT
> REAL_TIME" and the "sys" line of the "time" output for "STAT CPU_TIME".
> i.e. only system time is counted. I believe this was the intent of the
> original code, but want to verify before continuing.

it works on our test servers which are SuSE9.0/9.1.

I think there is a dependency on system utilities,  "Done" is not printed by
mongo.pl or any other program from the mongo package.

It would be nice to tell us what linux distro you are using and what mongo
phase fails. 

> Thanks,
> Steve Bergman

Thanks.

-- 
Alex.

Re: reiser4 plugins

[Hans Reiser]

Christoph Hellwig wrote:

>
>  The problem is not that it
>doesn't work but that it's really hard to maintain.  
>
You really like to talk about what you know nothing of, yes?

Reiser4 is FAR easier to maintain than V3.  Ask anyone on reiserfs-dev.  
It was designed to reduce our software maintenance costs. 

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote:
> This VFS interface is an integral part of Фvery filesystem, and it

VFS never was "an integral part" of ANY filesystem. my dog knows it.
its just unified INTERFACE TO any filesystem (including reiser4).


P.S. I imagine, how much flamed it would be if reiser4 made any intensive
changes in linux VFS code...



-- 
"the liberation loophole will make it clear.."
lex lyamin
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote:
> Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote:
> > This VFS interface is an integral part of ??very filesystem, and it
> 
> VFS never was "an integral part" of ANY filesystem. my dog knows it.
> its just unified INTERFACE TO any filesystem (including reiser4).

You's misquoting me.  IF you quoted the whole context it'd be pretty
sure that the part of the filesystem that intefaces with the VFS is
meant.

But one could even say VFS is integral part of a linux filesystem as
it does most of the work a filesystem driver does in other operating
systems.

> P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> changes in linux VFS code...

It really depends on how you sent them.  If you had a big patch without
explanations - sure.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 01:12:33PM +0200, Christoph Hellwig wrote:
> On Sat, Aug 28, 2004 at 02:59:29PM +0400, Alexander Lyamin wrote:
> > Thu, Aug 26, 2004 at 03:24:39PM +0200, Christoph Hellwig wrote:
> > > This VFS interface is an integral part of ??very filesystem, and it
> > 
> > VFS never was "an integral part" of ANY filesystem. my dog knows it.
> > its just unified INTERFACE TO any filesystem (including reiser4).
> 
> You's misquoting me.  IF you quoted the whole context it'd be pretty
> sure that the part of the filesystem that intefaces with the VFS is
> meant.

No. Its not me "misquoting", its just someone sound plain ?incoherrent?.
Even if I overquoted reply with whole message, its still sound ?incohherent?.

> 
> But one could even say VFS is integral part of a linux filesystem as
> it does most of the work a filesystem driver does in other operating
> systems.

theres no "linux filesystem". there are "linux filesystems".
thanks god.

But I it would be really grate if you'll elaborate your sentence with
example of VFS functionality (lack of it) on said "other operating systems"
and if you'll define "most of work".


> 
> > P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> > changes in linux VFS code...
> 
> It really depends on how you sent them.  If you had a big patch without
> explanations - sure.
It would work with small tweaks, but you just can take a look at reiser4
code and you'll understand that it just could not be chopped in
"set of small patches" altough it could be documented better ofcourse,
but its really well commented already.

some times, some approaches to  some problems  just would not work.

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 04:05:02PM +0400, Alexander Lyamin wrote:
> > But one could even say VFS is integral part of a linux filesystem as
> > it does most of the work a filesystem driver does in other operating
> > systems.
> 
> theres no "linux filesystem". there are "linux filesystems".
> thanks god.

a linux filesystem, not the linux filesystem, please read again.

> But I it would be really grate if you'll elaborate your sentence with
> example of VFS functionality (lack of it) on said "other operating systems"
> and if you'll define "most of work".

most trivial example is namespace locking, in *BSD, Windows, SVR4 and
derivates it's done in the lowlevel filesystem.  In plan9, Linux and
soon DragonlyBSD it's done in the VFS. 

> > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> > > changes in linux VFS code...
> > 
> > It really depends on how you sent them.  If you had a big patch without
> > explanations - sure.
> It would work with small tweaks, but you just can take a look at reiser4
> code and you'll understand that it just could not be chopped in
> "set of small patches" altough it could be documented better ofcourse,
> but its really well commented already.
> 
> some times, some approaches to  some problems  just would not work.

You still haven't even bother explaining what you want to do.  It's hard
to argue against vague uncertainity.

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 03:56:55PM +0200, Christoph Hellwig wrote:
> On Sat, Aug 28, 2004 at 04:05:02PM +0400, Alexander Lyamin wrote:
> > > But one could even say VFS is integral part of a linux filesystem as
> > > it does most of the work a filesystem driver does in other operating
> > > systems.
> > 
> > theres no "linux filesystem". there are "linux filesystems".
> > thanks god.
> 
> a linux filesystem, not the linux filesystem, please read again.
> 
> > But I it would be really grate if you'll elaborate your sentence with
> > example of VFS functionality (lack of it) on said "other operating systems"
> > and if you'll define "most of work".
> 
> most trivial example is namespace locking, in *BSD, Windows, SVR4 and
> derivates it's done in the lowlevel filesystem.  In plan9, Linux and
> soon DragonlyBSD it's done in the VFS. 

ok. good examples.

> 
> > > > P.S. I imagine, how much flamed it would be if reiser4 made any intensive
> > > > changes in linux VFS code...
> > > 
> > > It really depends on how you sent them.  If you had a big patch without
> > > explanations - sure.
> > It would work with small tweaks, but you just can take a look at reiser4
> > code and you'll understand that it just could not be chopped in
> > "set of small patches" altough it could be documented better ofcourse,
> > but its really well commented already.
> > 
> > some times, some approaches to  some problems  just would not work.
> 
> You still haven't even bother explaining what you want to do.  It's hard
> to argue against vague uncertainity.

 o files as directories -  no oppinion on that.
                            
 o metafiles -   AFAIK it was product of Nikita Danilov just playing and fooling.

Probably solution that will satisfy you is to have "legacy (dumb?) mode" which
is leaving all this fancy stuff out of sight.

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins

[Hans Reiser]

Alexander Lyamin wrote:

>
>                            
> o metafiles -   AFAIK it was product of Nikita Danilov just playing and fooling.
>  
>
No, it was not, it was an idea I had for so long I cannot remember when, 
maybe 1984, and I proposed it to darpa, and then nikita got told about 
it probably from being told to read the web page on our darpa project, 
and he took on the task of coding it.

Hans

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Linus Torvalds]

On Sat, 28 Aug 2004, Alexander Lyamin wrote:
> 
> VFS never was "an integral part" of ANY filesystem. my dog knows it.

That's not really true.

Name handling (dentry layer, mounting) is very much an integral part of
the filesystem. Almost everything else in the VFS is "helper functions",
ie a filesystem can choose to ignore buffer heads, page cache etc, but a 
filesystem really cannot ignore or override the VFS naming stuff.

(Arguably the page cache isn't part of the VFS layer at all, it's really a 
memory management thing, although it's so intertwined with the VFS helper 
functions that you can't really draw the line).

> its just unified INTERFACE TO any filesystem (including reiser4).

True, but that's not the whole truth. It's way more than just an
interface. It's a set of rules, and it's in many way the controlling
party.

		Linus

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Alexander Lyamin]

Sat, Aug 28, 2004 at 10:18:44AM -0700, Linus Torvalds wrote:
>
>
> On Sat, 28 Aug 2004, Alexander Lyamin wrote:
> >
> > VFS never was "an integral part" of ANY filesystem. my dog knows it.
>
> That's not really true.

Theres some truth. And its really arguably, but I think, I see your point.

And taking on the situation with reiser4 with "smartmost VFS" approach,
time should pass and with time and expirience on hands it will be evident what
should go in VFS. Now its may be too early.

Considering "amazing PR skills" of Hans Reiser it was the only viable way
to get this changes in VFS. Cause, ironically, mr. Hellwig that currently
demand it to be scrapped out or go in VFS would instakill Hans Reiser
(i know many people would:) if he only touched holy cow of VFS for any
reiser4 purpose.

As I told before, some technically right things just do not work in this
imperfect world. You have to count with social issues, and Hans Reiser
wont, he just avoid them whenever its possible. 

> Name handling (dentry layer, mounting) is very much an integral part of
> the filesystem. Almost everything else in the VFS is "helper functions",
> ie a filesystem can choose to ignore buffer heads, page cache etc, but a 
> filesystem really cannot ignore or override the VFS naming stuff.
> 
> (Arguably the page cache isn't part of the VFS layer at all, it's really a 
> memory management thing, although it's so intertwined with the VFS helper 
> functions that you can't really draw the line).
> 
> > its just unified INTERFACE TO any filesystem (including reiser4).
> 
> True, but that's not the whole truth. It's way more than just an
> interface. It's a set of rules, and it's in many way the controlling
> party.
> 
> 		Linus

"True, but thats not the whole truth."

-- 
"the liberation loophole will make it clear.."
lex lyamin

Re: reiser4 plugins (was: silent semantic changes with reiser4)

[Christoph Hellwig]

On Sat, Aug 28, 2004 at 11:03:50PM +0400, Alexander Lyamin wrote:
> And taking on the situation with reiser4 with "smartmost VFS" approach,
> time should pass and with time and expirience on hands it will be evident what
> should go in VFS. Now its may be too early.
> 
> Considering "amazing PR skills" of Hans Reiser it was the only viable way
> to get this changes in VFS. Cause, ironically, mr. Hellwig that currently
> demand it to be scrapped out or go in VFS would instakill Hans Reiser
> (i know many people would:) if he only touched holy cow of VFS for any
> reiser4 purpose.

Could you please stop talking such inproven crap?

Re: reiser4 plugins

[Hans Reiser]

I think it is reasonable to make the -nopseudos (turns off the metafiles 
) mount option mandatory, until the bugs are resolved.

Our testing did not find these metafile/VFS bugs because of the reason 
for all our bugs, we screwed up. 

There is a distinct difference between some persons and I, which is that 
some think all of reiser4 should be excluded until metafiles are 
implemented by VFS some long time from now, and I, in that I merely 
think buggy optional features should be turned off until they are 
fixed.  I, being renowned for my paranoia and asininity as I am, think 
these persons find it convenient as an excuse to keep us from competing, 
and I think that if we were slower there would be less hassle every time 
we try to get into the kernel. 

While reiser4 has some significant roughnesses remaining in its 
performance, I think the average user would find it performs better than 
other filesystems, and is stable enough for, say, a laptop, and I 
predict that by the time we have it stable enough for mission critical 
servers, all the roughness in various important corner cases will be 
gone.  

Persons benchmarking it with tarballs, please be sure to use tarballs 
created on reiser4, not ext2 tarballs, readdir order matters a lot for 
sorted directory filesystems.

Hans

Re: reiser4 plugins

[Herbert Poetzl]

On Sat, Aug 28, 2004 at 02:41:12PM -0700, Hans Reiser wrote:
> I think it is reasonable to make the -nopseudos (turns off the metafiles 
> ) mount option mandatory, until the bugs are resolved.
> 
> Our testing did not find these metafile/VFS bugs because of the reason 
> for all our bugs, we screwed up. 
> 
> There is a distinct difference between some persons and I, which is that 
> some think all of reiser4 should be excluded until metafiles are 
> implemented by VFS some long time from now, and I, in that I merely 
> think buggy optional features should be turned off until they are 
> fixed.  I, being renowned for my paranoia and asininity as I am, think 
> these persons find it convenient as an excuse to keep us from competing, 
> and I think that if we were slower there would be less hassle every time 
> we try to get into the kernel. 
> 
> While reiser4 has some significant roughnesses remaining in its 
> performance, I think the average user would find it performs better than 
> other filesystems, and is stable enough for, say, a laptop, and I 
> predict that by the time we have it stable enough for mission critical 
> servers, all the roughness in various important corner cases will be 
> gone.  
> 
> Persons benchmarking it with tarballs, please be sure to use tarballs 
> created on reiser4, not ext2 tarballs, readdir order matters a lot for 
> sorted directory filesystems.

hmm, so probably we have to wait until all
tar packagers moved to reiser4, so that the
available tar files are 'sorted properly' ...

best,
Herbert

> Hans
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: reiser4 plugins

[Hans Reiser]

Herbert Poetzl wrote:

>
>hmm, so probably we have to wait until all
>tar packagers moved to reiser4, so that the
>available tar files are 'sorted properly' ...
>
>best,
>Herbert
>
>  
>
Or wait for a repacker, which will probably happen sooner. Distros that 
use reiser4 will probably tar in reiser4 order.