Some more questions about Reiser4 ;)

Some more questions about Reiser4 ;)

[Fred -- Speed Up --]

[-- Attachment #1: Type: text/plain, Size: 3554 bytes --]

I posted one week ago some questions about Reiser4. I've just read your answers and I'll firstly want to thank all of the people developing Reiser4 and answering on the mailing list ;)

Thanks to Oleg I could understand what the object id stands for outside the FS. And parent id should be the part of the key related to what the file/item is related to : when we consider some file's attribute, it's stored in a separate object which has a single object id but whose parent id is the object id of the file it's related to, am I right ? And a file should behave the same way with its parent directory (this is likely to be a main concept of Reiser4 : objects are both files and folders).
Now an inode number identifies a physical file or folder considered from VFS (not an attribute or any metadata Reiser4 is implementing), it's the object id for the main file, from which we can get the attribute and/or the subfolders wether it is a directory or not.

Hans Reiser informed me about the release date :-D. Can you evaluate the developement stage, are you still implementing features or rather getting the software stable, or even doing tweaking stuff ... the website announces June 2003 : I'm pretty sure it won't be ready that soon :-D. I know developing such a project takes time, and the release date is not such an important thing as soon as it is released stable, powerfull and fast ;)


Now I got some question still remaining unanswered :

- Could you please describe the overall structure of Reiser4fs, with a diagram or a little text, where the superblock is, the wandering logs, the storage tree and semantic graph, just sort of a picture of the hard drive to know where and how data and structures are kept and ordered ... ;)

- How does your semantic graph work : is it a cache contaning parts of the semantic tree (containing the user-side paths : /foo/bar/...), and updated when the filesystem is being used to reduce paths resolution time and eventually cache some data, resolving path the "old fashion" way when the path is not in the graph, by browsing the storage tree looking for the desired folders and fetching their information, or does the semantic graph allready contain the whole path structure, so that it is not necessary to store directory (I mean, the way VFS understands what a directory is) informations (list of files and subfolders) in the storage tree ?

- Considering the semantic graph, as you choose to use a generic graph, you would be able to link paths together to shorten access to some files that are often used : does Reiser4 implement some algorithms that reduces access time to most acceded files (at least path resolution) other than caching paths ? For example, if "/foo/bar/1/2/3/file" file is being used very often, does the semantic graph make a link directly between the root and the file ?

- How do you keep the storage tree balanced, as it is constantly evolving ? Even with balancing at flush time, some important tree changes would take too much time to rebalance the tree : if this occurs, does the storage tree remain unbalanced, scheduling tree balancing later ? Could you graphically (or with a short text ...) tell how the tree evolves, and what the balancing operation really consists in ?

- When storing really big files, one extent pointer is not enough : does Reiser4 manage big files the same way ext2 does, with 2-level indirection pointers, storing only the first one in the tree ? If so, wouldn't this be considered as a Blob behaviour ?


Thank you ;)

Fred

Re: Some more questions about Reiser4 ;)

[Hans Reiser]

Fred -- Speed Up -- wrote:

>
>- Could you please describe the overall structure of Reiser4fs, with a diagram or a little text, where the superblock is, the wandering logs, the storage tree and semantic graph, just sort of a picture of the hard drive to know where and how data and structures are kept and ordered ... ;)
>
Elena, can you draw a diagram or somehow document the disk layout for 
reiser4.  We seem to describe the pieces but not the whole.

>
>- How does your semantic graph work : is it a cache contaning parts of the semantic tree (containing the user-side paths : /foo/bar/...), and updated when the filesystem is being used to reduce paths resolution time and eventually cache some data, resolving path the "old fashion" way when the path is not in the graph, by browsing the storage tree looking for the desired folders and fetching their information, or does the semantic graph allready contain the whole path structure, so that it is not necessary to store directory (I mean, the way VFS understands what a directory is) informations (list of files and subfolders) in the storage tree ?
>
VFS does caching of the semantic graph in something called dcache.  We 
cache data and semantics the same way in the FS without trying to 
understand what is data and what is semantics.  We often discuss how we 
should cache internal nodes of the tree longer than leaf nodes, but I 
think we have not implemented that yet in Reiser4.  Nikita, please look 
into that.

>
>- Considering the semantic graph, as you choose to use a generic graph, you would be able to link paths together to shorten access to some files that are often used : does Reiser4 implement some algorithms that reduces access time to most acceded files (at least path resolution) other than caching paths ? For example, if "/foo/bar/1/2/3/file" file is being used very often, does the semantic graph make a link directly between the root and the file ?
>
No, semantic layers do not concern themselves with access patterns and 
should not.

>
>- How do you keep the storage tree balanced, as it is constantly evolving ? Even with balancing at flush time, some important tree changes would take too much time to rebalance the tree : 
>
I don't agree.

>if this occurs, does the storage tree remain unbalanced, scheduling tree balancing later ? Could you graphically (or with a short text ...) tell how the tree evolves, and what the balancing operation really consists in ?
>
www.namesys.com/v4/v4.html describes the difference between height 
balancing and space usage balancing.  I will be writing more on space 
usage balancing over the coming months.

>
>- When storing really big files, one extent pointer is not enough : 
>
fragmentation not size of the file determines number of extents needed.

>does Reiser4 manage big files the same way ext2 does, with 2-level indirection pointers, storing only the first one in the tree ?
>
no.

> If so, wouldn't this be considered as a Blob behaviour ?
>
>
>Thank you ;)
>
>Fred
>
>  
>


-- 
Hans

Some more questions about Reiser4 ;)

[Fred -- Speed Up --]

What about me :-D

----- Original Message ----- 
From: "Fred -- Speed Up --" <speedup@free.fr>
To: <reiserfs-list@namesys.com>
Sent: Saturday, May 03, 2003 11:33 PM
Subject: Some more questions about Reiser4 ;)


I posted one week ago some questions about Reiser4. I've just read your
answers and I'll firstly want to thank all of the people developing Reiser4
and answering on the mailing list ;)

Thanks to Oleg I could understand what the object id stands for outside the
FS. And parent id should be the part of the key related to what the
file/item is related to : when we consider some file's attribute, it's
stored in a separate object which has a single object id but whose parent id
is the object id of the file it's related to, am I right ? And a file should
behave the same way with its parent directory (this is likely to be a main
concept of Reiser4 : objects are both files and folders).
Now an inode number identifies a physical file or folder considered from VFS
(not an attribute or any metadata Reiser4 is implementing), it's the object
id for the main file, from which we can get the attribute and/or the
subfolders wether it is a directory or not.

Hans Reiser informed me about the release date :-D. Can you evaluate the
developement stage, are you still implementing features or rather getting
the software stable, or even doing tweaking stuff ... the website announces
June 2003 : I'm pretty sure it won't be ready that soon :-D. I know
developing such a project takes time, and the release date is not such an
important thing as soon as it is released stable, powerfull and fast ;)


Now I got some question still remaining unanswered :

- Could you please describe the overall structure of Reiser4fs, with a
diagram or a little text, where the superblock is, the wandering logs, the
storage tree and semantic graph, just sort of a picture of the hard drive to
know where and how data and structures are kept and ordered ... ;)

- How does your semantic graph work : is it a cache contaning parts of the
semantic tree (containing the user-side paths : /foo/bar/...), and updated
when the filesystem is being used to reduce paths resolution time and
eventually cache some data, resolving path the "old fashion" way when the
path is not in the graph, by browsing the storage tree looking for the
desired folders and fetching their information, or does the semantic graph
allready contain the whole path structure, so that it is not necessary to
store directory (I mean, the way VFS understands what a directory is)
informations (list of files and subfolders) in the storage tree ?

- Considering the semantic graph, as you choose to use a generic graph, you
would be able to link paths together to shorten access to some files that
are often used : does Reiser4 implement some algorithms that reduces access
time to most acceded files (at least path resolution) other than caching
paths ? For example, if "/foo/bar/1/2/3/file" file is being used very often,
does the semantic graph make a link directly between the root and the file ?

- How do you keep the storage tree balanced, as it is constantly evolving ?
Even with balancing at flush time, some important tree changes would take
too much time to rebalance the tree : if this occurs, does the storage tree
remain unbalanced, scheduling tree balancing later ? Could you graphically
(or with a short text ...) tell how the tree evolves, and what the balancing
operation really consists in ?

- When storing really big files, one extent pointer is not enough : does
Reiser4 manage big files the same way ext2 does, with 2-level indirection
pointers, storing only the first one in the tree ? If so, wouldn't this be
considered as a Blob behaviour ?


Thank you ;)

Fred

Re: Some more questions about Reiser4 ;)

[Yury Umanets]

Fred -- Speed Up -- wrote:

>What about me :-D
>
>----- Original Message ----- 
>From: "Fred -- Speed Up --" <speedup@free.fr>
>To: <reiserfs-list@namesys.com>
>Sent: Saturday, May 03, 2003 11:33 PM
>Subject: Some more questions about Reiser4 ;)
>
>
>I posted one week ago some questions about Reiser4. I've just read your
>answers and I'll firstly want to thank all of the people developing Reiser4
>and answering on the mailing list ;)
>
>Thanks to Oleg I could understand what the object id stands for outside the
>FS. And parent id should be the part of the key related to what the
>file/item is related to : 
>

>when we consider some file's attribute, it's
>stored in a separate object which has a single object id but whose parent id
>is the object id of the file it's related to, am I right?
>
First of all, you should consider about object in reiser4 as abput as 
file, directory, symlinks, etc. Each object contains one or more items:
(1) Regular file contains stat data item, and not limited number of file 
body items. File body items are extents and tails. Empty file (size == 
0) contains only stat data item.
(2) Directory contains stat data item and not limited number of 
directory body items. Directory item contains directory entries, theirs 
hashes, keys of teh object entries point to.
(3) Symlink contains only stat data item (in the current implementation 
of the symlink40 plugin). If you like another symlink organization, you 
will probably intersting in writting your own symlink plugin

Each item of an object may be found by its key. The object id of the 
item is the same as object id of the file, it belongs to. Thus, in such 
a case all parts of some file may be found inside reiser4 tree.

Each object have its attributes.  But currently all object attributes 
are supposed to be stored inside stat data item of the object. And 
ofcourse attributes do not have own key.

> And a file should
>behave the same way with its parent directory (this is likely to be a main
>concept of Reiser4 : objects are both files and folders).
>

>Now an inode number identifies a physical file or folder considered from VFS
>(not an attribute or any metadata Reiser4 is implementing), it's the object
>id for the main file, from which we can get the attribute and/or the
>subfolders wether it is a directory or not.
>

>
>Hans Reiser informed me about the release date :-D. Can you evaluate the
>developement stage, are you still implementing features or rather getting
>the software stable, or even doing tweaking stuff ... the website announces
>June 2003 : I'm pretty sure it won't be ready that soon :-D. I know
>developing such a project takes time, and the release date is not such an
>important thing as soon as it is released stable, powerfull and fast ;)
>
Until fsck is done, filesystem cannot be used in production anyway :) We 
working on fsck now and another tools. If you are intersting, I will 
send you package or link, it may obtained at.

>
>
>Now I got some question still remaining unanswered :
>
>- Could you please describe the overall structure of Reiser4fs, with a
>diagram or a little text, where the superblock is, the wandering logs, the
>storage tree and semantic graph, just sort of a picture of the hard drive to
>know where and how data and structures are kept and ordered ... ;)
>
You are smart one :) It's huge work :) Read the source code please.

>
>- How does your semantic graph work : is it a cache contaning parts of the
>semantic tree (containing the user-side paths : /foo/bar/...), and updated
>when the filesystem is being used to reduce paths resolution time and
>eventually cache some data, resolving path the "old fashion" way when the
>path is not in the graph, by browsing the storage tree looking for the
>desired folders and fetching their information, or does the semantic graph
>allready contain the whole path structure, so that it is not necessary to
>store directory (I mean, the way VFS understands what a directory is)
>informations (list of files and subfolders) in the storage tree ?
>
The parsing the file path is fully VFS' job.

>
>- Considering the semantic graph, as you choose to use a generic graph, you
>would be able to link paths together to shorten access to some files that
>are often used : does Reiser4 implement some algorithms that reduces access
>time to most acceded files (at least path resolution) other than caching
>paths ? For example, if "/foo/bar/1/2/3/file" file is being used very often,
>does the semantic graph make a link directly between the root and the file ?
>

>
>- How do you keep the storage tree balanced, as it is constantly evolving ?
>Even with balancing at flush time, some important tree changes would take
>too much time to rebalance the tree : if this occurs, does the storage tree
>remain unbalanced, scheduling tree balancing later ? Could you graphically
>(or with a short text ...) tell how the tree evolves, and what the balancing
>operation really consists in ?
>
We should understand the difference between balancing and keeping tree 
consistent.
Tree is consistent if the following rule is hold true: all nodes may be 
accessed by means of using theirs left delimiting keys.
Tree is balanced if all used invariants are hold true (for instance, all 
nodes are half filled, or all nodes fully filled, etc). Also tree should 
not be singular.

Reiser4 in balancing time (any tree modification) does only care about 
tree consistency and keeps it in not singular state. Whereas flush does 
care about invariants (packing, etc).

>
>- When storing really big files, one extent pointer is not enough : does
>Reiser4 manage big files the same way ext2 does, with 2-level indirection
>pointers, storing only the first one in the tree ? If so, wouldn't this be
>considered as a Blob behaviour ?
>
Reiser4 uses extent items for that. Each extent item looks like the 
following:

------------------
start  | width
------------------
234    |  20
456    | 1034
567    | 345

and so on...

>
>
>Thank you ;)
>
>Fred
>
>
>
>  
>


-- 
Yury Umanets
"We're flying high, we're watching the world passes by..."

Re: Some more questions about Reiser4 ;)

[Fred -- Speed Up --]

Thank you for your answers ;)

You're right, I'd have to read the source code, but I don't really know
where I can find it : where do you keep the recent snapshots, the only thing
I've been able to download is your 2.5.60 patch and the february Reiser4
utilities.

> Until fsck is done, filesystem cannot be used in production anyway :) We
> working on fsck now and another tools. If you are intersting, I will
> send you package or link, it may obtained at.
It would be nice if you could sent me the url, thank you ;)

> The parsing the file path is fully VFS' job.
Probably ... but the Reiser4 doc specifies a "semantic layer" and a "storage
layer" : does the semantic layer only consist in VFS resolving path from
standard Reiser4 directory reading methods based on reading directory
entries and everytime resolving the whole path reading the Reiser4 physical
tree ?
Does Reiser4 implement some new ways of resolving paths, as I thought I've
read in the doc, to convert file paths to keys or does VFS alone achieve
this ?
When you're speaking of a graph used by the semantic layer, would it be part
of the VFS caching process ?
Another thing I could hardly understand : does Reiser4 keep directories
apart from files so that path can be cached or are they kept with the same
status normal files got ?

> We should understand the difference between balancing and keeping tree
> consistent.
> Tree is consistent if the following rule is hold true: all nodes may be
> accessed by means of using theirs left delimiting keys.
> Tree is balanced if all used invariants are hold true (for instance, all
> nodes are half filled, or all nodes fully filled, etc). Also tree should
> not be singular.
>
> Reiser4 in balancing time (any tree modification) does only care about
> tree consistency and keeps it in not singular state. Whereas flush does
> care about invariants (packing, etc).
I didn't noticed the difference, thank you for pointing this out. I
understand the "balancing" process, which is documented on your website,
meaning packing dirty memory items on flushing, but I don't understand what
you mean by "tree should not be singular" : does this mean a twig should
contain more than one item ?
The really interesting thing (the mysterious one :-D) is the
consistency-keeping process. I didn't really understood the left-delimiting
key concept, I guess it's something like a path in the storage tree, am I
right ? In a consistent tree, do all the inodes got the same left-delimiting
key depth ?


Real thank you for your answers Yuri ;)
I'm going to read my "kernel internals" book to know more about the VFS and
ext2 handlings, so you can expect some more questions (or answers ;))
comming :-D

Fred

Re: Some more questions about Reiser4 ;)

[Yury Umanets]

Fred -- Speed Up -- wrote:

>Thank you for your answers ;)
>
>You're right, I'd have to read the source code, but I don't really know
>where I can find it : where do you keep the recent snapshots, the only thing
>I've been able to download is your 2.5.60 patch and the february Reiser4
>utilities.
>
2.5.60 is enough. Disk format was not changed substantionaly since it. 
Also you can download reiser4progs and try to find out everything you 
are intersted in there.  You can get it at 
http://thebsh.namesys.com/snapshots/2003.05.10/

>
>  
>
>>Until fsck is done, filesystem cannot be used in production anyway :) We
>>working on fsck now and another tools. If you are intersting, I will
>>send you package or link, it may obtained at.
>>    
>>
>It would be nice if you could sent me the url, thank you ;)
>
http://thebsh.namesys.com/snapshots/2003.05.10/

>
>  
>
>>The parsing the file path is fully VFS' job.
>>    
>>
>Probably ... but the Reiser4 doc specifies a "semantic layer" and a "storage
>layer" : does the semantic layer only consist in VFS resolving path from
>standard Reiser4 directory reading methods based on reading directory
>entries and everytime resolving the whole path reading the Reiser4 physical
>tree ?
>
Storage layer (storage tree) is the tree balanced tree itself (node 
pointer items, keys, items on leaf level, etc).

Semantic layer is more higher level of abstraction which means, that the 
filesystem objects lie on storage level and has some relations with 
parent objects.
Suppose we have path /usr/src/linux. The object "linux" lies inside 
object "src", that is key locality component of the object "linux" the 
same as object id of the object "src".

>Does Reiser4 implement some new ways of resolving paths, as I thought I've
>read in the doc, to convert file paths to keys or does VFS alone achieve
>this ?
>
Filesystems in linux are not supposed to parse paths. When linux VFS is 
parsing a path, it is consulting with corresponding filesystem, in order 
to find needed components. For instance, VFS calls some lookup method of 
the directory object in order to find entry by name in it. And so on.

>When you're speaking of a graph used by the semantic layer, would it be part
>of the VFS caching process ?
>
Probably you mean dentry cache?

>Another thing I could hardly understand : does Reiser4 keep directories
>apart from files so that path can be cached or are they kept with the same
>status normal files got ?
>
I can't understand you.

>
>  
>
>>We should understand the difference between balancing and keeping tree
>>consistent.
>>Tree is consistent if the following rule is hold true: all nodes may be
>>accessed by means of using theirs left delimiting keys.
>>Tree is balanced if all used invariants are hold true (for instance, all
>>nodes are half filled, or all nodes fully filled, etc). Also tree should
>>not be singular.
>>
>>Reiser4 in balancing time (any tree modification) does only care about
>>tree consistency and keeps it in not singular state. Whereas flush does
>>care about invariants (packing, etc).
>>    
>>
>I didn't noticed the difference, thank you for pointing this out. I
>understand the "balancing" process, which is documented on your website,
>meaning packing dirty memory items on flushing, but I don't understand what
>you mean by "tree should not be singular" : 
>

>does this mean a twig should
>contain more than one item ?
>
This means, that tree is as minimal height as possible. Suppose we have 
tree of height 3. Root level (3) contains one item and 2 sevel has  one 
item. This tree is singular. That it is because,  we need only two 
levels in this case. Tree like this is possible in balancing time. Flush 
will care about that in its time.

>The really interesting thing (the mysterious one :-D) is the
>consistency-keeping process. I didn't really understood the left-delimiting
>key concept, I guess it's something like a path in the storage tree,
>
You are right :)

> am I
>right ? 
>

>In a consistent tree, do all the inodes got the same left-delimiting
>key depth ?
>
We do not have inodes in storage tree. We have nodes instead.

Left delimiting key means,  that if node pointer item in the index part 
of tree (levels from tree height through to 2) points to some node, then 
node pointer item key equal to leftmost key of the node node pointer 
points to.

>
>
>Real thank you for your answers Yuri ;)
>
You are welcome.

>I'm going to read my "kernel internals" book to know more about the VFS and
>ext2 handlings, so you can expect some more questions (or answers ;))
>comming :-D
>

>
>Fred
>
>
>
>  
>


-- 
Yury Umanets
"We're flying high, we're watching the world passes by..."

Re: Some more questions about Reiser4 ;)

[Fred -- Speed Up --]

> Storage layer (storage tree) is the tree balanced tree itself (node
> pointer items, keys, items on leaf level, etc).
> Semantic layer is more higher level of abstraction which means, that the
> filesystem objects lie on storage level and has some relations with
> parent objects.
> Suppose we have path /usr/src/linux. The object "linux" lies inside
> object "src", that is key locality component of the object "linux" the
> same as object id of the object "src".
Now the semantic layer handles with all the key stuff, being the link
between the storage layer and VFS, and a convenient tool for plugin
purposes, and the storage layer rules in-depth operations as files read and
write and the tree hanlding, am I right ?

> Filesystems in linux are not supposed to parse paths. When linux VFS is
> parsing a path, it is consulting with corresponding filesystem, in order
> to find needed components. For instance, VFS calls some lookup method of
> the directory object in order to find entry by name in it. And so on.
I just read my kernel book, in the VFS section, I hopelly won't ask you such
silly questions anymore :-D

> >When you're speaking of a graph used by the semantic layer, would it be
part
> >of the VFS caching process ?
> Probably you mean dentry cache?
Should be ;)
VFS seems to be very intrusive (I mean, it does a lot of the file-handling
process) ... doesn't it limit your initiatives in implementing new
structures ?

> >Another thing I could hardly understand : does Reiser4 keep directories
> >apart from files so that path can be cached or are they kept with the
same
> >status normal files got ?
> I can't understand you.
The reason why I asked this was because I didn't knew that VFS was the one
and only responsible for the path resolving ;)

> >I didn't noticed the difference, thank you for pointing this out. I
> >understand the "balancing" process, which is documented on your website,
> >meaning packing dirty memory items on flushing, but I don't understand
what
> >you mean by "tree should not be singular" :
> >does this mean a twig should
> >contain more than one item ?
> This means, that tree is as minimal height as possible. Suppose we have
> tree of height 3. Root level (3) contains one item and 2 sevel has  one
> item. This tree is singular. That it is because,  we need only two
> levels in this case. Tree like this is possible in balancing time. Flush
> will care about that in its time.
OK, now how do you balance the tree when items are deleted, or added ? This
operation must be very complex, as you don't have enough time to rebalance
the whole tree on every commit considering performance issues. I read tree
balancing (I mean keeping it consistent ;)) was done periodically, you're
saying it's done at flush time, but how can you achieve such a complex
operation in short enough time not to affect performance ?

> >The really interesting thing (the mysterious one :-D) is the
> >consistency-keeping process. I didn't really understood the
left-delimiting
> >key concept, I guess it's something like a path in the storage tree
> You are right :)
You can't guess how happy I am ||:-D

> >In a consistent tree, do all the inodes got the same left-delimiting
> >key depth ?
> >
> We do not have inodes in storage tree. We have nodes instead.
>
> Left delimiting key means,  that if node pointer item in the index part
> of tree (levels from tree height through to 2) points to some node, then
> node pointer item key equal to leftmost key of the node node pointer
> points to.
All right ;)


Thank you,
Fred

Re: Some more questions about Reiser4 ;)

[Hans Reiser]

Fred -- Speed Up -- wrote:

>
>  
>
>>>When you're speaking of a graph used by the semantic layer, would it be
>>>      
>>>
>part
>  
>
>>>of the VFS caching process ?
>>>      
>>>
>>Probably you mean dentry cache?
>>    
>>
>Should be ;)
>VFS seems to be very intrusive (I mean, it does a lot of the file-handling
>process) ... doesn't it limit your initiatives in implementing new
>structures ?\
>
Yes.  We manage somehow though.

>
>  
>
>>>Another thing I could hardly understand : does Reiser4 keep directories
>>>apart from files so that path can be cached or are they kept with the
>>>      
>>>
>same
>  
>
>>>status normal files got ?
>>>      
>>>
>>I can't understand you.
>>    
>>
>The reason why I asked this was because I didn't knew that VFS was the one
>and only responsible for the path resolving ;)
>
That is too strong a statement.  VFS delegates parts of the task to us, 
particularly directory lookup.

>
>  
>
>>>I didn't noticed the difference, thank you for pointing this out. I
>>>understand the "balancing" process, which is documented on your website,
>>>meaning packing dirty memory items on flushing, but I don't understand
>>>      
>>>
>what
>  
>
>>>you mean by "tree should not be singular" :
>>>does this mean a twig should
>>>contain more than one item ?
>>>      
>>>
>>This means, that tree is as minimal height as possible. Suppose we have
>>tree of height 3. Root level (3) contains one item and 2 sevel has  one
>>item. This tree is singular. That it is because,  we need only two
>>levels in this case. Tree like this is possible in balancing time. Flush
>>will care about that in its time.
>>    
>>
>OK, now how do you balance the tree when items are deleted, or added ? This
>operation must be very complex, 
>
yes, it is.

>as you don't have enough time to rebalance
>the whole tree on every commit considering performance issues. I read tree
>balancing (I mean keeping it consistent ;)) was done periodically, you're
>saying it's done at flush time, but how can you achieve such a complex
>operation in short enough time not to affect performance ?
>
we only squeeze what is being flushed to disk.  The CPU is faster than 
the disk drive, so it works.

>
>  
>
>>>The really interesting thing (the mysterious one :-D) is the
>>>consistency-keeping process. I didn't really understood the
>>>      
>>>
>left-delimiting
>  
>
>>>key concept, I guess it's something like a path in the storage tree
>>>      
>>>
>>You are right :)
>>    
>>
>You can't guess how happy I am ||:-D
>
>  
>
>>>In a consistent tree, do all the inodes got the same left-delimiting
>>>key depth ?
>>>
>>>      
>>>
>>We do not have inodes in storage tree. We have nodes instead.
>>
>>Left delimiting key means,  that if node pointer item in the index part
>>of tree (levels from tree height through to 2) points to some node, then
>>node pointer item key equal to leftmost key of the node node pointer
>>points to.
>>    
>>
>All right ;)
>
>
>Thank you,
>Fred
>
>
>
>  
>


-- 
Hans

Re: Some more questions about Reiser4 ;)

[Fred -- Speed Up --]

> >The reason why I asked this was because I didn't knew that VFS was the
one
> >and only responsible for the path resolving ;)
> >
> That is too strong a statement.  VFS delegates parts of the task to us,
> particularly directory lookup.
But VFS wants the directory reading task to return a list of files with
corresponding inodes numbers. Reiser4, in this case, has to check out the
tree and read the directory entries, but what about resolving a simple file
path : does VFS in this case want the files and subdirectory listings of
every directory the path consists in, or does VFS directly ask Reiser4 to
resolve the whole path ?
What does Reiser4 do to improve the path resolving and directory listing
performance ? Do you implement another cache even if the VFS allready got
one ?

> Elena, can you draw a diagram or somehow document the disk layout for
> reiser4.  We seem to describe the pieces but not the whole.
That would be a great thing ;)


Thank you for all your previous answers ... M. Reiser ;)

Fred

Re: Some more questions about Reiser4 ;)

[Hans Reiser]

Fred -- Speed Up -- wrote:

>>>The reason why I asked this was because I didn't knew that VFS was the
>>>      
>>>
>one
>  
>
>>>and only responsible for the path resolving ;)
>>>
>>>      
>>>
>>That is too strong a statement.  VFS delegates parts of the task to us,
>>particularly directory lookup.
>>    
>>
>But VFS wants the directory reading task to return a list of files with
>corresponding inodes numbers. Reiser4, in this case, has to check out the
>tree and read the directory entries, but what about resolving a simple file
>path : does VFS in this case want the files and subdirectory listings of
>every directory the path consists in, or does VFS directly ask Reiser4 to
>resolve the whole path ?
>What does Reiser4 do to improve the path resolving and directory listing
>performance ? Do you implement another cache even if the VFS allready got
>one ?
>
You are right.  It is a problem.  It is a problem we have sidestepped 
for a long time, but sometime in the next 5 years we will have to step 
into that code and regut VFS, and this will probably coincide with us 
introducing dramatic upwardly compatible changes in the semantics.

>
>  
>
>>Elena, can you draw a diagram or somehow document the disk layout for
>>reiser4.  We seem to describe the pieces but not the whole.
>>    
>>
>That would be a great thing ;)
>
>
>Thank you for all your previous answers ... M. Reiser ;)
>
>Fred
>
>
>
>  
>


-- 
Hans

Some more questions about Reiser4 ;)

[Fred -- Speed Up --]

Should I reformulate my questions, or is there an FAQ or doc allready
answering my questions ... ?
I'd need the answers until next tuesday, it would be nice if you gave me an
answer (even a short one ...) ;)

Fred

----- Original Message ----- 
From: "Fred -- Speed Up --" <speedup@free.fr>
To: <reiserfs-list@namesys.com>
Sent: Saturday, May 03, 2003 11:33 PM
Subject: Some more questions about Reiser4 ;)


I posted one week ago some questions about Reiser4. I've just read your
answers and I'll firstly want to thank all of the people developing Reiser4
and answering on the mailing list ;)

Thanks to Oleg I could understand what the object id stands for outside the
FS. And parent id should be the part of the key related to what the
file/item is related to : when we consider some file's attribute, it's
stored in a separate object which has a single object id but whose parent id
is the object id of the file it's related to, am I right ? And a file should
behave the same way with its parent directory (this is likely to be a main
concept of Reiser4 : objects are both files and folders).
Now an inode number identifies a physical file or folder considered from VFS
(not an attribute or any metadata Reiser4 is implementing), it's the object
id for the main file, from which we can get the attribute and/or the
subfolders wether it is a directory or not.

Hans Reiser informed me about the release date :-D. Can you evaluate the
developement stage, are you still implementing features or rather getting
the software stable, or even doing tweaking stuff ... the website announces
June 2003 : I'm pretty sure it won't be ready that soon :-D. I know
developing such a project takes time, and the release date is not such an
important thing as soon as it is released stable, powerfull and fast ;)


Now I got some question still remaining unanswered :

- Could you please describe the overall structure of Reiser4fs, with a
diagram or a little text, where the superblock is, the wandering logs, the
storage tree and semantic graph, just sort of a picture of the hard drive to
know where and how data and structures are kept and ordered ... ;)

- How does your semantic graph work : is it a cache contaning parts of the
semantic tree (containing the user-side paths : /foo/bar/...), and updated
when the filesystem is being used to reduce paths resolution time and
eventually cache some data, resolving path the "old fashion" way when the
path is not in the graph, by browsing the storage tree looking for the
desired folders and fetching their information, or does the semantic graph
allready contain the whole path structure, so that it is not necessary to
store directory (I mean, the way VFS understands what a directory is)
informations (list of files and subfolders) in the storage tree ?

- Considering the semantic graph, as you choose to use a generic graph, you
would be able to link paths together to shorten access to some files that
are often used : does Reiser4 implement some algorithms that reduces access
time to most acceded files (at least path resolution) other than caching
paths ? For example, if "/foo/bar/1/2/3/file" file is being used very often,
does the semantic graph make a link directly between the root and the file ?

- How do you keep the storage tree balanced, as it is constantly evolving ?
Even with balancing at flush time, some important tree changes would take
too much time to rebalance the tree : if this occurs, does the storage tree
remain unbalanced, scheduling tree balancing later ? Could you graphically
(or with a short text ...) tell how the tree evolves, and what the balancing
operation really consists in ?

- When storing really big files, one extent pointer is not enough : does
Reiser4 manage big files the same way ext2 does, with 2-level indirection
pointers, storing only the first one in the tree ? If so, wouldn't this be
considered as a Blob behaviour ?


Thank you ;)

Fred