ReConfigurable Directory Structure & Agrregation of files according to semantic.

ReConfigurable Directory Structure & Agrregation of files according to semantic.

[faraz ahmed]

Hi EveryBody;
Iam an final year engneering student and as part of
our final year project we have implemented a
filesysten which  
support more than 1 Directory Structures for the same
set of files. Each Directory Structure is a XML file
discribing the directory structure as well as the
attributes of the Files to be listed under Each
directory. The file system the mounts this XML file &
produces the directory listing by querying the FS for
files which match the criteria. The Project will soon
be made public under GPL. Following is the abstract.
Please guide me with your valuable suggestions.


//==============================================
1. Idea of our project:
//==============================================
The aim of this project is to implement a solution
that provides file-level host-based virtualization
that provides for better aggregation of
content/information based on semantics and properties.
File-system organization today very closely mirrors
storage paradigms rather than user-access paradigms
and semantic grouping. All file-system hierarchies are
containers that are expressed based on their physical
presence (a separate drive letter on Windows, or a
particular mount point based on the volume in Unix). 
We wish to implement a solution that will allow users
to organize their files based on their convenience. We
define this convenience in the following forms: 

The ability to organize the namespace based on certain
attribute properties (file-system metadata
virtualization). 
Ex Directory Listing as an output of high level
relational query based on file attributes.

The ability to de-link position of a file in the
hierarchy from its actual storage (file metadata
virtualization) 
Ex Directory Listing as an output of high level
relational query based on file attributes.

The ability to de-link position of a file in the
hierarchy from its actual storage (file metadata
virtualization) 
Ex Automatically Listing the files with the proper
attributes in the respective directories. Create &
forget file system now puts your MP3 file in the right
folder no matter where they are stored on the disk.

The ability to create and manipulate namespaces using
well-known metaphors (XML schema descriptions and
schema editors) 
Ex. Define the directory structure with directory
query as an XML file. Mount multiple such directory
structures (virtualization). There by allowing
multiple organizational views for the same set of
files.

The ability to continue using the standard metaphors
for manipulation and access to information
(file-system kernel API’s), thus maintaining current
large body of applications unbroken) .
Ex. Make all this features available as an standard
file system so existing apps wont fail

__________________________________
Do you Yahoo!?
Yahoo! Mail - More reliable, more storage, less spam
http://mail.yahoo.com

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Alexander G. M. Smith]

faraz ahmed wrote on Tue, 16 Mar 2004 14:45:23 -0800 (PST):
> Each Directory Structure is a XML file
> discribing the directory structure as well as the
> attributes of the Files to be listed under Each
> directory. The file system the mounts this XML file &
> produces the directory listing by querying the FS for
> files which match the criteria. The Project will soon
> be made public under GPL. Following is the abstract.
> Please guide me with your valuable suggestions.

We had a long discussion a few months ago (August to December 2003)
about similar techniques using variations of hard links to accomplish
a similar thing (putting a file in several different places).  Plus
the same discussion also involved attributes.  Have a look at the Reiser
archives (some awkward mailing list commands required), messages with
"Attributes" or "Hardlinks" or "Hard links" in the subject are relevant.

Microsoft is also doing something along those lines for their WinFS, a short
description is at http://www.tomshardware.com/storage/20040129/index.html

And if you want to see it in action, live queries and all, have a look at
BeOS, free version at http://www.bebits.com/app/2680 and read the BeOS Bible
book http://www.birdhouse.org/beos/bible/ to get a feel of how it is
actually used (I guess the Queries chapter would be most relevant).

- Alex

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Isaac Claymore]

[-- Attachment #1: Type: text/plain, Size: 4182 bytes --]

Hi,

I'm new to this idea, and am no fs expert. But it looks to me very
much like what Apple is doing on their iPods, I happen to have just
bought one iPod last weekend.

Internally, the audio files are stored on a HFS+ filesystem, under a 
hierarchy structure very weird to us humans. But, by using iPod or the
iTunes tool, the files are presented under various hierachy structures
organized by file meta data, like artist, album, composer... And, that's
far too convenient.

I guess this kind of stuff can be done in user space, well and
elegantly. Either a specialized application(e.g. iTunes), or some kind
of user space filesystem will do. I also think this is best suited
for storage of multimedia files, which naturelly come with much meta
data. And, I fail to figure out what benefit this can do to regular
system files, like those under /etc, /boot...

HTH~~


-Isaac

On Tue, Mar 16, 2004 at 02:45:23PM -0800, faraz ahmed wrote:
> Hi EveryBody;
> Iam an final year engneering student and as part of
> our final year project we have implemented a
> filesysten which  
> support more than 1 Directory Structures for the same
> set of files. Each Directory Structure is a XML file
> discribing the directory structure as well as the
> attributes of the Files to be listed under Each
> directory. The file system the mounts this XML file &
> produces the directory listing by querying the FS for
> files which match the criteria. The Project will soon
> be made public under GPL. Following is the abstract.
> Please guide me with your valuable suggestions.
> 
> 
> //==============================================
> 1. Idea of our project:
> //==============================================
> The aim of this project is to implement a solution
> that provides file-level host-based virtualization
> that provides for better aggregation of
> content/information based on semantics and properties.
> File-system organization today very closely mirrors
> storage paradigms rather than user-access paradigms
> and semantic grouping. All file-system hierarchies are
> containers that are expressed based on their physical
> presence (a separate drive letter on Windows, or a
> particular mount point based on the volume in Unix). 
> We wish to implement a solution that will allow users
> to organize their files based on their convenience. We
> define this convenience in the following forms: 
> 
> The ability to organize the namespace based on certain
> attribute properties (file-system metadata
> virtualization). 
> Ex Directory Listing as an output of high level
> relational query based on file attributes.
> 
> The ability to de-link position of a file in the
> hierarchy from its actual storage (file metadata
> virtualization) 
> Ex Directory Listing as an output of high level
> relational query based on file attributes.
> 
> The ability to de-link position of a file in the
> hierarchy from its actual storage (file metadata
> virtualization) 
> Ex Automatically Listing the files with the proper
> attributes in the respective directories. Create &
> forget file system now puts your MP3 file in the right
> folder no matter where they are stored on the disk.
> 
> The ability to create and manipulate namespaces using
> well-known metaphors (XML schema descriptions and
> schema editors) 
> Ex. Define the directory structure with directory
> query as an XML file. Mount multiple such directory
> structures (virtualization). There by allowing
> multiple organizational views for the same set of
> files.
> 
> The ability to continue using the standard metaphors
> for manipulation and access to information
> (file-system kernel API?s), thus maintaining current
> large body of applications unbroken) .
> Ex. Make all this features available as an standard
> file system so existing apps wont fail
> 
> __________________________________
> Do you Yahoo!?
> Yahoo! Mail - More reliable, more storage, less spam
> http://mail.yahoo.com

-- 

Regards, Isaac
()  ascii ribbon campaign - against html e-mail
/\                        - against microsoft attachments

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Alexander G. M. Smith]

Isaac Claymore wrote on Wed, 17 Mar 2004 10:49:31 +0800:
> I guess this kind of stuff can be done in user space, well and
> elegantly. Either a specialized application(e.g. iTunes), or some kind
> of user space filesystem will do. [...]

I (and others) think it would be better for this to be done at the file
system level.  That way you can leverage all sorts of other utilities and
programs.  Like using "cd" to change directory to the query results
virtual directory of all files by a particular artist, then being able to
use the usual file manipulation tools on them.  The general idea is to
reduce the number of separate name spaces to make life for the user and
programmer simpler and more efficient.

- Alex

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander G. M. Smith wrote:
...
| I (and others) think it would be better for this to be done at the file
| system level.  That way you can leverage all sorts of other utilities and


What about a reiser4 plugin?
What about a plugin that is mostly in userspace?
Even a generic reiser4 kernel plugin that allows userspace plugins to be
written?

I suggest v4 because you can start right now.  Maybe v6 would be better,
but at this rate, you'll be lucky to get a stable v4 by the time you're
done your project.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQFhlAXgHNmZLgCUhAQJFQQ//VKa0yoqu+WXEKynR/KsNZv+pGNDrKvUh
6uVewiftyE7wJIyhmqwOSwsfcNb9LjaV5TPt7h1bzw7g+dZyktIZIOM7564Zeb/V
ETbiE+S43lnprDxfGMUcrhp+n26mej48Onz5MOJI+Nb7JtJ6HkWs14lXfjNuyxg3
1arbX8PkH/igXknzMYvl4ghp6nrpyVf33kDSHjvq9jSA64Pott17Z1b1p0uUzUG/
WSXaqsosXATAR/mgfPP780693y9oxjIovWs0XBvOXhjDmpNWt3H829yFwfH+h4EO
23gj9kd3xmdXJiVSKpFwvZSVxm2wMDcMON92t0klTSthPB3E8t/f6lI/ndDhC1a0
xySmAwS52Mi1vhE+9Dc3w9vX90de/vUzHQoXhbVXDHtytP96VhumF9kco5QfqPPL
jLwa7k2ZcQS8maP4pUaYgtAdI8dOqNNyBxeEpTzzJ0WWBM1xa9OaeT+FOW1yQofj
2bO+ax+Mx+mXCw6RCWU2SvEGwjxLV1Rb+zcY+dx8E4cULw8K+PglOcaPFQKlW8CR
w7vvY8aXp/cTwQjLiNQPzzerudQtB4hLvMosmofisRr+95oiX6hLfdchQAWQung4
Ol9d9SJFZXnyxuzIUgeB1ELwF85iX0pAEcIyLqP4u3uwLIhEKCqBjvz+hpL/4MEo
Pvkl8kLPsQE=
=1erf
-----END PGP SIGNATURE-----

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Hubert Chan]

>>>>> "David" == David Masover <ninja@slaphack.com> writes:

David> Alexander G. M. Smith wrote:
Alexander> ...  I (and others) think it would be better for this to be
Alexander> done at the file system level.  That way you can leverage all
Alexander> sorts of other utilities and

David> What about a reiser4 plugin?

I would think that would count as being in the filesystem level.

David> What about a plugin that is mostly in userspace?

That just reminded me of lufs[1], which allows you to create a userspace
daemon that communicates with the lufs module to provide access to a
(virtual) filesystem.  They say that it's slow, because it uses UNIX
domain sockets for the communication, so it's more suitable for mounting
remote filesystems, where network lag would dominate.  (You also
wouldn't get all the nifty stuff that Reiser4 gives you, like
pseudo-files.)  But I bet that (at this point in time) it's probably
easier to write a lufs daemon than a reiser4 plugin, so it might be good
for prototyping.  Faraz might want to look at using that for his
project.

[1] http://lufs.sourceforge.net/lufs/

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[David Masover]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hubert Chan wrote:
|>>>>>"David" == David Masover <ninja@slaphack.com> writes:
|
|
| David> Alexander G. M. Smith wrote:
| Alexander> ...  I (and others) think it would be better for this to be
| Alexander> done at the file system level.  That way you can leverage all
| Alexander> sorts of other utilities and
|
| David> What about a reiser4 plugin?
|
| I would think that would count as being in the filesystem level.

Yet it doesn't involve writing a *new* filesystem, which is I think what
was being suggested?  Alexander?

| David> What about a plugin that is mostly in userspace?
|
| That just reminded me of lufs[1], which allows you to create a userspace
| daemon that communicates with the lufs module to provide access to a

A userspace daemon is what I was thinking, except that lufs is, again,
implementing an entire filesystem rather than piggybacking on one that
already exists.  Or is this new filesystem really so easy?  If so, I
want my stable reiser4!

No offense to anyone working on reiser4.  It looks impressive so far,
and I haven't even taken a peek at the code.  I'm impatient, but I don't
want to seem ungrateful.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQFoq/ngHNmZLgCUhAQLiNg//QOOTYVCMGzvWp7TPpJUMBUw3ty8rN8cT
XzCojUMKnece5O3NT75UHjkiRRKWiZxFxSkvwa7idfzg0fOajdCxdYqxnN3k+c6q
nZyp2cUeLPG/EU4oSIhEjEZYhj5IUENUzALhPftAOveUMc4vN0XvmpAESUZ1ApBp
efKH61wiWn7eq3NdHXMBbWroC9Rp2W1H36Gqe4lBwZFPPSo+XXHmjT2kRY5sjw9x
RA2UcoFiNbsfLiLviFAsfAiHSkCUop06Tub0js3sMAvWNxHPE6Fu6/DOUxmZsKom
tz10eAL+I0oRBy0+iVzwpEZ6aB2ic17cPS9KW9RBwzYWfVavJKrJ+BqZRgL0sARl
dW4A9XkKiyngAufrUli6WDJaihaWZtpP1APpBmM+eDdVQzihuVlL9ZCPaybtuhDn
O8dgwXKSdE2GCKG0lGEweKbow6mXDe8TTypvXzpSDVU4cyvBdUzhisb2HyHlUigx
rH1kJlTQvEmQIWnxkqQQxe5oX98dNnHxJLDtRyoiIN9ATGE0tEWbHZzvO8rGsOA3
wxtRwIr4xeXUKF8I0RN+3JI9rNqH2UNTzNTTIhOleVqO9O50IDJSR++XT0ZJ65Lu
WDctewQ16xm84IobMKqV3c+GVdRIpQjIdaChjnDYaovf+wstcQnzT/y9D58MOOma
+U37zogaPIs=
=k/hJ
-----END PGP SIGNATURE-----

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Alexander G. M. Smith]

David Masover wrote on Thu, 18 Mar 2004 17:04:30 -0600:
> | David> What about a reiser4 plugin?
> |
> | I would think that would count as being in the filesystem level.
> 
> Yet it doesn't involve writing a *new* filesystem, which is I think what
> was being suggested?  Alexander?

I was concerned that new features should also be available to all
programs, preferably through the existing file system API (open,
close, readdir, etc), rather than having a separate API layer (adding
a few functions is OK).  I don't care if it's a new file system under
the hood, or enhancements to an existing one; the important thing is
that all programs be able to use it easily.

For example, you could add a mkquery() to the usual API to make a
virtual directory that contains the results of a query (query
patterns specified as an argument to mkquery).  I'd be satisfied if
other software can open that virtual directory just like a regular
one, and use readdir on it.  Of course, creating a file in the
virtual directory would return an error code (or you could have
that operation automagically add attributes to the new file which
would make it satisfy the query).

The general idea is to avoid having multiple application programming
interfaces.  Keep it unified and you get more power through synergy.
Such as being able to use grep or ffmpeg on all files in a virtual
query results directory.

- Alex

P.S. Normally I wouldn't use "synergy", but it makes sense here.

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Hubert Chan]

>>>>> "David" == David Masover <ninja@slaphack.com> writes:

[...]

David> A userspace daemon is what I was thinking, except that lufs is,
David> again, implementing an entire filesystem rather than piggybacking
David> on one that already exists.

Not sure what you mean by piggybacking on top of an existing filesystem.
It looks like lufs itself is just a link between the kernel and some
userspace code.  It isn't intended to be a completely new filesystem.
From what I can tell, its main purpose is to allow you to mount remote
filesystems, e.g. through ssh.  My guess is that the userspace code just
provides functions to implement things like open, opendir, read, etc.,
which should be easy to translate into shell calls, over ssh.  So it's
definitely simpler than Reiser4.  But I think you would be able to use
it to prototype something like ~/music/by_artist, which just does some
shell and id3 magic to find all the artists in ~/music and display a
nice list.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Hubert Chan]

>>>>> "Isaac" == Isaac Claymore <clay@exavio.com.cn> writes:

[...]

Isaac> Internally, the audio files are stored on a HFS+ filesystem,
Isaac> under a hierarchy structure very weird to us humans. But, by
Isaac> using iPod or the iTunes tool, the files are presented under
Isaac> various hierachy structures organized by file meta data, like
Isaac> artist, album, composer... And, that's far too convenient.

The Neuros does something similar too.  Everything is stored on a FAT
filesystem, but it maintains a database of all the songs.  Of course,
that means that you have to use a special program to add songs, or else
the database won't get updated.  So you can't just drag-and-drop, or cp
the files over, which is why, as Alexander mentioned, it's best to do
this in the filesystem level.

Isaac> I guess this kind of stuff can be done in user space, well and
Isaac> elegantly. Either a specialized application(e.g. iTunes), or some
Isaac> kind of user space filesystem will do. I also think this is best
Isaac> suited for storage of multimedia files, which naturelly come with
Isaac> much meta data.

And document files too.  I'm looking forward to being able to being able
to scrap this strange hierarchy system that I'm currently using for all
my documents.  Email, too, would do well with this system.  Just toss
all the mail in a single folder, and have your MUA query the filesystem
for mails from the ReiserFS list, or mails from friends, etc.

Isaac> And, I fail to figure out what benefit this can do to regular
Isaac> system files, like those under /etc, /boot...

/boot is probably tiny enough that it wouldn't benefit too much.  For
/etc, it might be helpful for those files that are related to both LDAP
and PAM, or both Apache and PAM, or network configuration and bootup, or
printing and hotplugging, or ...  Currently, "ls /etc | wc" on my system
gives 400 files.  I'm sure someone is creative enough to come up with a
solution to that mess.  I agree that /etc probably wouldn't benefit as
much as ~/music would, but I'm sure someone can come up with something
useful.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Isaac Claymore]

[-- Attachment #1: Type: text/plain, Size: 3015 bytes --]

On Tue, Mar 16, 2004 at 11:10:04PM -0500, Hubert Chan wrote:
> >>>>> "Isaac" == Isaac Claymore <clay@exavio.com.cn> writes:
> 
> [...]
> 
> Isaac> Internally, the audio files are stored on a HFS+ filesystem,
> Isaac> under a hierarchy structure very weird to us humans. But, by
> Isaac> using iPod or the iTunes tool, the files are presented under
> Isaac> various hierachy structures organized by file meta data, like
> Isaac> artist, album, composer... And, that's far too convenient.
> 
> The Neuros does something similar too.  Everything is stored on a FAT
> filesystem, but it maintains a database of all the songs.  Of course,
> that means that you have to use a special program to add songs, or else
> the database won't get updated.  So you can't just drag-and-drop, or cp
> the files over, which is why, as Alexander mentioned, it's best to do
> this in the filesystem level.
> 

Thanks for the explanation, and now I see it as to why this'd done at
the fs level. I guess my mind's got numbed by traditional filesystems,
and it needs to be enlightened ;)

I also stumbled upon Gnome Storage, at:

http://www.gnome.org/~seth/storage/index.html

If I get it right, they seem to be doing it in user space, what's your
idea on this?


-Isaac

> Isaac> I guess this kind of stuff can be done in user space, well and
> Isaac> elegantly. Either a specialized application(e.g. iTunes), or some
> Isaac> kind of user space filesystem will do. I also think this is best
> Isaac> suited for storage of multimedia files, which naturelly come with
> Isaac> much meta data.
> 
> And document files too.  I'm looking forward to being able to being able
> to scrap this strange hierarchy system that I'm currently using for all
> my documents.  Email, too, would do well with this system.  Just toss
> all the mail in a single folder, and have your MUA query the filesystem
> for mails from the ReiserFS list, or mails from friends, etc.
> 
> Isaac> And, I fail to figure out what benefit this can do to regular
> Isaac> system files, like those under /etc, /boot...
> 
> /boot is probably tiny enough that it wouldn't benefit too much.  For
> /etc, it might be helpful for those files that are related to both LDAP
> and PAM, or both Apache and PAM, or network configuration and bootup, or
> printing and hotplugging, or ...  Currently, "ls /etc | wc" on my system
> gives 400 files.  I'm sure someone is creative enough to come up with a
> solution to that mess.  I agree that /etc probably wouldn't benefit as
> much as ~/music would, but I'm sure someone can come up with something
> useful.
> 
> -- 
> Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
> PGP/GnuPG key: 1024D/124B61FA
> Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
> Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

-- 

Regards, Isaac
()  ascii ribbon campaign - against html e-mail
/\                        - against microsoft attachments

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Hubert Chan]

>>>>> "Isaac" == Isaac Claymore <clay@exavio.com.cn> writes:

[...]

Isaac> I also stumbled upon Gnome Storage, at:

Isaac> http://www.gnome.org/~seth/storage/index.html

Isaac> If I get it right, they seem to be doing it in user space, what's
Isaac> your idea on this?

Yeah.  I was going to mention that, but I forgot.  Thanks for bringing
it up.  Yes, AFAICT, it's all in userspace.  Since it's done by GNOME
people, I assume that they will make a gnome-vfs module so that all
GNOME applications will be able to use it transparently.  But that
won't help other apps.

It seems to be based on a relational model, which Hans has been saying
is a bad way of organizing user data (and I would agree, but I'm no
database expert, so my opinion doesn't count as much).

It also seems to completely ditch the hierarchy system -- I find that a
hierarchy (or some sort of semi-hierarchy) can be a very useful filing
system for some types of data, or at least in splitting your data into
manageable chunks.

But I haven't dealt with Storage -- I've only briefly looked over it --
so this is all mostly speculation.  It looks like a neat system, but I
think that Reiser6 will blow its socks off.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 2478 bytes --]

On Tue, 16 Mar 2004 23:10:04 EST, Hubert Chan <hubert@uhoreg.ca>  said:
> And document files too.  I'm looking forward to being able to being able
> to scrap this strange hierarchy system that I'm currently using for all
> my documents.  Email, too, would do well with this system.  Just toss
> all the mail in a single folder, and have your MUA query the filesystem
> for mails from the ReiserFS list, or mails from friends, etc.

Ad-hoc query support in the file system (or even in user space) is always a
problematic issue, because there's so many corner cases that result in a DWIM
interface problem.

For example:

If you query your music filesystem for Eric Clapton, should it return "While My
Guitar Gently Weeps" by the Beatles?  If you ask it for "songs written by the
artist Prince", should it return "Manic Monday" by the Bangles (the album
credit says "Christopher")?  The music industry is *full* of that - and queries
like that Just Don't Work unless your metadata is accurate.

Bonus points for being able to handle "music by Metallica before they heaved
Mustaine overboard and he went off to make Megadeth" - what year did he leave,
and are the songs all *accurately* tagged for release dates?

If some idiot in Zanzibar says "ooh shiny" and clicks on an attachment they
shouldn't have, and starts spewing mail to you that has a friend's address in
the From: field, should "mail from friends" find it?

For that matter, how does my MUA know who "friends" are?  I have some people
that would count as "friends" who I correspond with on a much lower frequency
than some idiots that I'd rather never hear from again (but have to deal with
due to various obligations).  Equally problematic is when an old college
classmate drops me a note asking about our supercomputer, as an off-list reply
to something I said on a security mailing list (actually happened recently).
Is that a "security", or "friends", or "supercomputing", or "VT News", or all/
other?  And how does it know, other than simple word-indexing schemes (I
already use 'glimpse', but even that gets painful when your e-mail archive goes
back 15 years and totals over a gigabyte - compound searches take *forever*.

Semantic analysis is a royal pain - I can't expect the computer to be able to
figure out meanings in order to classify them, when *I* can't do it (I have at
least 10 or 15 pieces of mail that require a reply, but I haven't figured out
yet what the fleep the author was talking about..)


[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Hubert Chan]

>>>>> "Valdis" == Valdis Kletnieks <Valdis.Kletnieks@vt.edu> writes:

Valdis> On Tue, 16 Mar 2004 23:10:04 EST, Hubert Chan <hubert@uhoreg.ca>
Valdis> said:
Hubert> And document files too.  I'm looking forward to being able to
Hubert> being able to scrap this strange hierarchy system that I'm
Hubert> currently using for all my documents.  Email, too, would do well
Hubert> with this system.  Just toss all the mail in a single folder,
Hubert> and have your MUA query the filesystem for mails from the
Hubert> ReiserFS list, or mails from friends, etc.

Valdis> Ad-hoc query support in the file system (or even in user space)
Valdis> is always a problematic issue, because there's so many corner
Valdis> cases that result in a DWIM interface problem.

Yes.  When I wrote what I wrote, I was assuming that you have good
metadata that the filesystem can use.  I assume that semantic analysis,
DWIM, mind reading, etc., is out of the scope of the filesystem layer,
but anyone is free to implement a plugin that does what they want.

So when I say "mails from friends," I'm skipping most of the details,
which I think most people don't care about.  I'm really saying that I
would give the filesystem a list of email addresses of people that I
consider "friends," and get the filesystem to find all mails from those
addresses.  If someone from Zanzibar starts flooding me with the
virus-du-jour, then I refine my filter so that it excludes mails that
clamav detects viruses in.  (Actually, I would just configure clamav to
delete all mails that contain viruses.)  If my old college friend drops
me a mail, and I decide I want his mails to show up in my "friends"
query, then I add his email address to the list.  (Or add him to my
address book, mark him as "friend," and let the filesystem do the join.)

(Actually, I don't think I personally would use a "mails from friends"
query.  I would generally just use a "mails to/from person x.")

[...]

Valdis> And how does it know, other than simple word-indexing schemes (I
Valdis> already use 'glimpse', but even that gets painful when your
Valdis> e-mail archive goes back 15 years and totals over a gigabyte -
Valdis> compound searches take *forever*.

Well, Reiser4(? - with the appropriate plugin?) will let you add
arbitrary attributes through the everything-is-a-file-and-a-directory
mechanism, so you can add, for example, a "tags" attribute.  For all the
mails dealing with supercomputing, you add the supercomputing tag.  For
all security mails (that aren't already handled by your
security-mailing-list, etc. filters), you can add the security tag.
Then you tell the filesystem that the "tags" attribute is a handy thing
to index.  Again, I'm assuming that the user provides the filesystem
with useful metadata.  I don't assume that the filesystem can read your
mails and do automatic classification with 100% accuracy.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

On Mon, 22 Mar 2004 16:08:51 EST, Hubert Chan <hubert@uhoreg.ca>  said:

> to index.  Again, I'm assuming that the user provides the filesystem
> with useful metadata.  I don't assume that the filesystem can read your
> mails and do automatic classification with 100% accuracy.

There's 130,797 files in my Mail/ directory.  The only reason I use 'glimpse'
on it is because although MH provides 'sequences' to do some basic metadata
tagging, it requires me to add the metadata tags.  For instance, the real-life
note I mentioned would have to be added to the 'supercomputing', 'security,
'friends' and 'news' sequences.

Ask yourself how useful Google would be if the owner of the webpage had to
provide metadata to tell it how to classify it - in fact, "metadata/keyword
stuffing" turned out to be a major problem for Google.



[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Hubert Chan]

>>>>> "Valdis" == Valdis Kletnieks <Valdis.Kletnieks@vt.edu> writes:

Valdis> There's 130,797 files in my Mail/ directory.  The only reason I
Valdis> use 'glimpse' on it is because although MH provides 'sequences'
Valdis> to do some basic metadata tagging, it requires me to add the
Valdis> metadata tags.

You can always use something like ifile, or more static filters
(regexps, header values, etc.), to do automatic tagging, and just
manually tag when it gets things wrong.  (If you already have your mails
tagged/organized into directories, it should be fairly trivial to do the
conversion.)  If you really want something like glimpse functionality,
you can always create/get a plugin to do that.  It might not be fast,
but I can't imagine that it would be any slower than just using glimpse.
And it definitely won't be any worse (functionality-wise) than just
using glimpse.  That way you can do "cd ~/Mail/[glimpse query]", and see
all the mails that match.

Valdis> For instance, the real-life note I mentioned would have to be
Valdis> added to the 'supercomputing', 'security, 'friends' and 'news'
Valdis> sequences.

Yes, if you tag manually, but I assume most of those would be caught by
the automatic tagger.  Or you don't have to use tags, and use a
filesystem plugin just like you would use glimpse.

If you put everything in ~/Mail in a flat directory structure, I don't
think you'll be losing anything.  If you use procmail to put things in
folders, you can still use procmail to do automatic tagging, or some
similar scheme.  If you do simple word indexing, you can still do word
indexing through the filesystem.  I don't really see anything where you
would be losing functionality.

(And, of course, if you want to keep your old filing system, you're
free to do that as well.  Reiser6 isn't going to force you to do
anything you don't want to.)

Valdis> Ask yourself how useful Google would be if the owner of the
Valdis> webpage had to provide metadata to tell it how to classify it -
Valdis> in fact, "metadata/keyword stuffing" turned out to be a major
Valdis> problem for Google.

Well yes, because people lied about their keywords, so that they would
show up more.  I'm assuming that a user isn't going to lie about their
own keywords.

Yes, Google is useful because it can grep a page and try to figure out
what it's about.  But I'm not expecting that Google will be 100%
accurate.  If you want Google-like functionality, you can create a
plugin.  I don't think that Namesys is planning on creating their own
indexer, and assuming that it's going to work for absolutely everyone.
But I think this is out of the scope of what the original thread was
about, at least as far as I can tell.

P.S. no need to cc me.  I read the list.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[faraz ahmed]

Hi Isaac;
         The user space implementation u are talking
about has to be done for each file type seperately.
Example the Winamp playlist can only organize mp3's
wherelese our FS can aggregate any type of data
becuase  each file has attribs (name,value pairs) .
And the directory listing is done acc Relational
queryies on these attributes.
          Anyways the Winamp playlist can only be used
by winamp & other application have no way to use the
files organized in a playlist. Where as in our case
the 
aggregation is done by the file system, so all apps 
can use the organization without change.
          Exactly we do our processing in the user
space & just display the results under the FS.(Micro
kernel sorts of)
                                Regards.
                                 Faraz.

--- Isaac Claymore <clay@exavio.com.cn> wrote:
> Hi,
> 
> I'm new to this idea, and am no fs expert. But it
> looks to me very
> much like what Apple is doing on their iPods, I
> happen to have just
> bought one iPod last weekend.
> 
> Internally, the audio files are stored on a HFS+
> filesystem, under a 
> hierarchy structure very weird to us humans. But, by
> using iPod or the
> iTunes tool, the files are presented under various
> hierachy structures
> organized by file meta data, like artist, album,
> composer... And, that's
> far too convenient.
> 
> I guess this kind of stuff can be done in user
> space, well and
> elegantly. Either a specialized application(e.g.
> iTunes), or some kind
> of user space filesystem will do. I also think this
> is best suited
> for storage of multimedia files, which naturelly
> come with much meta
> data. And, I fail to figure out what benefit this
> can do to regular
> system files, like those under /etc, /boot...
> 
> HTH~~
> 
> 
> -Isaac
> 
> On Tue, Mar 16, 2004 at 02:45:23PM -0800, faraz
> ahmed wrote:
> > Hi EveryBody;
> > Iam an final year engneering student and as part
> of
> > our final year project we have implemented a
> > filesysten which  
> > support more than 1 Directory Structures for the
> same
> > set of files. Each Directory Structure is a XML
> file
> > discribing the directory structure as well as the
> > attributes of the Files to be listed under Each
> > directory. The file system the mounts this XML
> file &
> > produces the directory listing by querying the FS
> for
> > files which match the criteria. The Project will
> soon
> > be made public under GPL. Following is the
> abstract.
> > Please guide me with your valuable suggestions.
> > 
> > 
> > //==============================================
> > 1. Idea of our project:
> > //==============================================
> > The aim of this project is to implement a solution
> > that provides file-level host-based virtualization
> > that provides for better aggregation of
> > content/information based on semantics and
> properties.
> > File-system organization today very closely
> mirrors
> > storage paradigms rather than user-access
> paradigms
> > and semantic grouping. All file-system hierarchies
> are
> > containers that are expressed based on their
> physical
> > presence (a separate drive letter on Windows, or a
> > particular mount point based on the volume in
> Unix). 
> > We wish to implement a solution that will allow
> users
> > to organize their files based on their
> convenience. We
> > define this convenience in the following forms: 
> > 
> > The ability to organize the namespace based on
> certain
> > attribute properties (file-system metadata
> > virtualization). 
> > Ex Directory Listing as an output of high level
> > relational query based on file attributes.
> > 
> > The ability to de-link position of a file in the
> > hierarchy from its actual storage (file metadata
> > virtualization) 
> > Ex Directory Listing as an output of high level
> > relational query based on file attributes.
> > 
> > The ability to de-link position of a file in the
> > hierarchy from its actual storage (file metadata
> > virtualization) 
> > Ex Automatically Listing the files with the proper
> > attributes in the respective directories. Create &
> > forget file system now puts your MP3 file in the
> right
> > folder no matter where they are stored on the
> disk.
> > 
> > The ability to create and manipulate namespaces
> using
> > well-known metaphors (XML schema descriptions and
> > schema editors) 
> > Ex. Define the directory structure with directory
> > query as an XML file. Mount multiple such
> directory
> > structures (virtualization). There by allowing
> > multiple organizational views for the same set of
> > files.
> > 
> > The ability to continue using the standard
> metaphors
> > for manipulation and access to information
> > (file-system kernel API?s), thus maintaining
> current
> > large body of applications unbroken) .
> > Ex. Make all this features available as an
> standard
> > file system so existing apps wont fail
> > 
> > __________________________________
> > Do you Yahoo!?
> > Yahoo! Mail - More reliable, more storage, less
> spam
> > http://mail.yahoo.com
> 
> -- 
> 
> Regards, Isaac
> ()  ascii ribbon campaign - against html e-mail
> /\                        - against microsoft
> attachments
> 

> ATTACHMENT part 2 application/pgp-signature
name=signature.asc



__________________________________
Do you Yahoo!?
Yahoo! Mail - More reliable, more storage, less spam
http://mail.yahoo.com

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Hans Reiser]

Namesys is going to implement a semi-structured data  query language in 
the fs for version 6.  You should take a look at it at 
www.namesys.com/whitepaper.html, as there are definite reasons to go 
beyond the relational model.



faraz ahmed wrote:

>Hi Isaac;
>         The user space implementation u are talking
>about has to be done for each file type seperately.
>Example the Winamp playlist can only organize mp3's
>wherelese our FS can aggregate any type of data
>becuase  each file has attribs (name,value pairs) .
>And the directory listing is done acc Relational
>queryies on these attributes.
>          Anyways the Winamp playlist can only be used
>by winamp & other application have no way to use the
>files organized in a playlist. Where as in our case
>the 
>aggregation is done by the file system, so all apps 
>can use the organization without change.
>          Exactly we do our processing in the user
>space & just display the results under the FS.(Micro
>kernel sorts of)
>                                Regards.
>                                 Faraz.
>
>  
>


-- 
Hans

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Nishant Sharma]

[-- Attachment #1: Type: text/plain, Size: 6344 bytes --]

faraz ahmed wrote:

>Hi Isaac;
>         The user space implementation u are talking
>about has to be done for each file type seperately.
>Example the Winamp playlist can only organize mp3's
>wherelese our FS can aggregate any type of data
>becuase  each file has attribs (name,value pairs) .
>And the directory listing is done acc Relational
>queryies on these attributes.
>          Anyways the Winamp playlist can only be used
>by winamp & other application have no way to use the
>files organized in a playlist. Where as in our case
>the 
>aggregation is done by the file system, so all apps 
>can use the organization without change.
>
    I wonder if it is possible to put this "plugin infrastructure" in 
the user space rather than kernel
space, by putting it in the glibc implementation. That way, every 
application shall be able to access the
files' metadata information in a transparent manner, and we also get all 
the advantages of the user space.

>          Exactly we do our processing in the user
>space & just display the results under the FS.(Micro
>kernel sorts of)
>                                Regards.
>                                 Faraz.
>
>--- Isaac Claymore <clay@exavio.com.cn> wrote:
>  
>
>>Hi,
>>
>>I'm new to this idea, and am no fs expert. But it
>>looks to me very
>>much like what Apple is doing on their iPods, I
>>happen to have just
>>bought one iPod last weekend.
>>
>>Internally, the audio files are stored on a HFS+
>>filesystem, under a 
>>hierarchy structure very weird to us humans. But, by
>>using iPod or the
>>iTunes tool, the files are presented under various
>>hierachy structures
>>organized by file meta data, like artist, album,
>>composer... And, that's
>>far too convenient.
>>
>>I guess this kind of stuff can be done in user
>>space, well and
>>elegantly. Either a specialized application(e.g.
>>iTunes), or some kind
>>of user space filesystem will do. I also think this
>>is best suited
>>for storage of multimedia files, which naturelly
>>come with much meta
>>data. And, I fail to figure out what benefit this
>>can do to regular
>>system files, like those under /etc, /boot...
>>
>>HTH~~
>>
>>
>>-Isaac
>>
>>On Tue, Mar 16, 2004 at 02:45:23PM -0800, faraz
>>ahmed wrote:
>>    
>>
>>>Hi EveryBody;
>>>Iam an final year engneering student and as part
>>>      
>>>
>>of
>>    
>>
>>>our final year project we have implemented a
>>>filesysten which  
>>>support more than 1 Directory Structures for the
>>>      
>>>
>>same
>>    
>>
>>>set of files. Each Directory Structure is a XML
>>>      
>>>
>>file
>>    
>>
>>>discribing the directory structure as well as the
>>>attributes of the Files to be listed under Each
>>>directory. The file system the mounts this XML
>>>      
>>>
>>file &
>>    
>>
>>>produces the directory listing by querying the FS
>>>      
>>>
>>for
>>    
>>
>>>files which match the criteria. The Project will
>>>      
>>>
>>soon
>>    
>>
>>>be made public under GPL. Following is the
>>>      
>>>
>>abstract.
>>    
>>
>>>Please guide me with your valuable suggestions.
>>>
>>>
>>>//==============================================
>>>1. Idea of our project:
>>>//==============================================
>>>The aim of this project is to implement a solution
>>>that provides file-level host-based virtualization
>>>that provides for better aggregation of
>>>content/information based on semantics and
>>>      
>>>
>>properties.
>>    
>>
>>>File-system organization today very closely
>>>      
>>>
>>mirrors
>>    
>>
>>>storage paradigms rather than user-access
>>>      
>>>
>>paradigms
>>    
>>
>>>and semantic grouping. All file-system hierarchies
>>>      
>>>
>>are
>>    
>>
>>>containers that are expressed based on their
>>>      
>>>
>>physical
>>    
>>
>>>presence (a separate drive letter on Windows, or a
>>>particular mount point based on the volume in
>>>      
>>>
>>Unix). 
>>    
>>
>>>We wish to implement a solution that will allow
>>>      
>>>
>>users
>>    
>>
>>>to organize their files based on their
>>>      
>>>
>>convenience. We
>>    
>>
>>>define this convenience in the following forms: 
>>>
>>>The ability to organize the namespace based on
>>>      
>>>
>>certain
>>    
>>
>>>attribute properties (file-system metadata
>>>virtualization). 
>>>Ex Directory Listing as an output of high level
>>>relational query based on file attributes.
>>>
>>>The ability to de-link position of a file in the
>>>hierarchy from its actual storage (file metadata
>>>virtualization) 
>>>Ex Directory Listing as an output of high level
>>>relational query based on file attributes.
>>>
>>>The ability to de-link position of a file in the
>>>hierarchy from its actual storage (file metadata
>>>virtualization) 
>>>Ex Automatically Listing the files with the proper
>>>attributes in the respective directories. Create &
>>>forget file system now puts your MP3 file in the
>>>      
>>>
>>right
>>    
>>
>>>folder no matter where they are stored on the
>>>      
>>>
>>disk.
>>    
>>
>>>The ability to create and manipulate namespaces
>>>      
>>>
>>using
>>    
>>
>>>well-known metaphors (XML schema descriptions and
>>>schema editors) 
>>>Ex. Define the directory structure with directory
>>>query as an XML file. Mount multiple such
>>>      
>>>
>>directory
>>    
>>
>>>structures (virtualization). There by allowing
>>>multiple organizational views for the same set of
>>>files.
>>>
>>>The ability to continue using the standard
>>>      
>>>
>>metaphors
>>    
>>
>>>for manipulation and access to information
>>>(file-system kernel API?s), thus maintaining
>>>      
>>>
>>current
>>    
>>
>>>large body of applications unbroken) .
>>>Ex. Make all this features available as an
>>>      
>>>
>>standard
>>    
>>
>>>file system so existing apps wont fail
>>>
>>>__________________________________
>>>Do you Yahoo!?
>>>Yahoo! Mail - More reliable, more storage, less
>>>      
>>>
>>spam
>>    
>>
>>>http://mail.yahoo.com
>>>      
>>>
>>-- 
>>
>>Regards, Isaac
>>()  ascii ribbon campaign - against html e-mail
>>/\                        - against microsoft
>>attachments
>>
>>    
>>
>
>  
>
>>ATTACHMENT part 2 application/pgp-signature
>>    
>>
>name=signature.asc
>
>
>
>__________________________________
>Do you Yahoo!?
>Yahoo! Mail - More reliable, more storage, less spam
>http://mail.yahoo.com
>  
>


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3222 bytes --]

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Hubert Chan]

>>>>> "Nishant" == Nishant Sharma <nishantt@cse.iitd.ernet.in> writes:

Nishant>     I wonder if it is possible to put this "plugin
Nishant> infrastructure" in the user space rather than kernel space, by
Nishant> putting it in the glibc implementation. ...

I've thought about this too, but I think it would be harder to put this
in glibc.  Something like gnome-vfs is able to use plugins easily,
because it uses a URL-like syntax, and so it can tell what plugin to use
based on the format of the filename.  But for the type of stuff we
would want to do, glibc would have to make a bunch of kernel calls to
determine up to when, in the filename, the kernel can handle, and when
it has to start doing its processing.  So if you requested the file:
/home/hubert/music/foo.mp3/..albumcover/cover.jpg, I imagine the
sequence of events would have to be something like:
- glibc asks the kernel for /home/.../cover.jpg
- kernel returns "file not found"
- glibc asks the kernel for /home/.../..albumcover
- kernel returns "file not found"
- glibc asks the kernel for /home/.../foo.mp3
- kernel returns file handle
- glibc does its processing to extract the album cover.

-- 
Hubert Chan <hubert@uhoreg.ca> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

Re: ReConfigurable Directory Structure & Agrregation of files according to semantic.

[Nishant Sharma]

[-- Attachment #1: Type: text/plain, Size: 1478 bytes --]

Hubert Chan wrote:

>>>>>>"Nishant" == Nishant Sharma <nishantt@cse.iitd.ernet.in> writes:
>>>>>>            
>>>>>>
>
>Nishant>     I wonder if it is possible to put this "plugin
>Nishant> infrastructure" in the user space rather than kernel space, by
>Nishant> putting it in the glibc implementation. ...
>
>I've thought about this too, but I think it would be harder to put this
>in glibc.  Something like gnome-vfs is able to use plugins easily,
>because it uses a URL-like syntax, and so it can tell what plugin to use
>based on the format of the filename.  But for the type of stuff we
>would want to do, glibc would have to make a bunch of kernel calls to
>determine up to when, in the filename, the kernel can handle, and when
>it has to start doing its processing.  So if you requested the file:
>/home/hubert/music/foo.mp3/..albumcover/cover.jpg, I imagine the
>sequence of events would have to be something like:
>- glibc asks the kernel for /home/.../cover.jpg
>- kernel returns "file not found"
>- glibc asks the kernel for /home/.../..albumcover
>- kernel returns "file not found"
>- glibc asks the kernel for /home/.../foo.mp3
>- kernel returns file handle
>- glibc does its processing to extract the album cover.
>
    This particular problem can be solved by having a system call, 
which, when queried about
"/home/hubert/music/foo.mp3/..albumcover/cover.jpg", always gives the 
maximal path-match found, in
particular, "/home/hubert/music/foo.mp3/".

>
>  
>


[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/x-pkcs7-signature, Size: 3222 bytes --]