Multiple data streams...

Multiple data streams...

[Anders Widman]

   Is this supported, or will it be supported by ReiserFS?
   I  use  this  feature quite quite much.. Maybe this is something to
   add to ReiserFS?

   There is very brief info at Microsoft's website:

   http://www.microsoft.com/windows2000/techinfo/reskit/en-us/core/fncc_fil_khzt.asp



   

--------
PGP public key: https://tnonline.net/secure/pgp_key.txt

Re: Multiple data streams...

[Anders Widman]

>    Is this supported, or will it be supported by ReiserFS?
>    I  use  this  feature quite quite much.. Maybe this is something to
>    add to ReiserFS?

>    There is very brief info at Microsoft's website:

>   
> http://www.microsoft.com/windows2000/techinfo/reskit/en-us/core/fncc_fil_khzt.asp


   Some uses could be to be able to store locking information (such as
   user,  program  etc)  or extended security features within the same
   file as a separate data stream.

   There  are  of  course  numerous other things data streams could be
   used for.

   //Anders

--------
PGP public key: https://tnonline.net/secure/pgp_key.txt

Re: Multiple data streams...

[Russell Coker]

On Wed, 2 Apr 2003 19:28, Anders Widman wrote:
>    Is this supported, or will it be supported by ReiserFS?
>    I  use  this  feature quite quite much.. Maybe this is something to
>    add to ReiserFS?
>
>    There is very brief info at Microsoft's website:

Does MS support multiple data streams properly yet?

The last information I read about data streams indicated that commonly used 
operations (such as typing "copy" in a command-line session) would not work 
properly with multiple data streams.  Also many important programs such as 
anti-virus programs did not support it.

The Linux API does not support multiple data streams.  Support for extended 
attributes has recently been added, this will do most things that you might 
want to do with multiple streams and does it in a standard way that is 
supported.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: Multiple data streams...

[Anders Widman]

> On Wed, 2 Apr 2003 19:28, Anders Widman wrote:
>>    Is this supported, or will it be supported by ReiserFS?
>>    I  use  this  feature quite quite much.. Maybe this is something to
>>    add to ReiserFS?
>>
>>    There is very brief info at Microsoft's website:

> Does MS support multiple data streams properly yet?

  Yes.  Since  Windows  2000.  It  is also integrated in filemanager and
  programs like Microsoft Word.

  Easy   test:  right-click  any  file  on  a  NTFS  volume  and  choose
  "summary". That information is stored in a data stream.

  Another test you could to from a prompt:

     echo test>test.txt
     echo hello>test.txt:h
     echo bye>test.txt:b

     more <test.txt
     more <test.txt:h
     more <test.txt:b

   And  yes,  the  data  streams are preserved with copy/xcopy and normal
   copy/move  operations in explorer - even over SMB connections to other
   NTFS drives.

> The last information I read about data streams indicated that commonly used
> operations (such as typing "copy" in a command-line session) would not work
> properly with multiple data streams.  Also many important programs such as
> anti-virus programs did not support it.

> The Linux API does not support multiple data streams.  Support for extended
> attributes has recently been added, this will do most things that you might
> want to do with multiple streams and does it in a standard way that is
> supported.

  Could you give an example of this? I use multiple streams for adding
  thumbnails  or  text files  to pictures I am working on... - just to
  give an example.

  //Anders




--------
PGP public key: https://tnonline.net/secure/pgp_key.txt

Re: Multiple data streams...

[Oleg Drokin]

Hello!

On Wed, Apr 02, 2003 at 10:40:57PM +1000, Russell Coker wrote:

> The Linux API does not support multiple data streams.  Support for extended 
> attributes has recently been added, this will do most things that you might 
> want to do with multiple streams and does it in a standard way that is 
> supported.

Current API in the form I saw, greatly limits its usage.
E.g. get_attribute(name) kind of function can only return whole attribute
at a time. Also current ext3 realisation limits attributes size to be
not bigger than one block. I have not looked at XFS realisation yet.

Bye,
    Oleg

Re: Multiple data streams...

[Alexander G. M. Smith]

Russell Coker wrote on Wed, 2 Apr 2003 22:40:57 +1000:
> The Linux API does not support multiple data streams.  Support for extended
> attributes has recently been added, this will do most things that you might
> want to do with multiple streams and does it in a standard way that is
> supported.

In case people don't know, extra streams are used extensively in BeOS (for icons, e-mail attributes (subject, sender, date, etc), MIME types of files
and lots of other things).  Sometimes I wish their was a ReiserFS
implementation for BeOS so that performance would improve (small streams
are stored in the inode, but once that overflows, things get slow and
it uses at least two disk blocks for even the smallest stream data).

- Alex

Re: Multiple data streams...

[Jeff Mahoney]

[-- Attachment #1: Type: text/plain, Size: 2038 bytes --]

Anders Widman wrote:
>    Is this supported, or will it be supported by ReiserFS?
>    I  use  this  feature quite quite much.. Maybe this is something to
>    add to ReiserFS?
> 
>    There is very brief info at Microsoft's website:
> 
>    http://www.microsoft.com/windows2000/techinfo/reskit/en-us/core/fncc_fil_khzt.asp

	I have an implementation of extended attributes that has been included 
in the SuSE kernel for some time now. It's not at all suitable for 
filestreams with a lot of data, as the Linux extended attribute 
interface requires that the entire attribute be read and written in one 
chunk. However, for small things, such as mime type, email sender, and 
perhaps even small thumbprints, this wouldn't be a problem. The sizes of 
xattrs in my implementation aren't limited in size, as they are 
implemented as regular files in a hidden tree. [*]

	The patches are against the SuSE kernel at the moment, but if there's 
enough interest, I could split them out to depend on the generic  ACL/EA 
patches from Andreas Gruenbacher and Chris Mason's data logging and 
quota patches.

	I suppose that since the xattr files are normal files, it would be 
pretty easy to implement a "datastream" xattr that could be accessed 
using normal tools. The semantics could be similar to the windows 
implementation, e.g. /foo/bar/file:datastream1 (Lookup the file verbatim 
first, then try datastreams) However, since most of the userspace tools 
are unaware of xattrs, the same restrictions that apply for any other 
xattr apply: they'll be lost if you copy/backup with tools that don't 
know about them.

	-Jeff

[*] My design requirements included not forking the filesystem in an 
incompatible manner, this was the best way I came up with. If a 
filesystem with xattrs on it is mounted with them disabled or by a 
kernel that doesn't support them, the xattrs appear in a tree owned by 
root (mode 700), with all the files owned by the user owning the file 
that the xattr is associated with.

-- 
jeffm@suse.com
jeffm@csh.rit.edu

[-- Attachment #2: Type: application/pgp-signature, Size: 252 bytes --]

Re: Multiple data streams...

[Hans Reiser]

Anders Widman wrote:

>>   Is this supported, or will it be supported by ReiserFS?
>>   I  use  this  feature quite quite much.. Maybe this is something to
>>   add to ReiserFS?
>>    
>>
>
>  
>
>>   There is very brief info at Microsoft's website:
>>    
>>
>
>  
>
>>  
>>http://www.microsoft.com/windows2000/techinfo/reskit/en-us/core/fncc_fil_khzt.asp
>>    
>>
>
>
>   Some uses could be to be able to store locking information (such as
>   user,  program  etc)  or extended security features within the same
>   file as a separate data stream.
>
>   There  are  of  course  numerous other things data streams could be
>   used for.
>
>   //Anders
>
>--------
>PGP public key: https://tnonline.net/secure/pgp_key.txt
>
>
>
>  
>
Maybe this belongs in the faq....

read www.namesys.com/v4/v4.html where we describe how directories and 
files can be made to do all of this better.

-- 
Hans

Re: Multiple data streams...

[Hans Reiser]

Users should know that these will never be accepted into ReiserFS, and 
the reasons why are explained at www.namesys.com/v4/v4.html



Jeff Mahoney wrote:

> Anders Widman wrote:
>
>>    Is this supported, or will it be supported by ReiserFS?
>>    I  use  this  feature quite quite much.. Maybe this is something to
>>    add to ReiserFS?
>>
>>    There is very brief info at Microsoft's website:
>>
>>    
>> http://www.microsoft.com/windows2000/techinfo/reskit/en-us/core/fncc_fil_khzt.asp
>
>
>     I have an implementation of extended attributes that has been 
> included in the SuSE kernel for some time now. It's not at all 
> suitable for filestreams with a lot of data, as the Linux extended 
> attribute interface requires that the entire attribute be read and 
> written in one chunk. However, for small things, such as mime type, 
> email sender, and perhaps even small thumbprints, this wouldn't be a 
> problem. The sizes of xattrs in my implementation aren't limited in 
> size, as they are implemented as regular files in a hidden tree. [*]
>
>     The patches are against the SuSE kernel at the moment, but if 
> there's enough interest, I could split them out to depend on the 
> generic  ACL/EA patches from Andreas Gruenbacher and Chris Mason's 
> data logging and quota patches.
>
>     I suppose that since the xattr files are normal files, it would be 
> pretty easy to implement a "datastream" xattr that could be accessed 
> using normal tools. The semantics could be similar to the windows 
> implementation, e.g. /foo/bar/file:datastream1 (Lookup the file 
> verbatim first, then try datastreams) However, since most of the 
> userspace tools are unaware of xattrs, the same restrictions that 
> apply for any other xattr apply: they'll be lost if you copy/backup 
> with tools that don't know about them.
>
>     -Jeff
>
> [*] My design requirements included not forking the filesystem in an 
> incompatible manner, this was the best way I came up with. If a 
> filesystem with xattrs on it is mounted with them disabled or by a 
> kernel that doesn't support them, the xattrs appear in a tree owned by 
> root (mode 700), with all the files owned by the user owning the file 
> that the xattr is associated with.
>


-- 
Hans

Re: Multiple data streams...

[Chris Mason]

On Wed, 2003-04-02 at 11:53, Hans Reiser wrote:
> Users should know that these will never be accepted into ReiserFS, and 
> the reasons why are explained at www.namesys.com/v4/v4.html

They should also know that suse has committed to supporting the ACL and
xattr patches.  It's an important feature to our customers, so we more
or less had to choose between making something else the default and
giving reiserfs the features needed for it to compete in the enterprise
market.

We chose the latter ;-)

-chris

Re: Multiple data streams...

[Nikita Danilov]

Chris Mason writes:
 > On Wed, 2003-04-02 at 11:53, Hans Reiser wrote:
 > > Users should know that these will never be accepted into ReiserFS, and 
 > > the reasons why are explained at www.namesys.com/v4/v4.html
 > 
 > They should also know that suse has committed to supporting the ACL and
 > xattr patches.  It's an important feature to our customers, so we more
 > or less had to choose between making something else the default and
 > giving reiserfs the features needed for it to compete in the enterprise
 > market.
 > 
 > We chose the latter ;-)
 > 

Actually, locking ACLs out of mainline gives SuSe competitive advantage
over, say, some other distribution that would like to use Reiserfs. In
such situation that "other distribution" would probably opt for not
using reiserfs at all.

 > -chris
 > 

Nikita.

 > 
 > 
 > 

Re: Multiple data streams...

[Russell Coker]

On Wed, 2 Apr 2003 23:08, Anders Widman wrote:
> > The Linux API does not support multiple data streams.  Support for
> > extended attributes has recently been added, this will do most things
> > that you might want to do with multiple streams and does it in a standard
> > way that is supported.
>
>   Could you give an example of this? I use multiple streams for adding
>   thumbnails  or  text files  to pictures I am working on... - just to
>   give an example.

OS/2 used EA's for icons and description text in 1992.  There's no reason why 
EAs can't be used for this purpose today as well.

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: Multiple data streams...

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 446 bytes --]

On Wed, 02 Apr 2003 11:47:12 +0200, Anders Widman <andewid@tnonline.net>  said:

>    Some uses could be to be able to store locking information (such as
>    user,  program  etc)  or extended security features within the same
>    file as a separate data stream.
> 
>    There  are  of  course  numerous other things data streams could be
>    used for.

"It's deja vu all over again" -- Yogi Berra

MacOS had resource and data forks *WHEN*? ;)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: Multiple data streams...

[Hans Reiser]

Nikita Danilov wrote:

>Chris Mason writes:
> > On Wed, 2003-04-02 at 11:53, Hans Reiser wrote:
> > > Users should know that these will never be accepted into ReiserFS, and 
> > > the reasons why are explained at www.namesys.com/v4/v4.html
> > 
> > They should also know that suse has committed to supporting the ACL and
> > xattr patches.  It's an important feature to our customers, so we more
> > or less had to choose between making something else the default and
> > giving reiserfs the features needed for it to compete in the enterprise
> > market.
> > 
> > We chose the latter ;-)
> > 
>
>Actually, locking ACLs out of mainline gives SuSe competitive advantage
>over, say, some other distribution that would like to use Reiserfs. In
>such situation that "other distribution" would probably opt for not
>using reiserfs at all.
>
>
> > -chris
> > 
>
>Nikita.
>
> > 
> > 
> > 
>
>
>
>  
>
and if we introduce a non-compatible version of ACLs into the mainline....

There are the Unix wannabes and those who want to build something 
better.  There aren't actually a lot who want to build something better 
than what has passed before.... but then there never is.....

Reiser4 is not going to adopt the wait until some other OS has used a 
feature for 5 years, and then adopt it approach that Ted Tso suggested 
on one occasion (I think we were discussing upcoming reiser4 features;-) ).

This is a natural friction between those who want Linux to quickly 
become like Unix, and those who think Unix is not necessarily all that 
we could be.  I suspect that as Linux matures this is going to become 
more and more of an issue, because those who are able to copy features 
effectively are not those who are able to add new ones, but those who 
copy will be entrenched in the power structure.  It will be much like 
the competition between slow growth and fast growth vegetation in the 
forest: both approaches have their place in the ecosystem and both seek 
to displace the other.

What we should all remember though is that everyone is trying to make 
Linux better as best as they can, and so let us not make a big deal out 
of this ACL fork because little positive will come out of it (please 
take note of this point Nikita).

-- 
Hans

Re: Multiple data streams...

[Hans Reiser]

Russell Coker wrote:

>On Wed, 2 Apr 2003 23:08, Anders Widman wrote:
>  
>
>>>The Linux API does not support multiple data streams.  Support for
>>>extended attributes has recently been added, this will do most things
>>>that you might want to do with multiple streams and does it in a standard
>>>way that is supported.
>>>      
>>>
>>  Could you give an example of this? I use multiple streams for adding
>>  thumbnails  or  text files  to pictures I am working on... - just to
>>  give an example.
>>    
>>
>
>OS/2 used EA's for icons and description text in 1992.  There's no reason why 
>EAs can't be used for this purpose today as well.
>
>  
>
Implement inherited stat data, hidden directory entries, and files and 
directories and you don't need EAs.... yes?

-- 
Hans

Re: Multiple data streams...

[Chris Mason]

On Thu, 2003-04-03 at 13:37, Hans Reiser wrote:

> and if we introduce a non-compatible version of ACLs into the mainline....
> 

Grin, the sentence above is more than a little fuzzy.  The v3 ACLs in
the suse kernel are compatible with the acl standards used by the other
linux filesystems.  Any acls in v4 probably want to provide the same
interface, given that many applications in linux do or will support it.

> There are the Unix wannabes and those who want to build something 
> better.  There aren't actually a lot who want to build something better 
> than what has passed before.... but then there never is.....
> 

Shrug, acls as they are is better than no acls.  It's great that you
want to enhance and extend filesystems, their interfaces and performance
over all.  But while that is going on we need to make sure reiserfs
keeps up with the important features the other linux filesystems are
already including.

> What we should all remember though is that everyone is trying to make 
> Linux better as best as they can, and so let us not make a big deal out 
> of this ACL fork because little positive will come out of it (please 
> take note of this point Nikita).

I've no desire to create friction with the ACL fork at all ;-)  It's a
relatively small change that adds a cool feature without altering the
disk format, in other words it was designed to be non-intrusive.

I know that you were concerned about the v3 acl support being a
compatibility problem in v4, but given the acls/xattrs can easily be
exported to a neutral format and then reimported into whatever v4
supports, I don't think it will be a problem.  If it is, we'll work with
you to help fix it.

But none of that is the crucial issue ;-) Your previous message implied
the v3 acl patch was dead and I wanted to assure the people SuSE has
sold it to that we will continue to support it (and them).

-chris

Re: Multiple data streams...

[Hans Reiser]

Chris Mason wrote:

>
>I know that you were concerned about the v3 acl support being a
>compatibility problem in v4, but given the acls/xattrs can easily be
>exported to a neutral format and then reimported into whatever v4
>supports, I don't think it will be a problem.  If it is, we'll work with
>you to help fix it.
>
>
>  
>
We will both work to ensure problems are kept small in this area, and I 
think it is best to spend our time coding rather than worrying about 
this when we are not at any decision point in the process.   Hopefully 
it will not be substantive as a problem.

-- 
Hans

Re: Multiple data streams...

[Russell Coker]

On Fri, 4 Apr 2003 05:00, Hans Reiser wrote:
> >OS/2 used EA's for icons and description text in 1992.  There's no reason
> > why EAs can't be used for this purpose today as well.
>
> Implement inherited stat data, hidden directory entries, and files and
> directories and you don't need EAs.... yes?

That sounds like EAs the hard way...

-- 
http://www.coker.com.au/selinux/   My NSA Security Enhanced Linux packages
http://www.coker.com.au/bonnie++/  Bonnie++ hard drive benchmark
http://www.coker.com.au/postal/    Postal SMTP/POP benchmark
http://www.coker.com.au/~russell/  My home page

Re: Multiple data streams...

[Chris Mason]

On Thu, 2003-04-03 at 14:50, Hans Reiser wrote:
> Chris Mason wrote:
> 
> >
> >I know that you were concerned about the v3 acl support being a
> >compatibility problem in v4, but given the acls/xattrs can easily be
> >exported to a neutral format and then reimported into whatever v4
> >supports, I don't think it will be a problem.  If it is, we'll work with
> >you to help fix it.
> >
> >
> >  
> >
> We will both work to ensure problems are kept small in this area, and I 
> think it is best to spend our time coding rather than worrying about 
> this when we are not at any decision point in the process.   Hopefully 
> it will not be substantive as a problem.

Agreed ;-)  

-chris

Re: Multiple data streams...

[Hans Reiser]

Russell Coker wrote:

>On Fri, 4 Apr 2003 05:00, Hans Reiser wrote:
>  
>
>>>OS/2 used EA's for icons and description text in 1992.  There's no reason
>>>why EAs can't be used for this purpose today as well.
>>>      
>>>
>>Implement inherited stat data, hidden directory entries, and files and
>>directories and you don't need EAs.... yes?
>>    
>>
>
>That sounds like EAs the hard way...
>
>  
>
It is the difference between architecting and incrementally evolving in 
response to each separate feature request.

Sharing stat data and hiding directory entries are useful for a lot more 
than streams.

Atomic transactions are also useful for a lot more than EAs.....

I work by carefully defining easily recombinable primitives.  Others 
work by asking what does Unix have and copying it.

This is why it took far longer for us in the beginning, and why over the 
next 30 years we will develop faster than other filesystems.  That is 
why ext2 currently has more market share --- a lot of people benefited 
from them getting something out quick that was usable...

-- 
Hans

Re: Multiple data streams...

[Nikita Danilov]

Hans Reiser writes:
 > Nikita Danilov wrote:
 > 

[...]

 > 
 > What we should all remember though is that everyone is trying to make 
 > Linux better as best as they can, and so let us not make a big deal out 
 > of this ACL fork because little positive will come out of it (please 
 > take note of this point Nikita).

ok, ok. The rest will be silence.

 > 
 > -- 
 > Hans
 > 
 > 

Nikita.