named streams, extended attributes, and posix

named streams, extended attributes, and posix

[Michael Rothwell]

Now that 2.4 is out, it will probably be a few .x releases until 2.5
begins.

A discussion on Named Streams and Extended Attributes was put off until
2.5 earlier in the 2.4 development cycle. For compatibility with
existing, widely-deployed filesystems (e.g., NFS, XFS, BeFS, HFS, etc.),
Linux needs a standard way to expose and interact with these filesystem
features. A draft of a paper proposing a methd for accomplishing this on
Posix systems is available at:

http://www.flyingbuttmonkeys.com/streams-on-posix.txt
http://www.flyingbuttmonkeys.com/streams-on-posix.pdf

Please read and comment! :)

-M
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Michael Rothwell]

CORRECTION:

> existing, widely-deployed filesystems (e.g., NFS, XFS, BeFS, HFS, etc.),
NTFS-------------------------------------------^
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[James H. Cloos Jr.]

Michael> Please read and comment! :)

There should be some discussion on what to do about filenames which
contain colons in such a setup.  Moving a file w/ a colon from a fs
which does not support named streams to one which does should DTRT;
exactly what TRT is should be discussed.

-JimC
-- 
James H. Cloos, Jr.  <http://jhcloos.com/public_key>     1024D/ED7DAEA6 
<cloos@jhcloos.com>  E9E9 F828 61A4 6EA9 0F2B  63E7 997A 9F17 ED7D AEA6
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Michael Rothwell]

"James H. Cloos Jr." wrote:
> 
> Michael> Please read and comment! :)
> 
> There should be some discussion on what to do about filenames which
> contain colons in such a setup.  Moving a file w/ a colon from a fs
> which does not support named streams to one which does should DTRT;
> exactly what TRT is should be discussed.

It seems that if you move a file with a colon -- "file:colon" -- in the
name from Ext2 to "StreamFS," you would end up with a file named "file"
with a stream named "colon". When copying back, you would get
"file:colon" back.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Peter Samuelson]

[Michael Rothwell]
> It seems that if you move a file with a colon -- "file:colon" -- in
> the name from Ext2 to "StreamFS," you would end up with a file named
> "file" with a stream named "colon". When copying back, you would get
> "file:colon" back.

What if you copy both 'filename' and 'filename:ext' onto the same fs?
Do they get combined into one file?  That to me violates principle of
least surprise.  The fs should just mangle filenames it doesn't agree
with, like the existing legacy filesystems already do.

Any semantics by which 'filename:stream' and 'filename' refer to the
same file would be b0rken.  If instead you use 'filename/stream'
syntax, at least that is an illegal filename on *all* Linux
filesystems, so this particular point of confusion does not come up.

Peter
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Mo McKinlay]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yesterday, Peter Samuelson (peter@cadcamlab.org) wrote:

  >
  > [Michael Rothwell]
  > > It seems that if you move a file with a colon -- "file:colon" -- in
  > > the name from Ext2 to "StreamFS," you would end up with a file named
  > > "file" with a stream named "colon". When copying back, you would get
  > > "file:colon" back.
  >
  > What if you copy both 'filename' and 'filename:ext' onto the same fs?
  > Do they get combined into one file?  That to me violates principle of
  > least surprise.  The fs should just mangle filenames it doesn't agree
  > with, like the existing legacy filesystems already do.

  > Any semantics by which 'filename:stream' and 'filename' refer to the
  > same file would be b0rken.  If instead you use 'filename/stream'
  > syntax, at least that is an illegal filename on *all* Linux
  > filesystems, so this particular point of confusion does not come up.

Urgh.

We went through this last time around. What happens to directories  with
streams? If they are also dirname/stream, what happens when you have  a
file called 'stream' within 'dirname'?

Also, it makes it appear that files with streams are directories, which
they are not, so applying the directory accessor to them violates the
principle of least suprise and is misleading. Apply a new accessor (which
the colon would be great for, as it is already used for this purpose --
apart from the  fact that POSIX already allows applications to use it in
filenames).

- -- 
Mo McKinlay
mmckinlay@gnu.org
- -------------------------------------------------------------------------
GnuPG/PGP Key: pub  1024D/76A275F9 2000-07-22





-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjpk6V8ACgkQRcGgB3aidfkLgACfZZ1+5klZUB/NKTDK9PoRlV2H
ddcAoKBJSYA5/02Y8+dV9zyZHi5hUCeK
=ptqG
-----END PGP SIGNATURE-----

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Peter Samuelson]

[Mo McKinlay]
> We went through this last time around. What happens to directories
> with streams?

Yeah, I agree, 'file/stream' is lousy syntax as well.  If it weren't
for the possibility of having streams on directories, it would almost
be acceptible.  I still don't know which (':' or '/') is the worse
hack.

As I've said elsewhere in this thread, I can't think of *any* clean way
to shoehorn forks into nice, transparent posix calls.  It really wants
a new API.

Peter
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Mo McKinlay]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Yesterday, Peter Samuelson (peter@cadcamlab.org) wrote:

  > Yeah, I agree, 'file/stream' is lousy syntax as well.  If it weren't
  > for the possibility of having streams on directories, it would almost
  > be acceptible.  I still don't know which (':' or '/') is the worse
  > hack.

Me neither :/

  > As I've said elsewhere in this thread, I can't think of *any* clean way
  > to shoehorn forks into nice, transparent posix calls.  It really wants
  > a new API.

Likewise. This was my standpoint the last time around - a clear concise
portable API for accessing streams (even if it *started out*
Linux-specific) - without imposing silly semantics on existing
applications which currently ignore streams anyway.

Mo.

- -- 
Mo McKinlay
mmckinlay@gnu.org
- -------------------------------------------------------------------------
GnuPG/PGP Key: pub  1024D/76A275F9 2000-07-22





-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjpnYGQACgkQRcGgB3aidfmWBQCfXgNq/vqltt76mApoDiNI9HnH
ws8AoJZ2vvlH1iCAeUu7yktWWN0Bncc3
=gEmD
-----END PGP SIGNATURE-----

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Michael Rothwell]

Unfortunately, unix allows everything but "/" in filenames. This was
probably a mistake, as it makes it nearly impossible to augment the
namespace, but it is the reality.

Did you read the "new namespace" section of the paper?

It also talked a bit about supporting Extended Attributes, which are access
via an API and not with a filename separator. We could, perhaps, begin by
supporting EAs. That would take care of HFS, HPFS, XFS, and BeFS, and half
of NTFS. Then we could go on to tackle the Named Streams problem.

-M

----- Original Message -----
From: "Mo McKinlay" <mmckinlay@gnu.org>
To: "Peter Samuelson" <peter@cadcamlab.org>
Cc: "Mo McKinlay" <mmckinlay@gnu.org>; <linux-kernel@vger.kernel.org>
Sent: Thursday, January 18, 2001 1:30 PM
Subject: Re: named streams, extended attributes, and posix


> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Yesterday, Peter Samuelson (peter@cadcamlab.org) wrote:
>
>   > Yeah, I agree, 'file/stream' is lousy syntax as well.  If it weren't
>   > for the possibility of having streams on directories, it would almost
>   > be acceptible.  I still don't know which (':' or '/') is the worse
>   > hack.
>
> Me neither :/
>
>   > As I've said elsewhere in this thread, I can't think of *any* clean
way
>   > to shoehorn forks into nice, transparent posix calls.  It really wants
>   > a new API.
>
> Likewise. This was my standpoint the last time around - a clear concise
> portable API for accessing streams (even if it *started out*
> Linux-specific) - without imposing silly semantics on existing
> applications which currently ignore streams anyway.
>
> Mo.
>
> - --
> Mo McKinlay
> mmckinlay@gnu.org
> - ------------------------------------------------------------------------
-
> GnuPG/PGP Key: pub  1024D/76A275F9 2000-07-22
>
>
>
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.0.4 (GNU/Linux)
> Comment: For info see http://www.gnupg.org
>
> iEYEARECAAYFAjpnYGQACgkQRcGgB3aidfmWBQCfXgNq/vqltt76mApoDiNI9HnH
> ws8AoJZ2vvlH1iCAeUu7yktWWN0Bncc3
> =gEmD
> -----END PGP SIGNATURE-----
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> Please read the FAQ at http://www.tux.org/lkml/
>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Mo McKinlay]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Today, Michael Rothwell (rothwell@holly-springs.nc.us) wrote:

  > Unfortunately, unix allows everything but "/" in filenames. This was
  > probably a mistake, as it makes it nearly impossible to augment the
  > namespace, but it is the reality.

  > Did you read the "new namespace" section of the paper?

I've not, so pardon me if I make a bad assumption (and slap me for it,
too), but doesn't introducing a new namespace segregate the streams from
the files/directories, thus introducing an artifical separation which
isn't really there? (Pretty much why I'm more in favour of a specific API
for reading streams, extended attributes and whatnot, over any of the
other solutions thus suggested).

Mo.

- -- 
Mo McKinlay
mmckinlay@gnu.org
- -------------------------------------------------------------------------
GnuPG/PGP Key: pub  1024D/76A275F9 2000-07-22





-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjpoTQMACgkQRcGgB3aidfmurACdEb+w6gwUW7fc4FVdZ7umHlDs
/AgAoN8SOXejiKDd8G/KPVTP7qZwzhnO
=Ld9D
-----END PGP SIGNATURE-----

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Michael Rothwell]

Mo McKinlay wrote:
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Today, Michael Rothwell (rothwell@holly-springs.nc.us) wrote:
> 
>   > Unfortunately, unix allows everything but "/" in filenames. This was
>   > probably a mistake, as it makes it nearly impossible to augment the
>   > namespace, but it is the reality.
> 
>   > Did you read the "new namespace" section of the paper?
> 
> I've not, so pardon me if I make a bad assumption (and slap me for it,
> too), but doesn't introducing a new namespace segregate the streams from
> the files/directories, thus introducing an artifical separation which
> isn't really there? (Pretty much why I'm more in favour of a specific API
> for reading streams, extended attributes and whatnot, over any of the
> other solutions thus suggested).

Well, EAs are accessed via a special API. The paper also covers that.
Streams, however, are by nature accessed as files; this is what makes
them not EAs. EAs are set and retrieved atomically. Streams can be used
with open(), seek(), read(), write(), etc. This is actually more "unixy"
than EAs, as the usual set of Unix functions and tools can work with
streams unmodified; i.e., without knowledge of a special API. Of course,
cp() is the exception. It would have to be able to enumerate and copy
all the streams.

If you have time, please read over the paper so we don't get into the
same rut we got into last time. :)

http://www.flyingbuttmonkeys.com/streams-on-posix.txt
http://www.flyingbuttmonkeys.com/streams-on-posix.pdf

-M
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Mo McKinlay]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


  > Well, EAs are accessed via a special API. The paper also covers that.
  > Streams, however, are by nature accessed as files; this is what makes
  > them not EAs. EAs are set and retrieved atomically. Streams can be used
  > with open(), seek(), read(), write(), etc. This is actually more "unixy"
  > than EAs, as the usual set of Unix functions and tools can work with
  > streams unmodified; i.e., without knowledge of a special API. Of course,
  > cp() is the exception. It would have to be able to enumerate and copy
  > all the streams.

Hokay, I've skimmed the paper, most of it makes sense, although I still
think the additional separator is a bad idea (which I know isn't what I
said last time around, originally, but I've had a chance to ponder a
little more since :). Hence, I reckon we should have:

openstream(file, stream, flags)

Where 'file' should be an fd (although i'm sure the VFS gods will think of
plenty of reasons why this is a bad idea, at which point I'll
conventiently change my mind ;). Stream is simply the name of the stream,
flags are as with open() (O_RDONLY, et al). openstream() then returns an
fd which can be read/written/sendfiled/closed as the programmer wishes.

How daft does this sound?

Apart from the additional of a new open()-type call, your paper seems to
be fairly solid.

Mo.

- -- 
Mo McKinlay
mmckinlay@gnu.org
- -------------------------------------------------------------------------
GnuPG/PGP Key: pub  1024D/76A275F9 2000-07-22








-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjpoVmMACgkQRcGgB3aidfnLuACfSZqvswA0B1xnWilVWQcSHubM
yQAAn2T+RFRN3qznuQ8wOeWXxIBVGb45
=SUm5
-----END PGP SIGNATURE-----

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Michael Rothwell]

Mo McKinlay wrote:

> openstream(file, stream, flags)
> 
> Where 'file' should be an fd (although i'm sure the VFS gods will think of
> plenty of reasons why this is a bad idea, at which point I'll
> conventiently change my mind ;). Stream is simply the name of the stream,
> flags are as with open() (O_RDONLY, et al). openstream() then returns an
> fd which can be read/written/sendfiled/closed as the programmer wishes.

I'm not opposed to that, and think it is even a useful idea. Sort of
like fdopen().

> Apart from the additional of a new open()-type call, your paper seems to
> be fairly solid.

Thanks. I think having the option of the namespace augmentation would be
useful, in terms of supporting existing filesystems. On NTFS, ":" is not
a legal filename character anyway. The namespace augmentation suggested
in the paper would allow filesystems like NTFS to work as they should,
and all other filesystems to ignore it.

-M
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Mo McKinlay]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Today, Michael Rothwell (rothwell@holly-springs.nc.us) wrote:

  > Thanks. I think having the option of the namespace augmentation would be
  > useful, in terms of supporting existing filesystems. On NTFS, ":" is not
  > a legal filename character anyway. The namespace augmentation suggested
  > in the paper would allow filesystems like NTFS to work as they should,
  > and all other filesystems to ignore it.

This was the tack I originally took - but I realised it would eventually
cause problems - not least because rather than returning an error when a
user does something not supported like most actions do, the operation
behaves differently - which is a little confusing.

(Take symbolic linking, for example - if you ln -s on VFAT, you get
'operation not permitted' - named stream/EA operations on a filesystem
that doesn't support them should return the same, IMHO).

Also, I don't like the idea of bypassing POSIX in this manner (using ':'
as a delimeter), even if the underlying filesystem *may* not support it.

What's to say that ext4 (or whatever) won't support named streams, but
still allow ':'? Your solution as it stands would break in that situation
(assuming I've not missed something :)

Mo.

- -- 
Mo McKinlay
mmckinlay@gnu.org
- -------------------------------------------------------------------------
GnuPG/PGP Key: pub  1024D/76A275F9 2000-07-22





-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjpoW04ACgkQRcGgB3aidfncVgCgm19oUQqgGSW7XNCZwoWB/bIj
2W0AoK64xCfbjcamj3F5fDyBtVg8KQBa
=PEu2
-----END PGP SIGNATURE-----

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Michael Rothwell]

Mo McKinlay wrote:

> (Take symbolic linking, for example - if you ln -s on VFAT, you get
> 'operation not permitted' - named stream/EA operations on a filesystem
> that doesn't support them should return the same, IMHO).

And they would, if the chosen namespace was not supported.

> Also, I don't like the idea of bypassing POSIX in this manner (using ':'
> as a delimeter), even if the underlying filesystem *may* not support it.
> 
> What's to say that ext4 (or whatever) won't support named streams, but
> still allow ':'? Your solution as it stands would break in that situation
> (assuming I've not missed something :)

The filesystem, when registering that it supports the "named streams"
namespace, could specify its preferred delimiter to the VFS as well.
Ext4 could use /directory/file/stream, and NTFS could use
/directory/file:stream.

-M
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Mo McKinlay]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Today, Michael Rothwell (rothwell@holly-springs.nc.us) wrote:

  > The filesystem, when registering that it supports the "named streams"
  > namespace, could specify its preferred delimiter to the VFS as well.
  > Ext4 could use /directory/file/stream, and NTFS could use
  > /directory/file:stream.

Erk - nice from a programming point of view, horrible from a consistency
one. The nice thing about VFS is that it provides a consistent abstract
interface - I'd place a small amount of money on the fact that Al Viro
would probably flame you to high heaven for that last suggestion if he was
paying much attention to this thread :-)

- -- 
Mo McKinlay
mmckinlay@gnu.org
- -------------------------------------------------------------------------
GnuPG/PGP Key: pub  1024D/76A275F9 2000-07-22





-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjpoYe4ACgkQRcGgB3aidfn2RQCfa1nnClzSXxBCB0XnJ35RmOcm
ysoAoJSg+USBkDCp4PKcX5iD0JQQvXw9
=Lkci
-----END PGP SIGNATURE-----

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Michael Rothwell]

Mo McKinlay wrote:
> 
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> Today, Michael Rothwell (rothwell@holly-springs.nc.us) wrote:
> 
>   > The filesystem, when registering that it supports the "named streams"
>   > namespace, could specify its preferred delimiter to the VFS as well.
>   > Ext4 could use /directory/file/stream, and NTFS could use
>   > /directory/file:stream.
> 
> Erk - nice from a programming point of view, horrible from a consistency
> one. The nice thing about VFS is that it provides a consistent abstract
> interface - I'd place a small amount of money on the fact that Al Viro
> would probably flame you to high heaven for that last suggestion if he was
> paying much attention to this thread :-)

Oh, undoubtedly.  But NTFS already disallows several characters in valid
filenames. This also violates the "consistent abstract interface." But
it's reality.

-M
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Mo McKinlay]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Today, Michael Rothwell (rothwell@holly-springs.nc.us) wrote:

  > Oh, undoubtedly.  But NTFS already disallows several characters in valid
  > filenames. This also violates the "consistent abstract interface." But
  > it's reality.

Nono, that's not what I mean - each of the filesystems fails if it
doesn't support what you're trying to do, that's given - but having a
different delimeter registered by the filesystem (and hence the
possibility of every single filesystem using a different delimeter) brings
about a completely different kind of inconsistency.

Mo.

- -- 
Mo McKinlay
mmckinlay@gnu.org
- -------------------------------------------------------------------------
GnuPG/PGP Key: pub  1024D/76A275F9 2000-07-22





-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjpoZlQACgkQRcGgB3aidfnnNACcC99aXvrG1lsbEv5rr8wpGrTe
ZScAn1TCpbviEGzWn6UGHhqTgQVVSqVp
=UL3r
-----END PGP SIGNATURE-----

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Michael Rothwell]

Mo McKinlay wrote:

> Nono, that's not what I mean - each of the filesystems fails if it
> doesn't support what you're trying to do, that's given - but having a
> different delimeter registered by the filesystem (and hence the
> possibility of every single filesystem using a different delimeter) brings
> about a completely different kind of inconsistency.

True. Which is why I'd prefer a standard delimiter. ":" seems to be the
top candidate.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Mo McKinlay]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Today, Michael Rothwell (rothwell@holly-springs.nc.us) wrote:

  > Mo McKinlay wrote:
  >
  > > Nono, that's not what I mean - each of the filesystems fails if it
  > > doesn't support what you're trying to do, that's given - but having a
  > > different delimeter registered by the filesystem (and hence the
  > > possibility of every single filesystem using a different delimeter) brings
  > > about a completely different kind of inconsistency.

  > True. Which is why I'd prefer a standard delimiter. ":" seems to be the
  > top candidate.

I would too, but POSIX won't let us unless we start enforcing side-effect
rules for all filesystems. Hence why I came up with openstream() :)

Mo.

- -- 
Mo McKinlay
mmckinlay@gnu.org
- -------------------------------------------------------------------------
GnuPG/PGP Key: pub  1024D/76A275F9 2000-07-22





-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjpobHEACgkQRcGgB3aidfnOyQCeNwj2WYbQd059F/JurCkcruED
cWEAoMO0P8eH3BAzpRkzcP3RkVDiEXOl
=siV3
-----END PGP SIGNATURE-----

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Michael Rothwell]

Mo McKinlay wrote:
> I would too, but POSIX won't let us unless we start enforcing side-effect
> rules for all filesystems. Hence why I came up with openstream() :)

So, openstream() is probably the most painless way to get named streams
support into Linux in the immediate future. Openstream() will have to
fail on filesystems that do not support streams, ideally without
changing those filesystems. And as long as we're adding a system call to
deal with streams, we should consider adding the functions for extended
attributes at the same time.

>From http://www.flyingbuttmonkeys.com/streams-on-posix.txt ...

Minimum VFS Support
vop_eattr_get - read an EA
vop_eattr_set - set an EA
vop_eattr_remove - remove an EA
vop_eattr_list - list the EAs like vop_readdir would a directory.

Optional Support
vop_eattr_create - Create an EA or error if it exists.
vop_eattr_multi - perform a sequence of EA vops atomically.
vop_eattr_rename - change the name of an EA
vop_eattr_serialize - export all the EAs as a stream of entries.

Thoughts? You mught want to refer back to the paper to get the whole EAs
proposal...

-M
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Mo McKinlay]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Today, Michael Rothwell (rothwell@holly-springs.nc.us) wrote:

  > So, openstream() is probably the most painless way to get named streams
  > support into Linux in the immediate future. Openstream() will have to
  > fail on filesystems that do not support streams, ideally without
  > changing those filesystems.

Agree - I'd imagine with the same error that trying to symlink on a
filesystem which does not support symlinks produces ("Operation not
permitted"?).

  >  And as long as we're adding a system call to
  > deal with streams, we should consider adding the functions for extended
  > attributes at the same time.

Agree.

  > >From http://www.flyingbuttmonkeys.com/streams-on-posix.txt ...
  >
  > Minimum VFS Support
  > vop_eattr_get - read an EA
  > vop_eattr_set - set an EA
  > vop_eattr_remove - remove an EA
  > vop_eattr_list - list the EAs like vop_readdir would a directory.
  >
  > Optional Support
  > vop_eattr_create - Create an EA or error if it exists.
  > vop_eattr_multi - perform a sequence of EA vops atomically.
  > vop_eattr_rename - change the name of an EA
  > vop_eattr_serialize - export all the EAs as a stream of entries.
  >
  > Thoughts? You mught want to refer back to the paper to get the whole EAs
  > proposal...

I shall, and get back to you this evening (workworkworkwork... :)

Mo.

- -- 
Mo McKinlay
mmckinlay@gnu.org
- -------------------------------------------------------------------------
GnuPG/PGP Key: pub  1024D/76A275F9 2000-07-22





-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.4 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iEYEARECAAYFAjp1kGIACgkQRcGgB3aidfmBWQCfSODBdYvh0QhqwxHUWB3IGUxg
NY4AoLG1i01AmjYASQwwQMh58t2b1Ut7
=bwEX
-----END PGP SIGNATURE-----

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Albert D. Cahalan]

Michael Rothwell writes:
> ...
>> Today, Michael Rothwell (rothwell@holly-springs.nc.us) wrote:

>>> The filesystem, when registering that it supports the "named streams"
>>> namespace, could specify its preferred delimiter to the VFS as well.
>>> Ext4 could use /directory/file/stream, and NTFS could use
>>> /directory/file:stream.
...
> Oh, undoubtedly.  But NTFS already disallows several characters
> in valid filenames.

NTFS allows all 16-bit characters in filenames, including 0x0000.
Nothing is disallowed. The NT kernel's native API uses counted
Unicode strings. The strings can be huge too, like 128 kB.

So there isn't _any_ safe delimiter.

Win32 will choke on 0x0000 and a few other things, allowing a
clever person to create apparently inaccessible files.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Michael Rothwell]

What you say is true; but Win32 -- which pretty much all Windows apps use --
disallows the following:

\/:*?"<>|

... from that, they chose ":" as the stream delimiter, since the only other
place it is used is with the drive letters. For the user, and most
(non-native, i.e., Win32) apps, there are limitations on what a filename can
contain.

-M

----- Original Message -----
From: "Albert D. Cahalan" <acahalan@cs.uml.edu>
To: "Michael Rothwell" <rothwell@holly-springs.nc.us>
Cc: "Mo McKinlay" <mmckinlay@gnu.org>; "Peter Samuelson"
<peter@cadcamlab.org>; <linux-kernel@vger.kernel.org>
Sent: Saturday, January 20, 2001 11:27 PM
Subject: Re: named streams, extended attributes, and posix


> Michael Rothwell writes:
> > ...
> >> Today, Michael Rothwell (rothwell@holly-springs.nc.us) wrote:
>
> >>> The filesystem, when registering that it supports the "named streams"
> >>> namespace, could specify its preferred delimiter to the VFS as well.
> >>> Ext4 could use /directory/file/stream, and NTFS could use
> >>> /directory/file:stream.
> ...
> > Oh, undoubtedly.  But NTFS already disallows several characters
> > in valid filenames.
>
> NTFS allows all 16-bit characters in filenames, including 0x0000.
> Nothing is disallowed. The NT kernel's native API uses counted
> Unicode strings. The strings can be huge too, like 128 kB.
>
> So there isn't _any_ safe delimiter.
>
> Win32 will choke on 0x0000 and a few other things, allowing a
> clever person to create apparently inaccessible files.
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> Please read the FAQ at http://www.tux.org/lkml/
>

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Daniel Phillips]

Michael Rothwell wrote:
> Unfortunately, unix allows everything but "/" in filenames. This was
> probably a mistake, as it makes it nearly impossible to augment the
> namespace, but it is the reality.

For some reason totally beyond my comprehension // inside a file name is
taken to be the same as /, but if it wasn't it could be the stream
separator.  *sigh*

--
Daniel
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Thunder from the hill]

Daniel Phillips wrote:
> 
> Michael Rothwell wrote:
> > Unfortunately, unix allows everything but "/" in filenames. This was
> > probably a mistake, as it makes it nearly impossible to augment the
> > namespace, but it is the reality.
> 
> For some reason totally beyond my comprehension // inside a file name is
> taken to be the same as /, but if it wasn't it could be the stream
> separator.  *sigh*
It seems that you mix up forward and backward slashes. a // means //,
but a \\ means a single \. So if you want a double backslash, you have
to write \\\\. Thus, removing double backslashes from NETBIOS names via
perl is: $name =~ s/\\\\//;
So what...?

Cheers!
Thunder
---
I did a "cat /boot/vmlinuz >> /dev/audio" - and I think I heard god...
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[alex]

On Thu, Jan 25, 2001 at 02:07:11PM -0700, Thunder from the hill wrote:
> Daniel Phillips wrote:
> > For some reason totally beyond my comprehension // inside a file name is
> > taken to be the same as /, but if it wasn't it could be the stream
> > separator.  *sigh*
> It seems that you mix up forward and backward slashes. a // means //,
> but a \\ means a single \. So if you want a double backslash, you have

No, he's referring to the fact that multiple path separators ("/") in a file 
specification are collapsed to be equivalent to one.  Thus "/some//file///path"
is equivalent to "/some/file/path" as far as the system is concerned.

This is actually a very handy thing, IMHO, and avoids tons of 
trailing-slash/leading-slash special-case logic in apps, not to mention subtle 
bugs resulting from lack of same as I have encountered way too many times on 
other platforms...

I agree however, that it would perhaps have been nice if POSIX hadn't been 
quite so gung-ho about any-character-under-the-sun-is-ok-in-filenames so we had a couple of reserved characters to play with down the line..

Here's an idea: streams/etc are reached by appending "/.../xxx" or some 
such to paths, thus:
  for streamname on /dir/file, we have "/dir/file/.../streamname" 
    -- a few more characters to type but no big deal really,
  for a directory /dir/dir, we get /dir/dir/.../streamname" 
    -- "..." is a special subdirectory of any directories which have 
    attached streams.  If the name of such a directory is chosen well 
    (personally, I think "..." is a good choice as it goes well with "." and 
    ".." as a filesystem-intrinsic name), this is much less likely to 
    conflict with normal filesystem namespaces.

This also has the advantage of being extendable (using strings other than 
"...") for other applications or future additions.

-alex
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Michael Rothwell]

> What if you copy both 'filename' and 'filename:ext' onto the same fs?
> Do they get combined into one file?

ON Ext2, you get two files. On NTFS, you get one file, and a stream on that
file.

> Any semantics by which 'filename:stream' and 'filename' refer to the
> same file would be b0rken.  If instead you use 'filename/stream'

That does not allow streams on directories, otherwise I agree.

-M



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

Re: named streams, extended attributes, and posix

[Peter Samuelson]

  [Peter Samuelson]
> > What if you copy both 'filename' and 'filename:ext' onto the same
> > fs?  Do they get combined into one file?

[Michael Rothwell]
> ON Ext2, you get two files. On NTFS, you get one file, and a stream
> on that file.

Yeah.  I think that's broken.  It gets worse when you start doing other
posixy things.  Say you do 'mv foo bar:baz' on an ntfs partition.
Those strong in the force will recall that rename() is supposed to
atomically unlink 'bar:baz'.  Instead you seem to be asking it to add
an extra stream to 'bar'.  So should we still unlink the old 'bar'?
And then what if 'foo' is a multi-stream file?  Where do the extra
streams go?  Or do you just get a big fat -EINVAL for every special
case?

I think ultimately there is no posixy (or least-surprise) way to deal
with such things.  If we truly need forks, we need a new api to
manipulate them.  Cleverness with ':' or '/' only takes you so far.

Peter
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: named streams, extended attributes, and posix

[Leif Sawyer]

alex@foogod.com [alex@foogod.com] wrote:
> Here's an idea: streams/etc are reached by appending 
> "/.../xxx" or some such to paths, thus:
>   for streamname on /dir/file, we have "/dir/file/.../streamname" 
>  for a directory /dir/dir, we get /dir/dir/.../streamname" 
>    -- "..." is a special subdirectory of any directories which have 

An interesting point to note here would be that
the directory '...'  has been used for many years to 'hide' things
from un-skilled sysadmins.

In other words, warez ftp sites pop up all over the place, and this
directory name is pretty close to being the number one stash point,
right next to ".. "



-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/

RE: named streams, extended attributes, and posix

[Steven N. Hirsch]

On Thu, 25 Jan 2001, Leif Sawyer wrote:

> alex@foogod.com [alex@foogod.com] wrote:
> > Here's an idea: streams/etc are reached by appending 
> > "/.../xxx" or some such to paths, thus:
> >   for streamname on /dir/file, we have "/dir/file/.../streamname" 
> >  for a directory /dir/dir, we get /dir/dir/.../streamname" 
> >    -- "..." is a special subdirectory of any directories which have 
> 
> An interesting point to note here would be that
> the directory '...'  has been used for many years to 'hide' things
> from un-skilled sysadmins.
> 
> In other words, warez ftp sites pop up all over the place, and this
> directory name is pretty close to being the number one stash point,
> right next to ".. "

It's also the implicit root for the global DFS filesystem namespace (from
Transarc of AFS fame).





-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
Please read the FAQ at http://www.tux.org/lkml/