Re: Streams support in Linux

Matthew Wilcox
2018-08-25T19:31:13Z

On Sat, Aug 25, 2018 at 03:47:45PM +0100, Al Viro wrote:
> On Sat, Aug 25, 2018 at 06:51:07AM -0700, Matthew Wilcox wrote:
>
> > openat()
> > To access a named stream, we need to be able to get a file descriptor for
> > it. The new openat() syscall seems like the best way to accompish this;
> > specify a file descriptor, a new AT_NAMED_STREAM flag and a filename,
> > and the last component of the filename will be treated as the name of
> > the stream within the object. This permits us to distinguish between
> > a named stream on a directory and a file within a directory.
>
> What do you do for access via /proc/*/fd/*, when descriptor is a sodding
> stream?

I don't know. Is it useful to be able to access them that way?

> > renameat()
> > If olddirfd + oldpath refers to a stream then newdirfd + newpath must
> > refer to a stream within the same parent object. If that stream exists,
> > it is removed. If olddirfd + oldpath does not refer to a stream, then
> > newdirfd + newpath must not refer to a stream.
>
> And here the shit begins - what to do when parents have different dentries?
> Directories can't have hardlinks; regular files very much can.

I presume you mean from a locking point of view? In my mind, we check that
old and new resolve to the same inode and take a lock on that inode.

> Where do dentries of these things belong, anyway?

Do streams have to have dentries? They don't have names in the hierarchy.
Can they share their parent's dentry? Can they have a NULL dentry? Or
a magic dentry of some kind?

> > unlinkat()
> > This is how you remove an individual named stream
>
> Yeah... and what happens to access via other hardlinks to the same parent
> file?

I'm not sure I understand the problem here. You have, say /a/b/c
and /d which refer to the same inode. This inode has streams x and y.
If you unlinkat() /a/b/c::x then it is no longer openable. Any currently
existing open fds for x keep x alive until the last one is closed, no
matter which path was used to open them.

> > unlink()
> > Unlinking a file with named streams removes all named streams from that
> > file and then unlinks the file. Open streams will continue to exist in
> > the filesystem until they are closed, just as unlinked files do.
>
> Can they be looked up by openat() from such a starting point?

No, you can't use openat() on a stream fd. I should have been clearer
on that above; this proposal should have split the syscall list in two;
one for how the syscall behaves on a fd-for-a-file-with-streams and one
for how the syscall behaves on a stream-fd.

> The only new part in that proposal is requesting explicit AT_... flag to
> even get to those. In effect, you are multiplexing a lot of syscalls
> that way, which does, in theory, allow different locking scheme to be
> used from the very beginning. However, that doesn't take care of quite
> a few other problems; a lot of code fundamentally assumes that ability
> to have children mean having only one chain of ancestors.

Right, but that code also assumes it's dealing with files. Streams can't
have their own children (either files or streams), which makes the locking
easier as they only have one parent and can't be cross-parent-renamed.

> And frankly, we already have one pile of garbage of that sort; what's the
> benefit of adding another? About the only difference between those and
> xattrs is the ability to open them; is the resulting headache worth the
> trouble, seeing that naive programs can't use that shit and somebody
> arranging redirects for them can bloody well use pipe and stick around
> feeding xattr contents into it...

Eric says xattrs are limited to 64kB which is too small for fs-verity.