Re: seekable pipes

[Frederik Eaton <frederik@a5.repetae.net>]

On Tue, Jun 07, 2005 at 01:31:21PM -0700, Bryan Henderson wrote:
> I like it.  I like it a lot.

Good!

> But I'd expect it to do all of the open file operations, not just seek. (A 
> seek-only implementation is certainly acceptable, but the structure should 
> be for the whole concept).

Like ... ioctl, flock? What else isn't implementable in terms of read,
write, seek?

> The idea of doing seeks by using lseek() on both sides seems unnatural to 
> me.  A file doesn't magically change its position.  I'd just use ioctls on 
> the master side.

I guess I agree about lseek(); I'd considered ioctls but I thought the
advantage of a stream-based protocol using 'read' and 'write' would be
that it would be easy to forward over network connections, etc.

> This won't work for many of the things you'd like it to because many 
> application programs are too smart for their own good.  The proper way to 
> determine whether a file is seekable is to try a seek and see what 
> happens.  But many of them stat the file and check its type.  We might 
> want something really ugly like an ioctl that sets the stat type.

The stat type for <() files is "character special" - such files are
usually seekable. In any case, emacs and vim are cool. If an
application doesn't work because of what you describe, then it would
also have problems with other uses of that shell syntax, so I think
it's not a big issue for us.

On Tue, Jun 07, 2005 at 02:56:14PM -0700, x wrote:
...
> Given all this, I agree with Bryan that this makes
> more sense when done with IOCTLs on the server side.
> But the introduction of multiple readers brings in
> concurrency, which is not easy at all. In particular,
> my original idea allowed the kernel to buffer some
> amount of data, just like a regular pipe. But if there
> are multiple clients with read/write capability,
> you're basically forced to engage in substantial
> quantities of trickery, or else pass every client
> system call over to the server, and wait for the
> response. Not good.

My proposal involved buffering too. I said "for good random-access
performance I think the master should be able to provide a block of
data with an expiration time which is cached independently of slave
seek position." Why wouldn't this work?

> All that said, I'm happy to dedicate my meager
> intellectual resources to this project, but I think
> the amount of hackery required to do this right is
> probably beyond me.

I'm not sure if I would be qualified to work on it either, last time I
looked inside the kernel was a long time ago, and I'm busy. Is there a
wiki where we could work out a specification in case someone comes
along who decides it would make a good project?

Frederik

> --- Frederik Eaton <frederik@a5.repetae.net> wrote:
> 
> > Wow, what happened to Ian Turner's "seekable pipes"
> > proposal? 
> > 
> >
> http://marc.theaimsgroup.com/?l=linux-kernel&m=110153524529112&w=2

> > I have wanted to propose something very similar. I think it would
> > be a great way of implementing a lot of functionality that we
> > think of as belonging to more complicated userspace filesystems.
> > For instance, if the kernel had seekable pipes then I could write
> > a command that just serialized the protocol and allowed me to read
> > and write files over an 'ssh' session, with very simple syntax:

> > some-editor <(ssh-file host:path)
> > some-editor <(cmd-file ssh host open-file path)

> > or inside a tar file (maybe reading only), or on an FTP site, etc.

> > some-editor <(url-file ftp://host:path)

> > It might even make more sense than the AVFS way of doing things,
> > because commands can take multiple file arguments:

> > some-editor <(patched-file file.c file.patch) #
> > writes changes to file.patch

> > - difficult to do with AVFS file system modules which are
> > "mounted" on a single base file.

> > Kernel support could mean support for mmap and exec, which you
> > couldn't easily get from a library implementation.

> > Something like this could be the basis and core of a more complete
> > userspace file system protocol. But since the seekable pipe
> > concept also has independent applications I think it would be a
> > good thing to factor out and focus on getting right independently
> > of namespace-dependent functionality. Plus even without kernel
> > support for allowing processes to export virtual directories to
> > each other, seekable pipes would still give shared-library based
> > userspace filesystems such as AVFS-preload a way to present host
> > processes with a much more complete file system interface, by
> > solving their mmap and exec problems.

> > Ian admits that his idea of a protocol was too simple, since - if
> > you will allow me to use the terms 'master' and 'slave' - it
> > didn't allow multiple slaves for one master. And it didn't allow
> > the slave half of the pipe to write data, which I think would be
> > an important feature; also, for good random-access performance I
> > think the master should be able to provide a block of data with an
> > expiration time which is cached independently of slave seek
> > position.
> > 
> > But aside from that, the protocol need not be complex. Except for
> > the problem of creating the special file descriptors, the rest
> > could be done on the master side with read/write (read a request
> > for data, write a block of data with a description header).
> > Creating the descriptor pair could be done with an ioctl
> > "promoting" the pipe descriptors as in Ian's proposal, although
> > since seekable pipes wouldn't share much in common with regular
> > pipes, the kernel pipe data structures would probably be created
> > and then destroyed unnecessarily in this case - ugly if not a
> > performance issue. I guess there are several other possibilities.
> > Being able to name the slave as with a devpts-like solution could
> > have advantages but perhaps not enough to justify the complexity.
> > 
> > I feel that this kind of feature, in some form,
> > could be the shortest
> > path to a lot of neat applications. What do other
> > people think?
> > 
> > Frederik
> > 
> > -- 
> > http://ofb.net/~frederik/
> > 
> 
> 
> 
> 		
> __________________________________ 
> Discover Yahoo! 
> Stay in touch with email, IM, photo sharing and more. Check it out! 
> http://discover.yahoo.com/stayintouch.html
> -
> To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

-- 
http://ofb.net/~frederik/