Re: Make pipe data structure be a circular list of pages, rather

Linus Torvalds
2005-01-15T00:16:18Z

On Sat, 15 Jan 2005, Ingo Oeser wrote:
>
> Now imagine this (ACSCII art in monospace font):
>
> [ chip A ] ------(1)------ [ chip B ] [ CPU ] [memory] [disk]
> | | | | |
> +----------(2)-----------+---(2)-----+----(2)--+--------+
>
> Yes, I understand and support your vision. Now I would like to use path (1)
> the direct for this. Possible?

Yes, if I understand what you're thinking right. I'm just assuming (for
the sake of simplicity) that "A" generates all data, and "B" is the one
that uses it. If both A and B can generate/use data, it still works, but
you'd need two pipes per endpoint, just because pipes are fundamentally
uni-directional.

Also, for this particular example, I'll also assume that whatever buffers
that chips A/B use are "mappable" to the CPU too: they could be either
regular memory pages (that A/B DMA to/from), or they can be IO buffers on
the card itself that can be mapped into the CPU address space and
memcpy'ed.

That second simplification is something that at least my current example
code (which I don't think I've sent to you) kind of assumes. It's not
really _forced_ onto the design, but it avoids the need for a separate
"accessor" function to copy things around.

[ If you cannot map them into CPU space, then each "struct pipe_buffer" op
structure would need operations to copy them to/from kernel memory and
to copy them to/from user memory, so the second simplification basically
avoids having to have four extra "accessor" functions. ]

What you'd do to get your topology above is:
- create a "pipe" for both A and B (let's call them "fdA" and "fdB"
respectively).
- the driver is responsible for creating the "struct pipe_buffer" for any
data generated by chip A. If it's regular memory, it's a page
allocation and the appropriate DMA setup, and if it's IO memory on the
card, you'd need to generate fake "struct page *" and a mapping
function for them.
- if you want to send the data that A generates to both fdB _and_ write
it to disk (file X), you'd also need one anonymous pipe (fdC), and open
a fdX for writing to the file, and then just in your process
effectively do:

for (;;) {
n = tee(fdA, fdB, fbC);
splice(fdC, fdX, n);
}

and you'd get what I think you are after. The reason you need the "fdC"
is simply because the "tee()" operation would only work on pipes: you
can not duplicate any other data structure with "tee()" than a pipe
buffer, so you couldn't do a direct "tee(fdA, fdB, fdX)" in my world.

(The above example is _extremely_ simplified: in real life, you'd have to
take care of partial results from "splice()" etc error conditions, of
course).

NOTE! My first cut will _assume_ that all buffers are in RAM, so it
wouldn't support the notion of on-card memory. On-card memory does
complicate things - not just the accessor functions, but also the notion
of how to "splice()" (or "tee()" from one to the other). If you assume
buffers-in-RAM, then all pointers are the same, and a tee/splice really
just moves a pointer around. But if the pointer can have magic meaning
that depends on the source it came from, then splicing such a pointer to
another device must inevitably imply at least the possibility of some kind
of memory copy.

So I don't think you'll get _exactly_ what you want for a while, since
you'd have to go through in-memory buffers. But there's no huge conceptual
problem (just enough implementation issues to make it inconvenient) with
the concept of actually keeping the buffer on an external controller.

Linus