Re: [RFC] Representing hardware connections via MC

[Laurent Pinchart <laurent.pinchart@ideasonboard.com>]

Hi Mauro,

(Disclaimer: There are lots of thoughts in this e-mail, sometimes in a bit of 
a random order. I would thus recommend reading through it completely before 
starting to write a reply.)

On Wednesday 02 March 2016 12:40:29 Mauro Carvalho Chehab wrote:
> Em Wed, 02 Mar 2016 16:16:43 +0200 Sakari Ailus escreveu:
> > On Fri, Feb 26, 2016 at 09:13:17AM -0300, Mauro Carvalho Chehab wrote:
> >> We had some discussions on Feb, 12 about how to represent connectors via
> >> the Media Controller:
> >> 	
https://linuxtv.org/irc/irclogger_log/v4l?date=2016-02-12,Fri&sel=31#l27
> >> 
> >> We tried to finish those discussions on the last two weeks, but people
> >> doesn't seem to be available at the same time for the discussions. So,
> >> let's proceed with the discussions via e-mail.
> >> 
> >> So, I'd like to do such discussions via e-mail, as we need to close
> >> this question next week.
> > 
> > While I agree that we shouldn't waste time in designing new APIs, we also
> > musn't merge unfinished work, and especially not when it comes to user
> > space APIs. Rather, we have to come up with a sound user space API/ABI
> > and *then* get it to mainline and *not* the other way around.
> 
> Well, we've agreed with connectors during the MC workshop in August,
> 2015.
> 
> The problem is that, when mapping the connectors on a particular
> driver (saa7134), we noticed that things are not so simple, because
> of the way it implements S-Video.

I couldn't have said better :-) We've agreed on a design idea during the 
workshop, and now that we're finally trying to put it into use we notice it 
can't work due to issues we hadn't foreseen. There's nothing new under the 
sun, this happens all the time during development, it just highlights the 
importance of really testing designs (and I'd even argue that this is only the 
visible part of the iceberg, we're testing the design by trying to apply it 
manually to a couple of existing hardware platforms, but haven't started 
testing it with real userspace applications).

So, in my opinion, the bottom line is that we should not slow down but keep 
refining the design until we achieve a stable and sound result. That's a 
methodology that we've applied so far with mostly good results.

> > I just read the IRC discussion beginning from when I needed to leave, and
> > it looks like to me that we need to have a common understanding of the
> > relevant concepts before we can even reach a common understanding what is
> > being discussed. I'm not certain we're even at that level yet.
> > 
> > Besides that, concepts should not be mixed. Things such as
> > MEDIA_ENT_F_CONN_SVIDEO should not exist, as it suggests that there's an
> > S-video connector (which doesn't really exist). The signal is S-video and
> > the connector is variable, but often RCA.
> 
> Well, "CONN" can be an alias for "connection", with is what we're actually
> representing.

Now I really second Sakari's request, if you've understood F_CONN as 
F_CONNECTION and I as F_CONNECTOR, then we really need to define the terms 
we're using.

> "Signal" is something else. As discussed, a single S-Video connection has 2
> signals: Y and C.
> 
> > At least connector, input (and output I presume?) and signal need to be
> > defined. My problem with terms such as input and output at this level of
> > the API is that they're really generic terms, and we chose not to use
> > them in MC in the past exactly for that reason.
> 
> After all the discussions, I guess "CONN" for connection is the best way
> to represent it.

How do you define a connection ?

> > For instance, luminance and chrominance certainly are signals, but is
> > S-video a signal as well? I'd say so.
> 
> No. S-video has 2 signals only: luminace and chrominance. Nothing else.

For analog video we could define signal in the electrical sense, as a voltage 
or current carries over a wire or trace. S-Video would then have two signals, 
composite a single one, and component video three. A connector could have one 
pad per signal if we use that definition.

That definition is harder to apply to digital video though. I don't think we 
should model an HDMI connector with one pad for each electrical signal. Should 
we avoid using the word signal for digital video then, or give it a different 
meaning ?

I'm lacking a good term other than video signal to describe one or multiple 
electrical signals that together carry an analog video stream, such as the Y+C 
combination in S-Video or the R+G+B combination in component video. I'll use 
video signal for such that purpose in the rest of this e-mail, but please feel 
free to propose a better term.

> > But should we make a difference between the two?
> 
> For S-Video, we may not need to represent two pads.

Unless I'm mistaken, that's one of the fundamental questions we've been trying 
to answer through our discussions on this topic. And I really think we should 
answer it, it's the core of the problem we're trying to solve.

> For HDMI, for sure we'll need to represent multiple signals, as they're
> routed on a different way (CEC, for example is a bus, that can be connected
> to multiple HDMI inputs and outputs).

Yes, CEC is an odd beast, and I'm not sure exactly how it should be handled. 
We have to draw the line and decide what we want to include in the graph and 
we we want to leave out. While I would I think be inclined to include CEC, I 
currently think DDC shouldn't be represented, but that's obviously open for 
discussions. Let's not forget that we currently have no bus-type links in MC, 
all the links we've used so far are unidirectional. We'll have to find out how 
to represent CEC or other similar buses (I2S is another example).

> > Connectors are easy, everyone knows what they are. Having
> > read the meeting IRC log, I guess the word "input" was likely used to
> > refer signals transmitted over multiple physical wires, at least some of
> > the time. I'd prefer not to guess.
> > 
> >> QUESTION:
> >> ========
> >> 
> >> How to represent the hardware connection for inputs (and outputs) like:
> >> 	- Composite TV video;
> >> 	- stereo analog audio;
> >> 	- S-Video;
> >> 	- HDMI
> >> 
> >> Problem description:
> >> ===================
> >> 
> >> During the MC summit last year, we decided to add an entity called
> >> "connector" for such things. So, we added, so far, 3 types of
> >> connectors:
> >> 
> >> #define MEDIA_ENT_F_CONN_RF		(MEDIA_ENT_F_BASE + 10001)
> >> #define MEDIA_ENT_F_CONN_SVIDEO		(MEDIA_ENT_F_BASE + 10002)
> >> #define MEDIA_ENT_F_CONN_COMPOSITE	(MEDIA_ENT_F_BASE + 10003)
> >> 
> >> However, while implementing it, we saw that the mapping on hardware
> >> is actually more complex, as one physical connector may have multiple
> >> signals with can eventually used on a different way.
> >> 
> >> One simple example of this is the S-Video connector. It has internally
> >> two video streams, one for chrominance and another one for luminance.
> >> 
> >> It is very common for vendors to ship devices with a S-Video input
> >> and a "S-Video to RCA" cable.
> >> 
> >> At the driver's level, drivers need to know if such cable is
> >> plugged, as they need to configure a different input setting to
> >> enable either S-Video or composite decoding.
> >> 
> >> So, the V4L2 API usually maps "S-Video" on a different input
> >> than "Composite over S-Video". This can be seen, for example, at the
> >> saa7134 driver, who gained recently support for MC.
> >> 
> >> Additionally, it is interesting to describe the physical aspects
> >> of the connector (color, position, label, etc).
> >> 
> >> Proposal:
> >> ========
> >> 
> >> It seems that there was an agreement that the physical aspects of
> >> the connector should be mapped via the upcoming properties API,
> >> with the properties present only when it is possible to find them
> >> in the hardware. So, it seems that all such properties should be
> >> optional.
> >> 
> >> However, we didn't finish the meeting, as we ran out of time. Yet,
> >> I guess the last proposal there fulfills the requirements. So,
> >> let's focus our discussions on it. So, let me formulate it as a
> >> proposal
> >> 
> >> We should represent the entities based on the inputs. So, for the
> >> already implemented entities, we'll have, instead:
> >> 
> >> #define MEDIA_ENT_F_INPUT_RF		(MEDIA_ENT_F_BASE + 10001)
> >> #define MEDIA_ENT_F_INPUT_SVIDEO	(MEDIA_ENT_F_BASE + 10002)
> >> #define MEDIA_ENT_F_INPUT_COMPOSITE	(MEDIA_ENT_F_BASE + 10003)
> > 
> > I presume INPUT would be for both input and output? We already have
> > MEDIA_ENT_F_IO and that's something quite different.
> > 
> > I'd really like to have a better name for this.
> 
> See above.
> 
> > Another question indeed is whether modelling signals consisting of
> > multiple electrical signals is the way to go (vs. connectors, where the
> > discussion started). That's certainly another viewpoint.
> > 
> > However it disconnects the actual connectors from the signals. What the
> > user sees in a device are connectors, not the signals themselves.
> > Naturally that approach does have its issues as well (such as break-out
> > boxes), but connectors are hardware and I believe we're not making a
> > wrong choice if our basic unit of modelling is a connector. I can't say
> > the same on modelling sets of signals instead.

I tend to agree with that, unless we really can't find a way to use such a 
model cleanly, I would favour a connector-based approach. Given the 
craziness^Wingenuity of hardware engineers I don't think any model will cover 
100% of the cases without any quirk, so I'm more interested in how the model 
can represent the hardware cleanly in most cases and not be too quirky in the 
corner cases.

> > Applications that present information to the user would likely be much
> > better off with connectors, and such applications certainly are a valid
> > use case.
> 
> Huh? you said above that you're against using "connectors"... now you're
> saying that you prefer using it???
> 
> The problem with "connector" is: sometimes the same physical connector is
> used to carry on different types of signals. This is very common with the
> S-video connector. We need to be able to distinguish between:
> 
> 	- Composite via a composite RCA connector;
> 	- S-Video (with Y+C signals) via a S-Video 4-pin miniDIN connector;
> 	- Composite using a S-Video 4-pin miniDIN connector.

Let's assume we try to model connectors (not a decision set in stone, but we 
have to start somewhere), here are a few comments. I'm deliberately addressing 
the complex cases at the end only, so please don't judge the first proposals 
as not applicable due to that reason before having read through the whole e-
mail.

- RCA and BNC connectors are pretty easy in that they can carry a single 
signal. They thus have a single source pad, there's no big question about how 
to model them.

- When several RCA or BNC connectors are meant to be used together without any 
meaningful option to use them independently their relationship should be 
conveyed to the user. This could be done through a (to be defined) method to 
group entities (possibly through properties).

- Another option for grouped connectors would be to expose a single connector 
entity to represent the group. Properties would then be used to describe the 
type and other parameters of each physical connector inside the group. The 
entity could still have one source pad per signal, but to make full use of the 
grouped connectors abstraction a single source pad that would represent the 
combination of all the signals that make the video signal might be best. That 
would simplify the MC representation by lowering the number of pads and links, 
and thus potentially simplify userspace.

- Mini-DIN connectors, when they're used to carry a single video signal (such 
as an S-Video video signal on a mini-DIN 4 connector for instance), can be 
represented by a single connector entity. As in the grouped RCA or BNC 
connectors case, we have the option to use one source pad per pin or a single 
source pad to represent the group of pins. The pros and cons of each option 
are in my opinion identical to those of the grouped RCAR or BNC connectors 
case.

- The source pad or source pads of the connector entities obviously need to be 
linked, usually to a video decoder. Two good examples were provided by Hans 
for the adv7604 and adv7842 chips at

http://hverkuil.home.xs4all.nl/adv7604.png

This board has four physical connectors : HDMI1 (entity 16), HDMI2 (entity 
17), VGA (entity 15) and component video (entity 13, this could be multiple 
physical connectors, I'm not sure about it).

http://hverkuil.home.xs4all.nl/adv7842.png

This board has five physical connectors : HDMI (entity 17), DVI-I (split into 
entities 16 for the digital side and 15 for the analog side), component video 
(entity 13, this could be multiple physical connectors, I'm not sure about 
it), and two BNC connectors that can be used either together for one S-Video 
signal (use case modeled by entity 19) or separately for one composite signal 
on the first BNC connector (use case modeled by entity 18).

Similarly to the above cases where pads can be instantiated for each pin or 
for each group of pin, the same options are possible on the video decoder 
side. We can, as in the graphs above, use one pad per analog input pin, or one 
pad per group of input pins. This gives us four possibilities to model links 
depending on whether we use the one pad per pin (1:1) or one pad per pins 
group (1:n) option on the connector (left-hand side of ->) and video decoder 
(right-hand side of ->).

a) 1:1 -> 1:1 - A straightforward case, one link would be created per 
electrical signal. Nothing would explicitly show those links as being related 
though, unless we provide some kind of link grouping, possibly through 
properties (but that might not be a useful feature).

b) 1:1 -> 1:n - All the connector(s) source pads would be linked to the same 
video decoder sink pad. This seems confusing to me, as up to now we have only 
used sink fanout in case of mutually exclusive links that can be selected by 
userspace. We could obviously change that, but I see little value in this 
option.

c) 1:n -> 1:1 - The single connector source pad would be linked to multiple 
video decoder sink pads. As with option b) this seems confusing to me, as this 
denotes until now the same signal being routed to multiple destinations. Once 
again we could change that, but I see little value in this option.

d) 1:n -> 1:n - The single connector source pad would be linked to a single 
video decoder sink pad. The link would represent multiple electrical signals 
and a single video signal. Nothing would explicitly show make the electrical 
signals apparent, unless we add properties to links for such purpose (but 
again that might not be a useful feature).

I would recommend option a) or option d), I believe options b) and c) are not 
good fits for what we're trying to achieve.

- It should be noted that whether to use a single source pad or multiple 
source pads in the above cases (and thus whether to use option a) or option d) 
for the links) are not mutually exclusive options. We could, if desired, 
standardize both options and offer them either at the discretion of driver 
developers, or with guidelines regarding how and when to use each.

- The single pad option produces more compact graphs that model video signals, 
which is what I would expect userspace applications, and by extension their 
users, to be interested in. On the cons side the option is slightly less 
versatile, which could cause issues in more complex cases, especially the 
multiplex connectors use cases. I'll detail this below.

- The multiple pads option produces more verbose graphs that model electrical 
signals, which offers more versatility. On the cons side is the increased 
verbosity that can be confusing applications, and by extension their users, as 
the graph makes it cumbersome to extract the list of available inputs (in the 
V4L2 input sense) and their relationships, at least when directly modeling 
physical connectors as entities.

- The adv7842 example shows how the multiple pads model could be used to solve 
the multiplexed connectors problem. by trying to model logical inputs instead 
of physical connectors as entities. Entities 18 and 19 represent two mutually 
exclusive (due to sharing electrical signals) usages of the two BNC 
connectors. The mutual exclusion is shown by the two entities being linked to 
the same sink pad of the video decoder.

This has several drawbacks though.

a) The DVI-I connector is split into two unrelated entities, which I believe 
is confusing for applications as the graph can't be used to tell whether the 
two entities correspond to a single DVI-I connector or two separate DVI-A and 
DVI-D connectors. This can in my opinion be fixed by merging the two entities 
into a six source pads entity without impacting the other similar uses of the 
logical input model in the graph.

b) The two BNC connectors are not apparent, they're modeled by two entities 
that hint there would be a connector for composite video with a single pin and 
a connector for S-Video with two pins. Again, I believe this is confusing for 
applications for the reasons already explained above.

c) The two links to pad AIN10 of the video decoder are also problematic in my 
opinion. A single electrical signal is in this case modeled by two separate 
links, for the purpose of making the mutual exclusion between the two logical 
inputs apparent. The adv7842 driver will still need to expose the S_INPUT API, 
as we can't expect applications to switch to using link setup with existing 
drivers (nor should we probably for new drivers similar to the existing 
consumer-oriented drivers). What will thus be the dynamic behaviour of those 
links ?

d) The number of entities will grow exponentially if more analog signals are 
exposed. If the board had four BNC connectors, any of them usable for a 
composite video signal, and any two of them usable for an S-Video video 
signal, you will need 16 logical inputs with the above model. I hope I don't 
have to convince anyone that the resulting graph would be very confusing for 
applications.

- Let's also keep in mind that the MC graph won't come out of nowhere, it will 
need to be constructed based on a hardware description. For PCI boards this 
shouldn't be too much of a problem given that drivers are free to describe the 
board in any way they wish. However, for DT-based systems, DT bindings mandate 
a system model that describes the hardware. We already have DT bindings for 
physical connectors, proposing competing DT bindings that would model logical 
inputs doesn't seem to have high chances of being accepted. We could, of 
course, create an input-based graph based on the physical DT description, but 
that might be challenging. We need to verify that it would be doable before 
committing to a model that we might end up not being able to construct.

- As Hans stated, calling the entities inputs isn't a good idea as we also 
need to use the same model for outputs (given that outputs could create issues 
specific to them, has any given it a try by the way ?). Furthermore, as Sakari 
mentioned, the words input and output are very vague and would have a high 
chance of being confusing. For these reason I believe the word connector to be 
a better description (and I know this is being challenged by a counterproposal 
that uses the word connection as a generic term for both logical inputs and 
outputs). Using connector entities to model something that isn't a physical 
connector but a logical input or output should in my opinion be avoided.

- Using connector entities with a single pad per video signal (possibly 
grouping multiple physical signals), with option d) for the link model, would 
have the interesting benefit of creating a 1:1 mapping between sink pads of 
the video decoders and the inputs it exposes through the V4L2 INPUT API, at 
least in the non-multiplexed connectors cases.

- Connectors are obviously not specific to video, and I don't think we have 
run the current proposal through the ALSA core developers. I believe we 
shouldn't standardize such an important part of the API without making sure 
that it can be usable for audio as well as video.

- The problem of multiplexed connectors remains if we model physical 
connectors as entities. I believe that's solvable though, I'll try to comment 
more about that tomorrow as I still have to refine my thoughts on the subject 
(all the above already took quite some time to clarify and write down).

> From software perspective, Composite over S-Video is a different
> mux configuration than S-Video (see saa7134 driver, for example).
> 
> Yet, if we represent the connection entity as proposed, using one
> entity for each different input/output alternative, and PADs for
> signals, we cover all cases.
> 
> We can even associate the physical connector associated with the
> connection via the properties API in the future.

That's an essential feature of the MC API, I don't want to leave it out of the 
design. We don't have to implement it now, but I want to make sure it will 
work as expected when used with the MC mode we're trying to design for 
connectors (regardless of what model we choose).

I hope the above thoughts show that the problem isn't straightforward and 
needs to be handled with all the diligent care that an upstream API should 
receive.

-- 
Regards,

Laurent Pinchart