RFCl libv4l2 plugin API

RFCl libv4l2 plugin API

[Hans de Goede]

Hi All,

RFC: a plugin API for libv4l2
=============================

During the Plumbers conference in 2009 various parties expresse interest
in adding a plugin API to libv4l2. Some hardware can do some
processing steps in hardware, but this needs to be setup from userspace
and sometimes still need some regulation from userspace as streaming
happens, hardware specific libv4l2 plugins could be a solution here.

During the v4l summit in Helsinki this was discussed further and a
simple and very generic plugin API was pitched. This is a first draft
specification for this API.


Basic libv4l2 principles
------------------------

The basic unit libv4l2's API deals with is a /dev/video node represented
by a fd. A libv4l2 plugin will sit in between libv4l2 itself and the
actual /dev/video device node a fd refers to. It will be called each time
libv4l2 wants to do an operation (read/write/ioctl/mmap/munmap) on the
actual /dev/video node in question. When called the plugin can then choose
to do one of the following:
1. Pass the call unmodified to the fd, and return the return value unmodifed
    (iow do nothing)
2. Modify some arguments in the call and pass it through
3. Modify the return(ed) value(s) of a passed through call
4. Not do any operation on the fd at all but instead completely fake it
    (which opens the possibility for "fake" v4l devices)

Note that:
1. A plugin may decide between 1. - 4. on a per call basis depending on the
operations arguments and / or state. This esp. is important with the ioctl
operation where a plugin may want to intercept some ioctls but not all
2. If a plugin always wants to pass all calls through for a certain
operation, it is allowed to simply not define a callback function for that
operation
3. As libv4l2 does some "faking" itself esp. when it comes to format
conversion a single libv4l2 call may result in multiple calls to the plugin,
or to none at all.

So to summarize the above: Anything goes :)


libv4l2 plugin loading
----------------------

A libv4l2 plugin is a .so file which lives under /usr/lib[64]/libv4l/plugins.
This .so file defines a single symbol: libv4l2_plugin. This symbol is a
(constant) libv4l_plugin_data struct, which contains pointers to all the
callbacks the plugin wants to install.

libv4l2 plugin loading begins with the application using libv4l2 doing a
v4l2_open() for example v4l2_open("foobar", O_RDWR); When this happens
libv4l2 will dlopen all files found under /usr/lib[64]/libv4l/plugins 1 at a
time and call open callback passing through the applications parameters
unmodified. If a plugin is relevant for the specified device node, it can
indicate so by returning a value other then -1 from the open callback
(usually the result of an actual open call on the device node). If it
is not relevant for this specific video node it should return -1.

As soon as a plugin returns another value then -1 plugin loading stops
and that plugins callbacks are called for all operations on the /dev/video#
node in question. This means that plugins cannot be stacked!

Note that a plugin is bound to a specific fd, this way if there is for
example a system with a SOC connected camera and an external usb camera, the
SOC specific plugin will only be involved when an application is talking to
the SOC connected camera and not when the app talks to the USB webcam.

Note that a plugin is bound to a specific fd not to a device node! IOW it is
possible for a plugin to return not -1 from its open callback when called
on /dev/video0 with O_RDWR, and to return -1 when called on /dev/video0 with
O_RDONLY. Then its callbacks will only get called when operations are done
on the video node though the RW fd, and not when operations are done on the
same node through the RD_ONLY fd. This is not a good idea! It is only meant
to emphasize the binding of a plugin to the fd, rather then to a specific
device node.


libv4l2 plugins and private data
--------------------------------

libv4l2 plugins should *never* use any global variables. All data should be
bound to the specific fd to which the plugin is bound. This ensures that for
example a plugin for a specific type of usb webcam will also work when 2
identical cameras are plugged into a system (and both are used from the same
process).

A libv4l2 plugin can register plugin private data using:
void libv4l2_set_plugindata(int fd, void *plugin_data);

And can get this data out of libv4l2 again inside a callback using:
void *libv4l2_get_plugindata(int fd);

Note that a plugin should call libv4l2_set_plugindata only once per fd !
Calling it a second time will overwrite the previous value. The logical
place to use libv4l2_set_plugindata is from the plugin's open callback.


libv4l2-plugin.h
----------------

/* Plugin callback function struct */
struct libv4l2_plugin_data {
	int (*open)(const char *file, int oflag, ...);
	int (*close)(int fd);
	int (*ioctl)(int fd, unsigned long int request, ...);
	ssize_t (*read)(int fd, void *buffer, size_t n);
	void *(*mmap)(void *start, size_t length, int prot, int flags,
			int fd, int64_t offset);
	/* Note as munmap has no fd argument, defining a callback for munmap
	   will result in it getting called for *any* call to v4l2_munmap. So
	   if a plugin defines a callback for munmap (because for example it
	   returns fake mmap buffers from its mmap callback). Then it must
	   keep track of the addresses at which these buffers live and their
	   size and check the munmap arguments to see if the munmap call was
	   meant for it. */
	int (*munmap)(void *_start, size_t length);
};

/* Plugin utility functions */
void libv4l2_set_plugindata(int fd, void *plugin_data);
void *libv4l2_get_plugindata(int fd);

Regards,

Hans

Re: RFCl libv4l2 plugin API

[Sakari Ailus]

Hans de Goede wrote:
> Hi All,

Hi Hans,

Thanks for the RFC!

I'd have a few comments and questions.

The coding style for libv4l hasn't been defined as far as I understand.
Should kernel coding style be assumed, or something else?

> RFC: a plugin API for libv4l2
> =============================
> 
> During the Plumbers conference in 2009 various parties expresse interest
> in adding a plugin API to libv4l2. Some hardware can do some
> processing steps in hardware, but this needs to be setup from userspace
> and sometimes still need some regulation from userspace as streaming
> happens, hardware specific libv4l2 plugins could be a solution here.
> 
> During the v4l summit in Helsinki this was discussed further and a
> simple and very generic plugin API was pitched. This is a first draft
> specification for this API.
> 
> 
> Basic libv4l2 principles
> ------------------------
> 
> The basic unit libv4l2's API deals with is a /dev/video node represented
> by a fd. A libv4l2 plugin will sit in between libv4l2 itself and the
> actual /dev/video device node a fd refers to. It will be called each time
> libv4l2 wants to do an operation (read/write/ioctl/mmap/munmap) on the
> actual /dev/video node in question. When called the plugin can then choose
> to do one of the following:
> 1. Pass the call unmodified to the fd, and return the return value
> unmodifed
>    (iow do nothing)
> 2. Modify some arguments in the call and pass it through
> 3. Modify the return(ed) value(s) of a passed through call
> 4. Not do any operation on the fd at all but instead completely fake it
>    (which opens the possibility for "fake" v4l devices)
> 
> Note that:
> 1. A plugin may decide between 1. - 4. on a per call basis depending on the
> operations arguments and / or state. This esp. is important with the ioctl
> operation where a plugin may want to intercept some ioctls but not all
> 2. If a plugin always wants to pass all calls through for a certain
> operation, it is allowed to simply not define a callback function for that
> operation
> 3. As libv4l2 does some "faking" itself esp. when it comes to format
> conversion a single libv4l2 call may result in multiple calls to the
> plugin,
> or to none at all.
> 
> So to summarize the above: Anything goes :)

Plugin internal data bound to file descriptors make probably sense for
webcams, but we can have more complex setups than that. For example the
OMAP 3 ISP driver. Several video nodes may be used to capture video, and
there are dependencies between the nodes.

Essentially this kind of plugin can handle one media device at a time,
not a V4L2 device. For webcams etc. this would still be the same since
libv4l isn't aware of media devices.

However, the application is a V4L2 application which works on V4L2
devices and may well be unaware of the media device. I don't think this
is _necessarily_ a problem since the plugin can do whatever it sees fit
for user's request; it could even open another video node.

For example, there is likely one device node a general purpose
application is interested in in the OMAP 3 ISP driver.

Obviously, there can be only one of this kind of media device control
plugins at a time. Other plugins should have their say before that, so
that the device appears a regular V4L2 device for those plugins as well.

The order in which the plugins are executed matters also if they do
image processing. The result may well be different.

Perhaps it'd be good to experiment with one plugin per device first? :-)

What do you think?

Regards,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com

Re: RFCl libv4l2 plugin API

[Hans de Goede]

Hi Sakari,

On 10/28/2010 08:30 AM, Hans de Goede wrote:
> Hans de Goede wrote:
>  > Hi All,
>
> Hi Hans,
>
> Thanks for the RFC!
>
> I'd have a few comments and questions.
>
> The coding style for libv4l hasn't been defined as far as I understand.
> Should kernel coding style be assumed, or something else?

v4l-utils uses kernel coding style.

>
>  > RFC: a plugin API for libv4l2
>  > =============================
>  >
>  > During the Plumbers conference in 2009 various parties expresse interest
>  > in adding a plugin API to libv4l2. Some hardware can do some
>  > processing steps in hardware, but this needs to be setup from userspace
>  > and sometimes still need some regulation from userspace as streaming
>  > happens, hardware specific libv4l2 plugins could be a solution here.
>  >
>  > During the v4l summit in Helsinki this was discussed further and a
>  > simple and very generic plugin API was pitched. This is a first draft
>  > specification for this API.
>  >
>  >
>  > Basic libv4l2 principles
>  > ------------------------
>  >
>  > The basic unit libv4l2's API deals with is a /dev/video node represented
>  > by a fd. A libv4l2 plugin will sit in between libv4l2 itself and the
>  > actual /dev/video device node a fd refers to. It will be called each time
>  > libv4l2 wants to do an operation (read/write/ioctl/mmap/munmap) on the
>  > actual /dev/video node in question. When called the plugin can then choose
>  > to do one of the following:
>  > 1. Pass the call unmodified to the fd, and return the return value
>  > unmodifed
>  > (iow do nothing)
>  > 2. Modify some arguments in the call and pass it through
>  > 3. Modify the return(ed) value(s) of a passed through call
>  > 4. Not do any operation on the fd at all but instead completely fake it
>  > (which opens the possibility for "fake" v4l devices)
>  >
>  > Note that:
>  > 1. A plugin may decide between 1. - 4. on a per call basis depending on the
>  > operations arguments and / or state. This esp. is important with the ioctl
>  > operation where a plugin may want to intercept some ioctls but not all
>  > 2. If a plugin always wants to pass all calls through for a certain
>  > operation, it is allowed to simply not define a callback function for that
>  > operation
>  > 3. As libv4l2 does some "faking" itself esp. when it comes to format
>  > conversion a single libv4l2 call may result in multiple calls to the
>  > plugin,
>  > or to none at all.
>  >
>  > So to summarize the above: Anything goes :)
>
> Plugin internal data bound to file descriptors make probably sense for
> webcams, but we can have more complex setups than that. For example the
> OMAP 3 ISP driver. Several video nodes may be used to capture video, and
> there are dependencies between the nodes.
>
> Essentially this kind of plugin can handle one media device at a time,
> not a V4L2 device. For webcams etc. this would still be the same since
> libv4l isn't aware of media devices.
>
> However, the application is a V4L2 application which works on V4L2
> devices and may well be unaware of the media device. I don't think this
> is _necessarily_ a problem since the plugin can do whatever it sees fit
> for user's request; it could even open another video node.

Yes this is the whole idea, the app uses just one video node, like with
any regular v4l device, and then the plugin can open video nodes
for "sub-devices" to hook up all the plumbing to get a pipeline which
does what the app wants.

> For example, there is likely one device node a general purpose
> application is interested in in the OMAP 3 ISP driver.

Right.

> Obviously, there can be only one of this kind of media device control
> plugins at a time. Other plugins should have their say before that, so
> that the device appears a regular V4L2 device for those plugins as well.

The current "design" (the RFC) assumes only one plugin per /dev/video#
device, so no stacking of plugins on the same fd. It is possible to
have multiple plugins, but only one will get "attached" to a certain
fd. The idea here is that plugins are meant for doing certain hardware
specific stuff. And different /dev/video# nodes could relate
to completely different hardware, so we do need multiple plugins to
support this. But as the purpose of the plugins is to deal with certain
hardware specific things, having one plugin per type of hardware seems
enough.

> The order in which the plugins are executed matters also if they do
> image processing. The result may well be different.

As said, there can only be one plugin per fd.

> Perhaps it'd be good to experiment with one plugin per device first? :-)

Right, well one plugin per fd, as libv4l does everything at the fd level.

Note that currently besides this RFC the plugin support is completely
vaporware but it should be quite easy to implement the RFC, patches
welcome :)

Regards,

Hans

Re: RFCl libv4l2 plugin API

[Sakari Ailus]

Hans de Goede wrote:
> Hi Sakari,

Hi Hans,

> On 10/28/2010 08:30 AM, Hans de Goede wrote:
>> Hans de Goede wrote:
>>  > Hi All,
>>
>> Hi Hans,
>>
>> Thanks for the RFC!
>>
>> I'd have a few comments and questions.
>>
>> The coding style for libv4l hasn't been defined as far as I understand.
>> Should kernel coding style be assumed, or something else?
> 
> v4l-utils uses kernel coding style.

Good! A well defined coding style is always a plus. The current codebase
doesn't quite follow it at the moment, though. (And I don't have a patch
either.) New patches should still pass checkpatch.pl, I understand.

I Cc'd Yordan who is working on the plugin interface.

...
>> However, the application is a V4L2 application which works on V4L2
>> devices and may well be unaware of the media device. I don't think this
>> is _necessarily_ a problem since the plugin can do whatever it sees fit
>> for user's request; it could even open another video node.
> 
> Yes this is the whole idea, the app uses just one video node, like with
> any regular v4l device, and then the plugin can open video nodes
> for "sub-devices" to hook up all the plumbing to get a pipeline which
> does what the app wants.

Ok.

Video nodes that are related to the ISP driver may not be opened as long
as one of them is in use. They use the same resources --- the ISP.
Multiple opens are possible but for the use case (libv4l) it makes no
sense since libv4l is just for capture.

V4L2 allows multiple opens but if the MC setup is done from both that
may easily result in an invalid configuration. So for the time being, I
think we will be limited to just one application per (MC) device for the
time being.

That's specific to OMAP 3 plugin, however.

>> Obviously, there can be only one of this kind of media device control
>> plugins at a time. Other plugins should have their say before that, so
>> that the device appears a regular V4L2 device for those plugins as well.
> 
> The current "design" (the RFC) assumes only one plugin per /dev/video#
> device, so no stacking of plugins on the same fd. It is possible to
> have multiple plugins, but only one will get "attached" to a certain
> fd. The idea here is that plugins are meant for doing certain hardware
> specific stuff. And different /dev/video# nodes could relate
> to completely different hardware, so we do need multiple plugins to
> support this. But as the purpose of the plugins is to deal with certain
> hardware specific things, having one plugin per type of hardware seems
> enough.

Sounds good.

>> The order in which the plugins are executed matters also if they do
>> image processing. The result may well be different.
> 
> As said, there can only be one plugin per fd.

This makes things easy indeed.

Thanks for your comments.

Regards,

-- 
Sakari Ailus
sakari.ailus@maxwell.research.nokia.com