RFC: video device (stream) sharing

RFC: video device (stream) sharing

[Hans de Goede]

Hi All,

This has been discussed before and this is something Brandon and I would like
to discuss further at plumbers, so here is a first braindump, note that this
braindump is purely mine and not Brandon's in any way.

The basic idea is to have some sort of userspace proxy process which allows
sharing for example a webcam between 2 devices. For me there are 2 major
criteria which need to be matched to be able to do this:

1) No (as in 0) functionality regressions for the single use case, iow when
    only one app opens the device everything should work as before

2) No significant performance regressions for the single use case. Sure this
    may be a bit slower, but not much!

2) Will require some trickery with shared memory, etc. But the real hard problem
here is 1), so I will purely focus on 1) now.


My initial idea to solve 1) is that as soon as an application does anything
remotely stream (capture) related even something such as enum_fmt, it becomes the
stream owner. The stream owner is allowed to do everything. Any second application
which also wants to capture will only be shown the resolution and format currently
selected by the stream owner.

And here we immediately hit a problem. Imagine the following:
1) The user starts cheese at 640x480
2) The user starts application foo, which only sees 640x480 to and thus
    starts capturing at 640x480
3) The user changes the resolution in cheese to 320x240

Now we've got a problem, because cheese is allowed to do this, but we need 640x480
for application foo -> fail. And I'm not even talking about possible races when
cheese has become the stream owner, but has not yet choosen its format to stream in,
etc.


So the whole stream owner concept does not work. Instead, what would probably work, is
the following:
-limit the amount of reported supported formats (enum fmt) to formats which we can create
  by conversion from native formats
-report the full list of supported resolutions to all applications
-capture at the highest resolution requested by any of the applications
-downsample for applications which want a lower resolution


So this is how I suggest to handle this.

Regards,

Hans

Re: RFC: video device (stream) sharing

[Dongsoo, Nathaniel Kim]

On Fri, Aug 28, 2009 at 3:39 PM, Hans de Goede<hdegoede@redhat.com> wrote:
> Hi All,
>
> This has been discussed before and this is something Brandon and I would
> like
> to discuss further at plumbers, so here is a first braindump, note that this
> braindump is purely mine and not Brandon's in any way.
>
> The basic idea is to have some sort of userspace proxy process which allows
> sharing for example a webcam between 2 devices. For me there are 2 major
> criteria which need to be matched to be able to do this:
>
> 1) No (as in 0) functionality regressions for the single use case, iow when
>   only one app opens the device everything should work as before
>
> 2) No significant performance regressions for the single use case. Sure this
>   may be a bit slower, but not much!
>
> 2) Will require some trickery with shared memory, etc. But the real hard
> problem
> here is 1), so I will purely focus on 1) now.
>
>
> My initial idea to solve 1) is that as soon as an application does anything
> remotely stream (capture) related even something such as enum_fmt, it
> becomes the
> stream owner. The stream owner is allowed to do everything. Any second
> application
> which also wants to capture will only be shown the resolution and format
> currently
> selected by the stream owner.
>
> And here we immediately hit a problem. Imagine the following:
> 1) The user starts cheese at 640x480
> 2) The user starts application foo, which only sees 640x480 to and thus
>   starts capturing at 640x480
> 3) The user changes the resolution in cheese to 320x240
>
> Now we've got a problem, because cheese is allowed to do this, but we need
> 640x480
> for application foo -> fail. And I'm not even talking about possible races
> when
> cheese has become the stream owner, but has not yet choosen its format to
> stream in,
> etc.
>
>
> So the whole stream owner concept does not work. Instead, what would
> probably work, is
> the following:
> -limit the amount of reported supported formats (enum fmt) to formats which
> we can create
>  by conversion from native formats
> -report the full list of supported resolutions to all applications
> -capture at the highest resolution requested by any of the applications
> -downsample for applications which want a lower resolution
>

Using the highest resolution as a representative resolution might be a
great obstacle in case of using a high resolution camera like 8mega
pixel or over. the highest resolution of 8 mega pixel camera is
literally 8mega pixel which is 4096x1664 in common. And also should be
a great burden for users to downsample the image of  8mega pixel.

In embedded devices, I may suggest in this way.
Considering whether there is any resizer like codec device or not,
there might be a couple of cases like:
(Please note that I'm not aware of the way of how to handle codec
device properly in v4l2 sense. therefor just describing in abstract
way)

1) With codec device
If there is any kind of codec device embedded in the SoC platform
which can be used in global, the codec device might being assigned a
camera device as an input. And making codec device available for
multiple open, user could be using the same camera as an image source
and use different destination image resolution for each instance. And
this case, the codec device driver should arrange each instance for
different resolutions.

2) Without codec device
This case might be more neat and simpler.
The process which opens the camera device first should be the "stream
owner" but if there is any other process opening the device *again*,
any of those process should have EBUSY for return of s_fmt, s_parm and
any kind of setting ioctls. Then the device which is opened by
multiple processes, will stay in the exact setting of resolution, fps
state until it gets closed.

I know that there is no rules for codec device, but in my opinion if
the framework wants to provide a feature of single image source and
multiple destination format(resolution), codec device must be
involved.
Cheers,

Nate

>
> So this is how I suggest to handle this.
>
> Regards,
>
> Hans
> --
> To unsubscribe from this list: send the line "unsubscribe linux-media" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>



-- 
=
DongSoo, Nathaniel Kim
Engineer
Mobile S/W Platform Lab.
Digital Media & Communications R&D Centre
Samsung Electronics CO., LTD.
e-mail : dongsoo.kim@gmail.com
          dongsoo45.kim@samsung.com

Re: RFC: video device (stream) sharing

[Brandon Philips]

On 08:39 Fri 28 Aug 2009, Hans de Goede wrote:
> The basic idea is to have some sort of userspace proxy process which allows
> sharing for example a webcam between 2 devices. For me there are 2 major
> criteria which need to be matched to be able to do this:
> 
> 1) No (as in 0) functionality regressions for the single use case, iow when
>    only one app opens the device everything should work as before
>
> 2) No significant performance regressions for the single use case. Sure this
>    may be a bit slower, but not much!

Agreed and it should be possible to do with shm and a fast daemon.

> So the whole stream owner concept does not work. 

Agreed. This will need to be a per user session daemon like PulseAudio
is by default.

> -limit the amount of reported supported formats (enum fmt) to
>  formats which we can create by conversion from native formats

Agreed.

> -report the full list of supported resolutions to all applications
> -capture at the highest resolution requested by any of the
>  applications

All supported resolutions that don't cause a drop in frame capture
rate that the other applications are getting. I am not sure how this
heuristic will end up working and at this point I am a long way from
tackling this problem.

I am guessing that applications can provide hints on what they need
from the hardware.

But, alot of my interest is in enabling interesting things to be done
in that other operating systems can do:

 - Face tracking and panning
 - Software autofocus
 - "Take photo of the laptop theif" software running in background
 - Backup your camera output while Pidgin/Skype/etc is running

These sorts of applications will be fine with the same data format
that the current running application is using. 

> -downsample for applications which want a lower resolution

Yes.

I have been reading quite a bit of the PulseAudio code while I have
been working on my prototype. My hope is to ensure that we get the
implementation of this daemon and library right enough so as to avoid
layers upon layers of APIs

 http://rudd-o.com/en/linux-and-free-software/how-pulseaudio-works/images/pulseaudio-diagram.png?isImage=1

:D

Thanks for the input.

Cheers,

	Brandon