is it possible to use single IOCTL to setup media pipeline?

is it possible to use single IOCTL to setup media pipeline?

[Zhang, Ning A]

Hello everyone

when we need to setup media pipeline, eg, for camera capture, media-ctl 
needs to be called multiple time to setup media link and subdev
formats, or similar code in a single application. this will use
multiple IOCTLs on "/dev/mediaX" and "/dev/v4l2-subdevY".

to setup media pipeline in userspace requires to fully understanding
the topology of the media stack. but the fact is only media driver
developer could know how to setup media pipeline. each time driver
updates, this would break userspace application if application
engineers don't know this change. In this case, if a IOCTL is designed
to setup media pipeline, no need to update applications, after driver
is updated.

this will not only benefit for design a single IOCTL, this also helps
to hide the detail of media pipeline, by load a binary blob which holds
information about how to setup pipeline, or hide it in bootloader/ACPI
tables/device tree, etc.

another benefit is save time for setup media pipeline, if there is a
PKI like "time for open camera". as my test, this will saves hundreds
of milliseconds.

is this acceptable?

BR.
Ning.

Re: is it possible to use single IOCTL to setup media pipeline?

[Tomasz Figa]

Hi Ning,

On Thu, Nov 22, 2018 at 11:52 AM Zhang, Ning A <ning.a.zhang@intel.com> wrote:
>
> Hello everyone
>
> when we need to setup media pipeline, eg, for camera capture, media-ctl
> needs to be called multiple time to setup media link and subdev
> formats, or similar code in a single application. this will use
> multiple IOCTLs on "/dev/mediaX" and "/dev/v4l2-subdevY".
>
> to setup media pipeline in userspace requires to fully understanding
> the topology of the media stack. but the fact is only media driver
> developer could know how to setup media pipeline. each time driver
> updates, this would break userspace application if application
> engineers don't know this change.

That's obviously a bug in the driver. Kernel interfaces must not
change in a way that are not compatible with the userspace.

> In this case, if a IOCTL is designed
> to setup media pipeline, no need to update applications, after driver
> is updated.
>
> this will not only benefit for design a single IOCTL, this also helps
> to hide the detail of media pipeline, by load a binary blob which holds
> information about how to setup pipeline, or hide it in bootloader/ACPI
> tables/device tree, etc.

Media pipeline configuration is specific to the use case. If you
hardcode it in the driver or bootloader, the user will not be able to
use any other use case than the hardcoded blob, which is unacceptable
for Linux drivers.

Instead, it sounds like your userspace should be designed in a way
that the media topology configuration is loaded from a configuration
file that you could either get from your kernel driver developer or
just maintain yourself based on any changes the media developers do.
Of course that's unrelated to the backwards compatibility issue, which
should not happen normally. The configuration file would be helpful
for handling future extensions and new hardware platforms.

>
> another benefit is save time for setup media pipeline, if there is a
> PKI like "time for open camera". as my test, this will saves hundreds
> of milliseconds.

For this problem, the proper solution would be to create an ioctl that
can aggregate setting multiple parts of the topology in one go. For
example, V4L2 has VIDIOC_S_CTRL for setting a control, but there is
also VIDIOC_S_EXT_CTRLS, which lets you set multiple controls in one
call. Something like VIDIOC_S_EXT_CTRLS for configuring the media
topology would solve the performance problem.

Best regards,
Tomasz

Re: is it possible to use single IOCTL to setup media pipeline?

[Zhang, Ning A]

Tomasz, Thank you for your answers.

learned a lot.

BR.
Ning.

在 2018-11-22四的 16:06 +0900，Tomasz Figa写道：
> Hi Ning,
> 
> On Thu, Nov 22, 2018 at 11:52 AM Zhang, Ning A <ning.a.zhang@intel.co
> m> wrote:
> > 
> > Hello everyone
> > 
> > when we need to setup media pipeline, eg, for camera capture,
> > media-ctl
> > needs to be called multiple time to setup media link and subdev
> > formats, or similar code in a single application. this will use
> > multiple IOCTLs on "/dev/mediaX" and "/dev/v4l2-subdevY".
> > 
> > to setup media pipeline in userspace requires to fully
> > understanding
> > the topology of the media stack. but the fact is only media driver
> > developer could know how to setup media pipeline. each time driver
> > updates, this would break userspace application if application
> > engineers don't know this change.
> 
> That's obviously a bug in the driver. Kernel interfaces must not
> change in a way that are not compatible with the userspace.

I met this issue :(

> 
> > In this case, if a IOCTL is designed
> > to setup media pipeline, no need to update applications, after
> > driver
> > is updated.
> > 
> > this will not only benefit for design a single IOCTL, this also
> > helps
> > to hide the detail of media pipeline, by load a binary blob which
> > holds
> > information about how to setup pipeline, or hide it in
> > bootloader/ACPI
> > tables/device tree, etc.
> 
> Media pipeline configuration is specific to the use case. If you
> hardcode it in the driver or bootloader, the user will not be able to
> use any other use case than the hardcoded blob, which is unacceptable
> for Linux drivers.
> 
> Instead, it sounds like your userspace should be designed in a way
> that the media topology configuration is loaded from a configuration
> file that you could either get from your kernel driver developer or
> just maintain yourself based on any changes the media developers do.
> Of course that's unrelated to the backwards compatibility issue,
> which
> should not happen normally. The configuration file would be helpful
> for handling future extensions and new hardware platforms.

yes, if there are multiple user cases, then this is not optional.

> 
> > 
> > another benefit is save time for setup media pipeline, if there is
> > a
> > PKI like "time for open camera". as my test, this will saves
> > hundreds
> > of milliseconds.
> 
> For this problem, the proper solution would be to create an ioctl
> that
> can aggregate setting multiple parts of the topology in one go. For
> example, V4L2 has VIDIOC_S_CTRL for setting a control, but there is
> also VIDIOC_S_EXT_CTRLS, which lets you set multiple controls in one
> call. Something like VIDIOC_S_EXT_CTRLS for configuring the media
> topology would solve the performance problem.

this may be alternative choice.

> 
> 
> Best regards,
> Tomasz

Re: is it possible to use single IOCTL to setup media pipeline?

[Sakari Ailus]

Hi Tomasz,

On Thu, Nov 22, 2018 at 04:06:36PM +0900, Tomasz Figa wrote:
> Hi Ning,
> 
> On Thu, Nov 22, 2018 at 11:52 AM Zhang, Ning A <ning.a.zhang@intel.com> wrote:
> >
> > Hello everyone
> >
> > when we need to setup media pipeline, eg, for camera capture, media-ctl
> > needs to be called multiple time to setup media link and subdev
> > formats, or similar code in a single application. this will use
> > multiple IOCTLs on "/dev/mediaX" and "/dev/v4l2-subdevY".
> >
> > to setup media pipeline in userspace requires to fully understanding
> > the topology of the media stack. but the fact is only media driver
> > developer could know how to setup media pipeline. each time driver
> > updates, this would break userspace application if application
> > engineers don't know this change.
> 
> That's obviously a bug in the driver. Kernel interfaces must not
> change in a way that are not compatible with the userspace.

Alternatively, this could result from fixing a bug in a driver. Or adding
features that were not previously supported.

In this case there are often no perfect solutions to address all the issues
at hand.

> 
> > In this case, if a IOCTL is designed
> > to setup media pipeline, no need to update applications, after driver
> > is updated.
> >
> > this will not only benefit for design a single IOCTL, this also helps
> > to hide the detail of media pipeline, by load a binary blob which holds
> > information about how to setup pipeline, or hide it in bootloader/ACPI
> > tables/device tree, etc.
> 
> Media pipeline configuration is specific to the use case. If you
> hardcode it in the driver or bootloader, the user will not be able to
> use any other use case than the hardcoded blob, which is unacceptable
> for Linux drivers.
> 
> Instead, it sounds like your userspace should be designed in a way
> that the media topology configuration is loaded from a configuration
> file that you could either get from your kernel driver developer or
> just maintain yourself based on any changes the media developers do.
> Of course that's unrelated to the backwards compatibility issue, which
> should not happen normally. The configuration file would be helpful
> for handling future extensions and new hardware platforms.
> 
> >
> > another benefit is save time for setup media pipeline, if there is a
> > PKI like "time for open camera". as my test, this will saves hundreds
> > of milliseconds.
> 
> For this problem, the proper solution would be to create an ioctl that
> can aggregate setting multiple parts of the topology in one go. For
> example, V4L2 has VIDIOC_S_CTRL for setting a control, but there is
> also VIDIOC_S_EXT_CTRLS, which lets you set multiple controls in one
> call. Something like VIDIOC_S_EXT_CTRLS for configuring the media
> topology would solve the performance problem.

There have been ideas of moving all IOCTL handling to the media device, in
a way that would allow issuing them in the same fashion than controls. This
was discussed in last year's media summit actually. I think this could be
done once the request API is otherwise working for e.g. camera devices. I
don't expect this to materialise in near (or even medium) term though.

-- 
Regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: is it possible to use single IOCTL to setup media pipeline?

[Tomasz Figa]

On Thu, Nov 22, 2018 at 10:46 PM Sakari Ailus <sakari.ailus@iki.fi> wrote:
>
> Hi Tomasz,
>
> On Thu, Nov 22, 2018 at 04:06:36PM +0900, Tomasz Figa wrote:
> > Hi Ning,
> >
> > On Thu, Nov 22, 2018 at 11:52 AM Zhang, Ning A <ning.a.zhang@intel.com> wrote:
> > >
> > > Hello everyone
> > >
> > > when we need to setup media pipeline, eg, for camera capture, media-ctl
> > > needs to be called multiple time to setup media link and subdev
> > > formats, or similar code in a single application. this will use
> > > multiple IOCTLs on "/dev/mediaX" and "/dev/v4l2-subdevY".
> > >
> > > to setup media pipeline in userspace requires to fully understanding
> > > the topology of the media stack. but the fact is only media driver
> > > developer could know how to setup media pipeline. each time driver
> > > updates, this would break userspace application if application
> > > engineers don't know this change.
> >
> > That's obviously a bug in the driver. Kernel interfaces must not
> > change in a way that are not compatible with the userspace.
>
> Alternatively, this could result from fixing a bug in a driver. Or adding
> features that were not previously supported.
>
> In this case there are often no perfect solutions to address all the issues
> at hand.

An upstream kernel driver must maintain compatibility even in such
cases, which isn't the most convenient thing for driver developers,
but that's something we have to live with if we want to have users.

>
> >
> > > In this case, if a IOCTL is designed
> > > to setup media pipeline, no need to update applications, after driver
> > > is updated.
> > >
> > > this will not only benefit for design a single IOCTL, this also helps
> > > to hide the detail of media pipeline, by load a binary blob which holds
> > > information about how to setup pipeline, or hide it in bootloader/ACPI
> > > tables/device tree, etc.
> >
> > Media pipeline configuration is specific to the use case. If you
> > hardcode it in the driver or bootloader, the user will not be able to
> > use any other use case than the hardcoded blob, which is unacceptable
> > for Linux drivers.
> >
> > Instead, it sounds like your userspace should be designed in a way
> > that the media topology configuration is loaded from a configuration
> > file that you could either get from your kernel driver developer or
> > just maintain yourself based on any changes the media developers do.
> > Of course that's unrelated to the backwards compatibility issue, which
> > should not happen normally. The configuration file would be helpful
> > for handling future extensions and new hardware platforms.
> >
> > >
> > > another benefit is save time for setup media pipeline, if there is a
> > > PKI like "time for open camera". as my test, this will saves hundreds
> > > of milliseconds.
> >
> > For this problem, the proper solution would be to create an ioctl that
> > can aggregate setting multiple parts of the topology in one go. For
> > example, V4L2 has VIDIOC_S_CTRL for setting a control, but there is
> > also VIDIOC_S_EXT_CTRLS, which lets you set multiple controls in one
> > call. Something like VIDIOC_S_EXT_CTRLS for configuring the media
> > topology would solve the performance problem.
>
> There have been ideas of moving all IOCTL handling to the media device, in
> a way that would allow issuing them in the same fashion than controls. This
> was discussed in last year's media summit actually. I think this could be
> done once the request API is otherwise working for e.g. camera devices. I
> don't expect this to materialise in near (or even medium) term though.

I'm aware of those talks and we actually took this into consideration
when developing the Request API.

Hopefully we can speed up the work on this to make it happen medium
term at latest, if not short term, as it's going to be quite important
for the complex camera subsystems, especially in the days of all the
counter measures against the speculative execution vulnerabilities.

Best regards,
Tomasz