[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

Hi,
we are planning to add support to H.264/MPEG4 encoder inside the
i.MX27 to v4l2. It is mainly a hardware module that has the following
features:

- It needs two input buffers (current frame and previous frame).
- It produces a third buffer as output, containing the encoded frame,
and generates an IRQ when that happens.
- Previous three buffers need contiguous physical memory addresses and
probably some alignment requirements.
- It needs an external firmware to be loaded in another contiguous
memory buffer.

I would like to know what is your opinion on this, what v4l2 framework
should we use to deal with it, etc... I guess Multi Format Codec 5.1
driver for s5pv210 and exynos4 SoC is the most similar piece of HW
I've found so far but it has not yet entered mainline [1]

Note that mx2_camera driver is still using soc-camera framework and
soc-camera doesn't seem to be ready for integration with pad level API
[2]. For that reason we think we could develop this VPU driver
separately.

[1] http://www.spinics.net/lists/linux-media/msg35040.html
[2] http://www.open-technology.de/index.php?/categories/2-SoC-camera

-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Guennadi Liakhovetski]

Hi Javier

On Tue, 6 Sep 2011, javier Martin wrote:

> Hi,
> we are planning to add support to H.264/MPEG4 encoder inside the
> i.MX27 to v4l2. It is mainly a hardware module that has the following
> features:
> 
> - It needs two input buffers (current frame and previous frame).
> - It produces a third buffer as output, containing the encoded frame,
> and generates an IRQ when that happens.
> - Previous three buffers need contiguous physical memory addresses and
> probably some alignment requirements.
> - It needs an external firmware to be loaded in another contiguous
> memory buffer.
> 
> I would like to know what is your opinion on this, what v4l2 framework
> should we use to deal with it, etc... I guess Multi Format Codec 5.1
> driver for s5pv210 and exynos4 SoC is the most similar piece of HW
> I've found so far but it has not yet entered mainline [1]
> 
> Note that mx2_camera driver is still using soc-camera framework and
> soc-camera doesn't seem to be ready for integration with pad level API
> [2]. For that reason we think we could develop this VPU driver
> separately.

(The following is my understanding and opinion, all corrections welcome)

The MC API is important, when data can be passed directly between modules, 
using some internal busses. I.e., if you could just configure your VPU to 
receive the data directly from the camera capture module, without the use 
of memory buffers, then yes, you would want an MC-like API, and an ability 
to connect the camera module to the VPU using pads and links and configure 
each entity from the user-space separately, maintaining the format 
compatibility along the links.

In your case, however, the VPU uses memory buffers. I.e., what you want is 
some zero-copy buffer passing from the camera module to the VPU. I don't 
think the MC API is very helpful in this situation. With the currently 
available options you want to use USERPTR buffers and pass them from the 
source to the sink in your user-space application. There are also some 
buffer-sharing options approaching, but as long as this is just one 
application, that tosses buffers between two devices, you, probably, don't 
need those very much either. So, this looks like a "simple" mem2mem driver 
case to me.

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Robert Schwebel]

Hi,

On Tue, Sep 06, 2011 at 05:07:54PM +0200, javier Martin wrote:
> we are planning to add support to H.264/MPEG4 encoder inside the
> i.MX27 to v4l2. It is mainly a hardware module that has the following
> features:

We have driver prototypes for the MX27 VPU (which is basically the same
as on MX51/MX53, just an earlier version). Sascha Hauer can probably
elaborate about the details, but he is on holiday for this and next week
and will comment then.

rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Sakari Ailus]

javier Martin wrote:
> Hi,

Hi Javier,

> we are planning to add support to H.264/MPEG4 encoder inside the
> i.MX27 to v4l2. It is mainly a hardware module that has the following
> features:
> 
> - It needs two input buffers (current frame and previous frame).
> - It produces a third buffer as output, containing the encoded frame,
> and generates an IRQ when that happens.
> - Previous three buffers need contiguous physical memory addresses and
> probably some alignment requirements.

CMA (contiguous memory allocator). This isn't in mainline yet, so so'll
need to wait a little bit.

> - It needs an external firmware to be loaded in another contiguous
> memory buffer.

This needs to be loaded using request_firmware().

> I would like to know what is your opinion on this, what v4l2 framework
> should we use to deal with it, etc... I guess Multi Format Codec 5.1
> driver for s5pv210 and exynos4 SoC is the most similar piece of HW
> I've found so far but it has not yet entered mainline [1]

It sounds to me like that this device should be supported using the V4L2
interface.

> Note that mx2_camera driver is still using soc-camera framework and
> soc-camera doesn't seem to be ready for integration with pad level API
> [2]. For that reason we think we could develop this VPU driver
> separately.
> 
> [1] http://www.spinics.net/lists/linux-media/msg35040.html
> [2] http://www.open-technology.de/index.php?/categories/2-SoC-camera

As Guennadi noted, the Media controller framework should be used to
expose the control of more complex devices to user space than SoC camera
can support.

However, it sounds like to me that the video produced by the camera has
to be written to the system memory before it is processed by the
H.264/MPEG4 encoder. For this reason, I don't see there would be a need
to connect the camera driver to the encoder in kernel. Or am I wrong in
thinking that these two are separate devices?

Kind regards,

-- 
Sakari Ailus
sakari.ailus@iki.fi

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Guennadi Liakhovetski]

On Thu, 15 Sep 2011, Sakari Ailus wrote:

> javier Martin wrote:
> > Hi,
> 
> Hi Javier,
> 
> > we are planning to add support to H.264/MPEG4 encoder inside the
> > i.MX27 to v4l2. It is mainly a hardware module that has the following
> > features:
> > 
> > - It needs two input buffers (current frame and previous frame).
> > - It produces a third buffer as output, containing the encoded frame,
> > and generates an IRQ when that happens.
> > - Previous three buffers need contiguous physical memory addresses and
> > probably some alignment requirements.
> 
> CMA (contiguous memory allocator). This isn't in mainline yet, so so'll
> need to wait a little bit.
> 
> > - It needs an external firmware to be loaded in another contiguous
> > memory buffer.
> 
> This needs to be loaded using request_firmware().
> 
> > I would like to know what is your opinion on this, what v4l2 framework
> > should we use to deal with it, etc... I guess Multi Format Codec 5.1
> > driver for s5pv210 and exynos4 SoC is the most similar piece of HW
> > I've found so far but it has not yet entered mainline [1]
> 
> It sounds to me like that this device should be supported using the V4L2
> interface.
> 
> > Note that mx2_camera driver is still using soc-camera framework and
> > soc-camera doesn't seem to be ready for integration with pad level API
> > [2]. For that reason we think we could develop this VPU driver
> > separately.
> > 
> > [1] http://www.spinics.net/lists/linux-media/msg35040.html
> > [2] http://www.open-technology.de/index.php?/categories/2-SoC-camera
> 
> As Guennadi noted, the Media controller framework should be used to
> expose the control of more complex devices to user space than SoC camera
> can support.

Hm, did I say that? Even more, as I write, I'm testing an soc-camera 
extension to use the MC / pad-level API, implementing what I've proposed 
here

http://article.gmane.org/gmane.linux.drivers.video-input-infrastructure/38083

> However, it sounds like to me that the video produced by the camera has
> to be written to the system memory before it is processed by the
> H.264/MPEG4 encoder. For this reason, I don't see there would be a need
> to connect the camera driver to the encoder in kernel. Or am I wrong in
> thinking that these two are separate devices?

Now, that's much more like what I've said here:

http://article.gmane.org/gmane.linux.drivers.video-input-infrastructure/37826

Thanks
Guennadi
---
Guennadi Liakhovetski, Ph.D.
Freelance Open-Source Software Developer
http://www.open-technology.de/

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Sakari Ailus]

On Thu, Sep 15, 2011 at 07:48:36PM +0200, Guennadi Liakhovetski wrote:
> On Thu, 15 Sep 2011, Sakari Ailus wrote:
> 
> > javier Martin wrote:
> > > Hi,
> > 
> > Hi Javier,
> > 
> > > we are planning to add support to H.264/MPEG4 encoder inside the
> > > i.MX27 to v4l2. It is mainly a hardware module that has the following
> > > features:
> > > 
> > > - It needs two input buffers (current frame and previous frame).
> > > - It produces a third buffer as output, containing the encoded frame,
> > > and generates an IRQ when that happens.
> > > - Previous three buffers need contiguous physical memory addresses and
> > > probably some alignment requirements.
> > 
> > CMA (contiguous memory allocator). This isn't in mainline yet, so so'll
> > need to wait a little bit.
> > 
> > > - It needs an external firmware to be loaded in another contiguous
> > > memory buffer.
> > 
> > This needs to be loaded using request_firmware().
> > 
> > > I would like to know what is your opinion on this, what v4l2 framework
> > > should we use to deal with it, etc... I guess Multi Format Codec 5.1
> > > driver for s5pv210 and exynos4 SoC is the most similar piece of HW
> > > I've found so far but it has not yet entered mainline [1]
> > 
> > It sounds to me like that this device should be supported using the V4L2
> > interface.
> > 
> > > Note that mx2_camera driver is still using soc-camera framework and
> > > soc-camera doesn't seem to be ready for integration with pad level API
> > > [2]. For that reason we think we could develop this VPU driver
> > > separately.
> > > 
> > > [1] http://www.spinics.net/lists/linux-media/msg35040.html
> > > [2] http://www.open-technology.de/index.php?/categories/2-SoC-camera
> > 
> > As Guennadi noted, the Media controller framework should be used to
> > expose the control of more complex devices to user space than SoC camera
> > can support.
> 
> Hm, did I say that? Even more, as I write, I'm testing an soc-camera 
> extension to use the MC / pad-level API, implementing what I've proposed 
> here
> 
> http://article.gmane.org/gmane.linux.drivers.video-input-infrastructure/38083
> 
> > However, it sounds like to me that the video produced by the camera has
> > to be written to the system memory before it is processed by the
> > H.264/MPEG4 encoder. For this reason, I don't see there would be a need
> > to connect the camera driver to the encoder in kernel. Or am I wrong in
> > thinking that these two are separate devices?
> 
> Now, that's much more like what I've said here:
> 
> http://article.gmane.org/gmane.linux.drivers.video-input-infrastructure/37826

Thanks for the corrections; they're indeed right I must have read your
reply to Javier carelessly. :p

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi	jabber/XMPP/Gmail: sailus@retiisi.org.uk

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

OK,
thank you for your interest, it seems quite clear to me now.
However, I'll wait to see what Sascha has to say about this to avoid
duplication of work.


-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

Fabio,

On 8 June 2012 08:51, javier Martin <javier.martin@vista-silicon.com> wrote:
> Hi Fabio,
>
> On 7 June 2012 19:35, Fabio Estevam <festevam@gmail.com> wrote:
>> Hi Javier,
>>
>> On Thu, Jun 7, 2012 at 5:30 AM, javier Martin
>> <javier.martin@vista-silicon.com> wrote:
>>
>>> As i stated, the driver is still in an early development stage, it
>>> doesn't do anything useful yet. But this is the public git repository
>>> if you want to take a look:
>>>
>>> git repo: https://github.com/jmartinc/video_visstrim.git
>>> branch:  mx27-codadx6
>>
>> Thanks, I will take a look at your tree when I am back to the office next week.
>>
>> I also see that Linaro's tree has support for VPU for mx5/mx6:
>> http://git.linaro.org/gitweb?p=landing-teams/working/freescale/kernel.git;a=summary
>>
>> ,so we should probably think in unifying it with mx27 support there too.

If you refer to driver in [1] I have some concerns: i.MX27 VPU should
be implemented as a V4L2 mem2mem device since it gets raw pictures
from memory and outputs encoded frames to memory (some discussion
about the subject can be fond here [2]), as Exynos driver from Samsung
does. However, this driver you've mentioned doesn't do that: it just
creates several mapping regions so that the actual functionality is
implemented in user space by a library provided by Freescale, which
regarding i.MX27 it is also GPL.

What we are trying to do is implementing all the functionality in
kernel space using mem2mem V4L2 framework so that it can be accepted
in mainline.

Please, correct me if the driver you are talking about is not the one in [1].

Regards.

[1] http://git.linaro.org/gitweb?p=landing-teams/working/freescale/kernel.git;a=blob;f=drivers/mxc/vpu/mxc_vpu.c;h=27b09e56d5a3f6cb7eeba16fe5493cbec46c65cd;hb=d0f289f67f0ff403d92880c410b009f1fd4e69f3
[2] http://www.mail-archive.com/linux-media@vger.kernel.org/msg36555.html

-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

Hi Robert,

On 8 June 2012 09:26, Robert Schwebel <r.schwebel@pengutronix.de> wrote:
> Hi Javier,
>
> On Fri, Jun 08, 2012 at 09:21:13AM +0200, javier Martin wrote:
>> If you refer to driver in [1] I have some concerns: i.MX27 VPU should
>> be implemented as a V4L2 mem2mem device since it gets raw pictures
>> from memory and outputs encoded frames to memory (some discussion
>> about the subject can be fond here [2]), as Exynos driver from Samsung
>> does. However, this driver you've mentioned doesn't do that: it just
>> creates several mapping regions so that the actual functionality is
>> implemented in user space by a library provided by Freescale, which
>> regarding i.MX27 it is also GPL.
>>
>> What we are trying to do is implementing all the functionality in
>> kernel space using mem2mem V4L2 framework so that it can be accepted
>> in mainline.
>
> We will work on the VPU driver and it's migration towards a proper
> mem2mem device very soon, mainly on MX53, but of course MX27 should be
> taken care of by the same driver.
>
> So I'd suggest that we coordinate that work somehow.

Do you plan to provide both encoding and decoding support or just one of them?

-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Robert Schwebel]

On Fri, Jun 08, 2012 at 09:39:15AM +0200, javier Martin wrote:
> Do you plan to provide both encoding and decoding support or just one
> of them?

We need both.

rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Sascha Hauer]

On Fri, Jun 08, 2012 at 09:39:15AM +0200, javier Martin wrote:
> Hi Robert,
> 
> On 8 June 2012 09:26, Robert Schwebel <r.schwebel@pengutronix.de> wrote:
> > Hi Javier,
> >
> > On Fri, Jun 08, 2012 at 09:21:13AM +0200, javier Martin wrote:
> >> If you refer to driver in [1] I have some concerns: i.MX27 VPU should
> >> be implemented as a V4L2 mem2mem device since it gets raw pictures
> >> from memory and outputs encoded frames to memory (some discussion
> >> about the subject can be fond here [2]), as Exynos driver from Samsung
> >> does. However, this driver you've mentioned doesn't do that: it just
> >> creates several mapping regions so that the actual functionality is
> >> implemented in user space by a library provided by Freescale, which
> >> regarding i.MX27 it is also GPL.
> >>
> >> What we are trying to do is implementing all the functionality in
> >> kernel space using mem2mem V4L2 framework so that it can be accepted
> >> in mainline.
> >
> > We will work on the VPU driver and it's migration towards a proper
> > mem2mem device very soon, mainly on MX53, but of course MX27 should be
> > taken care of by the same driver.
> >
> > So I'd suggest that we coordinate that work somehow.
> 
> Do you plan to provide both encoding and decoding support or just one of them?

We have both encoding and decoding. It works on i.MX51/53, but was
originally written for i.MX27 aswell. I haven't tested i.MX27 for longer
now, so it might or might not work. Find the source here:

git://git.pengutronix.de/git/imx/gst-plugins-fsl-vpu.git

The main difference from the FSL code is that the whole VPU
functionality is in a kernel module which talks (mostly) v4l2. Our next
taks is to convert this into a real mem2mem device, right now it only
works with the included gstreamer plugin. You'll need a small kernel
patch to register the device and to add the clocks.

The gstreamer plugin is in a horrible state, but with the conversion to
mem2mem we hope to get rid of this entirely.

Sascha

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

On 8 June 2012 10:48, Sascha Hauer <s.hauer@pengutronix.de> wrote:
> On Fri, Jun 08, 2012 at 09:39:15AM +0200, javier Martin wrote:
>> Hi Robert,
>>
>> On 8 June 2012 09:26, Robert Schwebel <r.schwebel@pengutronix.de> wrote:
>> > Hi Javier,
>> >
>> > On Fri, Jun 08, 2012 at 09:21:13AM +0200, javier Martin wrote:
>> >> If you refer to driver in [1] I have some concerns: i.MX27 VPU should
>> >> be implemented as a V4L2 mem2mem device since it gets raw pictures
>> >> from memory and outputs encoded frames to memory (some discussion
>> >> about the subject can be fond here [2]), as Exynos driver from Samsung
>> >> does. However, this driver you've mentioned doesn't do that: it just
>> >> creates several mapping regions so that the actual functionality is
>> >> implemented in user space by a library provided by Freescale, which
>> >> regarding i.MX27 it is also GPL.
>> >>
>> >> What we are trying to do is implementing all the functionality in
>> >> kernel space using mem2mem V4L2 framework so that it can be accepted
>> >> in mainline.
>> >
>> > We will work on the VPU driver and it's migration towards a proper
>> > mem2mem device very soon, mainly on MX53, but of course MX27 should be
>> > taken care of by the same driver.
>> >
>> > So I'd suggest that we coordinate that work somehow.
>>
>> Do you plan to provide both encoding and decoding support or just one of them?
>
> We have both encoding and decoding. It works on i.MX51/53, but was
> originally written for i.MX27 aswell. I haven't tested i.MX27 for longer
> now, so it might or might not work. Find the source here:
>
> git://git.pengutronix.de/git/imx/gst-plugins-fsl-vpu.git

Much too late...

http://www.digipedia.pl/usenet/thread/18550/20724/

-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

Hi,
I've checked this matter with a colleague and we have several reasons
to doubt that the i.MX27 and the i.MX53 can share the same driver for
their Video Processing Units (VPU):

1. The VPU in the i.MX27 is a "codadx6" with support for H.264, H.263
and MPEG4-Part2 [1]. Provided Freescale is using the same IP provider
for i.MX53 and looking at the features that the VPU in this SoC
supports (1080p resolution, VP8) we are probably dealing with a "coda
9 series" [2].

2. An important part of the functionality for controlling the
"codadx6" is implemented using software messages between the main CPU
and the VPU, this means that a different firmware loaded in the VPU
can substantially change the way it is handled. As previously stated,
i.MX27 and i.MX53 have different IP blocks and because of this, those
messages will be very different.

For these reasons we suggest that we carry on developing different
drivers for the i.MX27 and the i.MX53. Though it's true that both
drivers could share some overhead given by the use of mem2mem
framework, I don't think this is a good enough reason the merge them.

By the way, driver for the VPU in the i.MX27 will be called
"codadx6"[3], I suggest you call your driver "coda9" to avoid
confusion.


[1] http://www.chipsnmedia.com/product_search/product_view.php?part_idx=30&idx=48
[2] http://www.chipsnmedia.com/product_search/product_view.php?part_idx=20&idx=53
[3] https://github.com/jmartinc/video_visstrim/tree/mx27-codadx6/drivers/media/video/codadx6


-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Sascha Hauer]

On Fri, Jun 08, 2012 at 11:02:31AM +0200, javier Martin wrote:
> Hi,
> I've checked this matter with a colleague and we have several reasons
> to doubt that the i.MX27 and the i.MX53 can share the same driver for
> their Video Processing Units (VPU):
> 
> 1. The VPU in the i.MX27 is a "codadx6" with support for H.264, H.263
> and MPEG4-Part2 [1]. Provided Freescale is using the same IP provider
> for i.MX53 and looking at the features that the VPU in this SoC
> supports (1080p resolution, VP8) we are probably dealing with a "coda
> 9 series" [2].
> 
> 2. An important part of the functionality for controlling the
> "codadx6" is implemented using software messages between the main CPU
> and the VPU, this means that a different firmware loaded in the VPU
> can substantially change the way it is handled. As previously stated,
> i.MX27 and i.MX53 have different IP blocks and because of this, those
> messages will be very different.
> 
> For these reasons we suggest that we carry on developing different
> drivers for the i.MX27 and the i.MX53. Though it's true that both
> drivers could share some overhead given by the use of mem2mem
> framework, I don't think this is a good enough reason the merge them.
> 
> By the way, driver for the VPU in the i.MX27 will be called
> "codadx6"[3], I suggest you call your driver "coda9" to avoid
> confusion.

Well, our driver works on i.MX27 and i.MX5. Yes, it needs some
abstraction for different register layouts and different features, but
the cores behave sufficiently similar that it makes sense to share the
code in a single driver.

Sascha


-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

On 8 June 2012 11:23, Sascha Hauer <s.hauer@pengutronix.de> wrote:
> On Fri, Jun 08, 2012 at 11:02:31AM +0200, javier Martin wrote:
>> Hi,
>> I've checked this matter with a colleague and we have several reasons
>> to doubt that the i.MX27 and the i.MX53 can share the same driver for
>> their Video Processing Units (VPU):
>>
>> 1. The VPU in the i.MX27 is a "codadx6" with support for H.264, H.263
>> and MPEG4-Part2 [1]. Provided Freescale is using the same IP provider
>> for i.MX53 and looking at the features that the VPU in this SoC
>> supports (1080p resolution, VP8) we are probably dealing with a "coda
>> 9 series" [2].
>>
>> 2. An important part of the functionality for controlling the
>> "codadx6" is implemented using software messages between the main CPU
>> and the VPU, this means that a different firmware loaded in the VPU
>> can substantially change the way it is handled. As previously stated,
>> i.MX27 and i.MX53 have different IP blocks and because of this, those
>> messages will be very different.
>>
>> For these reasons we suggest that we carry on developing different
>> drivers for the i.MX27 and the i.MX53. Though it's true that both
>> drivers could share some overhead given by the use of mem2mem
>> framework, I don't think this is a good enough reason the merge them.
>>
>> By the way, driver for the VPU in the i.MX27 will be called
>> "codadx6"[3], I suggest you call your driver "coda9" to avoid
>> confusion.
>
> Well, our driver works on i.MX27 and i.MX5. Yes, it needs some
> abstraction for different register layouts and different features, but
> the cores behave sufficiently similar that it makes sense to share the
> code in a single driver.

Hi Sascha,

>From our point of view the current situation is the following:
We have a very reliable driver for the VPU which is not mainline but
it's been used for two years in our products. This driver only
supports encoding in the i.MX27 chip.
In parallel, you have a a multichip driver in progress which is not
mainline either, not fully V4L2 compatible and not tested for i.MX27.
[1]
At the same time, we have a driver in progress for i.MX27 encoding
only which follows V4L2 mem2mem framework. [2].

The first thing to decide would be which of both drivers we take as a
base for final mainline developing.
In our view, cleaning up driver from Pengutronix [1] would imply a lot
of effort of maintaining code that we cannot really test (i.MX5,
i.MX6) whereas if we continue using [2] we would have something valid
for i.MX27 much faster. Support for decoding and other chips could be
added later.

The second thing would be whether we keep on developing or whether we
should wait for you to have something in mainline. We have already
allocated resources to the development of the driver and long-term
testing to achieve product level reliability. Pengutronix does not
seem to be committed to developing the features relevant to our
product (lack of YUV420 support for i.MX27 camera driver[6]) nor
committed to any deadline (lack of answers or development on dmaengine
for i.MX27[4][5]). Moreover, development effort is only 50% of the
cost and we would still have to spend a lot of time checking the video
stream manually in different real-rife conditions (only extensive
testing allowed us to catch the "P frame marked as IDR" bug [7]).

As a conclusion we propose that we keep on developing our driver for
encoding in the i.MX27 VPU under the following conditions:
- We will provide a more generic name for the driver than "codadx6",
maybe something as "imx_vpu".
- We will do an extra effort and will study your code in [1] in order
to provide a modular approach that makes adding new functionality (new
chips or decoding) as easy as possible while making obvious that
further patches do not break support for encoding in the i.MX27.

Does it sound reasonable for you?

Regards.

[1] git://git.pengutronix.de/git/imx/gst-plugins-fsl-vpu.git
[2] https://github.com/jmartinc/video_visstrim/tree/mx27-codadx6/drivers/media/video/codadx6
[3] http://www.spinics.net/lists/linux-media/msg37920.html
[4] http://www.spinics.net/lists/arm-kernel/msg159196.html
[5] http://lists.infradead.org/pipermail/linux-arm-kernel/2012-March/087842.html
[6] http://patchwork.linuxtv.org/patch/8826/
[7] http://article.gmane.org/gmane.linux.drivers.video-input-infrastructure/49166

-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

Re: [RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Sascha Hauer]

On Fri, Jun 08, 2012 at 01:32:27PM +0200, javier Martin wrote:
> On 8 June 2012 11:23, Sascha Hauer <s.hauer@pengutronix.de> wrote:
> > On Fri, Jun 08, 2012 at 11:02:31AM +0200, javier Martin wrote:
> 
> Hi Sascha,
> 
> From our point of view the current situation is the following:
> We have a very reliable driver for the VPU which is not mainline but
> it's been used for two years in our products. This driver only
> supports encoding in the i.MX27 chip.
> In parallel, you have a a multichip driver in progress which is not
> mainline either, not fully V4L2 compatible and not tested for i.MX27.
> [1]
> At the same time, we have a driver in progress for i.MX27 encoding
> only which follows V4L2 mem2mem framework. [2].
> 
> The first thing to decide would be which of both drivers we take as a
> base for final mainline developing.
> In our view, cleaning up driver from Pengutronix [1] would imply a lot
> of effort of maintaining code that we cannot really test (i.MX5,
> i.MX6) whereas if we continue using [2] we would have something valid
> for i.MX27 much faster. Support for decoding and other chips could be
> added later.
> 
> The second thing would be whether we keep on developing or whether we
> should wait for you to have something in mainline. We have already
> allocated resources to the development of the driver and long-term
> testing to achieve product level reliability. Pengutronix does not
> seem to be committed to developing the features relevant to our
> product (lack of YUV420 support for i.MX27 camera driver[6]) nor
> committed to any deadline (lack of answers or development on dmaengine
> for i.MX27[4][5]). Moreover, development effort is only 50% of the
> cost and we would still have to spend a lot of time checking the video
> stream manually in different real-rife conditions (only extensive
> testing allowed us to catch the "P frame marked as IDR" bug [7]).
> 
> As a conclusion we propose that we keep on developing our driver for
> encoding in the i.MX27 VPU under the following conditions:
> - We will provide a more generic name for the driver than "codadx6",
> maybe something as "imx_vpu".

Since we already know that this is a codadx6 we should name it codadx
instead of anything vpu specific. I also suggest codadx instead of
anything more specific like codadx6 since we should rather motivate
people to merge other coda cores into this driver.

> - We will do an extra effort and will study your code in [1] in order
> to provide a modular approach that makes adding new functionality (new
> chips or decoding) as easy as possible while making obvious that
> further patches do not break support for encoding in the i.MX27.

This sounds like a plan. I am happy that you are willing to keep an eye
on our driver. It would be a pity to have unnecessary barriers for
merging codadx9 stuff later.

Sascha


-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |