[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

Fabio,

On 8 June 2012 08:51, javier Martin <javier.martin@vista-silicon.com> wrote:
> Hi Fabio,
>
> On 7 June 2012 19:35, Fabio Estevam <festevam@gmail.com> wrote:
>> Hi Javier,
>>
>> On Thu, Jun 7, 2012 at 5:30 AM, javier Martin
>> <javier.martin@vista-silicon.com> wrote:
>>
>>> As i stated, the driver is still in an early development stage, it
>>> doesn't do anything useful yet. But this is the public git repository
>>> if you want to take a look:
>>>
>>> git repo: https://github.com/jmartinc/video_visstrim.git
>>> branch: ?mx27-codadx6
>>
>> Thanks, I will take a look at your tree when I am back to the office next week.
>>
>> I also see that Linaro's tree has support for VPU for mx5/mx6:
>> http://git.linaro.org/gitweb?p=landing-teams/working/freescale/kernel.git;a=summary
>>
>> ,so we should probably think in unifying it with mx27 support there too.

If you refer to driver in [1] I have some concerns: i.MX27 VPU should
be implemented as a V4L2 mem2mem device since it gets raw pictures
from memory and outputs encoded frames to memory (some discussion
about the subject can be fond here [2]), as Exynos driver from Samsung
does. However, this driver you've mentioned doesn't do that: it just
creates several mapping regions so that the actual functionality is
implemented in user space by a library provided by Freescale, which
regarding i.MX27 it is also GPL.

What we are trying to do is implementing all the functionality in
kernel space using mem2mem V4L2 framework so that it can be accepted
in mainline.

Please, correct me if the driver you are talking about is not the one in [1].

Regards.

[1] http://git.linaro.org/gitweb?p=landing-teams/working/freescale/kernel.git;a=blob;f=drivers/mxc/vpu/mxc_vpu.c;h=27b09e56d5a3f6cb7eeba16fe5493cbec46c65cd;hb=d0f289f67f0ff403d92880c410b009f1fd4e69f3
[2] http://www.mail-archive.com/linux-media at vger.kernel.org/msg36555.html

-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

Hi Robert,

On 8 June 2012 09:26, Robert Schwebel <r.schwebel@pengutronix.de> wrote:
> Hi Javier,
>
> On Fri, Jun 08, 2012 at 09:21:13AM +0200, javier Martin wrote:
>> If you refer to driver in [1] I have some concerns: i.MX27 VPU should
>> be implemented as a V4L2 mem2mem device since it gets raw pictures
>> from memory and outputs encoded frames to memory (some discussion
>> about the subject can be fond here [2]), as Exynos driver from Samsung
>> does. However, this driver you've mentioned doesn't do that: it just
>> creates several mapping regions so that the actual functionality is
>> implemented in user space by a library provided by Freescale, which
>> regarding i.MX27 it is also GPL.
>>
>> What we are trying to do is implementing all the functionality in
>> kernel space using mem2mem V4L2 framework so that it can be accepted
>> in mainline.
>
> We will work on the VPU driver and it's migration towards a proper
> mem2mem device very soon, mainly on MX53, but of course MX27 should be
> taken care of by the same driver.
>
> So I'd suggest that we coordinate that work somehow.

Do you plan to provide both encoding and decoding support or just one of them?

-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Robert Schwebel]

On Fri, Jun 08, 2012 at 09:39:15AM +0200, javier Martin wrote:
> Do you plan to provide both encoding and decoding support or just one
> of them?

We need both.

rsc
-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Sascha Hauer]

On Fri, Jun 08, 2012 at 09:39:15AM +0200, javier Martin wrote:
> Hi Robert,
> 
> On 8 June 2012 09:26, Robert Schwebel <r.schwebel@pengutronix.de> wrote:
> > Hi Javier,
> >
> > On Fri, Jun 08, 2012 at 09:21:13AM +0200, javier Martin wrote:
> >> If you refer to driver in [1] I have some concerns: i.MX27 VPU should
> >> be implemented as a V4L2 mem2mem device since it gets raw pictures
> >> from memory and outputs encoded frames to memory (some discussion
> >> about the subject can be fond here [2]), as Exynos driver from Samsung
> >> does. However, this driver you've mentioned doesn't do that: it just
> >> creates several mapping regions so that the actual functionality is
> >> implemented in user space by a library provided by Freescale, which
> >> regarding i.MX27 it is also GPL.
> >>
> >> What we are trying to do is implementing all the functionality in
> >> kernel space using mem2mem V4L2 framework so that it can be accepted
> >> in mainline.
> >
> > We will work on the VPU driver and it's migration towards a proper
> > mem2mem device very soon, mainly on MX53, but of course MX27 should be
> > taken care of by the same driver.
> >
> > So I'd suggest that we coordinate that work somehow.
> 
> Do you plan to provide both encoding and decoding support or just one of them?

We have both encoding and decoding. It works on i.MX51/53, but was
originally written for i.MX27 aswell. I haven't tested i.MX27 for longer
now, so it might or might not work. Find the source here:

git://git.pengutronix.de/git/imx/gst-plugins-fsl-vpu.git

The main difference from the FSL code is that the whole VPU
functionality is in a kernel module which talks (mostly) v4l2. Our next
taks is to convert this into a real mem2mem device, right now it only
works with the included gstreamer plugin. You'll need a small kernel
patch to register the device and to add the clocks.

The gstreamer plugin is in a horrible state, but with the conversion to
mem2mem we hope to get rid of this entirely.

Sascha

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

On 8 June 2012 10:48, Sascha Hauer <s.hauer@pengutronix.de> wrote:
> On Fri, Jun 08, 2012 at 09:39:15AM +0200, javier Martin wrote:
>> Hi Robert,
>>
>> On 8 June 2012 09:26, Robert Schwebel <r.schwebel@pengutronix.de> wrote:
>> > Hi Javier,
>> >
>> > On Fri, Jun 08, 2012 at 09:21:13AM +0200, javier Martin wrote:
>> >> If you refer to driver in [1] I have some concerns: i.MX27 VPU should
>> >> be implemented as a V4L2 mem2mem device since it gets raw pictures
>> >> from memory and outputs encoded frames to memory (some discussion
>> >> about the subject can be fond here [2]), as Exynos driver from Samsung
>> >> does. However, this driver you've mentioned doesn't do that: it just
>> >> creates several mapping regions so that the actual functionality is
>> >> implemented in user space by a library provided by Freescale, which
>> >> regarding i.MX27 it is also GPL.
>> >>
>> >> What we are trying to do is implementing all the functionality in
>> >> kernel space using mem2mem V4L2 framework so that it can be accepted
>> >> in mainline.
>> >
>> > We will work on the VPU driver and it's migration towards a proper
>> > mem2mem device very soon, mainly on MX53, but of course MX27 should be
>> > taken care of by the same driver.
>> >
>> > So I'd suggest that we coordinate that work somehow.
>>
>> Do you plan to provide both encoding and decoding support or just one of them?
>
> We have both encoding and decoding. It works on i.MX51/53, but was
> originally written for i.MX27 aswell. I haven't tested i.MX27 for longer
> now, so it might or might not work. Find the source here:
>
> git://git.pengutronix.de/git/imx/gst-plugins-fsl-vpu.git

Much too late...

http://www.digipedia.pl/usenet/thread/18550/20724/

-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

Hi,
I've checked this matter with a colleague and we have several reasons
to doubt that the i.MX27 and the i.MX53 can share the same driver for
their Video Processing Units (VPU):

1. The VPU in the i.MX27 is a "codadx6" with support for H.264, H.263
and MPEG4-Part2 [1]. Provided Freescale is using the same IP provider
for i.MX53 and looking at the features that the VPU in this SoC
supports (1080p resolution, VP8) we are probably dealing with a "coda
9 series" [2].

2. An important part of the functionality for controlling the
"codadx6" is implemented using software messages between the main CPU
and the VPU, this means that a different firmware loaded in the VPU
can substantially change the way it is handled. As previously stated,
i.MX27 and i.MX53 have different IP blocks and because of this, those
messages will be very different.

For these reasons we suggest that we carry on developing different
drivers for the i.MX27 and the i.MX53. Though it's true that both
drivers could share some overhead given by the use of mem2mem
framework, I don't think this is a good enough reason the merge them.

By the way, driver for the VPU in the i.MX27 will be called
"codadx6"[3], I suggest you call your driver "coda9" to avoid
confusion.


[1] http://www.chipsnmedia.com/product_search/product_view.php?part_idx=30&idx=48
[2] http://www.chipsnmedia.com/product_search/product_view.php?part_idx=20&idx=53
[3] https://github.com/jmartinc/video_visstrim/tree/mx27-codadx6/drivers/media/video/codadx6


-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Sascha Hauer]

On Fri, Jun 08, 2012 at 11:02:31AM +0200, javier Martin wrote:
> Hi,
> I've checked this matter with a colleague and we have several reasons
> to doubt that the i.MX27 and the i.MX53 can share the same driver for
> their Video Processing Units (VPU):
> 
> 1. The VPU in the i.MX27 is a "codadx6" with support for H.264, H.263
> and MPEG4-Part2 [1]. Provided Freescale is using the same IP provider
> for i.MX53 and looking at the features that the VPU in this SoC
> supports (1080p resolution, VP8) we are probably dealing with a "coda
> 9 series" [2].
> 
> 2. An important part of the functionality for controlling the
> "codadx6" is implemented using software messages between the main CPU
> and the VPU, this means that a different firmware loaded in the VPU
> can substantially change the way it is handled. As previously stated,
> i.MX27 and i.MX53 have different IP blocks and because of this, those
> messages will be very different.
> 
> For these reasons we suggest that we carry on developing different
> drivers for the i.MX27 and the i.MX53. Though it's true that both
> drivers could share some overhead given by the use of mem2mem
> framework, I don't think this is a good enough reason the merge them.
> 
> By the way, driver for the VPU in the i.MX27 will be called
> "codadx6"[3], I suggest you call your driver "coda9" to avoid
> confusion.

Well, our driver works on i.MX27 and i.MX5. Yes, it needs some
abstraction for different register layouts and different features, but
the cores behave sufficiently similar that it makes sense to share the
code in a single driver.

Sascha


-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[javier Martin]

On 8 June 2012 11:23, Sascha Hauer <s.hauer@pengutronix.de> wrote:
> On Fri, Jun 08, 2012 at 11:02:31AM +0200, javier Martin wrote:
>> Hi,
>> I've checked this matter with a colleague and we have several reasons
>> to doubt that the i.MX27 and the i.MX53 can share the same driver for
>> their Video Processing Units (VPU):
>>
>> 1. The VPU in the i.MX27 is a "codadx6" with support for H.264, H.263
>> and MPEG4-Part2 [1]. Provided Freescale is using the same IP provider
>> for i.MX53 and looking at the features that the VPU in this SoC
>> supports (1080p resolution, VP8) we are probably dealing with a "coda
>> 9 series" [2].
>>
>> 2. An important part of the functionality for controlling the
>> "codadx6" is implemented using software messages between the main CPU
>> and the VPU, this means that a different firmware loaded in the VPU
>> can substantially change the way it is handled. As previously stated,
>> i.MX27 and i.MX53 have different IP blocks and because of this, those
>> messages will be very different.
>>
>> For these reasons we suggest that we carry on developing different
>> drivers for the i.MX27 and the i.MX53. Though it's true that both
>> drivers could share some overhead given by the use of mem2mem
>> framework, I don't think this is a good enough reason the merge them.
>>
>> By the way, driver for the VPU in the i.MX27 will be called
>> "codadx6"[3], I suggest you call your driver "coda9" to avoid
>> confusion.
>
> Well, our driver works on i.MX27 and i.MX5. Yes, it needs some
> abstraction for different register layouts and different features, but
> the cores behave sufficiently similar that it makes sense to share the
> code in a single driver.

Hi Sascha,

>From our point of view the current situation is the following:
We have a very reliable driver for the VPU which is not mainline but
it's been used for two years in our products. This driver only
supports encoding in the i.MX27 chip.
In parallel, you have a a multichip driver in progress which is not
mainline either, not fully V4L2 compatible and not tested for i.MX27.
[1]
At the same time, we have a driver in progress for i.MX27 encoding
only which follows V4L2 mem2mem framework. [2].

The first thing to decide would be which of both drivers we take as a
base for final mainline developing.
In our view, cleaning up driver from Pengutronix [1] would imply a lot
of effort of maintaining code that we cannot really test (i.MX5,
i.MX6) whereas if we continue using [2] we would have something valid
for i.MX27 much faster. Support for decoding and other chips could be
added later.

The second thing would be whether we keep on developing or whether we
should wait for you to have something in mainline. We have already
allocated resources to the development of the driver and long-term
testing to achieve product level reliability. Pengutronix does not
seem to be committed to developing the features relevant to our
product (lack of YUV420 support for i.MX27 camera driver[6]) nor
committed to any deadline (lack of answers or development on dmaengine
for i.MX27[4][5]). Moreover, development effort is only 50% of the
cost and we would still have to spend a lot of time checking the video
stream manually in different real-rife conditions (only extensive
testing allowed us to catch the "P frame marked as IDR" bug [7]).

As a conclusion we propose that we keep on developing our driver for
encoding in the i.MX27 VPU under the following conditions:
- We will provide a more generic name for the driver than "codadx6",
maybe something as "imx_vpu".
- We will do an extra effort and will study your code in [1] in order
to provide a modular approach that makes adding new functionality (new
chips or decoding) as easy as possible while making obvious that
further patches do not break support for encoding in the i.MX27.

Does it sound reasonable for you?

Regards.

[1] git://git.pengutronix.de/git/imx/gst-plugins-fsl-vpu.git
[2] https://github.com/jmartinc/video_visstrim/tree/mx27-codadx6/drivers/media/video/codadx6
[3] http://www.spinics.net/lists/linux-media/msg37920.html
[4] http://www.spinics.net/lists/arm-kernel/msg159196.html
[5] http://lists.infradead.org/pipermail/linux-arm-kernel/2012-March/087842.html
[6] http://patchwork.linuxtv.org/patch/8826/
[7] http://article.gmane.org/gmane.linux.drivers.video-input-infrastructure/49166

-- 
Javier Martin
Vista Silicon S.L.
CDTUC - FASE C - Oficina S-345
Avda de los Castros s/n
39005- Santander. Cantabria. Spain
+34 942 25 32 60
www.vista-silicon.com

[RFC] Support for H.264/MPEG4 encoder (VPU) in i.MX27.

[Sascha Hauer]

On Fri, Jun 08, 2012 at 01:32:27PM +0200, javier Martin wrote:
> On 8 June 2012 11:23, Sascha Hauer <s.hauer@pengutronix.de> wrote:
> > On Fri, Jun 08, 2012 at 11:02:31AM +0200, javier Martin wrote:
> 
> Hi Sascha,
> 
> From our point of view the current situation is the following:
> We have a very reliable driver for the VPU which is not mainline but
> it's been used for two years in our products. This driver only
> supports encoding in the i.MX27 chip.
> In parallel, you have a a multichip driver in progress which is not
> mainline either, not fully V4L2 compatible and not tested for i.MX27.
> [1]
> At the same time, we have a driver in progress for i.MX27 encoding
> only which follows V4L2 mem2mem framework. [2].
> 
> The first thing to decide would be which of both drivers we take as a
> base for final mainline developing.
> In our view, cleaning up driver from Pengutronix [1] would imply a lot
> of effort of maintaining code that we cannot really test (i.MX5,
> i.MX6) whereas if we continue using [2] we would have something valid
> for i.MX27 much faster. Support for decoding and other chips could be
> added later.
> 
> The second thing would be whether we keep on developing or whether we
> should wait for you to have something in mainline. We have already
> allocated resources to the development of the driver and long-term
> testing to achieve product level reliability. Pengutronix does not
> seem to be committed to developing the features relevant to our
> product (lack of YUV420 support for i.MX27 camera driver[6]) nor
> committed to any deadline (lack of answers or development on dmaengine
> for i.MX27[4][5]). Moreover, development effort is only 50% of the
> cost and we would still have to spend a lot of time checking the video
> stream manually in different real-rife conditions (only extensive
> testing allowed us to catch the "P frame marked as IDR" bug [7]).
> 
> As a conclusion we propose that we keep on developing our driver for
> encoding in the i.MX27 VPU under the following conditions:
> - We will provide a more generic name for the driver than "codadx6",
> maybe something as "imx_vpu".

Since we already know that this is a codadx6 we should name it codadx
instead of anything vpu specific. I also suggest codadx instead of
anything more specific like codadx6 since we should rather motivate
people to merge other coda cores into this driver.

> - We will do an extra effort and will study your code in [1] in order
> to provide a modular approach that makes adding new functionality (new
> chips or decoding) as easy as possible while making obvious that
> further patches do not break support for encoding in the i.MX27.

This sounds like a plan. I am happy that you are willing to keep an eye
on our driver. It would be a pity to have unnecessary barriers for
merging codadx9 stuff later.

Sascha


-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |