[PATCH] media: media-dev: Add media devices for EXYNOS5

[PATCH] media: media-dev: Add media devices for EXYNOS5

[Sungchun Kang]

Since the EXYNOS5 SoCs have various multimedia IPs
such as Gscaler, FIMC-LITE, and MIXER, and so on.
Additionally, media controller interface is needed
to configure connection between them and to control each IPs.

This patch adds support media device for EXYNOS5 SoCs.
Actually, there are three media devices such as below
diagram which are using media control framework.
Since they are not belong to one hardware block, we
need to manage it for connecting with each devices.

Follwing is detailed list of them:

* Gscaler: general scaler
  Support memory to memory interface
  Support output interface from memory to display device(LCD, TV)
  Support capture interface from device(FIMC-LITE, FIMD) to memory

* MIPI-CSIS
  Support interconnection(subdev interface) between devices

* FIMC-LITE
  Support capture interface from device(Sensor, MIPI-CSIS) to memory
  Support interconnection(subdev interface) between devices

* MIXER
  Support output interface from memory to device(HDMI)
  Support interconnection(subdev interface) between devices

* FIMD
  Support framebuffer interface
  Support subdev interface to display frames sent from Gscaler

* Rotator
  Support memory to memory interface

* m2m-scaler
  Support only memory to memory interface

* And so on...

 1) media 0
  LCD Output path consists of Gscaler and FIMD(display controller).
  +----------------+     +------+
  | Gscaler-output | --> | FIMD | --> LCD
  +----------------+     +------+

  HDMI Output path consists of Gscaler, Mixer and HDMI.
  +----------------+     +-------+     +------+
  | Gscaler-output | --> | MIXER | --> | HDMI | --> TV
  +----------------+     +-------+     +------+

+--------+     +-----------+     +-----------+     +-----------------+

 2) media 1
  Camera Capture path consists of MIPI-CSIS, FIMC-LITE and Gscaler
  +--------+     +-----------+     +-----------------+
  | Sensor | --> | FIMC-LITE | --> | Gscaler-capture |
  +--------+     +-----------+     +-----------------+

  +--------+     +-----------+     +-----------+     +-----------------+
  | Sensor | --> | MIPI-CSIS | --> | FIMC-LITE | --> | Gscaler-capture |
  +--------+     +-----------+     +-----------+     +-----------------+

Signed-off-by: Sungchun Kang <sungchun.kang@samsung.com>
---
 drivers/media/video/exynos/mdev/Kconfig       |    8 ++
 drivers/media/video/exynos/mdev/Makefile      |    2 +
 drivers/media/video/exynos/mdev/exynos-mdev.c |  115 ++++++++++++++++++
 include/media/exynos_mc.h                     |  160 +++++++++++++++++++++++++
 4 files changed, 285 insertions(+), 0 deletions(-)
 create mode 100644 drivers/media/video/exynos/mdev/Kconfig
 create mode 100644 drivers/media/video/exynos/mdev/Makefile
 create mode 100644 drivers/media/video/exynos/mdev/exynos-mdev.c
 create mode 100644 include/media/exynos_mc.h

diff --git a/drivers/media/video/exynos/mdev/Kconfig b/drivers/media/video/exynos/mdev/Kconfig
new file mode 100644
index 0000000..15134b0
--- /dev/null
+++ b/drivers/media/video/exynos/mdev/Kconfig
@@ -0,0 +1,8 @@
+config EXYNOS_MEDIA_DEVICE
+	bool
+	depends on MEDIA_EXYNOS
+	select MEDIA_CONTROLLER
+	select VIDEO_V4L2_SUBDEV_API
+	default y
+	help
+	  This is a v4l2 driver for exynos media device.
diff --git a/drivers/media/video/exynos/mdev/Makefile b/drivers/media/video/exynos/mdev/Makefile
new file mode 100644
index 0000000..175a4bc
--- /dev/null
+++ b/drivers/media/video/exynos/mdev/Makefile
@@ -0,0 +1,2 @@
+mdev-objs := exynos-mdev.o
+obj-$(CONFIG_EXYNOS_MEDIA_DEVICE)	+= mdev.o
diff --git a/drivers/media/video/exynos/mdev/exynos-mdev.c b/drivers/media/video/exynos/mdev/exynos-mdev.c
new file mode 100644
index 0000000..a76e7c3
--- /dev/null
+++ b/drivers/media/video/exynos/mdev/exynos-mdev.c
@@ -0,0 +1,115 @@
+/* drviers/media/video/exynos/mdev/exynos-mdev.c
+ *
+ * Copyright (c) 2011 Samsung Electronics Co., Ltd.
+ *		http://www.samsung.com
+ *
+ * EXYNOS5 SoC series media device driver
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+*/
+
+#include <linux/bug.h>
+#include <linux/device.h>
+#include <linux/errno.h>
+#include <linux/i2c.h>
+#include <linux/kernel.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/types.h>
+#include <linux/slab.h>
+#include <linux/version.h>
+#include <media/v4l2-ctrls.h>
+#include <media/media-device.h>
+#include <media/exynos_mc.h>
+
+static int __devinit mdev_probe(struct platform_device *pdev)
+{
+	struct v4l2_device *v4l2_dev;
+	struct exynos_md *mdev;
+	int ret;
+
+	mdev = kzalloc(sizeof(struct exynos_md), GFP_KERNEL);
+	if (!mdev)
+		return -ENOMEM;
+
+	mdev->id = pdev->id;
+	mdev->pdev = pdev;
+	spin_lock_init(&mdev->slock);
+
+	snprintf(mdev->media_dev.model, sizeof(mdev->media_dev.model), "%s%d",
+		 dev_name(&pdev->dev), mdev->id);
+
+	mdev->media_dev.dev = &pdev->dev;
+
+	v4l2_dev = &mdev->v4l2_dev;
+	v4l2_dev->mdev = &mdev->media_dev;
+	snprintf(v4l2_dev->name, sizeof(v4l2_dev->name), "%s",
+		 dev_name(&pdev->dev));
+
+	ret = v4l2_device_register(&pdev->dev, &mdev->v4l2_dev);
+	if (ret < 0) {
+		v4l2_err(v4l2_dev, "Failed to register v4l2_device: %d\n", ret);
+		goto err_v4l2_reg;
+	}
+	ret = media_device_register(&mdev->media_dev);
+	if (ret < 0) {
+		v4l2_err(v4l2_dev, "Failed to register media device: %d\n", ret);
+		goto err_mdev_reg;
+	}
+
+	platform_set_drvdata(pdev, mdev);
+	v4l2_info(v4l2_dev, "Media%d[0x%08x] was registered successfully\n",
+		  mdev->id, (unsigned int)mdev);
+	return 0;
+
+err_mdev_reg:
+	v4l2_device_unregister(&mdev->v4l2_dev);
+err_v4l2_reg:
+	kfree(mdev);
+	return ret;
+}
+
+static int __devexit mdev_remove(struct platform_device *pdev)
+{
+	struct exynos_md *mdev = platform_get_drvdata(pdev);
+
+	if (!mdev)
+		return 0;
+	media_device_unregister(&mdev->media_dev);
+	v4l2_device_unregister(&mdev->v4l2_dev);
+	kfree(mdev);
+	return 0;
+}
+
+static struct platform_driver mdev_driver = {
+	.probe		= mdev_probe,
+	.remove		= __devexit_p(mdev_remove),
+	.driver = {
+		.name	= MDEV_MODULE_NAME,
+		.owner	= THIS_MODULE,
+	}
+};
+
+int __init mdev_init(void)
+{
+	int ret = platform_driver_register(&mdev_driver);
+	if (ret)
+		err("platform_driver_register failed: %d\n", ret);
+	return ret;
+}
+
+void __exit mdev_exit(void)
+{
+	platform_driver_unregister(&mdev_driver);
+}
+
+module_init(mdev_init);
+module_exit(mdev_exit);
+
+MODULE_AUTHOR("Hyunwoong Kim <khw0178.kim@samsung.com>");
+MODULE_DESCRIPTION("EXYNOS5 SoC series media device driver");
+MODULE_LICENSE("GPL");
diff --git a/include/media/exynos_mc.h b/include/media/exynos_mc.h
new file mode 100644
index 0000000..54875fb
--- /dev/null
+++ b/include/media/exynos_mc.h
@@ -0,0 +1,160 @@
+/* linux/inclue/media/exynos_mc.h
+ *
+ * Copyright (c) 2011 Samsung Electronics Co., Ltd.
+ *		http://www.samsung.com
+ *
+ * header file for exynos media device driver
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ */
+
+#ifndef GSC_MDEVICE_H_
+#define GSC_MDEVICE_H_
+
+#include <linux/clk.h>
+#include <linux/platform_device.h>
+#include <linux/mutex.h>
+#include <linux/device.h>
+#include <media/media-device.h>
+#include <media/media-entity.h>
+#include <media/v4l2-device.h>
+#include <media/v4l2-subdev.h>
+
+#define err(fmt, args...) \
+	printk(KERN_ERR "%s:%d: " fmt "\n", __func__, __LINE__, ##args)
+
+#define MDEV_MODULE_NAME "exynos-mdev"
+#define MAX_GSC_SUBDEV		4
+#define MDEV_MAX_NUM	3
+
+#define GSC_OUT_PAD_SINK	0
+#define GSC_OUT_PAD_SOURCE	1
+
+#define GSC_CAP_PAD_SINK	0
+#define GSC_CAP_PAD_SOURCE	1
+
+#define FLITE_PAD_SINK		0
+#define FLITE_PAD_SOURCE_PREV	1
+#define FLITE_PAD_SOURCE_CAMCORD	2
+#define FLITE_PAD_SOURCE_MEM		3
+#define FLITE_PADS_NUM		4
+
+#define CSIS_PAD_SINK		0
+#define CSIS_PAD_SOURCE		1
+#define CSIS_PADS_NUM		2
+
+#define MAX_CAMIF_CLIENTS	2
+
+#define MXR_SUBDEV_NAME		"s5p-mixer"
+
+#define GSC_MODULE_NAME			"exynos-gsc"
+#define GSC_SUBDEV_NAME			"exynos-gsc-sd"
+#define FIMD_MODULE_NAME		"s5p-fimd1"
+#define FIMD_ENTITY_NAME		"s3c-fb-window"
+#define FLITE_MODULE_NAME		"exynos-fimc-lite"
+#define CSIS_MODULE_NAME		"s5p-mipi-csis"
+
+#define GSC_CAP_GRP_ID			(1 << 0)
+#define FLITE_GRP_ID			(1 << 1)
+#define CSIS_GRP_ID			(1 << 2)
+#define SENSOR_GRP_ID			(1 << 3)
+#define FIMD_GRP_ID			(1 << 4)
+
+#define SENSOR_MAX_ENTITIES		MAX_CAMIF_CLIENTS
+#define FLITE_MAX_ENTITIES		2
+#define CSIS_MAX_ENTITIES		2
+
+enum mdev_node {
+	MDEV_OUTPUT,
+	MDEV_CAPTURE,
+	MDEV_ISP,
+};
+
+enum mxr_data_from {
+	FROM_GSC_SD,
+	FROM_MXR_VD,
+};
+
+struct exynos_media_ops {
+	int (*power_off)(struct v4l2_subdev *sd);
+};
+
+struct exynos_entity_data {
+	const struct exynos_media_ops *media_ops;
+	enum mxr_data_from mxr_data_from;
+};
+
+/**
+ * struct exynos_md - Exynos media device information
+ * @media_dev: top level media device
+ * @v4l2_dev: top level v4l2_device holding up the subdevs
+ * @pdev: platform device this media device is hooked up into
+ * @slock: spinlock protecting @sensor array
+ * @id: media device IDs
+ * @gsc_sd: each pointer of g-scaler's subdev array
+ */
+struct exynos_md {
+	struct media_device	media_dev;
+	struct v4l2_device	v4l2_dev;
+	struct platform_device	*pdev;
+	struct v4l2_subdev	*gsc_sd[MAX_GSC_SUBDEV];
+	struct v4l2_subdev	*gsc_cap_sd[MAX_GSC_SUBDEV];
+	struct v4l2_subdev	*csis_sd[CSIS_MAX_ENTITIES];
+	struct v4l2_subdev	*flite_sd[FLITE_MAX_ENTITIES];
+	struct v4l2_subdev	*sensor_sd[SENSOR_MAX_ENTITIES];
+	u16			id;
+	spinlock_t slock;
+};
+
+static int dummy_callback(struct device *dev, void *md)
+{
+	/* non-zero return stops iteration */
+	return -1;
+}
+
+static inline void *module_name_to_driver_data(char *module_name)
+{
+	struct device_driver *drv;
+	struct device *dev;
+	void *driver_data;
+
+	drv = driver_find(module_name, &platform_bus_type);
+	if (drv) {
+		dev = driver_find_device(drv, NULL, NULL, dummy_callback);
+		driver_data = dev_get_drvdata(dev);
+		put_driver(drv);
+		return driver_data;
+	} else
+		return NULL;
+}
+
+/* print entity information for debug*/
+static inline void entity_info_print(struct media_entity *me, struct device *dev)
+{
+	u16 num_pads = me->num_pads;
+	u16 num_links = me->num_links;
+	int i;
+
+	dev_dbg(dev, "entity name : %s\n", me->name);
+	dev_dbg(dev, "number of pads = %d\n", num_pads);
+	for (i = 0; i < num_pads; ++i) {
+		dev_dbg(dev, "pad[%d] flag : %s\n", i,
+			(me->pads[i].flags == 1) ? "SINK" : "SOURCE");
+	}
+
+	dev_dbg(dev, "number of links = %d\n", num_links);
+
+	for (i = 0; i < num_links; ++i) {
+		dev_dbg(dev, "link[%d] info  =  ", i);
+		dev_dbg(dev, "%s : %s[%d]  --->  %s : %s[%d]\n",
+			me->links[i].source->entity->name,
+			me->links[i].source->flags == 1 ? "SINK" : "SOURCE",
+			me->links[i].source->index,
+			me->links[i].sink->entity->name,
+			me->links[i].sink->flags == 1 ? "SINK" : "SOURCE",
+			me->links[i].sink->index);
+	}
+}
+#endif
-- 
1.7.1

Re: [PATCH] media: media-dev: Add media devices for EXYNOS5

[Sylwester Nawrocki]

Hi Sungchun,

On 02/15/2012 07:02 AM, Sungchun Kang wrote:
> Since the EXYNOS5 SoCs have various multimedia IPs
> such as Gscaler, FIMC-LITE, and MIXER, and so on.
> Additionally, media controller interface is needed
> to configure connection between them and to control each IPs.
> 
> This patch adds support media device for EXYNOS5 SoCs.
> Actually, there are three media devices such as below
> diagram which are using media control framework.
> Since they are not belong to one hardware block, we
> need to manage it for connecting with each devices.
> 
> Follwing is detailed list of them:
> 
> * Gscaler: general scaler
>    Support memory to memory interface
>    Support output interface from memory to display device(LCD, TV)
>    Support capture interface from device(FIMC-LITE, FIMD) to memory
> 
> * MIPI-CSIS
>    Support interconnection(subdev interface) between devices

Is there any difference in s5p/exynos4 and exynos5 MIPI-CSIS devices ?
I suspect there isn't and the existing MIPI-CSIS driver can be used for
exynos5 too. 

> 
> * FIMC-LITE
>    Support capture interface from device(Sensor, MIPI-CSIS) to memory
>    Support interconnection(subdev interface) between devices

This device is also present on exynos4212/4412 SoCs. Can you tell what's
difference between FIMC-LITE on Exynos4 and Exynos5 ?
Either we need separate FIMC-LITE drivers or we need a shared one. I'd like
to clarify this first.

> * MIXER
>    Support output interface from memory to device(HDMI)
>    Support interconnection(subdev interface) between devices
> 
> * FIMD
>    Support framebuffer interface
>    Support subdev interface to display frames sent from Gscaler

What about Exynos DRM driver ? Do you have any plans to integrate the V4L2
and the DRM driver ? IMHO DRM is more appropriate for some tasks on display
side, like 2D operations, multiple outputs, windows, blending, etc.

> * Rotator
>    Support memory to memory interface
> 
> * m2m-scaler
>    Support only memory to memory interface
> 
> * And so on...
> 
>   1) media 0
>    LCD Output path consists of Gscaler and FIMD(display controller).
>    +----------------+     +------+
>    | Gscaler-output | -->  | FIMD | -->  LCD
>    +----------------+     +------+
> 
>    HDMI Output path consists of Gscaler, Mixer and HDMI.
>    +----------------+     +-------+     +------+
>    | Gscaler-output | -->  | MIXER | -->  | HDMI | -->  TV
>    +----------------+     +-------+     +------+
> 
> +--------+     +-----------+     +-----------+     +-----------------+
> 
>   2) media 1
>    Camera Capture path consists of MIPI-CSIS, FIMC-LITE and Gscaler
>    +--------+     +-----------+     +-----------------+
>    | Sensor | -->  | FIMC-LITE | -->  | Gscaler-capture |
>    +--------+     +-----------+     +-----------------+
> 
>    +--------+     +-----------+     +-----------+     +-----------------+
>    | Sensor | -->  | MIPI-CSIS | -->  | FIMC-LITE | -->  | Gscaler-capture |
>    +--------+     +-----------+     +-----------+     +-----------------+
> 
> Signed-off-by: Sungchun Kang<sungchun.kang@samsung.com>
> ---
>   drivers/media/video/exynos/mdev/Kconfig       |    8 ++
>   drivers/media/video/exynos/mdev/Makefile      |    2 +
>   drivers/media/video/exynos/mdev/exynos-mdev.c |  115 ++++++++++++++++++
>   include/media/exynos_mc.h                     |  160 +++++++++++++++++++++++++
>   4 files changed, 285 insertions(+), 0 deletions(-)
>   create mode 100644 drivers/media/video/exynos/mdev/Kconfig
>   create mode 100644 drivers/media/video/exynos/mdev/Makefile
>   create mode 100644 drivers/media/video/exynos/mdev/exynos-mdev.c
>   create mode 100644 include/media/exynos_mc.h
> 
> diff --git a/drivers/media/video/exynos/mdev/Kconfig b/drivers/media/video/exynos/mdev/Kconfig
> new file mode 100644
> index 0000000..15134b0
> --- /dev/null
> +++ b/drivers/media/video/exynos/mdev/Kconfig
> @@ -0,0 +1,8 @@
> +config EXYNOS_MEDIA_DEVICE
> +	bool
> +	depends on MEDIA_EXYNOS
> +	select MEDIA_CONTROLLER
> +	select VIDEO_V4L2_SUBDEV_API
> +	default y
> +	help
> +	  This is a v4l2 driver for exynos media device.
> diff --git a/drivers/media/video/exynos/mdev/Makefile b/drivers/media/video/exynos/mdev/Makefile
> new file mode 100644
> index 0000000..175a4bc
> --- /dev/null
> +++ b/drivers/media/video/exynos/mdev/Makefile
> @@ -0,0 +1,2 @@
> +mdev-objs := exynos-mdev.o
> +obj-$(CONFIG_EXYNOS_MEDIA_DEVICE)	+= mdev.o
> diff --git a/drivers/media/video/exynos/mdev/exynos-mdev.c b/drivers/media/video/exynos/mdev/exynos-mdev.c
> new file mode 100644
> index 0000000..a76e7c3
> --- /dev/null
> +++ b/drivers/media/video/exynos/mdev/exynos-mdev.c
> @@ -0,0 +1,115 @@
> +/* drviers/media/video/exynos/mdev/exynos-mdev.c
> + *
> + * Copyright (c) 2011 Samsung Electronics Co., Ltd.
> + *		http://www.samsung.com
> + *
> + * EXYNOS5 SoC series media device driver
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> +*/
> +
> +#include<linux/bug.h>
> +#include<linux/device.h>
> +#include<linux/errno.h>
> +#include<linux/i2c.h>
> +#include<linux/kernel.h>
> +#include<linux/list.h>
> +#include<linux/module.h>
> +#include<linux/platform_device.h>
> +#include<linux/pm_runtime.h>
> +#include<linux/types.h>
> +#include<linux/slab.h>
> +#include<linux/version.h>
> +#include<media/v4l2-ctrls.h>
> +#include<media/media-device.h>
> +#include<media/exynos_mc.h>
> +
> +static int __devinit mdev_probe(struct platform_device *pdev)
> +{
> +	struct v4l2_device *v4l2_dev;
> +	struct exynos_md *mdev;
> +	int ret;
> +
> +	mdev = kzalloc(sizeof(struct exynos_md), GFP_KERNEL);
> +	if (!mdev)
> +		return -ENOMEM;
> +
> +	mdev->id = pdev->id;
> +	mdev->pdev = pdev;
> +	spin_lock_init(&mdev->slock);
> +
> +	snprintf(mdev->media_dev.model, sizeof(mdev->media_dev.model), "%s%d",
> +		 dev_name(&pdev->dev), mdev->id);
> +
> +	mdev->media_dev.dev =&pdev->dev;
> +
> +	v4l2_dev =&mdev->v4l2_dev;
> +	v4l2_dev->mdev =&mdev->media_dev;
> +	snprintf(v4l2_dev->name, sizeof(v4l2_dev->name), "%s",
> +		 dev_name(&pdev->dev));
> +
> +	ret = v4l2_device_register(&pdev->dev,&mdev->v4l2_dev);
> +	if (ret<  0) {
> +		v4l2_err(v4l2_dev, "Failed to register v4l2_device: %d\n", ret);
> +		goto err_v4l2_reg;
> +	}
> +	ret = media_device_register(&mdev->media_dev);
> +	if (ret<  0) {
> +		v4l2_err(v4l2_dev, "Failed to register media device: %d\n", ret);
> +		goto err_mdev_reg;
> +	}

At this point you have registered a media device which isn't functional, 
i.e. it doesn't have all expected entities. What is the rationale behind 
such decision ? If you look at the s5p-fimc driver, it ensures all entities 
and their drivers are present and then registers a media device itself.
The data describing which entities are present in the system is contained
in the media device's platform_data. This is also helpful for instantiation
from the device tree, the media entities would be described by child nodes
of an aggregate node - which would bound to the platform device driver
registering a media device.

At this point exynos5 drivers are incompatible with s5p-fimc driver, 
effectively preventing the modules reuse.

> +
> +	platform_set_drvdata(pdev, mdev);
> +	v4l2_info(v4l2_dev, "Media%d[0x%08x] was registered successfully\n",
> +		  mdev->id, (unsigned int)mdev);
> +	return 0;
> +
> +err_mdev_reg:
> +	v4l2_device_unregister(&mdev->v4l2_dev);
> +err_v4l2_reg:
> +	kfree(mdev);
> +	return ret;
> +}
> +
> +static int __devexit mdev_remove(struct platform_device *pdev)
> +{
> +	struct exynos_md *mdev = platform_get_drvdata(pdev);
> +
> +	if (!mdev)
> +		return 0;
> +	media_device_unregister(&mdev->media_dev);
> +	v4l2_device_unregister(&mdev->v4l2_dev);
> +	kfree(mdev);
> +	return 0;
> +}
> +
> +static struct platform_driver mdev_driver = {
> +	.probe		= mdev_probe,
> +	.remove		= __devexit_p(mdev_remove),
> +	.driver = {
> +		.name	= MDEV_MODULE_NAME,
> +		.owner	= THIS_MODULE,
> +	}
> +};
> +
> +int __init mdev_init(void)
> +{
> +	int ret = platform_driver_register(&mdev_driver);
> +	if (ret)
> +		err("platform_driver_register failed: %d\n", ret);
> +	return ret;
> +}
> +
> +void __exit mdev_exit(void)
> +{
> +	platform_driver_unregister(&mdev_driver);
> +}
> +
> +module_init(mdev_init);
> +module_exit(mdev_exit);
> +
> +MODULE_AUTHOR("Hyunwoong Kim<khw0178.kim@samsung.com>");
> +MODULE_DESCRIPTION("EXYNOS5 SoC series media device driver");
> +MODULE_LICENSE("GPL");
<snip>

---

Thanks,
Sylwester

RE: [PATCH] media: media-dev: Add media devices for EXYNOS5

[Sungchun Kang]

Hi sylwester,
On 03/01/2012 07:44 AM, Sylwester Nawrocki wrote:
> Hi Sungchun,
> 
> On 02/15/2012 07:02 AM, Sungchun Kang wrote:
> > Since the EXYNOS5 SoCs have various multimedia IPs such as Gscaler,
> > FIMC-LITE, and MIXER, and so on.
> > Additionally, media controller interface is needed to configure
> > connection between them and to control each IPs.
> >
> > This patch adds support media device for EXYNOS5 SoCs.
> > Actually, there are three media devices such as below diagram which
> > are using media control framework.
> > Since they are not belong to one hardware block, we need to manage
> it
> > for connecting with each devices.
> >
> > Follwing is detailed list of them:
> >
> > * Gscaler: general scaler
> >    Support memory to memory interface
> >    Support output interface from memory to display device(LCD, TV)
> >    Support capture interface from device(FIMC-LITE, FIMD) to memory
> >
> > * MIPI-CSIS
> >    Support interconnection(subdev interface) between devices
> 
> Is there any difference in s5p/exynos4 and exynos5 MIPI-CSIS devices ?
> I suspect there isn't and the existing MIPI-CSIS driver can be used
> for
> exynos5 too.
>
It is same hardware, and driver is almost identical.
But I copied from s5p-fimc directory to exynos directory, because we use
media controller framework in each other hardware. - reference from my first mail

Surely, you are author of mipi-csis driver in s5p-fimc directory and I re-used most of your code.
So, I will not submit mipi-csis driver.

> >
> > * FIMC-LITE
> >    Support capture interface from device(Sensor, MIPI-CSIS) to memory
> >    Support interconnection(subdev interface) between devices
> 
> This device is also present on exynos4212/4412 SoCs. Can you tell
> what's difference between FIMC-LITE on Exynos4 and Exynos5 ?
> Either we need separate FIMC-LITE drivers or we need a shared one. I'd
> like to clarify this first.
> 
> > * MIXER
> >    Support output interface from memory to device(HDMI)
> >    Support interconnection(subdev interface) between devices
> >
> > * FIMD
> >    Support framebuffer interface
> >    Support subdev interface to display frames sent from Gscaler
> 
> What about Exynos DRM driver ? Do you have any plans to integrate the
> V4L2 and the DRM driver ? IMHO DRM is more appropriate for some tasks
> on display side, like 2D operations, multiple outputs, windows,
> blending, etc.
> 
I don't know about DRM, can you explain to me about it, or let me know mail-thread?

> > * Rotator
> >    Support memory to memory interface
> >
> > * m2m-scaler
> >    Support only memory to memory interface
> >
> > * And so on...
> >
> >   1) media 0
> >    LCD Output path consists of Gscaler and FIMD(display controller).
> >    +----------------+     +------+
> >    | Gscaler-output | -->  | FIMD | -->  LCD
> >    +----------------+     +------+
> >
> >    HDMI Output path consists of Gscaler, Mixer and HDMI.
> >    +----------------+     +-------+     +------+
> >    | Gscaler-output | -->  | MIXER | -->  | HDMI | -->  TV
> >    +----------------+     +-------+     +------+
> >
> > +--------+     +-----------+     +-----------+     +-----------------+
> >
> >   2) media 1
> >    Camera Capture path consists of MIPI-CSIS, FIMC-LITE and Gscaler
> >    +--------+     +-----------+     +-----------------+
> >    | Sensor | -->  | FIMC-LITE | -->  | Gscaler-capture |
> >    +--------+     +-----------+     +-----------------+
> >
> >    +--------+     +-----------+     +-----------+     +-----------------
> +
> >    | Sensor | -->  | MIPI-CSIS | -->  | FIMC-LITE | -->  | Gscaler-
> capture |
> >    +--------+     +-----------+     +-----------+     +-----------------
> +
> >
> > Signed-off-by: Sungchun Kang<sungchun.kang@samsung.com>
> > ---
> >   drivers/media/video/exynos/mdev/Kconfig       |    8 ++
> >   drivers/media/video/exynos/mdev/Makefile      |    2 +
> >   drivers/media/video/exynos/mdev/exynos-mdev.c |  115
> ++++++++++++++++++
> >   include/media/exynos_mc.h                     |  160
> +++++++++++++++++++++++++
> >   4 files changed, 285 insertions(+), 0 deletions(-)
> >   create mode 100644 drivers/media/video/exynos/mdev/Kconfig
> >   create mode 100644 drivers/media/video/exynos/mdev/Makefile
> >   create mode 100644 drivers/media/video/exynos/mdev/exynos-mdev.c
> >   create mode 100644 include/media/exynos_mc.h
> >
> > diff --git a/drivers/media/video/exynos/mdev/Kconfig
> > b/drivers/media/video/exynos/mdev/Kconfig
> > new file mode 100644
> > index 0000000..15134b0
> > --- /dev/null
> > +++ b/drivers/media/video/exynos/mdev/Kconfig
> > @@ -0,0 +1,8 @@
> > +config EXYNOS_MEDIA_DEVICE
> > +	bool
> > +	depends on MEDIA_EXYNOS
> > +	select MEDIA_CONTROLLER
> > +	select VIDEO_V4L2_SUBDEV_API
> > +	default y
> > +	help
> > +	  This is a v4l2 driver for exynos media device.
> > diff --git a/drivers/media/video/exynos/mdev/Makefile
> > b/drivers/media/video/exynos/mdev/Makefile
> > new file mode 100644
> > index 0000000..175a4bc
> > --- /dev/null
> > +++ b/drivers/media/video/exynos/mdev/Makefile
> > @@ -0,0 +1,2 @@
> > +mdev-objs := exynos-mdev.o
> > +obj-$(CONFIG_EXYNOS_MEDIA_DEVICE)	+= mdev.o
> > diff --git a/drivers/media/video/exynos/mdev/exynos-mdev.c
> > b/drivers/media/video/exynos/mdev/exynos-mdev.c
> > new file mode 100644
> > index 0000000..a76e7c3
> > --- /dev/null
> > +++ b/drivers/media/video/exynos/mdev/exynos-mdev.c
> > @@ -0,0 +1,115 @@
> > +/* drviers/media/video/exynos/mdev/exynos-mdev.c
> > + *
> > + * Copyright (c) 2011 Samsung Electronics Co., Ltd.
> > + *		http://www.samsung.com
> > + *
> > + * EXYNOS5 SoC series media device driver
> > + *
> > + * This program is free software; you can redistribute it and/or
> > +modify
> > + * it under the terms of the GNU General Public License version 2
> as
> > + * published by the Free Software Foundation.
> > +*/
> > +
> > +#include<linux/bug.h>
> > +#include<linux/device.h>
> > +#include<linux/errno.h>
> > +#include<linux/i2c.h>
> > +#include<linux/kernel.h>
> > +#include<linux/list.h>
> > +#include<linux/module.h>
> > +#include<linux/platform_device.h>
> > +#include<linux/pm_runtime.h>
> > +#include<linux/types.h>
> > +#include<linux/slab.h>
> > +#include<linux/version.h>
> > +#include<media/v4l2-ctrls.h>
> > +#include<media/media-device.h>
> > +#include<media/exynos_mc.h>
> > +
> > +static int __devinit mdev_probe(struct platform_device *pdev) {
> > +	struct v4l2_device *v4l2_dev;
> > +	struct exynos_md *mdev;
> > +	int ret;
> > +
> > +	mdev = kzalloc(sizeof(struct exynos_md), GFP_KERNEL);
> > +	if (!mdev)
> > +		return -ENOMEM;
> > +
> > +	mdev->id = pdev->id;
> > +	mdev->pdev = pdev;
> > +	spin_lock_init(&mdev->slock);
> > +
> > +	snprintf(mdev->media_dev.model, sizeof(mdev->media_dev.model),
> "%s%d",
> > +		 dev_name(&pdev->dev), mdev->id);
> > +
> > +	mdev->media_dev.dev =&pdev->dev;
> > +
> > +	v4l2_dev =&mdev->v4l2_dev;
> > +	v4l2_dev->mdev =&mdev->media_dev;
> > +	snprintf(v4l2_dev->name, sizeof(v4l2_dev->name), "%s",
> > +		 dev_name(&pdev->dev));
> > +
> > +	ret = v4l2_device_register(&pdev->dev,&mdev->v4l2_dev);
> > +	if (ret<  0) {
> > +		v4l2_err(v4l2_dev, "Failed to register v4l2_device: %d\n",
> ret);
> > +		goto err_v4l2_reg;
> > +	}
> > +	ret = media_device_register(&mdev->media_dev);
> > +	if (ret<  0) {
> > +		v4l2_err(v4l2_dev, "Failed to register media device: %d\n",
> ret);
> > +		goto err_mdev_reg;
> > +	}
> 
> At this point you have registered a media device which isn't
> functional, i.e. it doesn't have all expected entities. What is the
> rationale behind such decision ? If you look at the s5p-fimc driver,
> it ensures all entities and their drivers are present and then
> registers a media device itself.
> The data describing which entities are present in the system is
> contained in the media device's platform_data. This is also helpful
> for instantiation from the device tree, the media entities would be
> described by child nodes of an aggregate node - which would bound to
> the platform device driver registering a media device.
> 
In exynos5, there are 3 media devices.(media0~2)
First, media devices is probed according to "Makefile".
And the last probed driver called v4l2_device_register_subdev_nodes using late_initcall.
I know this is not great method. But in order to use mc, I think this is out of our control.

> At this point exynos5 drivers are incompatible with s5p-fimc driver,
> effectively preventing the modules reuse.
> 
In exynos5, the fimc hardware is removed.

> > +
> > +	platform_set_drvdata(pdev, mdev);
> > +	v4l2_info(v4l2_dev, "Media%d[0x%08x] was registered
> successfully\n",
> > +		  mdev->id, (unsigned int)mdev);
> > +	return 0;
> > +
> > +err_mdev_reg:
> > +	v4l2_device_unregister(&mdev->v4l2_dev);
> > +err_v4l2_reg:
> > +	kfree(mdev);
> > +	return ret;
> > +}
> > +
> > +static int __devexit mdev_remove(struct platform_device *pdev) {
> > +	struct exynos_md *mdev = platform_get_drvdata(pdev);
> > +
> > +	if (!mdev)
> > +		return 0;
> > +	media_device_unregister(&mdev->media_dev);
> > +	v4l2_device_unregister(&mdev->v4l2_dev);
> > +	kfree(mdev);
> > +	return 0;
> > +}
> > +
> > +static struct platform_driver mdev_driver = {
> > +	.probe		= mdev_probe,
> > +	.remove		= __devexit_p(mdev_remove),
> > +	.driver = {
> > +		.name	= MDEV_MODULE_NAME,
> > +		.owner	= THIS_MODULE,
> > +	}
> > +};
> > +
> > +int __init mdev_init(void)
> > +{
> > +	int ret = platform_driver_register(&mdev_driver);
> > +	if (ret)
> > +		err("platform_driver_register failed: %d\n", ret);
> > +	return ret;
> > +}
> > +
> > +void __exit mdev_exit(void)
> > +{
> > +	platform_driver_unregister(&mdev_driver);
> > +}
> > +
> > +module_init(mdev_init);
> > +module_exit(mdev_exit);
> > +
> > +MODULE_AUTHOR("Hyunwoong Kim<khw0178.kim@samsung.com>");
> > +MODULE_DESCRIPTION("EXYNOS5 SoC series media device driver");
> > +MODULE_LICENSE("GPL");
> <snip>
> 
> ---
> 
> Thanks,
> Sylwester

Re: [PATCH] media: media-dev: Add media devices for EXYNOS5

[Sylwester Nawrocki]

Hi Sungchun,

On 03/05/2012 11:53 AM, Sungchun Kang wrote:
> On 03/01/2012 07:44 AM, Sylwester Nawrocki wrote:
>> On 02/15/2012 07:02 AM, Sungchun Kang wrote:
>>> Since the EXYNOS5 SoCs have various multimedia IPs such as Gscaler,
>>> FIMC-LITE, and MIXER, and so on.
>>> Additionally, media controller interface is needed to configure
>>> connection between them and to control each IPs.
>>>
>>> This patch adds support media device for EXYNOS5 SoCs.
>>> Actually, there are three media devices such as below diagram which
>>> are using media control framework.
>>> Since they are not belong to one hardware block, we need to manage
>> it
>>> for connecting with each devices.
>>>
>>> Follwing is detailed list of them:
>>>
>>> * Gscaler: general scaler
>>>     Support memory to memory interface
>>>     Support output interface from memory to display device(LCD, TV)
>>>     Support capture interface from device(FIMC-LITE, FIMD) to memory
>>>
>>> * MIPI-CSIS
>>>     Support interconnection(subdev interface) between devices
>>
>> Is there any difference in s5p/exynos4 and exynos5 MIPI-CSIS devices ?
>> I suspect there isn't and the existing MIPI-CSIS driver can be used
>> for
>> exynos5 too.
>>
> It is same hardware, and driver is almost identical.
> But I copied from s5p-fimc directory to exynos directory, because we use
> media controller framework in each other hardware. - reference from my first mail

I see, you register the subdevs in a bit different way. You're not justified
to copy the code all around though :-) Code duplication in short term might
look like a quick and easy solution, but it's more a recipe for disaster.
Certainly there is no place for such in the mainline.

Technically, supporting your sub-devices registration method in the s5p-csis
module is rather trivial (thanks to that I've designed it as an independent 
driver), except that the initialization scheme you've adopted doesn't seem
right to me.

Do you have plans on how to instantiate your exynos5 media drivers, including
sensor/video encoder subdevs, from the device tree ?
I bet you'll find easier to do it in the case, where the media device driver
coordinates (sub)devices' probing/initialization.

> Surely, you are author of mipi-csis driver in s5p-fimc directory and I 
> re-used most of your code.
> So, I will not submit mipi-csis driver.

But you need it, so we should agree on the directory/files layout and 
the kernel APIs to allow the common modules to be freely reused.
 
I've seen patches for the other devices, like HDMI/TV out, JPEG, etc. They 
are basically extending the existing drivers/media/video/s5p-* drivers.

I wonder, if we could do something similar with FIMC, e.g. rename 
drivers/media/video/s5p-fimc to drivers/media/video/fimc and implement there
an uniform user interface for as many SoC revisions as possible ?

Common headers would have been in include/media, and include/video if needed.

What do you think ?

>>> * FIMC-LITE
>>>     Support capture interface from device(Sensor, MIPI-CSIS) to memory
>>>     Support interconnection(subdev interface) between devices
>>
>> This device is also present on exynos4212/4412 SoCs. Can you tell
>> what's difference between FIMC-LITE on Exynos4 and Exynos5 ?
>> Either we need separate FIMC-LITE drivers or we need a shared one. I'd
>> like to clarify this first.
>>
>>> * MIXER
>>>     Support output interface from memory to device(HDMI)
>>>     Support interconnection(subdev interface) between devices
>>>
>>> * FIMD
>>>     Support framebuffer interface
>>>     Support subdev interface to display frames sent from Gscaler
>>
>> What about Exynos DRM driver ? Do you have any plans to integrate the
>> V4L2 and the DRM driver ? IMHO DRM is more appropriate for some tasks
>> on display side, like 2D operations, multiple outputs, windows,
>> blending, etc.
>>
> I don't know about DRM, can you explain to me about it, or let me know mail-thread?

I'm no expert in the DRM area, you might get better results contacting 
Mr. Inki Dae, Seung-Wo Kim or Joonyoung Shim directly.

Please check this site for an ELCE presentation slides:
http://elinux.org/ELCE_2011_Presentations_redirect#Table_of_Presentations
("DRM Driver Development For Embedded Systems")

>>> * Rotator
>>>     Support memory to memory interface
>>>
>>> * m2m-scaler
>>>     Support only memory to memory interface
>>>
>>> * And so on...
>>>
>>>    1) media 0
>>>     LCD Output path consists of Gscaler and FIMD(display controller).
>>>     +----------------+     +------+
>>>     | Gscaler-output | -->   | FIMD | -->   LCD
>>>     +----------------+     +------+
>>>
>>>     HDMI Output path consists of Gscaler, Mixer and HDMI.
>>>     +----------------+     +-------+     +------+
>>>     | Gscaler-output | -->   | MIXER | -->   | HDMI | -->   TV
>>>     +----------------+     +-------+     +------+
>>>
>>> +--------+     +-----------+     +-----------+     +-----------------+
>>>
>>>    2) media 1
>>>     Camera Capture path consists of MIPI-CSIS, FIMC-LITE and Gscaler
>>>     +--------+     +-----------+     +-----------------+
>>>     | Sensor | -->   | FIMC-LITE | -->   | Gscaler-capture |
>>>     +--------+     +-----------+     +-----------------+
>>>
>>>     +--------+     +-----------+     +-----------+     +-----------------
>> +
>>>     | Sensor | -->   | MIPI-CSIS | -->   | FIMC-LITE | -->   | Gscaler-
>> capture |
>>>     +--------+     +-----------+     +-----------+     +-----------------
>> +
>>>
>>> Signed-off-by: Sungchun Kang<sungchun.kang@samsung.com>
>>> ---
>>>    drivers/media/video/exynos/mdev/Kconfig       |    8 ++
>>>    drivers/media/video/exynos/mdev/Makefile      |    2 +
>>>    drivers/media/video/exynos/mdev/exynos-mdev.c |  115
>> ++++++++++++++++++
>>>    include/media/exynos_mc.h                     |  160
>> +++++++++++++++++++++++++
>>>    4 files changed, 285 insertions(+), 0 deletions(-)
>>>    create mode 100644 drivers/media/video/exynos/mdev/Kconfig
>>>    create mode 100644 drivers/media/video/exynos/mdev/Makefile
>>>    create mode 100644 drivers/media/video/exynos/mdev/exynos-mdev.c
>>>    create mode 100644 include/media/exynos_mc.h
>>>
>>> diff --git a/drivers/media/video/exynos/mdev/Kconfig
>>> b/drivers/media/video/exynos/mdev/Kconfig
>>> new file mode 100644
>>> index 0000000..15134b0
>>> --- /dev/null
>>> +++ b/drivers/media/video/exynos/mdev/Kconfig
>>> @@ -0,0 +1,8 @@
>>> +config EXYNOS_MEDIA_DEVICE
>>> +	bool
>>> +	depends on MEDIA_EXYNOS
>>> +	select MEDIA_CONTROLLER
>>> +	select VIDEO_V4L2_SUBDEV_API
>>> +	default y
>>> +	help
>>> +	  This is a v4l2 driver for exynos media device.
>>> diff --git a/drivers/media/video/exynos/mdev/Makefile
>>> b/drivers/media/video/exynos/mdev/Makefile
>>> new file mode 100644
>>> index 0000000..175a4bc
>>> --- /dev/null
>>> +++ b/drivers/media/video/exynos/mdev/Makefile
>>> @@ -0,0 +1,2 @@
>>> +mdev-objs := exynos-mdev.o
>>> +obj-$(CONFIG_EXYNOS_MEDIA_DEVICE)	+= mdev.o
>>> diff --git a/drivers/media/video/exynos/mdev/exynos-mdev.c
>>> b/drivers/media/video/exynos/mdev/exynos-mdev.c
>>> new file mode 100644
>>> index 0000000..a76e7c3
>>> --- /dev/null
>>> +++ b/drivers/media/video/exynos/mdev/exynos-mdev.c
>>> @@ -0,0 +1,115 @@
>>> +/* drviers/media/video/exynos/mdev/exynos-mdev.c
>>> + *
>>> + * Copyright (c) 2011 Samsung Electronics Co., Ltd.
>>> + *		http://www.samsung.com
>>> + *
>>> + * EXYNOS5 SoC series media device driver
>>> + *
>>> + * This program is free software; you can redistribute it and/or
>>> +modify
>>> + * it under the terms of the GNU General Public License version 2
>> as
>>> + * published by the Free Software Foundation.
>>> +*/
>>> +
>>> +#include<linux/bug.h>
>>> +#include<linux/device.h>
>>> +#include<linux/errno.h>
>>> +#include<linux/i2c.h>
>>> +#include<linux/kernel.h>
>>> +#include<linux/list.h>
>>> +#include<linux/module.h>
>>> +#include<linux/platform_device.h>
>>> +#include<linux/pm_runtime.h>
>>> +#include<linux/types.h>
>>> +#include<linux/slab.h>
>>> +#include<linux/version.h>
>>> +#include<media/v4l2-ctrls.h>
>>> +#include<media/media-device.h>
>>> +#include<media/exynos_mc.h>
>>> +
>>> +static int __devinit mdev_probe(struct platform_device *pdev) {
>>> +	struct v4l2_device *v4l2_dev;
>>> +	struct exynos_md *mdev;
>>> +	int ret;
>>> +
>>> +	mdev = kzalloc(sizeof(struct exynos_md), GFP_KERNEL);
>>> +	if (!mdev)
>>> +		return -ENOMEM;
>>> +
>>> +	mdev->id = pdev->id;
>>> +	mdev->pdev = pdev;
>>> +	spin_lock_init(&mdev->slock);
>>> +
>>> +	snprintf(mdev->media_dev.model, sizeof(mdev->media_dev.model),
>> "%s%d",
>>> +		 dev_name(&pdev->dev), mdev->id);
>>> +
>>> +	mdev->media_dev.dev =&pdev->dev;
>>> +
>>> +	v4l2_dev =&mdev->v4l2_dev;
>>> +	v4l2_dev->mdev =&mdev->media_dev;
>>> +	snprintf(v4l2_dev->name, sizeof(v4l2_dev->name), "%s",
>>> +		 dev_name(&pdev->dev));
>>> +
>>> +	ret = v4l2_device_register(&pdev->dev,&mdev->v4l2_dev);
>>> +	if (ret<   0) {
>>> +		v4l2_err(v4l2_dev, "Failed to register v4l2_device: %d\n",
>> ret);
>>> +		goto err_v4l2_reg;
>>> +	}
>>> +	ret = media_device_register(&mdev->media_dev);
>>> +	if (ret<   0) {
>>> +		v4l2_err(v4l2_dev, "Failed to register media device: %d\n",
>> ret);
>>> +		goto err_mdev_reg;
>>> +	}
>>
>> At this point you have registered a media device which isn't
>> functional, i.e. it doesn't have all expected entities. What is the
>> rationale behind such decision ? If you look at the s5p-fimc driver,
>> it ensures all entities and their drivers are present and then
>> registers a media device itself.
>> The data describing which entities are present in the system is
>> contained in the media device's platform_data. This is also helpful
>> for instantiation from the device tree, the media entities would be
>> described by child nodes of an aggregate node - which would bound to
>> the platform device driver registering a media device.
>>
> In exynos5, there are 3 media devices.(media0~2)

Can you please list how the devices are partitioned onto each media device ?
Perhaps you could provide a graph using the media-ctl tool from Laurent ?

Here are some details:
 http://linux.davincidsp.com/pipermail/davinci-linux-open-source/2011-September/023373.html

And the media-ctl sources are available here: 
 http://git.ideasonboard.org/?p=media-ctl.git;a=summary

> First, media devices is probed according to "Makefile".
> And the last probed driver called v4l2_device_register_subdev_nodes using 
> late_initcall.
> I know this is not great method. But in order to use mc, I think this is out
> of our control.

Does it mean you don't support camera, display,.. drivers as modules at all ? 

If you haven't anticipated using those drivers as the loadable kernel 
modules then there might long way before you until they are merged upstream. 
We will eventually have kernels covering multiple SoC and the LKM support is
an important feature. You can have different plans for your internal 
implementations, but for the mainline I don't think I'm going to agree
to ignore it.

We probably need some notification mechanism that would be registered by each 
media device driver and then the subdevs would use it, so that an aggregate 
driver can complete initialization.

>> At this point exynos5 drivers are incompatible with s5p-fimc driver,
>> effectively preventing the modules reuse.
>>
> In exynos5, the fimc hardware is removed.

It's rather irrelevant. What's important is that some IPs are used in unchanged
form within multiple SoC series. I was referring to s5p-fimc as the whole camera
interface driver for s5pv210, exynos4210/4x12 SoC, with MIPI-CSIS included.

--
Regards,
Sylwester