Software-only image processing for Intel "complex" cameras

Software-only image processing for Intel "complex" cameras

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 873 bytes --]

Hi!

On Nokia N900, I have similar problems as Intel IPU3 hardware.

Meeting notes say that pure software implementation is not fast
enough, but that it may be useful for debugging. It would be also
useful for me on N900, and probably useful for processing "raw" images
from digital cameras.

There is sensor part, and memory-to-memory part, right? What is
the format of data from the sensor part? What operations would be
expensive on the CPU? If we did everthing on the CPU, what would be
maximum resolution where we could still manage it in real time?

Would it be possible to get access to machine with IPU3, or would
there be someone willing to test libv4l2 patches?

Thanks and best regards,

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Software-only image processing for Intel "complex" cameras

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 1117 bytes --]

Le mercredi 20 juin 2018 à 22:38 +0200, Pavel Machek a écrit :
> Hi!
> 
> On Nokia N900, I have similar problems as Intel IPU3 hardware.
> 
> Meeting notes say that pure software implementation is not fast
> enough, but that it may be useful for debugging. It would be also
> useful for me on N900, and probably useful for processing "raw"
> images
> from digital cameras.
> 
> There is sensor part, and memory-to-memory part, right? What is
> the format of data from the sensor part? What operations would be
> expensive on the CPU? If we did everthing on the CPU, what would be
> maximum resolution where we could still manage it in real time?

The IPU3 sensor produce a vendor specific form of bayer. If we manage
to implement support for this format, it would likely be done in
software. I don't think anyone can answer your other questions has no
one have ever implemented this, hence measure performance.

> 
> Would it be possible to get access to machine with IPU3, or would
> there be someone willing to test libv4l2 patches?
> 
> Thanks and best regards,
> 
> 									
> Pavel

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Software-only image processing for Intel "complex" cameras

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1175 bytes --]

Hi!

> > On Nokia N900, I have similar problems as Intel IPU3 hardware.
> > 
> > Meeting notes say that pure software implementation is not fast
> > enough, but that it may be useful for debugging. It would be also
> > useful for me on N900, and probably useful for processing "raw"
> > images
> > from digital cameras.
> > 
> > There is sensor part, and memory-to-memory part, right? What is
> > the format of data from the sensor part? What operations would be
> > expensive on the CPU? If we did everthing on the CPU, what would be
> > maximum resolution where we could still manage it in real time?
> 
> The IPU3 sensor produce a vendor specific form of bayer. If we manage
> to implement support for this format, it would likely be done in
> software. I don't think anyone can answer your other questions has no
> one have ever implemented this, hence measure performance.

I believe Intel has some estimates.

What is the maximum resolution of camera in the current Dell systems?

Thanks,
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

RE: Software-only image processing for Intel "complex" cameras

[Mario.Limonciello]

> -----Original Message-----
> From: Pavel Machek [mailto:pavel@ucw.cz]
> Sent: Wednesday, June 20, 2018 4:12 PM
> To: Nicolas Dufresne
> Cc: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com;
> niklas.soderlund@ragnatech.se; jerry.w.hu@intel.com; Limonciello, Mario
> Subject: Re: Software-only image processing for Intel "complex" cameras
> 
> Hi!
> 
> > > On Nokia N900, I have similar problems as Intel IPU3 hardware.
> > >
> > > Meeting notes say that pure software implementation is not fast
> > > enough, but that it may be useful for debugging. It would be also
> > > useful for me on N900, and probably useful for processing "raw"
> > > images
> > > from digital cameras.
> > >
> > > There is sensor part, and memory-to-memory part, right? What is
> > > the format of data from the sensor part? What operations would be
> > > expensive on the CPU? If we did everthing on the CPU, what would be
> > > maximum resolution where we could still manage it in real time?
> >
> > The IPU3 sensor produce a vendor specific form of bayer. If we manage
> > to implement support for this format, it would likely be done in
> > software. I don't think anyone can answer your other questions has no
> > one have ever implemented this, hence measure performance.
> 
> I believe Intel has some estimates.
> 
> What is the maximum resolution of camera in the current Dell systems?
> 

5M camera sensor HW spec:
2592x1944

8M camera sensor HW spec:
3264x2448

Thanks,

Re: Software-only image processing for Intel "complex" cameras

[Mauro Carvalho Chehab]

Em Thu, 21 Jun 2018 13:41:37 +0000
<Mario.Limonciello@dell.com> escreveu:

> > -----Original Message-----
> > From: Pavel Machek [mailto:pavel@ucw.cz]
> > Sent: Wednesday, June 20, 2018 4:12 PM
> > To: Nicolas Dufresne
> > Cc: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com;
> > niklas.soderlund@ragnatech.se; jerry.w.hu@intel.com; Limonciello, Mario
> > Subject: Re: Software-only image processing for Intel "complex" cameras
> > 
> > Hi!
> >   
> > > > On Nokia N900, I have similar problems as Intel IPU3 hardware.
> > > >
> > > > Meeting notes say that pure software implementation is not fast
> > > > enough, but that it may be useful for debugging. It would be also
> > > > useful for me on N900, and probably useful for processing "raw"
> > > > images
> > > > from digital cameras.
> > > >
> > > > There is sensor part, and memory-to-memory part, right? What is
> > > > the format of data from the sensor part? What operations would be
> > > > expensive on the CPU? If we did everthing on the CPU, what would be
> > > > maximum resolution where we could still manage it in real time?  
> > >
> > > The IPU3 sensor produce a vendor specific form of bayer. If we manage
> > > to implement support for this format, it would likely be done in
> > > software. I don't think anyone can answer your other questions has no
> > > one have ever implemented this, hence measure performance.  
> > 
> > I believe Intel has some estimates.
> > 
> > What is the maximum resolution of camera in the current Dell systems?
> >   
> 
> 5M camera sensor HW spec:
> 2592x1944
> 
> 8M camera sensor HW spec:
> 3264x2448

Looking at the ipu3 driver, I'm wandering how the library would identify
the system components. The way VIDIOC_QUERYCAP is currently implemented
doesn't help at all:

static int cio2_v4l2_querycap(struct file *file, void *fh,
			      struct v4l2_capability *cap)
{
	struct cio2_device *cio2 = video_drvdata(file);

	strlcpy(cap->driver, CIO2_NAME, sizeof(cap->driver));
	strlcpy(cap->card, CIO2_DEVICE_NAME, sizeof(cap->card));
	snprintf(cap->bus_info, sizeof(cap->bus_info),
		 "PCI:%s", pci_name(cio2->pci_dev));

	return 0;
}

In order to allow the library to know more about the hardware, it
would likely need to expose some model number to userspace. Ok, userspace
could always call dmidecode, but that requires root privileges. We
really don't want media apps to require root.

So, perhaps caps->cap (or a MC caps equivalent call) should, instead be filled
with values obtained from the BIOS DMI tables with some logic based on
enum dmi_field:

enum dmi_field {
	DMI_NONE,
	DMI_BIOS_VENDOR,
	DMI_BIOS_VERSION,
	DMI_BIOS_DATE,
	DMI_SYS_VENDOR,
	DMI_PRODUCT_NAME,
	DMI_PRODUCT_VERSION,
	DMI_PRODUCT_SERIAL,
	DMI_PRODUCT_UUID,
	DMI_PRODUCT_FAMILY,
	DMI_BOARD_VENDOR,
	DMI_BOARD_NAME,
	DMI_BOARD_VERSION,
	DMI_BOARD_SERIAL,
	DMI_BOARD_ASSET_TAG,
	DMI_CHASSIS_VENDOR,
	DMI_CHASSIS_TYPE,
	DMI_CHASSIS_VERSION,
	DMI_CHASSIS_SERIAL,
	DMI_CHASSIS_ASSET_TAG,
	DMI_STRING_MAX,
};

e. g. something like: 

	board_vendor = dmi_get_system_info(DMI_BOARD_VENDOR);
	board_name = dmi_get_system_info(DMI_BOARD_NAME);
	board_version = dmi_get_system_info(DMI_BOARD_NAME);
	product_name = dmi_get_system_info(DMI_PRODUCT_NAME);
	product_version = dmi_get_system_info(DMI_PRODUCT_VERSION);

	sprintf(dev->cap, "%s:%s:%s:%s", board_vendor, board_name, board_version, product_name, product_version);

(the real code should check if the values are filled, as not all BIOS vendors use the same DMI fields)

With that, the library can auto-adjust without needing to run anything as
root.



Cheers,
Mauro

RE: Software-only image processing for Intel "complex" cameras

[Mario.Limonciello]

> -----Original Message-----
> From: Mauro Carvalho Chehab [mailto:mchehab+samsung@kernel.org]
> Sent: Thursday, June 21, 2018 1:50 PM
> To: Limonciello, Mario
> Cc: pavel@ucw.cz; nicolas@ndufresne.ca; linux-media@vger.kernel.org;
> sakari.ailus@linux.intel.com; niklas.soderlund@ragnatech.se;
> jerry.w.hu@intel.com
> Subject: Re: Software-only image processing for Intel "complex" cameras
> 
> Em Thu, 21 Jun 2018 13:41:37 +0000
> <Mario.Limonciello@dell.com> escreveu:
> 
> > > -----Original Message-----
> > > From: Pavel Machek [mailto:pavel@ucw.cz]
> > > Sent: Wednesday, June 20, 2018 4:12 PM
> > > To: Nicolas Dufresne
> > > Cc: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com;
> > > niklas.soderlund@ragnatech.se; jerry.w.hu@intel.com; Limonciello, Mario
> > > Subject: Re: Software-only image processing for Intel "complex" cameras
> > >
> > > Hi!
> > >
> > > > > On Nokia N900, I have similar problems as Intel IPU3 hardware.
> > > > >
> > > > > Meeting notes say that pure software implementation is not fast
> > > > > enough, but that it may be useful for debugging. It would be also
> > > > > useful for me on N900, and probably useful for processing "raw"
> > > > > images
> > > > > from digital cameras.
> > > > >
> > > > > There is sensor part, and memory-to-memory part, right? What is
> > > > > the format of data from the sensor part? What operations would be
> > > > > expensive on the CPU? If we did everthing on the CPU, what would be
> > > > > maximum resolution where we could still manage it in real time?
> > > >
> > > > The IPU3 sensor produce a vendor specific form of bayer. If we manage
> > > > to implement support for this format, it would likely be done in
> > > > software. I don't think anyone can answer your other questions has no
> > > > one have ever implemented this, hence measure performance.
> > >
> > > I believe Intel has some estimates.
> > >
> > > What is the maximum resolution of camera in the current Dell systems?
> > >
> >
> > 5M camera sensor HW spec:
> > 2592x1944
> >
> > 8M camera sensor HW spec:
> > 3264x2448
> 
> Looking at the ipu3 driver, I'm wandering how the library would identify
> the system components. The way VIDIOC_QUERYCAP is currently implemented
> doesn't help at all:
> 
> static int cio2_v4l2_querycap(struct file *file, void *fh,
> 			      struct v4l2_capability *cap)
> {
> 	struct cio2_device *cio2 = video_drvdata(file);
> 
> 	strlcpy(cap->driver, CIO2_NAME, sizeof(cap->driver));
> 	strlcpy(cap->card, CIO2_DEVICE_NAME, sizeof(cap->card));
> 	snprintf(cap->bus_info, sizeof(cap->bus_info),
> 		 "PCI:%s", pci_name(cio2->pci_dev));
> 
> 	return 0;
> }
> 
> In order to allow the library to know more about the hardware, it
> would likely need to expose some model number to userspace. Ok, userspace
> could always call dmidecode, but that requires root privileges. We
> really don't want media apps to require root.
> 
> So, perhaps caps->cap (or a MC caps equivalent call) should, instead be filled
> with values obtained from the BIOS DMI tables with some logic based on
> enum dmi_field:
> 
> enum dmi_field {
> 	DMI_NONE,
> 	DMI_BIOS_VENDOR,
> 	DMI_BIOS_VERSION,
> 	DMI_BIOS_DATE,
> 	DMI_SYS_VENDOR,
> 	DMI_PRODUCT_NAME,
> 	DMI_PRODUCT_VERSION,
> 	DMI_PRODUCT_SERIAL,
> 	DMI_PRODUCT_UUID,
> 	DMI_PRODUCT_FAMILY,
> 	DMI_BOARD_VENDOR,
> 	DMI_BOARD_NAME,
> 	DMI_BOARD_VERSION,
> 	DMI_BOARD_SERIAL,
> 	DMI_BOARD_ASSET_TAG,
> 	DMI_CHASSIS_VENDOR,
> 	DMI_CHASSIS_TYPE,
> 	DMI_CHASSIS_VERSION,
> 	DMI_CHASSIS_SERIAL,
> 	DMI_CHASSIS_ASSET_TAG,
> 	DMI_STRING_MAX,
> };
> 
> e. g. something like:
> 
> 	board_vendor = dmi_get_system_info(DMI_BOARD_VENDOR);
> 	board_name = dmi_get_system_info(DMI_BOARD_NAME);
> 	board_version = dmi_get_system_info(DMI_BOARD_NAME);
> 	product_name = dmi_get_system_info(DMI_PRODUCT_NAME);
> 	product_version = dmi_get_system_info(DMI_PRODUCT_VERSION);
> 
> 	sprintf(dev->cap, "%s:%s:%s:%s", board_vendor, board_name,
> board_version, product_name, product_version);
> 
> (the real code should check if the values are filled, as not all BIOS vendors use the
> same DMI fields)
> 
> With that, the library can auto-adjust without needing to run anything as
> root.
> 
Well actually most of those fields you're interested in are already exposed to userspace
through sysfs /sys/class/dmi/id/

Can't the library just pull them from there?

The one field that isn't exposed is actually the one I think you should key off of though:
Product SKU number.  So I would propose as part of this change that should start to get
exposed to userspace too.

The reasoning is I'm a little concerned in taking an approach that goes off of marketing model number
specifically because it's creating an assumption that all systems with that model number
have the exact same components.

It's possible for two systems to have the same model number but to second source for
example.  This might not affect complex cameras, but I just want to make sure it's taken
into consideration.  At least going off of Product SKU will better narrow it down.

Re: Software-only image processing for Intel "complex" cameras

[Mauro Carvalho Chehab]

Em Thu, 21 Jun 2018 18:58:37 +0000
<Mario.Limonciello@dell.com> escreveu:

> > -----Original Message-----
> > From: Mauro Carvalho Chehab [mailto:mchehab+samsung@kernel.org]
> > Sent: Thursday, June 21, 2018 1:50 PM
> > To: Limonciello, Mario
> > Cc: pavel@ucw.cz; nicolas@ndufresne.ca; linux-media@vger.kernel.org;
> > sakari.ailus@linux.intel.com; niklas.soderlund@ragnatech.se;
> > jerry.w.hu@intel.com
> > Subject: Re: Software-only image processing for Intel "complex" cameras
> > 
> > Em Thu, 21 Jun 2018 13:41:37 +0000
> > <Mario.Limonciello@dell.com> escreveu:
> >   
> > > > -----Original Message-----
> > > > From: Pavel Machek [mailto:pavel@ucw.cz]
> > > > Sent: Wednesday, June 20, 2018 4:12 PM
> > > > To: Nicolas Dufresne
> > > > Cc: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com;
> > > > niklas.soderlund@ragnatech.se; jerry.w.hu@intel.com; Limonciello, Mario
> > > > Subject: Re: Software-only image processing for Intel "complex" cameras
> > > >
> > > > Hi!
> > > >  
> > > > > > On Nokia N900, I have similar problems as Intel IPU3 hardware.
> > > > > >
> > > > > > Meeting notes say that pure software implementation is not fast
> > > > > > enough, but that it may be useful for debugging. It would be also
> > > > > > useful for me on N900, and probably useful for processing "raw"
> > > > > > images
> > > > > > from digital cameras.
> > > > > >
> > > > > > There is sensor part, and memory-to-memory part, right? What is
> > > > > > the format of data from the sensor part? What operations would be
> > > > > > expensive on the CPU? If we did everthing on the CPU, what would be
> > > > > > maximum resolution where we could still manage it in real time?  
> > > > >
> > > > > The IPU3 sensor produce a vendor specific form of bayer. If we manage
> > > > > to implement support for this format, it would likely be done in
> > > > > software. I don't think anyone can answer your other questions has no
> > > > > one have ever implemented this, hence measure performance.  
> > > >
> > > > I believe Intel has some estimates.
> > > >
> > > > What is the maximum resolution of camera in the current Dell systems?
> > > >  
> > >
> > > 5M camera sensor HW spec:
> > > 2592x1944
> > >
> > > 8M camera sensor HW spec:
> > > 3264x2448  
> > 
> > Looking at the ipu3 driver, I'm wandering how the library would identify
> > the system components. The way VIDIOC_QUERYCAP is currently implemented
> > doesn't help at all:
> > 
> > static int cio2_v4l2_querycap(struct file *file, void *fh,
> > 			      struct v4l2_capability *cap)
> > {
> > 	struct cio2_device *cio2 = video_drvdata(file);
> > 
> > 	strlcpy(cap->driver, CIO2_NAME, sizeof(cap->driver));
> > 	strlcpy(cap->card, CIO2_DEVICE_NAME, sizeof(cap->card));
> > 	snprintf(cap->bus_info, sizeof(cap->bus_info),
> > 		 "PCI:%s", pci_name(cio2->pci_dev));
> > 
> > 	return 0;
> > }
> > 
> > In order to allow the library to know more about the hardware, it
> > would likely need to expose some model number to userspace. Ok, userspace
> > could always call dmidecode, but that requires root privileges. We
> > really don't want media apps to require root.
> > 
> > So, perhaps caps->cap (or a MC caps equivalent call) should, instead be filled
> > with values obtained from the BIOS DMI tables with some logic based on
> > enum dmi_field:
> > 
> > enum dmi_field {
> > 	DMI_NONE,
> > 	DMI_BIOS_VENDOR,
> > 	DMI_BIOS_VERSION,
> > 	DMI_BIOS_DATE,
> > 	DMI_SYS_VENDOR,
> > 	DMI_PRODUCT_NAME,
> > 	DMI_PRODUCT_VERSION,
> > 	DMI_PRODUCT_SERIAL,
> > 	DMI_PRODUCT_UUID,
> > 	DMI_PRODUCT_FAMILY,
> > 	DMI_BOARD_VENDOR,
> > 	DMI_BOARD_NAME,
> > 	DMI_BOARD_VERSION,
> > 	DMI_BOARD_SERIAL,
> > 	DMI_BOARD_ASSET_TAG,
> > 	DMI_CHASSIS_VENDOR,
> > 	DMI_CHASSIS_TYPE,
> > 	DMI_CHASSIS_VERSION,
> > 	DMI_CHASSIS_SERIAL,
> > 	DMI_CHASSIS_ASSET_TAG,
> > 	DMI_STRING_MAX,
> > };
> > 
> > e. g. something like:
> > 
> > 	board_vendor = dmi_get_system_info(DMI_BOARD_VENDOR);
> > 	board_name = dmi_get_system_info(DMI_BOARD_NAME);
> > 	board_version = dmi_get_system_info(DMI_BOARD_NAME);
> > 	product_name = dmi_get_system_info(DMI_PRODUCT_NAME);
> > 	product_version = dmi_get_system_info(DMI_PRODUCT_VERSION);
> > 
> > 	sprintf(dev->cap, "%s:%s:%s:%s", board_vendor, board_name,
> > board_version, product_name, product_version);
> > 
> > (the real code should check if the values are filled, as not all BIOS vendors use the
> > same DMI fields)
> > 
> > With that, the library can auto-adjust without needing to run anything as
> > root.
> >   
> Well actually most of those fields you're interested in are already exposed to userspace
> through sysfs /sys/class/dmi/id/
> 
> Can't the library just pull them from there?

Good point. Yeah, the library could use them.

> The one field that isn't exposed is actually the one I think you should key off of though:
> Product SKU number.  So I would propose as part of this change that should start to get
> exposed to userspace too.
> 
> The reasoning is I'm a little concerned in taking an approach that goes off of marketing model number
> specifically because it's creating an assumption that all systems with that model number
> have the exact same components.
> 
> It's possible for two systems to have the same model number but to second source for
> example.  This might not affect complex cameras, but I just want to make sure it's taken
> into consideration.  At least going off of Product SKU will better narrow it down.

Yeah, the library should be able to uniquely match whatever components are
inside a system, and just the marketing model number might not be enough.

Right now, only dmidecode seems to get SKU number.

On a quick look, it seems that a patch adding support for it was already
proposed:
	https://lkml.org/lkml/2018/4/24/1166

And got accepted:
	https://lkml.org/lkml/2018/4/27/96 

So, it seems we'll be covered for newer Kernel versions.

Cheers,
Mauro

Re: Software-only image processing for Intel "complex" cameras

[Mauro Carvalho Chehab]

Em Fri, 22 Jun 2018 06:08:50 +0900
Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:

> Em Thu, 21 Jun 2018 18:58:37 +0000
> <Mario.Limonciello@dell.com> escreveu:
> 
> > > -----Original Message-----
> > > From: Mauro Carvalho Chehab [mailto:mchehab+samsung@kernel.org]
> > > Sent: Thursday, June 21, 2018 1:50 PM
> > > To: Limonciello, Mario
> > > Cc: pavel@ucw.cz; nicolas@ndufresne.ca; linux-media@vger.kernel.org;
> > > sakari.ailus@linux.intel.com; niklas.soderlund@ragnatech.se;
> > > jerry.w.hu@intel.com
> > > Subject: Re: Software-only image processing for Intel "complex" cameras
> > > 
> > > Em Thu, 21 Jun 2018 13:41:37 +0000
> > > <Mario.Limonciello@dell.com> escreveu:
> > >   
> > > > > -----Original Message-----
> > > > > From: Pavel Machek [mailto:pavel@ucw.cz]
> > > > > Sent: Wednesday, June 20, 2018 4:12 PM
> > > > > To: Nicolas Dufresne
> > > > > Cc: linux-media@vger.kernel.org; sakari.ailus@linux.intel.com;
> > > > > niklas.soderlund@ragnatech.se; jerry.w.hu@intel.com; Limonciello, Mario
> > > > > Subject: Re: Software-only image processing for Intel "complex" cameras
> > > > >
> > > > > Hi!
> > > > >  
> > > > > > > On Nokia N900, I have similar problems as Intel IPU3 hardware.
> > > > > > >
> > > > > > > Meeting notes say that pure software implementation is not fast
> > > > > > > enough, but that it may be useful for debugging. It would be also
> > > > > > > useful for me on N900, and probably useful for processing "raw"
> > > > > > > images
> > > > > > > from digital cameras.
> > > > > > >
> > > > > > > There is sensor part, and memory-to-memory part, right? What is
> > > > > > > the format of data from the sensor part? What operations would be
> > > > > > > expensive on the CPU? If we did everthing on the CPU, what would be
> > > > > > > maximum resolution where we could still manage it in real time?  
> > > > > >
> > > > > > The IPU3 sensor produce a vendor specific form of bayer. If we manage
> > > > > > to implement support for this format, it would likely be done in
> > > > > > software. I don't think anyone can answer your other questions has no
> > > > > > one have ever implemented this, hence measure performance.  
> > > > >
> > > > > I believe Intel has some estimates.
> > > > >
> > > > > What is the maximum resolution of camera in the current Dell systems?
> > > > >  
> > > >
> > > > 5M camera sensor HW spec:
> > > > 2592x1944
> > > >
> > > > 8M camera sensor HW spec:
> > > > 3264x2448  
> > > 
> > > Looking at the ipu3 driver, I'm wandering how the library would identify
> > > the system components. The way VIDIOC_QUERYCAP is currently implemented
> > > doesn't help at all:
> > > 
> > > static int cio2_v4l2_querycap(struct file *file, void *fh,
> > > 			      struct v4l2_capability *cap)
> > > {
> > > 	struct cio2_device *cio2 = video_drvdata(file);
> > > 
> > > 	strlcpy(cap->driver, CIO2_NAME, sizeof(cap->driver));
> > > 	strlcpy(cap->card, CIO2_DEVICE_NAME, sizeof(cap->card));
> > > 	snprintf(cap->bus_info, sizeof(cap->bus_info),
> > > 		 "PCI:%s", pci_name(cio2->pci_dev));
> > > 
> > > 	return 0;
> > > }
> > > 
> > > In order to allow the library to know more about the hardware, it
> > > would likely need to expose some model number to userspace. Ok, userspace
> > > could always call dmidecode, but that requires root privileges. We
> > > really don't want media apps to require root.
> > > 
> > > So, perhaps caps->cap (or a MC caps equivalent call) should, instead be filled
> > > with values obtained from the BIOS DMI tables with some logic based on
> > > enum dmi_field:
> > > 
> > > enum dmi_field {
> > > 	DMI_NONE,
> > > 	DMI_BIOS_VENDOR,
> > > 	DMI_BIOS_VERSION,
> > > 	DMI_BIOS_DATE,
> > > 	DMI_SYS_VENDOR,
> > > 	DMI_PRODUCT_NAME,
> > > 	DMI_PRODUCT_VERSION,
> > > 	DMI_PRODUCT_SERIAL,
> > > 	DMI_PRODUCT_UUID,
> > > 	DMI_PRODUCT_FAMILY,
> > > 	DMI_BOARD_VENDOR,
> > > 	DMI_BOARD_NAME,
> > > 	DMI_BOARD_VERSION,
> > > 	DMI_BOARD_SERIAL,
> > > 	DMI_BOARD_ASSET_TAG,
> > > 	DMI_CHASSIS_VENDOR,
> > > 	DMI_CHASSIS_TYPE,
> > > 	DMI_CHASSIS_VERSION,
> > > 	DMI_CHASSIS_SERIAL,
> > > 	DMI_CHASSIS_ASSET_TAG,
> > > 	DMI_STRING_MAX,
> > > };
> > > 
> > > e. g. something like:
> > > 
> > > 	board_vendor = dmi_get_system_info(DMI_BOARD_VENDOR);
> > > 	board_name = dmi_get_system_info(DMI_BOARD_NAME);
> > > 	board_version = dmi_get_system_info(DMI_BOARD_NAME);
> > > 	product_name = dmi_get_system_info(DMI_PRODUCT_NAME);
> > > 	product_version = dmi_get_system_info(DMI_PRODUCT_VERSION);
> > > 
> > > 	sprintf(dev->cap, "%s:%s:%s:%s", board_vendor, board_name,
> > > board_version, product_name, product_version);
> > > 
> > > (the real code should check if the values are filled, as not all BIOS vendors use the
> > > same DMI fields)
> > > 
> > > With that, the library can auto-adjust without needing to run anything as
> > > root.
> > >   
> > Well actually most of those fields you're interested in are already exposed to userspace
> > through sysfs /sys/class/dmi/id/
> > 
> > Can't the library just pull them from there?
> 
> Good point. Yeah, the library could use them.
> 
> > The one field that isn't exposed is actually the one I think you should key off of though:
> > Product SKU number.  So I would propose as part of this change that should start to get
> > exposed to userspace too.
> > 
> > The reasoning is I'm a little concerned in taking an approach that goes off of marketing model number
> > specifically because it's creating an assumption that all systems with that model number
> > have the exact same components.
> > 
> > It's possible for two systems to have the same model number but to second source for
> > example.  This might not affect complex cameras, but I just want to make sure it's taken
> > into consideration.  At least going off of Product SKU will better narrow it down.
> 
> Yeah, the library should be able to uniquely match whatever components are
> inside a system, and just the marketing model number might not be enough.
> 
> Right now, only dmidecode seems to get SKU number.
> 
> On a quick look, it seems that a patch adding support for it was already
> proposed:
> 	https://lkml.org/lkml/2018/4/24/1166
> 
> And got accepted:
> 	https://lkml.org/lkml/2018/4/27/96 
> 
> So, it seems we'll be covered for newer Kernel versions.

Just occurred to me that this is a major difference about how complex
camera hardware detection should happen. On ARM-based devices, the
hardware topology comes from Open Firmware. It uniquely describes how
the board is wired. That reflects the way the media controller exposes
the hardware, as each platform driver reads a set of properties via
of_property_foo() functions.

On "normal" PCI drivers (bttv, cx88, ...), there's a PCI table filled with
PCI vendor IDs and some complex logic with uses that in order to identify
the hardware components and to load the associated drivers.

With IPU3, the device appears as a PCI device. The driver doesn't do
any of_property_foo() call, nor it has a device table with would be used to
identify what sensor is associated to what device.

So, on a quick look there, I was unable to see how it detects what sensors
are connected to an specific hardware instance and how it would load the
associated sensor drivers.

For us to be able to stick with an example, let's assume the Dell Latitude 5285
laptop [1]. What sensors are present on it? What's its PCI ID?

[1] https://www.spinics.net/lists/linux-usb/msg167478.html
    https://www.spinics.net/lists/linux-usb/msg163617.html


Jerry/Jian,

Could you please shed a light about how a Dell 5285 hardware would be
detected by the IPU3 driver and what MC graph is created by the current
driver?

Cheers,
Mauro

RE: Software-only image processing for Intel "complex" cameras

[Zheng, Jian Xu]

Hi Mauro,

> -----Original Message-----
> From: linux-media-owner@vger.kernel.org [mailto:linux-media-
> owner@vger.kernel.org] On Behalf Of Mauro Carvalho Chehab
> Sent: Friday, June 22, 2018 5:41 AM
> To: Mario.Limonciello@dell.com
> Cc: pavel@ucw.cz; nicolas@ndufresne.ca; linux-media@vger.kernel.org;
> sakari.ailus@linux.intel.com; niklas.soderlund@ragnatech.se; Hu, Jerry W
> <jerry.w.hu@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>
> Subject: Re: Software-only image processing for Intel "complex" cameras
> 
> Em Fri, 22 Jun 2018 06:08:50 +0900
> Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> 
> > Em Thu, 21 Jun 2018 18:58:37 +0000
> > <Mario.Limonciello@dell.com> escreveu:
> >
> Jerry/Jian,
> 
> Could you please shed a light about how a Dell 5285 hardware would be
> detected by the IPU3 driver and what MC graph is created by the current
> driver?

Sure, Mauro. I need to check the information on the Dell 5285.
IPU3 driver are detected by PCI vendor id and device id.

IPU3 CIO2 MC graph is:
Sensor A -> IPU3 CSI2 0(subdev) -> IPU3 CIO2 0 (video node)
Sensor B -> IPU3 CSI2 1(subdev) -> IPU3 CIO2 1 (video node)

IPU3 IMGu MC graph is:
IMGu subdev inputs:
ipu3-imgu input [Pad0] => ipu3-imgu [Pad0]
ipu3-imgu parameters [Pad0] => ipu3-imgu [Pad1]

IMGu subdev outputs:
ipu3-imgu [Pad2]  => ipu3-imgu output [Pad0] 
ipu3-imgu [Pad3]  => ipu3-imgu viewfinder [Pad0] 
ipu3-imgu [Pad4]  => ipu3-imgu postview [Pad0] 
ipu3-imgu [Pad5]  => ipu3-imgu 3a stat [Pad0]

Best Regards,
Jianxu(Clive) Zheng

Re: Software-only image processing for Intel "complex" cameras

[Mauro Carvalho Chehab]

Em Mon, 25 Jun 2018 09:48:56 +0000
"Zheng, Jian Xu" <jian.xu.zheng@intel.com> escreveu:

> Hi Mauro,
> 
> > -----Original Message-----
> > From: linux-media-owner@vger.kernel.org [mailto:linux-media-
> > owner@vger.kernel.org] On Behalf Of Mauro Carvalho Chehab
> > Sent: Friday, June 22, 2018 5:41 AM
> > To: Mario.Limonciello@dell.com
> > Cc: pavel@ucw.cz; nicolas@ndufresne.ca; linux-media@vger.kernel.org;
> > sakari.ailus@linux.intel.com; niklas.soderlund@ragnatech.se; Hu, Jerry W
> > <jerry.w.hu@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>
> > Subject: Re: Software-only image processing for Intel "complex" cameras
> > 
> > Em Fri, 22 Jun 2018 06:08:50 +0900
> > Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
> >   
> > > Em Thu, 21 Jun 2018 18:58:37 +0000
> > > <Mario.Limonciello@dell.com> escreveu:
> > >  
> > Jerry/Jian,
> > 
> > Could you please shed a light about how a Dell 5285 hardware would be
> > detected by the IPU3 driver and what MC graph is created by the current
> > driver?  
> 
> Sure, Mauro. I need to check the information on the Dell 5285.
> IPU3 driver are detected by PCI vendor id and device id.
> 
> IPU3 CIO2 MC graph is:
> Sensor A -> IPU3 CSI2 0(subdev) -> IPU3 CIO2 0 (video node)
> Sensor B -> IPU3 CSI2 1(subdev) -> IPU3 CIO2 1 (video node)

How does it detect what driver should be loaded for Sensor A and B?

Does the ACPI table identifies the sensors? If so, how the driver
associates an specific PCI vendor ID with the corresponding sensor
settings?

> 
> IPU3 IMGu MC graph is:
> IMGu subdev inputs:
> ipu3-imgu input [Pad0] => ipu3-imgu [Pad0]
> ipu3-imgu parameters [Pad0] => ipu3-imgu [Pad1]
> 
> IMGu subdev outputs:
> ipu3-imgu [Pad2]  => ipu3-imgu output [Pad0] 
> ipu3-imgu [Pad3]  => ipu3-imgu viewfinder [Pad0] 
> ipu3-imgu [Pad4]  => ipu3-imgu postview [Pad0] 
> ipu3-imgu [Pad5]  => ipu3-imgu 3a stat [Pad0]
> 
> Best Regards,
> Jianxu(Clive) Zheng



Thanks,
Mauro

Re: Software-only image processing for Intel "complex" cameras

[Javier Martinez Canillas]

Hi Mauro,

On Mon, Jun 25, 2018 at 3:10 PM, Mauro Carvalho Chehab
<mchehab+samsung@kernel.org> wrote:
> Em Mon, 25 Jun 2018 09:48:56 +0000
> "Zheng, Jian Xu" <jian.xu.zheng@intel.com> escreveu:
>
>> Hi Mauro,
>>
>> > -----Original Message-----
>> > From: linux-media-owner@vger.kernel.org [mailto:linux-media-
>> > owner@vger.kernel.org] On Behalf Of Mauro Carvalho Chehab
>> > Sent: Friday, June 22, 2018 5:41 AM
>> > To: Mario.Limonciello@dell.com
>> > Cc: pavel@ucw.cz; nicolas@ndufresne.ca; linux-media@vger.kernel.org;
>> > sakari.ailus@linux.intel.com; niklas.soderlund@ragnatech.se; Hu, Jerry W
>> > <jerry.w.hu@intel.com>; Zheng, Jian Xu <jian.xu.zheng@intel.com>
>> > Subject: Re: Software-only image processing for Intel "complex" cameras
>> >
>> > Em Fri, 22 Jun 2018 06:08:50 +0900
>> > Mauro Carvalho Chehab <mchehab+samsung@kernel.org> escreveu:
>> >
>> > > Em Thu, 21 Jun 2018 18:58:37 +0000
>> > > <Mario.Limonciello@dell.com> escreveu:
>> > >
>> > Jerry/Jian,
>> >
>> > Could you please shed a light about how a Dell 5285 hardware would be
>> > detected by the IPU3 driver and what MC graph is created by the current
>> > driver?
>>
>> Sure, Mauro. I need to check the information on the Dell 5285.
>> IPU3 driver are detected by PCI vendor id and device id.
>>
>> IPU3 CIO2 MC graph is:
>> Sensor A -> IPU3 CSI2 0(subdev) -> IPU3 CIO2 0 (video node)
>> Sensor B -> IPU3 CSI2 1(subdev) -> IPU3 CIO2 1 (video node)
>

I don't think your questions below were answered. I'll try to answer
since I've been looking at this, but Sakari or others can correct me
if I got something wrong.

> How does it detect what driver should be loaded for Sensor A and B?
>

It depends on the firmware interface used to describe the hardware
topology, for ACPI the sensors are described in a DSDT table and match
using an entry in the struct acpi_device_id table in the camera
driver. For DT the sensors are described in a DT node with a
compatible string that matches an entry in the struct of_device_id
table, or an entry in the struct i2c_device_id table as a fallback.

> Does the ACPI table identifies the sensors? If so, how the driver
> associates an specific PCI vendor ID with the corresponding sensor
> settings?
>

Camera sensors are usually not PCI devices, but I2C devices. And yes,
these are described in an ACPI table (DSDT) as mentioned before. For
example in a laptop I've the following on an disassembled DSDT table:

        Device (CAM0)
        {
            Name (_ADR, Zero)  // _ADR: Address
            Name (_HID, "OVTI5648")  // _HID: Hardware ID
            Name (_CID, "OVTI5648")  // _CID: Compatible ID
            Name (_DDN, "OV5648-CRDD")  // _DDN: DOS Device Name
            Name (_UID, Zero)  // _UID: Unique ID
            Name (_DEP, Package (0x02)  // _DEP: Dependencies
            {
                PMI0,
                I2C4
            })

Now how the drivers get loaded / registered and how the media entities
are associated is somewhat orthogonal (although the information to do
both is usually encoded in the same place).

In the case of OF, the relation between the bridge device and the
sensors is encoded in the DT using the video interface / graph binding
[0, 1]. For ACPI, the same information is encoded in an ACPI table
using a _DSD extension [2], which is quite similar to the DT binding.

Since both hardware descriptions use the same properties (port,
endpoint, etc) and semantics, drivers can use the V4L2 fwnode kAPI to
parse this information regardless of the underlying firmware interface
used to encoded this relationship. This allows the same camera drivers
to be used with either DT or ACPI.

So a bridge driver will register a v4l2 async notifier on probe and
camera drivers will register the subdevices using the v4l2 async
interface. That way the v4l2 async layer can bind the two and call the
bridge device .bound (each time that a pending device is registered)
and .complete (when there are no pending devices remaining to be
registered) callbacks.

A problem I found though is that [2] is Linux specific, so the
firmware for a machine meant to be used with Windows (i.e: Microsoft
Surface 2-in-1 computers) will have a DSDT table that's not compatible
with the v4l2 fwnode layer. One option is to override [4] the DSDT
table with one that contains the information expected by Linux, for
example by having the table in the initramfs [5]. The problem is that
it seems the distros (at least in the Fedora case) don't want to get
into the business of shipping machine specific ACPI tables.

Another option could be to have a shim layer in the kernel that
translates the information in the DSDT as used by Windows to what's
expected by Linux. This is an option that I haven't investigated yet,
but wanted to know your opinion if this could be a feasible approach.
Or if you have any other suggestions on how we should proceed with
this.

[0]: https://www.kernel.org/doc/Documentation/devicetree/bindings/media/video-interfaces.txt
[1]: https://www.kernel.org/doc/Documentation/devicetree/bindings/graph.txt
[2]: https://www.kernel.org/doc/Documentation/acpi/dsd/graph.txt
[3]: https://linuxtv.org/downloads/v4l-dvb-apis/kapi/v4l2-fwnode.html
[4]: https://www.kernel.org/doc/Documentation/acpi/dsdt-override.txt
[5]: https://www.kernel.org/doc/Documentation/acpi/initrd_table_override.txt

Best regards,
Javier

Re: Software-only image processing for Intel "complex" cameras

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]

Hi!

> > > e. g. something like:
> > > 
> > > 	board_vendor = dmi_get_system_info(DMI_BOARD_VENDOR);
> > > 	board_name = dmi_get_system_info(DMI_BOARD_NAME);
> > > 	board_version = dmi_get_system_info(DMI_BOARD_NAME);
> > > 	product_name = dmi_get_system_info(DMI_PRODUCT_NAME);
> > > 	product_version = dmi_get_system_info(DMI_PRODUCT_VERSION);
> > > 
> > > 	sprintf(dev->cap, "%s:%s:%s:%s", board_vendor, board_name,
> > > board_version, product_name, product_version);
> > > 
> > > (the real code should check if the values are filled, as not all BIOS vendors use the
> > > same DMI fields)
> > > 
> > > With that, the library can auto-adjust without needing to run anything as
> > > root.
> > >   
> > Well actually most of those fields you're interested in are already exposed to userspace
> > through sysfs /sys/class/dmi/id/
> > 
> > Can't the library just pull them from there?
> 
> Good point. Yeah, the library could use them.

This could be done, but it would be better if libraries could query
neccessary information from v4l2 drivers.

If DMI was used, _library_ would need to know about new hardware,
which is  not desirable.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Software-only image processing for Intel "complex" cameras

[Mauro Carvalho Chehab]

Em Wed, 20 Jun 2018 16:57:59 -0400
Nicolas Dufresne <nicolas@ndufresne.ca> escreveu:

> The IPU3 sensor produce a vendor specific form of bayer. If we manage
> to implement support for this format, it would likely be done in
> software. I don't think anyone can answer your other questions has no
> one have ever implemented this, hence measure performance.

Decoding that bayer-like specific format is something that we currently
do at libv4lconvert. IMHO, we should have support for it inside the
library, even if the device is capable of decoding it via a mem2mem
device. If not for usage in "production", being able to read it from
the sensor directly helps to test it. It might also be used on some
applications where there would be no need for 3A algorithms to run
(for example, on industrial cameras where the light and the distance of
the objects don't change).


Cheers,
Mauro

Re: Software-only image processing for Intel "complex" cameras

[Tomasz Figa]

Hi Pavel,

On Thu, Jun 21, 2018 at 5:38 AM Pavel Machek <pavel@ucw.cz> wrote:
>
> Hi!
>
> On Nokia N900, I have similar problems as Intel IPU3 hardware.
>
> Meeting notes say that pure software implementation is not fast
> enough, but that it may be useful for debugging. It would be also
> useful for me on N900, and probably useful for processing "raw" images
> from digital cameras.
>
> There is sensor part, and memory-to-memory part, right? What is
> the format of data from the sensor part? What operations would be
> expensive on the CPU? If we did everthing on the CPU, what would be
> maximum resolution where we could still manage it in real time?

We can still use the memory-to-memory part (IMGU), even without 3A. It
would just do demosaicing at default parameters and give us a YUV
(NV12) frame. We could use some software component to analyze the YUV
output and adjust sensor parameters accordingly. Possibly the part we
already have in libv4l2 could just work?

The expensive operation would be analyzing the frame itself. I suppose
you need to build some histogram representing brightness and white
balance of the frame and then infer necessary sensor adjustments from
that.

>
> Would it be possible to get access to machine with IPU3, or would
> there be someone willing to test libv4l2 patches?

I should be able to help with some basic testing, preferably limited
to command line tools (but I might be able to create a test
environment for X11 tools if really necessary).

Best regards,
Tomasz

Re: Software-only image processing for Intel "complex" cameras

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 2134 bytes --]

Hi!

> > On Nokia N900, I have similar problems as Intel IPU3 hardware.
> >
> > Meeting notes say that pure software implementation is not fast
> > enough, but that it may be useful for debugging. It would be also
> > useful for me on N900, and probably useful for processing "raw" images
> > from digital cameras.
> >
> > There is sensor part, and memory-to-memory part, right? What is
> > the format of data from the sensor part? What operations would be
> > expensive on the CPU? If we did everthing on the CPU, what would be
> > maximum resolution where we could still manage it in real time?
> 
> We can still use the memory-to-memory part (IMGU), even without 3A. It
> would just do demosaicing at default parameters and give us a YUV
> (NV12) frame. We could use some software component to analyze the YUV
> output and adjust sensor parameters accordingly. Possibly the part we
> already have in libv4l2 could just work?

As soon as you get YUV, yes, libv4l2 should be able to work with that.

OTOH using the memory-to-memory part is going to be tricky. What
format is the data before demosaicing? Something common like BGGR10?

> The expensive operation would be analyzing the frame itself. I suppose
> you need to build some histogram representing brightness and white
> balance of the frame and then infer necessary sensor adjustments from
> that.

That does not really have to be expensive. You can sample ... say
10000 pixels from the image, and get good-enough data for 3A.

> > Would it be possible to get access to machine with IPU3, or would
> > there be someone willing to test libv4l2 patches?
> 
> I should be able to help with some basic testing, preferably limited
> to command line tools (but I might be able to create a test
> environment for X11 tools if really necessary).

Could you just compile libv4l2 with sdlcam demo on the target, and
then ssh -X there from some sort of reasonable system?

Best regards,
								Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: Software-only image processing for Intel "complex" cameras

[Tomasz Figa]

Hi Pavel,

On Fri, Jun 29, 2018 at 6:18 PM Pavel Machek <pavel@ucw.cz> wrote:
>
> Hi!
>
> > > On Nokia N900, I have similar problems as Intel IPU3 hardware.
> > >
> > > Meeting notes say that pure software implementation is not fast
> > > enough, but that it may be useful for debugging. It would be also
> > > useful for me on N900, and probably useful for processing "raw" images
> > > from digital cameras.
> > >
> > > There is sensor part, and memory-to-memory part, right? What is
> > > the format of data from the sensor part? What operations would be
> > > expensive on the CPU? If we did everthing on the CPU, what would be
> > > maximum resolution where we could still manage it in real time?
> >
> > We can still use the memory-to-memory part (IMGU), even without 3A. It
> > would just do demosaicing at default parameters and give us a YUV
> > (NV12) frame. We could use some software component to analyze the YUV
> > output and adjust sensor parameters accordingly. Possibly the part we
> > already have in libv4l2 could just work?
>
> As soon as you get YUV, yes, libv4l2 should be able to work with that.
>
> OTOH using the memory-to-memory part is going to be tricky.

Why? I don't see any reason why it would be tricky. You just feed the
right CAPTURE node with YUV buffers and the right OUTPUT node with raw
buffers and that should work.

> What
> format is the data before demosaicing? Something common like BGGR10?

It's documented in detail in V4L2 documentation:
https://www.kernel.org/doc/html/latest/media/uapi/v4l/pixfmt-srggb10-ipu3.html

>
> > The expensive operation would be analyzing the frame itself. I suppose
> > you need to build some histogram representing brightness and white
> > balance of the frame and then infer necessary sensor adjustments from
> > that.
>
> That does not really have to be expensive. You can sample ... say
> 10000 pixels from the image, and get good-enough data for 3A.

Yes, but you need to do it relatively frequently to react for scene
changes and also even 10000 pixels (depending on distribution
) might mean fetching 10000*cacheline bytes of data.

> > > Would it be possible to get access to machine with IPU3, or would
> > > there be someone willing to test libv4l2 patches?
> >
> > I should be able to help with some basic testing, preferably limited
> > to command line tools (but I might be able to create a test
> > environment for X11 tools if really necessary).
>
> Could you just compile libv4l2 with sdlcam demo on the target, and
> then ssh -X there from some sort of reasonable system?

Yes, that should work I guess. That would be a Chrome OS device (which
doesn't include X11), so that would mean compiling some X11 libs (and
probably some more dependencies) as well, but that's not impossible.

Best regards,
Tomasz

Re: Software-only image processing for Intel "complex" cameras

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 3381 bytes --]

Hi!

> > > > On Nokia N900, I have similar problems as Intel IPU3 hardware.
> > > >
> > > > Meeting notes say that pure software implementation is not fast
> > > > enough, but that it may be useful for debugging. It would be also
> > > > useful for me on N900, and probably useful for processing "raw" images
> > > > from digital cameras.
> > > >
> > > > There is sensor part, and memory-to-memory part, right? What is
> > > > the format of data from the sensor part? What operations would be
> > > > expensive on the CPU? If we did everthing on the CPU, what would be
> > > > maximum resolution where we could still manage it in real time?
> > >
> > > We can still use the memory-to-memory part (IMGU), even without 3A. It
> > > would just do demosaicing at default parameters and give us a YUV
> > > (NV12) frame. We could use some software component to analyze the YUV
> > > output and adjust sensor parameters accordingly. Possibly the part we
> > > already have in libv4l2 could just work?
> >
> > As soon as you get YUV, yes, libv4l2 should be able to work with that.
> >
> > OTOH using the memory-to-memory part is going to be tricky.
> 
> Why? I don't see any reason why it would be tricky. You just feed the
> right CAPTURE node with YUV buffers and the right OUTPUT node with raw
> buffers and that should work.

I have seen insides of libv4l2. I believe unpacking by hand will be
simpler / less tricky than passing buffers around.

Of course, long term both will be needed.

> > What
> > format is the data before demosaicing? Something common like BGGR10?
> 
> It's documented in detail in V4L2 documentation:
> > https://www.kernel.org/doc/html/latest/media/uapi/v4l/pixfmt-srggb10-ipu3.html

Ah, thanks for pointer. That's not too bad.

> > > The expensive operation would be analyzing the frame itself. I suppose
> > > you need to build some histogram representing brightness and white
> > > balance of the frame and then infer necessary sensor adjustments from
> > > that.
> >
> > That does not really have to be expensive. You can sample ... say
> > 10000 pixels from the image, and get good-enough data for 3A.
> 
> Yes, but you need to do it relatively frequently to react for scene
> changes and also even 10000 pixels (depending on distribution
> ) might mean fetching 10000*cacheline bytes of data.

Yeah, there are tradeoffs...

> > > > Would it be possible to get access to machine with IPU3, or would
> > > > there be someone willing to test libv4l2 patches?
> > >
> > > I should be able to help with some basic testing, preferably limited
> > > to command line tools (but I might be able to create a test
> > > environment for X11 tools if really necessary).
> >
> > Could you just compile libv4l2 with sdlcam demo on the target, and
> > then ssh -X there from some sort of reasonable system?
> 
> Yes, that should work I guess. That would be a Chrome OS device (which
> doesn't include X11), so that would mean compiling some X11 libs (and
> probably some more dependencies) as well, but that's not impossible.

I believe you want to put Debian chroot in there... or get machine
with IPU3 where you can install Debian directly.
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]