Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Adam Baker]

Responses to both Kilgore and Hans are included below.

On Tuesday 18 November 2008, kilgota@banach.math.auburn.edu wrote:

> Hi,
>
> I am not a specialist on kernel camera drivers, but I do know a few things
> about SQ cameras, having done the support in Gphoto for them. I am quite
> willing to offer some help on this, as the other thing that I am doing
> right now a very slow effort to figure out the decompression algorithm for
> another camera, from another company. Here follow some comments and
> questions, in no particular order:

Thanks for the offer of help - if nothing else it will be useful to have 
someone who has a selection of these cameras available for testing as I've 
only got one myself.

>
> 1. The SQ905C cameras (0x2770:0x905c) are not identical but in fact are
> very similar. The commands are similar, but no single one of them is an
> exact match. It would have been feasible to combine the two drivers in
> libgphoto2, but it would have been messy. I think that to keep them
> separate was the right decision for that reason. But there is a lot of
> similarity, so it seems to me that it is certainly a better idea to
> produce video drivers for both SQ905 and SQ905C at the same time. Or,
> perhaps since the list of commands used for video is much smaller, then
> perhaps they could be combined after all.

I think it might be best to produce one driver first and then see if it makes 
more sense to add sq905c or to use the sq905 driver as a template for an 
sq905c one. Not having access to an sq905c camera I'm not volunteering for 
that phase though.

>
> 3. Even so, you should both take seriously the question whether these
> cameras actually need a kernel driver, or whether it would actually be
> useful. Perhaps it would be a better idea to do something totally in
> userspace and figure out how to hook that into a frontend program like
> xawtv or some other program which is associated with V4L instead of
> getting input from a kernel-supported device? The reasons I say this are:
>
> a. There is IMHO a too big proliferation of kernel modules especially in
> this area of video. It seems every new device that comes along needs its
> own separate module. It would be nice if it is possible to slow down on
> this. In general, I think it is a *good idea* if more is done in userspace
> and as little as is needed is done in the kernel.

I've been thinking about that for quite a while now. The problem is that 
applications are written to use V4L2 and a V4L2 camera has a device node. I 
did consider for a while the idea of creating a kernel space driver that just 
exposed an interface that a user space libusb app could provide video data to 
but concluded it seemed like a lot of work for little benefit.

>
> b. As I understand, the reason for needing a kernel driver is because of
> speed issues. Partly correlated with this, the libusb support for
> isochronous data transmission (used by most webcams) has been missing.
> There are attempts, I understand, to add that. But I have not heard it is
> done. However, the question of isochronous transport is not relevant here,
> at all.

>From my PoV the main reason for wanting a kernel driver is so that a V4L2 
aware app can find the device. I don't expect performance to be a big issue, 
especially as this is only a USB 1.1 device

>
> 4. As to kernel support, there is the further problem that the camera does
> not appear visibly different to the computer if it is in still mode or in
> webcam mode. Some devices, when they change from one mode to another, also
> change the USB Product number or give some other signal that they are now
> in a different mode. But, no, these do not do any of that. It is just a
> question of what command is given _to_ the camera. If you give a certain
> sequence of bytes to the camera, it trips the shutter and the camera then
> expects you to download the frame it just shot. That's it. So the problem
> is now which will grab the camera when it is plugged in -- libgphoto2 or
> the kernel module? If the kernel module grabs the camera, then libgphoto2
> is userspace and will lose. So then the user who was intending to download
> all the snapshots on the camera, swears under his breath and has to unplug
> the camera, become root, remove the module, blacklist the module, become
> user, replug the camera, and then get the photos off. Or, if the module is
> blacklisted by default, then the user plugs in the camera, runs a video
> capture/display program, gets nothing but an error message saying there is
> no device, swears under his breath, remembers that he has to do "su
> modprobe xxxx" and only then can run the capture program. It is far the
> best to think of ways to avoid nonsense situations like this before they
> happen to users.

Solving that problem was one of my blocking points for pushing a module into 
mainline. Thankfully the libgphoto2 team have near enough solved it. Current 
operation is that when the camera is plugged the kernel driver takes control 
of it. If you then start a libgphoto based app it will attempt to claim the 
device and as long as the driver isn't in use it should succeed. The only bit 
that doesn't work perfectly is that it won't revert to webcam mode after you 
stop the libgphoto app, you need to unplug the camera and plug it back in to 
use it as a webcam again. As far as I can tell that is a shortcoming of 
libusb for which a fix is being considered.

>
> 5. As to the command which is required to get video out of one of these
> cameras, the command is already in use in both of the camera support
> libraries in libgphoto2. It is used there for taking a single snapshot,
> with the command option gphoto2 --capture-preview. So all that is needed
> is to be able to run this command repeatedly, process the data each time
> into a frame, and have some method for displaying the sequence of frames,
> instead of doing these things just once and saving the captured frame, as
> it is done in Gphoto.

The necessary commands are also working in the current out of kernel V4L 
driver (based I believe on some of your earlier work) so I'm not too worried 
on that front.

>
> Needless to say, even though I have some reservations about kernel
> support, as presented in items 3 and 4, I offer my cooperation. I do
> believe, however that the points under item 3 should be thought about
> seriously.

I hope my comments above have at least reduced your reservations

> A few further comments are interspersed below, which address some specific
> points.
>
> Theodore Kilgore
>
> > Message: 7
> > Date: Mon, 17 Nov 2008 22:53:33 +0000
> > From: Adam Baker <linux@baker-net.org.uk>
> > Subject: [sqcam-devel] Advice wanted on producing an in kernel sq905
> > 	driver
> > To: video4linux-list@redhat.com
> > Cc: sqcam-devel@lists.sourceforge.net
> > Message-ID: <200811172253.33396.linux@baker-net.org.uk>
> > Content-Type: text/plain;  charset="us-ascii"
> >
> > Hi V4L readers,
> >
> > There currently exists an out of kernel driver for the SQ technologies
> > sq905 USB webcam chipset used by a number of low cost webcams. This
> > driver has a number of issues which would count against it being included
> > in kernel as is but I'm considering trying to create something that could
> > be suitable. I have a number of questions on how best to proceed.
>
> Please do refer to item 5 above. That is a problem which needs to be
> solved.

Did you mean 4, not 5?

> >
> > First off some thoughts on how I'd like to proceed.
> >
> > The chip exposes the Bayer sensor output directly to the driver so the
> > current driver uses code borrowed from libgphoto2 to do the Bayer decode
> > in kernel. Obviously this is wrong and now we have libv4l it should use
> > that instead.
>
> Ah, Bayer demosaicing. I wonder if my new demosaicing algorithm for
> libgphoto2 could run fast enough to work for a webcam, or at least for a
> slow webcam. It does give much better output for still shots.
>

The out of kernel code has just copy and pasted from libgphoto2 for the Bayer 
demosaicing but if I rely on libv4l this becomes Hans de Goede's problem to 
select the best algorithm, not mine.

> >
> > 3) The current driver needs to do some up/down and left/right flips of
> > the data to get it in the right order to do the Bayer decode that depend
> > on the version info the camera returns to its init sequence.
>
> That is correct. If you don't know which camera it is, then you cannot
> correctly process the images because of these orientation issues. The
> Bayer tiling scheme also is dependent on the model version number, so if
> this is not accounted for then the colors will be wrong as well as the
> orientation.
>
> Should that code remain
>
> > in the driver and if not how should the driver communicate what is
> > needed.
>
> My two cents is that you have to read the version number and make the
> appropriate use of that information. There is no other way to communicate
> what is needed.
>

I clearly have to read the version and either flip the data myself or inform 
libv4l that it needs to be flipped. The do as much work in userspace as 
possible argument says I should just provide an indicator that it needs 
flipping but I don't know how to do that.


> >> First off some thoughts on how I'd like to proceed.
> >>
> >> The chip exposes the Bayer sensor output directly to the driver so the
> >> current driver uses code borrowed from libgphoto2 to do the Bayer decode
> >> in kernel. Obviously this is wrong and now we have libv4l it should use
> >> that instead.
> >
> > Correct, there is nothing special you need to do for that, just pass
> > frames with the raw bayer data to userspace and set the pixelformat to
> > one of: V4L2_PIX_FMT_SBGGR8 /* v4l2_fourcc('B', 'A', '8', '1'), 8 bit
> > BGBG.. GRGR.. */ V4L2_PIX_FMT_SGBRG8 /* v4l2_fourcc('G', 'B', 'R', 'G'),
> > 8 bit GBGB.. RGRG.. */ V4L2_PIX_FMT_SGRBG8 /*
> > v4l2_fourcc('G','R','B','G'), 8 bit GRGR.. BGBG.. */ V4L2_PIX_FMT_SRGGB8
> > /* v4l2_fourcc('R','G','G','B'), 8 bit RGRG.. GBGB.. */
> >
> > Note the last 2 currently are only defined internally in libv4l and not
> > in linux/videodev2.h as no drivers use them yet, but if you need one of
> > them adding it to linux/videodev2.h is fine.

The different known variants of this camera use either BGGR or GBRG so I seem 
to be off the hook there.

> >
> > And then make sure your applications are either patched to use libv4l, or
> > use the LD_PRELOAD libc wrapper (see libv4l docs).
> >

I've been testing with LD_PRELOAD but discovered it was rejecting the camera 
because the current driver doesn't support mmap for V4L2 so doesn't set the 
streaming flag.

> >> Now my questions
> >>
> >> 1) What kernel should I base any patches I produce on? The obvious
> >> choice would seem to be
> >> git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-2.6.git but
> >> it seems as if Linus's 2.6 tree is more up to date for gspca.
> >
> > Please use:
> > http://linuxtv.org/hg/~jfrancois/gspca/

OK

> >
> > Note that like al v4l-dvb trees this is a standalone tree, not a complete
> > kernel, assuming you've got the headers installed for your current kernel
> > you can just do make menuconfig, make, make install from this tree (from
> > the main dir, not from the linux subdir) to update your current kernels
> > video4linux modules to the latest, then add your own driver and rince
> > repeat :)
> >
> >> 2) The existing driver needed to perform a gamma adjustment after
> >> performing the Bayer decode. I couldn't find anything in libv4l that
> >> obviously did that so I'm guessing it isn't needed for existing devices
> >> - should that be added to libv4l if needed and if so how should the
> >> driver indicate it is needed - could it be indicated with a new value
> >> for v4l2_colorspace?
> >
> > Hmm interesting for now lets ignore this and first get it up and running
> > without this, and the revisit this. I'm asking for this because gamma
> > correction might be interesting for other cams too, so I would like to
> > see a generic solution for this, which will take some design work, etc.

OK, I wondered if anyone else might be able to make use of it. The 
implementation in the current driver is just a single fixed correction. There 
is a version at http://philippe.corbes.free.fr/sqcam/ in which someone has 
implemented 8 gamma tables selected by VIDIOCSPICT (that driver is pure V4L1) 
but libv4l is allowed to use floating point so could calculate the table on 
the fly upon receiving an ioctl. This doesn't sound too hard to add to libv4l 
if needed - the only problem is in selecting an appropriate initial value as 
that may be camera dependent.

> >
> >> 3) The current driver needs to do some up/down and left/right flips of
> >> the data to get it in the right order to do the Bayer decode that depend
> >> on the version info the camera returns to its init sequence. Should that
> >> code remain in the driver and if not how should the driver communicate
> >> what is needed.
> >
> > It should communicate what is needed all possible bayer orders are
> > covered by the 4 formats I gave above, and libv4l knows how to handle all
> > 4 of them, for the defines of the last 2 see
> > libv4lconvert/libv4lconvert-priv.h

This isn't just a question of Bayer selection, the image will be upside down 
and possibly mirror imaged too. The question is mainly one of how can the 
driver communicate such info to libv4l.

> >
> >> 4) The current driver doesn't support streaming mode for V4L2 (only
> >> V4L1) and libv4l doesn't support cameras that don't support streaming so
> >> is there an easy way to test out if the camera output will work well
> >> with libv4l before starting work?
> >
> > This should not be a problem. gspca now can work with both bulk and
> > isochronyous usb transfers using cams, although your cam most likely is
> > an isoc one (bulk mode cams are rare).
>
> It may be rare, but this is one of those rare specimens. These cameras do
> not even have isochronous endpoints so there is no option to use what does
> not even exist.

Correct.

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Antonio Ospite]

[-- Attachment #1.1: Type: text/plain, Size: 2068 bytes --]

On Wed, 19 Nov 2008 00:20:15 +0000
Adam Baker <linux@baker-net.org.uk> wrote:

> On Tuesday 18 November 2008, kilgota@banach.math.auburn.edu wrote:
> 
> >
> > 3. Even so, you should both take seriously the question whether these
> > cameras actually need a kernel driver, or whether it would actually be
> > useful. Perhaps it would be a better idea to do something totally in
> > userspace and figure out how to hook that into a frontend program like
> > xawtv or some other program which is associated with V4L instead of
> > getting input from a kernel-supported device? The reasons I say this are:
> >
> > a. There is IMHO a too big proliferation of kernel modules especially in
> > this area of video. It seems every new device that comes along needs its
> > own separate module. It would be nice if it is possible to slow down on
> > this. In general, I think it is a *good idea* if more is done in userspace
> > and as little as is needed is done in the kernel.
> 
> I've been thinking about that for quite a while now. The problem is that 
> applications are written to use V4L2 and a V4L2 camera has a device node. I 
> did consider for a while the idea of creating a kernel space driver that just 
> exposed an interface that a user space libusb app could provide video data to 
> but concluded it seemed like a lot of work for little benefit.
> 

There were some v4l1 solutions for that already, but nothing that can
go to linux mainline without rewriting. See:
http://www.lavrsen.dk/twiki/bin/view/Motion/VideoFourLinuxLoopbackDevice

There is also a v4l2 solution, but I've never tried it:
http://sourceforge.net/projects/v4l2vd/

Having such a virtual device in mainline would be very good.

Regards,
   Antonio Ospite

-- 
A: Because it messes up the order in which people normally read text.
Q: Why is top-posting such a bad thing?
A: Top-posting.
Q: What is the most annoying thing in e-mail?

  Web site: http://www.studenti.unina.it/~ospite
Public key: http://www.studenti.unina.it/~ospite/aopubkey.asc

[-- Attachment #1.2: Type: application/pgp-signature, Size: 197 bytes --]

[-- Attachment #2: Type: text/plain, Size: 164 bytes --]

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Hans de Goede]

Adam Baker wrote:

<huge snip>

>>> First off some thoughts on how I'd like to proceed.
>>>
>>> The chip exposes the Bayer sensor output directly to the driver so the
>>> current driver uses code borrowed from libgphoto2 to do the Bayer decode
>>> in kernel. Obviously this is wrong and now we have libv4l it should use
>>> that instead.
>> Ah, Bayer demosaicing. I wonder if my new demosaicing algorithm for
>> libgphoto2 could run fast enough to work for a webcam, or at least for a
>> slow webcam. It does give much better output for still shots.
>>
> 
> The out of kernel code has just copy and pasted from libgphoto2 for the Bayer 
> demosaicing but if I rely on libv4l this becomes Hans de Goede's problem to 
> select the best algorithm, not mine.
> 

libv4l currently uses linear interpolated demosaicing, the 4 (or 2) surrounding 
pixels with the missing color components are taken and then straight forward 
averaged. At the borders only those pixels which are actually present get used 
ofcourse.

This works pretty good and is reasonable fast, I don't want to add a zillion 
demosaicing algorithms to libv4l and keep in mind that webcams which produce 
raw bayer in general are real cheap and have matching have poor image quality, 
so using some highly advanced bayer algorithm is not going to help. Also keep 
in mind some demosaicing algorithms are patented.

I'm always open for convincing examples of differences in algorithms, esp. when 
backed up with a proposed patch and benchmarks showing the additional costs.

>>> 3) The current driver needs to do some up/down and left/right flips of
>>> the data to get it in the right order to do the Bayer decode that depend
>>> on the version info the camera returns to its init sequence.
>> That is correct. If you don't know which camera it is, then you cannot
>> correctly process the images because of these orientation issues. The
>> Bayer tiling scheme also is dependent on the model version number, so if
>> this is not accounted for then the colors will be wrong as well as the
>> orientation.
>>
>> Should that code remain
>>
>>> in the driver and if not how should the driver communicate what is
>>> needed.
>> My two cents is that you have to read the version number and make the
>> appropriate use of that information. There is no other way to communicate
>> what is needed.
>>
> 
> I clearly have to read the version and either flip the data myself or inform 
> libv4l that it needs to be flipped. The do as much work in userspace as 
> possible argument says I should just provide an indicator that it needs 
> flipping but I don't know how to do that.
> 

Ah interesting, 2 things:

1) Do not do the flipping in the kernel libv4l already has all the necessary
    code.

2) Currently there is no API for telling libv4l to do the flipping, but I think
    we should design one. Currently for the few cams which need software
    flipping (or rather software rotate 180 as that is what libv4l does), are
    detected by libv4l by doing string comparisons on the v4l2_capability struct
    card string. We could cheat and let the sq905 driver put something special
    in there for cams with upside down mounted sensors, but given that this is a
    re-occuring problem, defining a proper API for this would be good I think.

<snip>

>>> And then make sure your applications are either patched to use libv4l, or
>>> use the LD_PRELOAD libc wrapper (see libv4l docs).
>>>
> 
> I've been testing with LD_PRELOAD but discovered it was rejecting the camera 
> because the current driver doesn't support mmap for V4L2 so doesn't set the 
> streaming flag.
> 

Correct libv4l only works with v4l2 devices which do mmap (and will happily 
emulate read() access on top of that).


>>>> 2) The existing driver needed to perform a gamma adjustment after
>>>> performing the Bayer decode. I couldn't find anything in libv4l that
>>>> obviously did that so I'm guessing it isn't needed for existing devices
>>>> - should that be added to libv4l if needed and if so how should the
>>>> driver indicate it is needed - could it be indicated with a new value
>>>> for v4l2_colorspace?
>>> Hmm interesting for now lets ignore this and first get it up and running
>>> without this, and the revisit this. I'm asking for this because gamma
>>> correction might be interesting for other cams too, so I would like to
>>> see a generic solution for this, which will take some design work, etc.
> 
> OK, I wondered if anyone else might be able to make use of it. The 
> implementation in the current driver is just a single fixed correction. There 
> is a version at http://philippe.corbes.free.fr/sqcam/ in which someone has 
> implemented 8 gamma tables selected by VIDIOCSPICT (that driver is pure V4L1) 
> but libv4l is allowed to use floating point so could calculate the table on 
> the fly upon receiving an ioctl. This doesn't sound too hard to add to libv4l 
> if needed - the only problem is in selecting an appropriate initial value as 
> that may be camera dependent.
> 

Yes, what we are seeing with libv4l and webcams is that there are a lot of 
things, which libv4l needs to know about the cam and the current API doesn't 
give us, like:
-does this cam need software whitebalance
-does this cam need software auto exposure
-does this cam need gamma correction, and what initial gamma to use
-if the sensor is mounted upside down, and the hardware cannot flip the image
  itself

And this are just a few examples. I've been thinking about ways to use the 
existing API for some of these, but others are more trouble some to fit into 
the existing API.

I will send an RFC for this problem to video4linux-list@redhat.com, please 
comment on it :)

>>>> 3) The current driver needs to do some up/down and left/right flips of
>>>> the data to get it in the right order to do the Bayer decode that depend
>>>> on the version info the camera returns to its init sequence. Should that
>>>> code remain in the driver and if not how should the driver communicate
>>>> what is needed.
>>> It should communicate what is needed all possible bayer orders are
>>> covered by the 4 formats I gave above, and libv4l knows how to handle all
>>> 4 of them, for the defines of the last 2 see
>>> libv4lconvert/libv4lconvert-priv.h
> 
> This isn't just a question of Bayer selection, the image will be upside down 
> and possibly mirror imaged too. The question is mainly one of how can the 
> driver communicate such info to libv4l.
> 

Ack.

Regards,

Hans

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:

<snip>

> 
>> libv4l currently uses linear interpolated demosaicing, the 4 (or 2) 
>> surrounding pixels with the missing color components are taken and 
>> then straight forward averaged. At the borders only those pixels which 
>> are actually present get used ofcourse.
> 
> Perhaps you ought at least to use what is in gp_bayer_accrue() as well 
> (see libgphoto2/libgphoto2/bayer.c) because it cuts down on the zipper 
> effect.
> 

Interesting, although I must say the libgphoto code certainly is not written 
with speed in mind, but we can fix that. Still doing something like 
gp_bayer_accrue() will cause a significant additional CPU load, please keep in 
mind that:
1) We often also need to decompress the bayer data first using some form of
    Huffman decompression as usb11 bandwidth is not enough for raw bayer
2) Then we have 352x288 (luckily all raw bayer cams seem to be this low) at
    30 fps
3) Which we then need to demosaic
4) And do white balance *
5) And normalize *
6) And gamma correct *
7) And flip / rotate the image
8) And the app may want to do stuff with it like display it, which might
    involve pixelformat conversion, and / or software scaling
9) And if the app wants to record broadcast, the app often also needs to do
    some form of framerate modification to get a steady framerate (webcam
    framerate is exposure setting dependend)
10) And the app may do even more such as HG264 compression for video
    conferencing
11) And this may all be part of the user just having a chat window open, while
    he is mostly using his PC for something else.

*) Not yet implemented

So all in all we need to be carefull with CPU usage here.


>> I'm always open for convincing examples of differences in algorithms, 
>> esp. when backed up with a proposed patch and benchmarks showing the 
>> additional costs.
> 
> The algorithm in libgphoto2/libgphoto2/ahd_bayer.c requires 3X the time 
> which is required for the old algorithm in bayer.c

Ugh, and I'm quite sure the algorithm in bayer.c is much much slower then what 
is in libv4l, not only because of gp_bayer_accrue(), but because it is not 
written very efficient in general IMHO (lots of if's instead of seperate 
methods for different cases like borders, different bayer orders).

> . For still photos 
> with some rather low quality cameras, it gives much better results. I 
> have written a paper about this algorithm, and also some demonstration 
> photos from some different cameras, and you can find the paper and the 
> demonstration photos at www.auburn.edu/~kilgota. As to whether the speed 
> hit of 3X is crucial for a webcam, I am not sure.

Nor am I, but I think we should be careful with not causing too much cpuload, 
also think of things like netbooks, which do not come with a very powerful cpu.


<snip>

>>> I clearly have to read the version and either flip the data myself or 
>>> inform libv4l that it needs to be flipped. The do as much work in 
>>> userspace as possible argument says I should just provide an 
>>> indicator that it needs flipping but I don't know how to do that.
>>>
>>
>> Ah interesting, 2 things:
>>
>> 1) Do not do the flipping in the kernel libv4l already has all the 
>> necessary
>>   code.
>>
>> 2) Currently there is no API for telling libv4l to do the flipping, 
>> but I think
>>   we should design one. Currently for the few cams which need software
>>   flipping (or rather software rotate 180 as that is what libv4l 
>> does), are
>>   detected by libv4l by doing string comparisons on the 
>> v4l2_capability struct
>>   card string. We could cheat and let the sq905 driver put something 
>> special
>>   in there for cams with upside down mounted sensors, but given that 
>> this is a
>>   re-occuring problem, defining a proper API for this would be good I 
>> think.
> 
> 
> Well, for these particular cameras the problem is twofold. First, the 
> image is always upside down (and the way to fix that is with a byte 
> reversal of the raw image). Second, it is also true for *some* of them 
> that the image has reversed left and right. So what you have to do is 
> read the firmware, know which firmware number corresponds to what, and 
> then have a standard way to tell the app precisely what it should do. 
> Something like a variable called "orientation" and then the appropriate 
> one of the bitmap values 00, 01, 10, or 11, or, in ordinary numbers
> 
> 0    oriented correctly, no action needed
> 1    reversal of byte string needed (giving 180 degree rotation)
> 2    reverse bytes on lines (i. e. do de-mirroring)
> 3    do both 1 and 2 (notice that 3 = 1|2)
> 

Erm, 3 just boils down to only doing vflip, rotate 180 == vflip + hflip,
so rotate180 + hflip == vflip + hflip + hflip == vflip

Regards,

Hans

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:
> 

<snip>

>>>
>>> 0    oriented correctly, no action needed
>>> 1    reversal of byte string needed (giving 180 degree rotation)
>>> 2    reverse bytes on lines (i. e. do de-mirroring)
>>> 3    do both 1 and 2 (notice that 3 = 1|2)
>>>
>>
>> Erm, 3 just boils down to only doing vflip, rotate 180 == vflip + hflip,
>> so rotate180 + hflip == vflip + hflip + hflip == vflip
> 
> Yeah, right. In a still photo processing it does not matter, of course, 
> so I was not paying attention to such things. Well, then, instead of 
> using 0, 1, 2, and 3 one could use 0, 1, 2 and so on, in which
>     0    do nothing
>     1    vflip
>     2    hflip
>     3    rotate 180    (1|2)
>     4    apply decompression
>     5    4|1
>     6    4|2
>     7    4|3
>     upper nibble gives Bayer tiling code of 0 thru 7.
>     If upper nibble is 8 or more, this could signify one of several
>     ways that the data could have already been processed, such as if
>     a camera emits finished JPEG frames. Or, this could have been
>     done, too, with using 8 or more in the lower nibble.
> 
> So one way to do this kind of thing might be to create in the kernel 
> module a bitmapped char variable, similar to the above, and send it out 
> to the app, during initialization, and the app is supposed to understand 
> it.
> 

Eh, no you are confusing different things now and stuffing them all into 1 byte.

We already have a pixelformat integer in the API, which can contain things like 
"this is raw bayer, first line rgrg second line gbgb" or like:
"this is bayer compressed by the spca561 bridge, compression level 1" or like:
"this is jpeg data without a quantization table", etc.

We call all this pixelformat, a pixelformat should contain all info to get from 
the raw data to say plain rgb. The image may then still be upside down / 
mirrored / whatever.

Currently we have no clear API to allow a driver to tell userspace that it is 
sending out upside down images, I've send a proposal for this to the 
video4linux-list@redhat.com

> OK, so I will look again at the Bayer algorithm and see if my suspicion 
> is correct, that it is possible to come out with something which is both 
> faster than the ahd_bayer.c code and gives better results than straight 
> bilinear interpolation, which can give such ugly artifacting. Probably, 
> the most effective use of my time right now is to work on that, because 
> I am pretty familiar with the code that I wrote.
> 
> As far as the rest of things are concerned, it would probably help if I 
> know where to go and look for your existing code about this stuff.

Easiest way to get my latest code is here:
http://people.atrpms.net/~hdegoede/libv4l-0.5.5.tar.gz

You are looking for libv4lconvert/bayer.c

As you can see I've got separate code to render the borders and to render the 
center. I think it would be interesting to use an optimized version of your 
gp_bayer_accrue() for the center rendering. Optimalizations I'm thinking about are:
1) no need to check for having all 4 pixels, you always do
2) just pass in the 4 pixel values instead of passing in coordinates
3) make separate green and blue_red versions.

Maybe also only do the horizontal / vertical line detection for green without 
the fallback to the red_blue code when no line is detected (in that case just 
return the avg of all 4).

Regards,

Hans

p.s.

If you are experienced with reverse engineering decompression algorithms 9for 
raw bayer), I've got multiple issues where you could help me.

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:
> 

<snip>

> 
>> Easiest way to get my latest code is here:
>> http://people.atrpms.net/~hdegoede/libv4l-0.5.5.tar.gz
>>
>> You are looking for libv4lconvert/bayer.c
> 
> Thanks. I got it. By your leave, the first thing I will do is to try to 
> speed up my own code. There are, I think, some real possibilities to do 
> that. However, without looking at any details of what you did in your 
> bayer.c, I will ask a question about organization of the way you did 
> things. What happens in libgphoto2 is, first a function is run which 
> "spreads out" the (uncompressed) raw data to a block of data of size 
> 3*w*h, so that each byte of actual data is in its natural place in the 
> finished image and the rest of the locations in the finished image are 
> left filled with 00 bytes. Then after this what the algorithm does is to 
> work on this copy of the still-not-finished image and to finish it, 
> entering each piece of computed or estimated data in its proper 
> location. There might be other ways to go about the entire project of 
> doing the Bayer demosaicing. Therefore, I ask whether you have done 
> things the same way, or did you take one of those other possible routes 
> to the final result? For example, in
> 
> static void bayer_to_rgbbgr24(const unsigned char *bayer,
>   unsigned char *bgr, int width, int height, unsigned int pixfmt,
>         int start_with_green, int blue_line)
> 
> (which looks like one of the relevant functions) is bgr of size w*h, 
> (size of uncompressed input) still, or of size 3*w*h (size of 
> interpolated RGB output)? I could figure out all of this stuff in 3 
> days, but probably not in 5 minutes.
> 

bayer is the raw bayer input here, and that is ofcourse w*h, bgr is w*3*h, and 
when this function is done contains complete rgb triplets for all pixels. This 
function does the "spread out" and the interpolating in one pass.

<snip>

>> p.s.
>>
>> If you are experienced with reverse engineering decompression 
>> algorithms 9for raw bayer), I've got multiple issues where you could 
>> help me.
>>
> 
> Haha. I was about to ask you the same. On my end, this is *the* problem. 

Hehe

<snip>

> As to the cameras I mentioned before, and who figured out the 
> decompression:
> 
> SQ905 and 905C     myself (took several years to get it all done right)
> 
> SN9C2028     macam project, with some improvements by myself after they 
> did the hard work. The code from the macam project was based, in turn, 
> on the work of Bertrik Sikkens on the SN9C10X cameras. The two 
> algorithms are similar but not identical.
> 

Interesting, so you are familiar with the sn9c10x bayer compression, as you 
then may remeber it uses differential compression with the following huffman codes:
0
100
101
1101
1111
11001
110000
1110xxxx /* direct set */

But sometimes it also sends:
110001xx

And we have no idea what this code means, does the sn9c2028 also have something 
similar and have you figured it out?

> MR97310 Bertrik figured out some of the basic stuff but it was never 
> right, until I was finally (after years) able to figure out what was 
> wrong. Again this is a differential Huffman encoding which is similar to 
> the one the Sonix cameras are using, but again different in details.
> 
> You want quick, go and ask Bertrik. He is, in my experience, either very 
> clever or very lucky.
> 

Ok, I'll ask him.

Thanks & Regards,

Hans

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:
> 
> Hans,
> 
> Two topics.
> 
> First one:
> I tried out some simplification and shortening of the code in 
> ahd_bayer.c and I managed to cut down on the time a bit. The time for it 
> to run on a 320x240 image (from a Sonix SN9C2028) that I keep around for 
> testing is
> 
> Total time of AHD :     40 ms (time before doing the algorithm changes)
> 
> Total time of AHD :     20 ms (after doing the algorithm simplification)
> 

Uhg, with 352x288 @ 30 fps (and yes some cams can do 30 fps at 352x288) this 
translates to 792 ms of each second being used just for the demosaic phase.

> Total time of bilin :     10 ms (done with the old algorithm that Lutz 
> M. wrote for libgphoto2 before the "accrue" edge detection was added)
> 
> Total time of accrue :  10 ms  (done with the "accrue" edge detection 
> which is found at present in ahd_bayer.c)
> 

Hmm, still not too good, using an spca561 based cam, which sends compressed 
bayer, displaying 352x288 @ 30 fps, takes 35% CPU load, with my CPU running at 
1Ghz. Note this includes everything, decompression, demosaic and actually 
displaying the pictures.

Could you drop in the bilinear code from libv4l bayer.c and see how that 
performs on your file ?

> What was timed is the interpolation function only;

So these times do not include the spread out ? Hmm.

> Relative image quality of the methods:
> 
> 1. straight bilinear interpolation is lousy on this image. Lots of 
> zippering.
> 

Ok, then maybe we do need to get something better.

> 2. accrue is not much better. Cures zippering but leaves stripes of 
> color artifacting beside vertical or horizontal edges.
> 

Are those "stripes of color artifacting" also present with plain bilinear

> 3. Very simplified AHD (choice algorithm at each pixel tossed, only an 
> average of the vertical and horizontal approximation used at each color 
> in each pixel). Zippering, again, but the rest of the image leaves a 
> much better impression.
> 
> 4. The simplified AHD which is found in ahd_bayer.c (joint work with 
> Eero Salminen at H.U.T.) gives of course far the best quality, but 
> probably it is just too slow for webcam type of stuff.
> 
> It would probably be possible further to streamline the code for the 
> very simplified AHD and then I suspect it would run a little bit faster. 

It might just be that your system is slow, but if it turns out that bayer.c 
from libv4l is much faster then the bilinear code from libgphoto2, then IMHO 
going to AHD is speedwise not a good idea.

> Second item:
> 
> I notice that you intend for the kernel modules which support webcams to 
> do only basic stuff, and then to pass everything over to userspace. 
> Specifically, you intend to pass everything over to libv4l2, and various 
> apps for doing stuff with webcam output are then envisioned to sit on 
> top of libv4l2 and use it. Therefore some dumb, or, for all I know, not 
> so dumb questions:
> 
> 1. What is to prevent the kernel module from interacting with (for 
> example) both libv4l2 and libusb? Why could not the kernel module (for 
> example) pass isochronous stuff through libv4l2 and bulk stuff through 
> libusb?
> 

Erm, when you have a kernel module there is no need to use libusb, the kernel 
has its own functions for usb.

Basicly whenever we have a video input device, we want to have a kernel driver, 
so as to present a standard /dev/videoX device to userspace, so that all apps 
written for v4l can use the device.

However we do not want to do video format conversion (let alone decompression) 
in kernel space, so if the data is in an exotic format we simply indicate this 
in the pixelformat member of the exchanged structs, and let userspace deal with it.

This is where libv4l comes in into play, libv4l is a convenience lib for 
applications, which can do conversion for them so that the app writers do not 
have to add support for each exotic webcam format themselves.

Currently we are planning on improving the format conversion by adding 
additional steps like whitebalance and normalizing.

> 2. What prevents libusb, in turn, from interacting with a kernel module. 
> Or, if something does prevent it, then what is preventing something like 
> libgphoto2 from interacting with the kernel module?
> 

libusb does not interact with a kernel module for a specific device, it 
interacts directly with the device through the usb subsystem, here in lies a 
problem for the sq905 as only an in kernel driver, or libusb can access the 
device at a time. I've understood from others in this thread there are 
solutions for this, but I'm not familiar with them.

Regards,

Hans

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:
> 
<snip>

>> Uhg, with 352x288 @ 30 fps (and yes some cams can do 30 fps at 
>> 352x288) this translates to 792 ms of each second being used just for 
>> the demosaic phase.
> 
> Yes, I know that some cameras can do 30 fps. I would also assume there 
> are cameras which can do much more than that. So one needs to be more 
> clever.

Well assuming 352x288 @ 30 fps for cams *which send raw bayer* seems like a 
good assumption, anything higher resolution usually uses jpg.

<snip>

>>
>>> Relative image quality of the methods:
>>>
>>> 1. straight bilinear interpolation is lousy on this image. Lots of 
>>> zippering.
>>>
>>
>> Ok, then maybe we do need to get something better.
> 
> If you want to see the results of this on the same photo that I was 
> using for the test, then you should go to my web page 
> www.auburn.edu/~kilgota and look at the comparison photos and also have 
> a look at the article which has done comparisons of the various methods 
> on the same photos, too.
> 

Ok, I went and looked at the article, I do see there is a huge difference 
between plain bi-linear and bi-linear with acrue edge detection. I must say the 
difference between bi-linear with acrue edge detection, and ahd is less 
convincing though.

>> It might just be that your system is slow, but if it turns out that 
>> bayer.c from libv4l is much faster then the bilinear code from 
>> libgphoto2, then IMHO going to AHD is speedwise not a good idea.
> 
> My system is an AMD Sempron 2600+ or 2800+ (forget which, and I am at 
> work, not at home) with 1 M of RAM. Not the fastest thing around, but 
> not slow at all, actually. We will just have to see what the libv4l code 
> does. I don't know yet.

Ok, that should actually be pretty comparable to one of the 2 cores of my dual 
core amd64 when clocked at 1Ghz (the default setting when not more is required, 
yeah for speedstep). So I think that libv4l's bilinear is much faster than, but 
that is not strange as it was written with speed in mind.


> Where I am coming from is, libgphoto2 uses libusb. When a kernel module 
> has "taken over" the device, then as things currently stand, or have 
> stood until recently, libgphoto2 has no access to the device unless and 
> until the kernel module has been rmmod-ed. I understand that there is a 
> partial solution for this. I am trying to figure out how there is a 
> complete solution which would make everyone happy, including users who 
> just want to plug in their cameras.
> 

I understand, but I'm afraid I don't have any answers here, I'm not familiar 
enough with the relevant usb stuff.

>> Basicly whenever we have a video input device, we want to have a 
>> kernel driver, so as to present a standard /dev/videoX device to 
>> userspace, so that all apps written for v4l can use the device.
> 
> Yes, that is quite clear. But insofar as the act of loading the kernel 
> module which creates that device causes the still camera functionality 
> of the same camera to be simultaneously inaccessible to the usual apps 
> which deal with stillcam mode stuff, we have a problem. The worst aspect 
> of the problem is, of course, that we have a bunch of people whom we 
> would like to convince to use our nice operating system, and they do not 
> understand why there has to be a problem because they do not face that 
> kind of problem if they are not using Linux.
> 

All I can say is: again I understand.

>>
>> However we do not want to do video format conversion (let alone 
>> decompression) in kernel space, so if the data is in an exotic format 
>> we simply indicate this in the pixelformat member of the exchanged 
>> structs, and let userspace deal with it.
>>
>> This is where libv4l comes in into play, libv4l is a convenience lib 
>> for applications, which can do conversion for them so that the app 
>> writers do not have to add support for each exotic webcam format 
>> themselves.
> 
> This is obviously a good idea. I can't imagine that anyone would have a 
> problem with it. Certainly, I don't. But my questions were about how to 
> address the userspace-kernelspace conflict which takes place when we try 
> to support dual-purpose devices, in which one of the modes requires 
> kernel support and the other mode does not.
> 

All I can say is: again I understand.

>> libusb does not interact with a kernel module for a specific device, 
>> it interacts directly with the device through the usb subsystem,
> 
> Indeed. The two lines above capture precisely what needs to be 
> addressed. The current situation certainly is described by 'does not.' I 
> am well aware of the 'does not.' Otherwise, I probably would not ask 
> 'why not' or, more precisely, 'what prevents.' So, back to the original 
> problem:
> 
> For a dual-mode device such as most of the cameras that I support in 
> libgphoto2 in stillcam mode, and you are trying to support in webcam 
> mode, this 'does not' causes nasty problems for the user, doesn't it?

Agreed.

> So 
> we should not ignore such difficulties but try to figure out how to 
> avoid them or work around them.

Agreed.

> Thus, possible solutions (at least so I 
> could imagine) are that libusb is changed, so that it is willing to 
> interact with the kernel module, or that the kernel module is so 
> constructed that it can run in some kind of 'pass-through' mode and 
> libusb can go ahead and access the device anyway, or libgphoto2 camera 
> drivers are changed so as to address a kernel module if it is present 
> instead of addressing the device directly through libusb.

I agree (and still understand) but I'm not aware of a way for a device driver 
to give up his claim on an usb device, perhaps such an option exists in the 
kernel usb subsystem, then adding an API to ask v4l to release the device 
should be possible, although IMHO not the best way forward, how I envision this 
to work is:

usb-drivers get 2 new optional driver callbacks which the usb-subsystem can 
call, called :

1) release_device() this returns 0 if the device is not in use, and disallows 
any new users from using the device through the driver (iow disallow new open's 
of /dev/videoX in our case).

2) reclaim_device(), the driver may now use the device again and allow users to 
open it.


The usb kernel<->userspace API used by libusb gets a new API, which makes it 
possible for libusb to send a release request to a driver holding a device

If the driver has release_device implemented and returns 0, the usb subsystem 
will allow libusb to use the device even though there is a driver for it,

When libusb is done with the device (closes the relevant filedescriptors, the 
drivers resume_access() method gets called.

Notes:
1 this is all a figment of my imagination / vaporware.
2 I think we must come up with something more KISS


Ok, so looking for a simpler solution, there is an kernel usb subsystem method 
called:
usb_driver_release_interface()

And another called :
usb_driver_claim_interface()

Using these 2 a driver could release a device when requested and then later 
(when told to) reclaim it.

No idea how well this will work though, and when we solve this completely at 
the driver level we will need some (none generic) driver API for libgphoto2 to 
ask us to release the device (and tell us whem it is ok to reclaim it).

Regards,

Hans

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: Report on time needed for completing v4l Bayer interpolation.

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:
> 
> Hans,
> 
> I made your Bayer code to work. It was not terribly difficult, but did 
> need a bit of fixing and fooling.
> 
> So now the gp_bayer_decode function (in byr2ppm.c)says
> 
> int
> gp_bayer_decode (unsigned char *input, int w, int h, unsigned char *output,
>                  BayerTile tile)
> {
> //      gp_bayer_expand (input, w, h, output, tile);
> //      gp_bayer_interpolate (output, w, h, tile);
>         clock_t time_start=clock();
>                 fprintf(stderr, "Starting bayer v4l-interpolation\n");
> 
> 
> 
> v4lconvert_bayer_to_rgb24(input,
>  output, w, h, tile);
> 
>         clock_t time_end=clock();
>         fprintf(stderr,"Total time of v4l Bayer :\t%i ms\n",
>                                     (int)(time_end-time_start)/1000);
> 
> 
>         return (0);
> }
> 
> and the result for the same photo as before is
> 
> Total time of v4l Bayer :       0 ms
> 
> In other words, the time is reduced by a factor of approximately ten 
> from the code in the original libgphoto2/bayer.c (the straight bilinear 
> version).
> 

Well 10x might be a bit exaggerating, see my comments about clock() precision 
below.

> With a larger 640x480 image the time required is variously 0 ms or 
> sometimes 10 ms, so again a dramatic improvement.
> 

Hmm, if it jumps between 0 and 10 then clock() apparently has a 10 ms 
resolution I know the resolution of clock() wasn't all that great, but this is 
kinda bad actually.

> What would you suggest to do in order to get a more meaningful 
> measurement, at this point? Remove the dividend 1000 completely?
> 

That wont help, I advice you to use gettimeofday instead that much much better 
precision (sub microsecond on most systems).


> As far as the output image is concerned, the results are as I would 
> expect, though. To me, the output appears to be pretty much identical to 
> what the unmodified bilinear interpolation gives, which was in the 
> "original" libgphoto2/bayer.c. That is, there is lots of zipper effect 
> or ladder effect which is visible at vertical and horizontal edges all 
> over the photo, and this artifacting can be easily seen even at the 
> original 320x240 resolution without doing any magnification.
> 

Ack.

> Incidentally, the photo is the "infamous number 23" and you can see the 
> same photo on the website, as 023.ppm (now that I got the permissions 
> problem fixed). In the paper there are also some magnified views of a 
> small portion of the same photo, which show the effects of the three 
> different algorithms that show up especially at the edges of the balcony 
> railing and at the tops of the buildings in the distance.
> 
> Bottom line: I will see if I can figure out anything to get rid of the 
> zippering without excessively jacking up the time needed. Probably the 
> first thing to test is the accrue functionality, or a modification of that.

Ok and Thanks!

Regards,

Hans

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: Apparent inconsistency in the labels of Bayer tilings

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:
> 
> Thanks for the recommendation about gettimeofday. I will try putting 
> that in. Right now, I wonder if you know something about the following:
> 
> In order to make your Bayer demosaicing algorithm work, I had to do the 
> following because otherwise the colors were mapped backwards:
> 
> 1. create new entries in your libv4lconvert-priv.h like this
> 
> #ifndef V4L2_PIX_FMT_SBGGR8
> #define V4L2_PIX_FMT_SBGGR8 v4l2_fourcc('B','G','G','R')
> #endif
> 

Erm, that one is defined in linux/videodev2.h and has been for a long time:
#define V4L2_PIX_FMT_SBGGR8  v4l2_fourcc('B', 'A', '8', '1') /*  8  BGBG.. 
GRGR.. */

But I see I did not include it in libv4lconvert-priv.h, my bad.

Anyways to which fourcc you define V4L2_PIX_FMT_SBGGR8 should not matter, as 
for the meaning of the defines, as the command above shows in v4l BGGR means 
first line: BGBG.. second line: GRGR.., and then third line BGBG.. again, etc.

That seems pretty straightforward to me, how are libgphoto's defines supposed 
to be interpreted?

> #ifndef BAYER_TILE_BGGR
> #define BAYER_TILE_BGGR V4L2_PIX_FMT_SBGGR8
> #endif
> 
> and then use these. But in libgphoto2 the same photo from the same 
> camera has to use
> 
> BAYER_TILE_RGGB
> 
> So in other words when changing over the Bayer algorithm I also had to 
> change the label for the tiling from RGGB over to BGGR.
> 
> What gives? Do you happen to know? The two labels clearly disagree to 
> such an extent that they cannot both be right. Of course, the whole 
> thing is a matter of convention, presumably settled a long time ago by 
> someone who was neither of us. Also, the labeling of the Bayer tiles in 
> Gphoto was done long before I came along and thus I am not responsible 
> for it. So I explicitly do not give and do not have an opinion about who 
> is right.

The v4l convention also stems from before I came a long, I merely added the 
other 3 possible bayer patterns to the list.

Regards,

Hans


--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Adam Baker]

On Friday 21 November 2008, kilgota@banach.math.auburn.edu wrote:
> Where I am coming from is, libgphoto2 uses libusb. When a kernel module
> has "taken over" the device, then as things currently stand, or have stood
> until recently, libgphoto2 has no access to the device unless and until
> the kernel module has been rmmod-ed. I understand that there is a partial
> solution for this. I am trying to figure out how there is a complete
> solution which would make everyone happy, including users who just want to
> plug in their cameras.


The current solution is because of line 256 of libgphoto2_port/usb/libusb.c in 
version 2.4.3 of libgphoto2 (It has actually been in there for quite a while, 
the relevant code was added in svn rev 7283 over 4 years ago although the 
lack of releases around that time (if svn release tags are correct) may mean 
it took years until distributions started shipping it).

Basically what happens is that libgphoto2 calls usb_detach_kernel_driver_np 
which hands the device over to libusb. Unfortunately libusb doesn't include a 
corresponding attach method for libgphoto2 to use when it has finished so it 
can't re-instate the kernel driver.

There is a patch for libusb at 
http://osdir.com/ml/lib.libusb.devel.general/2007-04/msg00239.html to 
implemnt attach which the relevant maintainer claimed he had applied but 
doesn't appear to exist in libusb svn. I'll raise the issue on the 
libusb-devel list.

Adam

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Adam Baker]

On Friday 21 November 2008, Adam Baker wrote:
> There is a patch for libusb at
> http://osdir.com/ml/lib.libusb.devel.general/2007-04/msg00239.html to
> implemnt attach which the relevant maintainer claimed he had applied but
> doesn't appear to exist in libusb svn. I'll raise the issue on the

Further investigation reveals that the reason this patch hasn't been applied 
is because libusb-0.1 is in bug fixes only mode while a new libusb variant is 
being developed. It did get applied to a previous development branch but that 
branch got killed.

I've asked the question anyway.

Adam

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Adam Baker]

On Friday 21 November 2008, kilgota@banach.math.auburn.edu wrote:
> > Unfortunately libusb doesn't include a
> > corresponding attach method for libgphoto2 to use when it has finished so
> > it can't re-instate the kernel driver.
>
> True. But what I am wondering is, if it would be possible to write the
> kernel driver in such a way that it does not need to get detached, then it
> would be possible to solve the problem in both directions,

I don't believe that that is possible because if I understand correctly libusb 
detects that a driver has been loaded that has claimed that device and so it 
needs to be involved in the communication to get the kernel driver to allow 
access.

To acheive what you are suggesting the kernel driver would have to return 
false from its xxx_probe routine and not create it's endpoints but still call 
video_register_device to create the /dev/videox dev file. The problem is I 
don't believe there exists a mechanism whereby it could then obtain access to 
the USB device within it's xxx_open routine and it certainlywouldn't receive 
USB disconnect events so couldn't clean up properly.

The libusb attach/detach mechanism definitely looks like the cleanest solution 
if we can get a shipping version of libusb that implements attach.

Adam

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: The userspace-kernelspace thing. Continuation of sq905 driver discussion.

[Adam Baker]

On Saturday 22 November 2008, kilgota@banach.math.auburn.edu wrote:
> On Sat, 22 Nov 2008, Adam Baker wrote:
> > On Friday 21 November 2008, kilgota@banach.math.auburn.edu wrote:
> >>> Unfortunately libusb doesn't include a
> >>> corresponding attach method for libgphoto2 to use when it has finished
> >>> so it can't re-instate the kernel driver.
> >>
> >> True. But what I am wondering is, if it would be possible to write the
> >> kernel driver in such a way that it does not need to get detached, then
> >> it would be possible to solve the problem in both directions,
> >
> > I don't believe that that is possible because if I understand correctly
> > libusb detects that a driver has been loaded that has claimed that device
> > and so it needs to be involved in the communication to get the kernel
> > driver to allow access.
>
> I understand this to be an accurate description of the current state of
> affairs. Under a different scenario, though, libgphoto2 asks the kernel
> for access to the device. The kernel either gives it directly, or, if the
> device is controlled by a module, then passes the request to the module.
> If the module is a "nice" module and understands that the hardware device
> is not in use by some other program, it cheerfully honors the request. If
> such a thing were to be implemented, then of course there would have to be
> a change in the way that libusb tries to get access to a device, too.
>

That is basically what usb_detach_kernel_driver_np does. The only problem is 
that the current version of libusb does not provide a corresponding attach 
method to reinstate the kernel driver afterwards. Having got a response on 
the libusb mailing list I can now also confirm that the current libusb API 
never will.

The attach method is however a completely standalone piece of code so 
libgphoto2 could if desired just define it's own attach method if the 
platform provides a detach method but no attach method.

> > To acheive what you are suggesting the kernel driver would have to return
> > false from its xxx_probe routine and not create it's endpoints but still
> > call video_register_device to create the /dev/videox dev file.
>
> Hmmm. Well, not necessarily. If automatically loaded at boot it could just
> sit there and do nothing unless it gets a request, couldn't it? Or, if
> loaded later on because the camera is plugged in, it could then do the
> same thing. Then, if a v4l type of program wants something from it, it
> creates the video device. Otherwise, it does not. And if the v4l type
> program is exited, then the video device goes away and the kernel driver
> waits for the next request, from whatever source.
>
>

It needs to create the /dev/video device so that video aware apps can see a 
device to offer to the user to use. Otherwise they have no way of talking to 
the driver to say they want it to wake up.

> The problem is I
>
> > don't believe there exists a mechanism whereby it
>
> by "it" I assume you mean the kernel driver?

Correct

>
> could then obtain access to
>
> > the USB device within it's xxx_open routine and it certainlywouldn't
> > receive USB disconnect events so couldn't clean up properly.
> >
> > The libusb attach/detach mechanism definitely looks like the cleanest
> > solution if we can get a shipping version of libusb that implements
> > attach.
>
> Well, It would be fun to test that. Is there an SVN version? I hereby
> volunteer. If it works, then, problem solved.

There is an svn version but it is in a dead branch. As stated above I think 
putting the attach code into libgphoto2 is the best we can do.

>
> I am not very much scared of such things, but if I am supposed to be
> scared, then, well, I have 2 other machines sitting within a two meter
> radius. They are old boxes. So if one of them catches fire I will just
> have to run and get the fire extinguisher. Adtually, I run a risk every
> time I boot one of them up already. They are both dual boot (as this
> machine is not because it is for serious business). One of them has a
> Win98 partition on it and the other one has a Win2K partition. Neither of
> them is hooked to any network because of that. They are used, of course,
> for installing Win drivers and running stuff like usbsnoop. But I already
> take my life in my hands every time I fire one of those up. So, scared of
> experimental code? Who, me? Nah.

I don't think the patch at
http://osdir.com/ml/lib.libusb.devel.general/2007-04/msg00239.html
is anything to be scared of, at least not on a machine that already has a non 
vendor packaged libgphoto2

Regards

Adam

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:
> 

<snip>

> Unfortunately libusb doesn't include a
>> corresponding attach method for libgphoto2 to use when it has finished 
>> so it
>> can't re-instate the kernel driver.
> 
> True. But what I am wondering is, if it would be possible to write the 
> kernel driver in such a way that it does not need to get detached, then 
> it would be possible to solve the problem in both directions, not just 
> one. Somehow, arrange things so that if an app requires the kernel 
> driver, then the kernel driver obliges. But if the app wants to approach 
> the device through libusb, then libusb, upon discovering that the kernel 
> driver is loaded, asks the kernel driver for access and then the kernel 
> driver, under certain reasonable conditions, obliges. For example, in 
> the case of a dual-mode camera I would probably tend to consider that 
> the kernel module ought not to surrender while a webcam app is actually 
> in use, but in other circumstances ought to allow "pass through." I 
> mean, if I am a userspace app which peaceably and above-board wants 
> access to a still camera through libusb then do I care if there is a 
> device /dev/videoXX which is associated with my USB vendor:product 
> number? Actually, not. I only care that if I am an app which never wants 
> even to go near to any /dev/videoXX then nobody is going to try to force 
> me to do that. In particular, why does the kernel driver for the device 
> have to force me to access /dev/videoXX even if I never requested any 
> such thing? So it could work like this:
> 

I've been thinking along similar lines (keeping /dev/videoX present when using 
the still image function). But thinking about this some more I don't think this 
is worth the trouble. A camera which can do both still images and function as 
webcam really is a multifunction device, with one function active at a time, 
this is just like any usb device with multiple usb configurations, and when you 
change the configuration certain functionality becomes not available and other 
becomes available. If this cam would be correctly using usb configuaration 
profiles for this, the /dev/videoX would also disappear.

Also by just unloading the driver removing /dev/videoX things stay KISS.

Last it is not that strange for the webcam not to show up as a webcam to choose 
from for use in a webcam app, when you've got a photo app open for importing 
photo's (and when the import dialog is closed the device should be given back 
to the /dev/videoX driver).

So all in all I believe that having some mechanism:
-to unload the driver when libusb wants access *AND*
-reload it when libusb is done

Is enough, and is nice and KISS, which I like.

Regards,

Hans

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:
> 

<snip>

>>>> Correct, there is nothing special you need to do for that, just pass
>>>> frames with the raw bayer data to userspace and set the pixelformat to
>>>> one of: V4L2_PIX_FMT_SBGGR8 /* v4l2_fourcc('B', 'A', '8', '1'), 8 bit
>>>> BGBG.. GRGR.. */ V4L2_PIX_FMT_SGBRG8 /* v4l2_fourcc('G', 'B', 'R', 
>>>> 'G'),
>>>> 8 bit GBGB.. RGRG.. */ V4L2_PIX_FMT_SGRBG8 /*
>>>> v4l2_fourcc('G','R','B','G'), 8 bit GRGR.. BGBG.. */ 
>>>> V4L2_PIX_FMT_SRGGB8
>>>> /* v4l2_fourcc('R','G','G','B'), 8 bit RGRG.. GBGB.. */
> 
> Hmmm. If the userspace app wants the data presented like this, then I 
> would have to take back my previous remark. If the userspace app is 
> supposed to know something about the particular camera, though, my 
> previous remark stands.
> 

No this is just an example, if the cam can send compressed data too for 
example, then we need to define a new V4L2_PIX_FMT_FOO for that, and add suport 
for decompressing that to raw bayer to libv4l. Which could be an issue as 
libv4l is LGPL, to allow using it from any app, so we need to get permission 
from the author(s) to get the decompression code relicensed.

Regards,

Hans

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Adam Baker]

[-- Attachment #1: Type: text/plain, Size: 2136 bytes --]

On Monday 24 November 2008, kilgota@banach.math.auburn.edu wrote:
> Well, there is another ideal solution which probably cannot work for ten
> thousand other very good reasons. That would be to eliminate the need for
> the kernel to create a "device" before a webcam is fired up.
>

Unfortunately that would mean changing lots of existing applications.

> So in other words the ability of libusb to load up a kernel module is
> another trick which may alleviate the problem for some people, but does
> not solve the problem. Not yet.

libusb doesn't actually reload the kernel module. The patch I posted a 
reference to just causes the same scan for modules to run as would have run 
when the device was first plugged in.

This approach can even cope with the complex case that no-one has yet 
considered, wanting to use one sq905 based camera to record video while you 
download stills from another.

On an embedded system that didn't want camera drivers it would never use you 
would probably build the kernel without V4L support and nothing anyone does 
is then going to load the module.

To hopefully convince you this can easily provide all the functionality you 
want I've attempted to do some testing and in order to avoid changing 
libgphoto I've written a little standalone app (attached) to do the cleanup. 
I'm taking advantage of the fact I know we want ifno 0 for this cam but 
libgphoto2 has already found the correct value.

I can now happily run the following sequence

Plug in camera, driver loads and /dev/video0 is created

run gphoto2 -L and it tells me there are photos on the camera and /dev/video0 
dissappears (and this is a model that deletes photos when used as a web cam)

run my rescan app which I'd suggest libgphoto2 should do automatically if it 
has called usb_detach_kernel_driver_np
./usbscan /dev/bus/usb/005/008
and /dev/video0 re-appears and I can run up xawtv and see a picture

If I now run up gphoto2 again all my images have gone as I actually used the 
webcam functionality.

I've changed the ioctl's in the patch from using libusb private values to 
values from a kernel provided header file.

[-- Attachment #2: usbscan.c --]
[-- Type: text/x-csrc, Size: 552 bytes --]

#include <sys/ioctl.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdlib.h>
#include <linux/usbdevice_fs.h>

int main(int argc, char *argv[])
{
    int fd;
    struct usbdevfs_ioctl command;

    if (argc != 2) exit(1);
    fd=open(argv[1],O_RDWR);

    if (fd < 0)
    {
        perror(argv[1]);
        exit(1);
    }

    command.ifno = 0;
    command.ioctl_code = USBDEVFS_CONNECT;
    command.data = NULL;

    if (ioctl(fd, USBDEVFS_IOCTL, &command))
       perror("ioctl");
       
    close(fd);

    return 0;
}

[-- Attachment #3: Type: text/plain, Size: 164 bytes --]

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Adam Baker]

On Tuesday 25 November 2008, kilgota@banach.math.auburn.edu wrote:
> On Tue, 25 Nov 2008, Adam Baker wrote:
<snip>
> > This approach can even cope with the complex case that no-one has yet
> > considered, wanting to use one sq905 based camera to record video while
> > you download stills from another.
>
> That is interesting if it works, yes. But how could a thing like that
> work?

Because it the libusb detach call doesn't unload the driver, just detach it 
from the usb device supplied as an argument. If there were another device 
using the same driver it would remain attached.

> If the kernel is without V4L support, then clearly no module will get
> loaded. But I would say that
>
> "embedded system or small system" != "don't want camera drivers"

Certainly there exist embedded systems that may want camera drivers but then 
they have to have the memory to support them so there is no benefit to not 
loading the driver.

>
> > To hopefully convince you this can easily provide all the functionality
> > you want I've attempted to do some testing and in order to avoid changing
> > libgphoto I've written a little standalone app (attached) to do the
> > cleanup. I'm taking advantage of the fact I know we want ifno 0 for this
> > cam but libgphoto2 has already found the correct value.
> >
<snip>
> >
> > If I now run up gphoto2 again all my images have gone as I actually used
> > the webcam functionality.
>
> Convincing, yes.
>
> > I've changed the ioctl's in the patch from using libusb private values to
> > values from a kernel provided header file.
>
> All right, I will have a look. Unfortunately, I understand that in order
> really to test it I need to have some piece of hardware which is supported
> through libgphoto2 in one mode and by a module in the other mode. An old
> SQ905 camera will do that, but what module are you using to support it
> right now? The old one? Is it compatible with recent kernels? Or is it
> rewritten somewhere? Sorry, but while I can easily understand the
> underlying problems here I am not up to speed at all about webcam apps or,
> for that matter, the latest status of V4L.

Yes, I am using the existing sqcam driver from 
http://sqcam.cvs.sourceforge.net/viewvc/sqcam/sqcam26/

I think the newest kernel I've tested it with is 2.6.26. I've certainly tried 
with one where V4L1 support is optional which I think started from 2.6.26 but 
I never committed the changes needed to allow the driver to build if V4L1 
support is disabled so you need to check CONFIG_VIDEO_V4L1 is set on recent 
kernels.

>
>
> So what I understand is this code will re-scan the USB bus, and, when it
> does, the module gets reloaded if the camera is plugged in. It seems to me
> that a very logical place to put code like this would be in a webcam app,
> not in libgphoto2. I suspect that this is not what you want me to say.
> But, as I said, I will have a look at the code.

It doesn't re-scan the whole bus, just rescan the driver list to see if one 
matches that device.

Regards

Adam

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list

Re: [sqcam-devel] Advice wanted on producing an in kernel sq905 driver

[Hans de Goede]

kilgota@banach.math.auburn.edu wrote:
> 
> 
> On Mon, 24 Nov 2008, Hans de Goede wrote:
<snip>

>> I've been thinking along similar lines (keeping /dev/videoX present 
>> when using the still image function). But thinking about this some 
>> more I don't think this is worth the trouble. A camera which can do 
>> both still images and function as webcam really is a multifunction 
>> device, with one function active at a time, this is just like any usb 
>> device with multiple usb configurations,
> 
> Ah, but the problem is that USB devices which are "good citizens" 
> usually have a different Product number when running in different modes.
> 

Well, that would make things consistent then with my proposal for cams who hide 
multiple functions behind one ID, if one function is used, the other 
disappears, including the /dev/videoX device.

>> and when you change the configuration certain functionality becomes 
>> not available and other becomes available. If this cam would be 
>> correctly using usb configuaration profiles for this, the /dev/videoX 
>> would also disappear.
>>
>> Also by just unloading the driver removing /dev/videoX things stay KISS.
> 
> Well, I am not sure that this is simpler, actually, which is why I 
> brought the subject up for discussion.
> 

Well, drivers need to handle disconnect anyways, so this requires no additional 
code from the driver, how simple do you want it to be ?

>>
>> Last it is not that strange for the webcam not to show up as a webcam 
>> to choose from for use in a webcam app, when you've got a photo app 
>> open for importing photo's (and when the import dialog is closed the 
>> device should be given back to the /dev/videoX driver).
>>
>> So all in all I believe that having some mechanism:
>> -to unload the driver when libusb wants access *AND*
>> -reload it when libusb is done
> 
> This does not address the question of whether the owner of the machine 
> wants the kernel module installed, or not. I can imagine that the owner 
> might not be interested at all in loading the module if said owner is 
> not interested in running the camera as a webcam. I can also imagine 
> that the outcome is
> 

If you look at any up2date Linux distro, all drivers for all available hardware 
will get loaded. This is just how things work now a days, and this is the right 
thing todo for 99.99 % of all users. And I really don't care about that other 
0.01 %

> 1. owner gets still photos off the camera.
> 2. owner exits from photo-getting app
> 3. libgphoto2 releases the camera, and libusb loads the kernel module -- 
> which was, prior to (1), *not* loaded.

No, as Adam explained libgphoto would trigger a rescanning of the usb bus, if 
the driver was not loaded at plugin / boot because it was blacklisted for 
example, the rescan wont load it either.

But this is a corner case. One should never design / optimize for corner case. 
Design / optimize for the straight path / normal expected use scenario and then 
  see what needs and can be done for special cases.

> 
>>
>> Is enough, and is nice and KISS, which I like.
> 
> 
> Yeah. I just love to have modules installed to support hardware which I 
> do not intend to use. Don't we all? ;)

Erm, yes we do all love that, because then when I do decide to buy a bluetooth 
gizmo, even though I did not have any use for the entire bluetooth stack being 
loaded on my desktop before, now all of a sudden it works out of the box.

I'm sorry but not loading modules is something from a recent past, we just 
don't do that anymore. Just like almost no one builds its own kernels now.

> What is "simple" about that?

That if I decide to change my mind and use the functionality after all things 
will just work.

> If I 
> am running some low-powered piece of hardware, why would I want all 
> kinds of support for devices X, Y, and Z installed in the kernel unless 
> and until I am actually going to use one of them?
> 

If you run on a low-powered piece of hardware, you will need a custom distro 
anyways.

> The ideal solution probably can not work for ten thousand good reasons, 
> but what ought to happen is that a libgphoto2-based app could turn off 
> or remove the module (unless the module is in use), and an app requiring 
> the kernel module could cause it to be installed, unless the device is 
> in use by some app already. This would actually be most ideal and would 
> make it unnecessary for libusb to install kernel modules which, perhaps, 
> no one is going to use during the next week after installation.
> 

No, that will never work, because now a days we have a model where drivers gets 
loaded when hardware is detected, not when apps need the functionality. This 
allows applications to ask the system which devices are present. Since 
applications now ask the system which devices are present, they cannot tell it 
to load the driver, otherwise we get a classic chicken and egg problem.

> So in other words the ability of libusb to load up a kernel module is 
> another trick which may alleviate the problem for some people, but does 
> not solve the problem. Not yet.

I disagree, it is not a trick, it is a nice and very *simple* solution. Maybe 
I'm wring, time will tell then and then we can start thinking about more 
complex solutions, for now this seems the way forward to me.

Regards,

Hans

--
video4linux-list mailing list
Unsubscribe mailto:video4linux-list-request@redhat.com?subject=unsubscribe
https://www.redhat.com/mailman/listinfo/video4linux-list