Re: [PATCH] input: Add a detailed multi-touch finger data report

Re: [PATCH] input: Add a detailed multi-touch finger data report

[Jim Gettys]

Henrik Rydberg wrote:


Sorry I'm slightly late to this discussion.

> > is there hardware that can do finger identification? (i.e. thumb vs. index
> > finger)? 

Yes, in the extreme case.  There are research table systems which image the surface,
and you get an image of the hand above the surface; they then compute the fingers position
relative to the general hand outline, and which ones/how much of the hand
is touching (which are in-focus). These can also tell you something about proximity
(not yet touching the surface), very much the way magnetic tablet technologies can tell
you proximity. After processing, you get which finger(s) are touching (or have
proximity), associated with a right or left hand, for information.

> Should we accommodate for this?

Should we bother right now? The extreme often becomes the norm with time, but we need
to draw a line at a sane place.  The closely related case to identifying fingers that
I think *is* worth accommodating immediately I describe below.

> 
> I believe we should start with events that fit the general idea of
> detailed finger information, and which can be produced by at least one
> existing kernel driver, so that we can test it immediately. I believe
> the proposed set pretty much covers it. I would love to be wrong. :-)
> 
> Regarding identification, one of the harder problems involved in
> making use of finger data is that of matching an anonymous finger at a
> certain position to an identified finger, tagged with a number.  This
> is very important in order to know if the fingers moved, which finger
> did the tapping, how much rotation was made, etc. Generally, this is
> the (euclidian) bipartite matching problem, and is one of the major
> computations a multi-touch X driver needs to perform. I can imagine
> such identification features eventually ending up on a chip. Maybe
> someone more knowledgeable in hardware can give us a hint.
> 
> 
> 

I agree with Peter's point that modeling all of this (fingers, markers, etc)
as multiple pointers will cause madness to ensue.

The way I distinguish devices in my mind is by "sensors".  If there are multiple
touches, markers, fingers, users, all using the same sensor (of the same resolution),
then the information should start off life together in the same input stream: this way
the relative time ordering of events all makes sense.

Some systems (e.g. Merl's Diamond Touch), give you an ID associated with the user
(in that case, it works by knowing where you are sitting by capacitive coupling). In
this case, it is actually where the person is sitting, rather than a particular person.

Another case that will be common soon is to be able to sense and identify
markers on the surface (which can be distinguished from each other).  I know of at 
least three hardware systems able to do this. One of these will be in commodity hardware
soon enough to worry about immediately.  So having and ID reported with a touch is clearly
needed, whether thumb, index finger, or some marker. 

Whether such markers would have any user identity directly associated with them is less than
clear, though we'll certainly start giving them such identity either by convention or
fiat somewhere in the system as the events get processed.

We may also face co-located sensors, where two sensors
are geometrically on top of each other (but might even report different coordinates of
differing resolutions), but co-aligned.  I'm thinking of the Dell Latitude XT in this case, 
though I don't yet know enough about it to know if in fact its pen uses a different sensor 
than the capacitive multi-touch screen.  I'm still trying to get precise details on this device.

Another question is whether an ellipse models a touch adequately at the moment; other sensors
may report more complex geometric information.  There is a slippery slope here, of course.
In the extreme case noted above, research systems give you a full image, which seems like overkill.

I also note the current input system does not provide any mechanism or hint 
to associate an input device with a particular frame buffer or with each other.  Maybe it should, 
maybe it shouldn't... Opinions?

Hope this helps.  The problem here is to draw a line *before* we win our complexity merit badge,
while leaving things open to be extended as more instances of real hardware appears and we have
more experience.
                        - Jim

-- 
Jim Gettys <jg@laptop.org>
One Laptop Per Child

Re: [PATCH] input: Add a detailed multi-touch finger data report

[Jim Gettys]

Henrik Rydberg wrote:

Sorry I'm slightly late to this discussion.

> > is there hardware that can do finger identification? (i.e. thumb 
> > vs. index finger)? 

Yes, in the extreme case.  There are research table systems which image
the surface, and you get an image of the hand above the surface; they
then compute the fingers position relative to the general hand outline,
and which ones/how much of the hand is touching (which are in-focus).
These can also tell you something about proximity (not yet touching the
surface), very much the way magnetic tablet technologies can tell you
proximity. After processing, you get which finger(s) are touching (or
have proximity), associated with a right or left hand, for information.

> Should we accommodate for this?

Should we bother right now? The extreme often becomes the norm with
time, but we need to draw a line at a sane place.  The closely related
case to identifying fingers that I think *is* worth accommodating
immediately I describe below.

> 
> I believe we should start with events that fit the general idea of
> detailed finger information, and which can be produced by at least one
> existing kernel driver, so that we can test it immediately. I believe
> the proposed set pretty much covers it. I would love to be wrong. :-)
> 
> Regarding identification, one of the harder problems involved in
> making use of finger data is that of matching an anonymous finger at a
> certain position to an identified finger, tagged with a number.  This
> is very important in order to know if the fingers moved, which finger
> did the tapping, how much rotation was made, etc. Generally, this is
> the (euclidian) bipartite matching problem, and is one of the major
> computations a multi-touch X driver needs to perform. I can imagine
> such identification features eventually ending up on a chip. Maybe
> someone more knowledgeable in hardware can give us a hint.
> 
> 
> 

I agree with Peter's point that modeling all of this (fingers, markers,
etc) as multiple pointers will cause madness to ensue.

I distinguish devices in my mind is by "sensors".  If there are multiple
touches, markers, fingers, users, all using the same sensor (of
the same resolution), then the information should start off life
together in the same input stream: this way the relative time ordering
of events all makes sense.

Some systems (e.g. Merl's Diamond Touch), also give you an ID 
associated with the user for each touch (in that case, it works
by knowing where you are sitting by capacitive coupling with your
seat). 

Another case that will be common *soon* is to be able to sense and 
identify markers on the surface (which can be distinguished from each
other).  I know of at least three hardware systems able to do this. One
of these will be in commodity hardware soon enough to worry about
immediately.  So having and ID reported with a touch is clearly
needed, whether thumb, index finger, or some marker. 

Whether such markers would have any user identity or other association
with them is less than clear, though we'll certainly start giving them
such identity either by convention or fiat somewhere in the system as
the events get processed.

We may also face co-located sensors, where two sensors
are geometrically on top of each other (but might even report different
coordinates of differing resolutions), but co-aligned.  I'm thinking of
the Dell Latitude XT in this case,  though I don't yet know enough about
it to know if in fact its pen uses a different sensor than the
capacitive multi-touch screen.  I'm still trying to get precise details
on this device.

Another question is whether an ellipse models a touch adequately at the
moment; other sensors may report more complex geometric information.
There is a slippery slope here, of course. In the extreme case noted
above, research systems give you a full image, which seems like
overkill.

I also note the current input system does not provide any mechanism or
hint to associate an input device with a particular frame buffer or with
each other.  Maybe it should, maybe it shouldn't... Opinions?

Hope this helps.  The problem here is to draw a line *before* we win our
complexity merit badge, while leaving things open to be extended as we
have more instances of real hardware and we have more experience.
                        - Jim
-- 
Jim Gettys <jg@laptop.org>
One Laptop Per Child

Re: [PATCH] input: Add a detailed multi-touch finger data report

[Henrik Rydberg]

> Some systems (e.g. Merl's Diamond Touch), give you an ID associated
> with the user (in that case, it works by knowing where you are
> sitting by capacitive coupling). In this case, it is actually where
> the person is sitting, rather than a particular person.

This is similar to what I have experienced with the bcm5974. The chip
outputs some identification based on position, but in the end, the
information needed is of the kind 'which finger is being lifted', not
'where is it lifted'.  Obtaining such tracking information requires
the additional assumption of continuous movement, which makes the
usefulness of position-based identifiers somewhat limited. I left out
some finger details from the spec for that reason.

> Another case that will be common soon is to be able to sense and
> identify markers on the surface (which can be distinguished from
> each other).  I know of at least three hardware systems able to do
> this. One of these will be in commodity hardware soon enough to
> worry about immediately.

Like putting pins on a map and being able to tell where each pin is?

> So having and ID reported with a touch is clearly needed, whether
> thumb, index finger, or some marker.

If a chip can actually classify fingers as index or thumb, it would
definitely qualify as detailed finger information. Cool.

> Whether such markers would have any user identity directly
> associated with them is less than clear, though we'll certainly
> start giving them such identity either by convention or fiat
> somewhere in the system as the events get processed.  We may also
> face co-located sensors, where two sensors are geometrically on top
> of each other (but might even report different coordinates of
> differing resolutions), but co-aligned.  I'm thinking of the Dell
> Latitude XT in this case, though I don't yet know enough about it to
> know if in fact its pen uses a different sensor than the capacitive
> multi-touch screen.

This sounds similar to the finger classification, although here
distinguishing a pen from a finger.

Looking at these three cases, it seems adding something like
ABS_MT_TOOL_TYPE to the protocol right away makes sense. The wording
here is chosen with the distinction between (pin1, pin2, index, thumb,
pen) and (pointing-finger, clicking-finger) in mind.

> Another question is whether an ellipse models a touch adequately at
> the moment; other sensors may report more complex geometric
> information.  There is a slippery slope here, of course.  In the
> extreme case noted above, research systems give you a full image,
> which seems like overkill.  I also note the current input system
> does not provide any mechanism or hint to associate an input device
> with a particular frame buffer or with each other.  Maybe it should,
> maybe it shouldn't... Opinions?  Hope this helps.  The problem here
> is to draw a line *before* we win our complexity merit badge, while
> leaving things open to be extended as more instances of real
> hardware appears and we have more experience.

Right. :-) I believe the ellipse model is adequate, because it is the
simplest model that allows for utilization of the orientation of a
single finger, for instance to turn a knob. At this point, it seems
like tough enough a challenge.

Re: [PATCH] input: Add a detailed multi-touch finger data report

[Jim Gettys]

On Fri, 2008-11-14 at 01:55 +0100, Henrik Rydberg wrote:
> > Some systems (e.g. Merl's Diamond Touch), give you an ID associated
> > with the user (in that case, it works by knowing where you are
> > sitting by capacitive coupling). In this case, it is actually where
> > the person is sitting, rather than a particular person.
> 
> This is similar to what I have experienced with the bcm5974. The chip
> outputs some identification based on position, but in the end, the
> information needed is of the kind 'which finger is being lifted', not
> 'where is it lifted'.  Obtaining such tracking information requires
> the additional assumption of continuous movement, which makes the
> usefulness of position-based identifiers somewhat limited. I left out
> some finger details from the spec for that reason.
> 
> > Another case that will be common soon is to be able to sense and
> > identify markers on the surface (which can be distinguished from
> > each other).  I know of at least three hardware systems able to do
> > this. One of these will be in commodity hardware soon enough to
> > worry about immediately.
> 
> Like putting pins on a map and being able to tell where each pin is?

Yes.  And what color each pin is (at least logically, if not literally).
You can distinguish that they are different on some hardware I'm aware
of.

> 
> > So having and ID reported with a touch is clearly needed, whether
> > thumb, index finger, or some marker.
> 
> If a chip can actually classify fingers as index or thumb, it would
> definitely qualify as detailed finger information. Cool.

Certainly there are research systems that do this today (Microsoft's
surface system, to give a concrete example).  They can classify the
fingers on a hand, and tell you the which finger on which hand from the
image the camera sees.

That table can give you proximity as well; so you know the fingers are
roughly how close, even before they touch. (They do this by looking at
how sharp the edges of the fingers are).

Whether there will be commodity hardware able to do this soon is less
clear; but arguably, all it is is a webcam pointing up at the bottom of
the tabletop.

> 
> > Whether such markers would have any user identity directly
> > associated with them is less than clear, though we'll certainly
> > start giving them such identity either by convention or fiat
> > somewhere in the system as the events get processed.  We may also
> > face co-located sensors, where two sensors are geometrically on top
> > of each other (but might even report different coordinates of
> > differing resolutions), but co-aligned.  I'm thinking of the Dell
> > Latitude XT in this case, though I don't yet know enough about it to
> > know if in fact its pen uses a different sensor than the capacitive
> > multi-touch screen.
> 
> This sounds similar to the finger classification, although here
> distinguishing a pen from a finger.

As I said, I don't know if the Latitude hardware is really logically a
single sensor or two, yet, though I hope to know soon.  It can certainly
distinguish your fingers from its stylus.  The stylus clearly also has
proximity information, from playing with it.

The Latitude XT's stylus has a pressure sensor, like a Wacom; I presume
it will be able to have multiple different flavor styluses like Wacom's
do too (e.g. erasers, calligraphic pens, etc...).  Certainly our
software should be able to use many different styluses and distinguish
them, as happens with Wacom tablets today.

The Latitude XT is interesting as it has the first multitouch laptop
screen available, at any price (still not cheap, but some of us can
actually buy them and play with them, which I can't do in the case of a
table display).

> 
> Looking at these three cases, it seems adding something like
> ABS_MT_TOOL_TYPE to the protocol right away makes sense. The wording
> here is chosen with the distinction between (pin1, pin2, index, thumb,
> pen) and (pointing-finger, clicking-finger) in mind.
> 

Yeah, something like this. 

> > Another question is whether an ellipse models a touch adequately at
> > the moment; other sensors may report more complex geometric
> > information.  There is a slippery slope here, of course.  In the
> > extreme case noted above, research systems give you a full image,
> > which seems like overkill.  I also note the current input system
> > does not provide any mechanism or hint to associate an input device
> > with a particular frame buffer or with each other.  Maybe it should,
> > maybe it shouldn't... Opinions?  Hope this helps.  The problem here
> > is to draw a line *before* we win our complexity merit badge, while
> > leaving things open to be extended as more instances of real
> > hardware appears and we have more experience.
> 
> Right. :-) I believe the ellipse model is adequate, because it is the
> simplest model that allows for utilization of the orientation of a
> single finger, for instance to turn a knob. At this point, it seems
> like tough enough a challenge.
> 

Maybe; but modeling this as a parallelogram or trapezoid may be easier
to deal with computationally when you stroke a path from the data, to
which you may have fit a spline....

We learned the hard way about "lumpy lines" in the old X core graphics
about why you don't want circular (or ellipsoidal) pens; you end up with
painfully complex mathematics when stroking a path of such things.  And
the lines were so ugly it made wide lines completely useless.  Cairo
will avoid the lumpy line problem due to anti-aliasing, but the math is
still painful.

The attempt to do Postscript "Red book" described circular pens wastes
like a few hundred K bytes of code in the X server. In short, trying to
solve to the edge of a circle or ellipse is a high order equation. If
you approximate this with a polygon, Hobby showed the resulting lines
are not only much more pleasing when stroked, but are mathematically
much easier to deal with.  And Adobe cheated anyway, it turned out....
We should have fixed the X spec, but never did.  Thankfully, Cairo
finally exists.

So I think something other than an ellipse is probably better....
                           - Jim

-- 
Jim Gettys <jg@laptop.org>
One Laptop Per Child

Re: [PATCH] input: Add a detailed multi-touch finger data report

[Henrik Rydberg]

>> Right. :-) I believe the ellipse model is adequate, because it is the
>> simplest model that allows for utilization of the orientation of a
>> single finger, for instance to turn a knob. At this point, it seems
>> like tough enough a challenge.
>>
>
> Maybe; but modeling this as a parallelogram or trapezoid may be easier
> to deal with computationally when you stroke a path from the data, to
> which you may have fit a spline....

I believe the word "ellipse" is taken too literally here. Orientation
and eccentricity are two parameters describing a non-circular
object. I does not have to be realized as ellipses on a screen.