[PATCH 2/2] input: Add a detailed multi-touch finger data report protocol (rev2)

[PATCH 2/2] input: Add a detailed multi-touch finger data report protocol (rev2)

[Henrik Rydberg]

In order to utilize the full power of the new multi-touch devices, a
way to report detailed finger data to user space is needed. This patch
adds a multi-touch (MT) protocol which allows drivers to report details
for an arbitrary number of fingers.

The driver sends a SYN_MT_REPORT event via the input_mt_sync() function
when a complete finger has been reported.

In order to stay compatible with existing applications, the data
reported in a finger packet must not be recognized as single-touch
events. In addition, all finger data must bypass input filtering,
since subsequent events of the same type refer to different fingers.

A set of ABS_MT events with the desired properties are defined. The
events are divided into categories, to allow for partial implementation.
The minimum set consists of ABS_MT_TOUCH_MAJOR, ABS_MT_POSITION_X and
ABS_MT_POSITION_Y, which allows for multiple fingers to be tracked.
If the device supports it, the ABS_MT_WIDTH_MAJOR may be used to provide
the size of the approaching finger. Anisotropy and direction may be
specified with ABS_MT_TOUCH_MINOR, ABS_MT_WIDTH_MINOR and
ABS_MT_ORIENTATION. Devices with more granular information may specify
general shapes as blobs, i.e., as a sequence of rectangular shapes
grouped together by a ABS_MT_BLOB_ID. Finally, the ABS_MT_TOOL_TYPE
may be used to specify whether the touching tool is a finger or a pen.

Signed-off-by: Henrik Rydberg <rydberg@euromail.se>
---
 drivers/input/input.c |   13 +++++++++++++
 include/linux/input.h |   27 +++++++++++++++++++++++++++
 2 files changed, 40 insertions(+), 0 deletions(-)

diff --git a/drivers/input/input.c b/drivers/input/input.c
index 8b865ba..2c2a95e 100644
--- a/drivers/input/input.c
+++ b/drivers/input/input.c
@@ -33,6 +33,15 @@ MODULE_LICENSE("GPL");
  * EV_ABS events which should not be cached are listed here.
  */
 static __initdata unsigned int input_abs_bypass_init_data[] = {
+	ABS_MT_TOUCH_MAJOR,
+	ABS_MT_TOUCH_MINOR,
+	ABS_MT_WIDTH_MAJOR,
+	ABS_MT_WIDTH_MINOR,
+	ABS_MT_ORIENTATION,
+	ABS_MT_POSITION_X,
+	ABS_MT_POSITION_Y,
+	ABS_MT_TOOL_TYPE,
+	ABS_MT_BLOB_ID,
 	0
 };

@@ -165,6 +174,10 @@ static void input_handle_event(struct input_dev *dev,
 				disposition = INPUT_PASS_TO_HANDLERS;
 			}
 			break;
+		case SYN_MT_REPORT:
+			dev->sync = 0;
+			disposition = INPUT_PASS_TO_HANDLERS;
+			break;
 		}
 		break;

diff --git a/include/linux/input.h b/include/linux/input.h
index 53a1f52..bcfebd9 100644
--- a/include/linux/input.h
+++ b/include/linux/input.h
@@ -106,6 +106,7 @@ struct input_absinfo {

 #define SYN_REPORT		0
 #define SYN_CONFIG		1
+#define SYN_MT_REPORT		2

 /*
  * Keys and buttons
@@ -646,6 +647,19 @@ struct input_absinfo {
 #define ABS_TOOL_WIDTH		0x1c
 #define ABS_VOLUME		0x20
 #define ABS_MISC		0x28
+
+#define ABS_MT_TOUCH		0x30	/* Diameter of touching circle */
+#define ABS_MT_TOUCH_MAJOR	0x30	/* Major axis of touching ellipse */
+#define ABS_MT_TOUCH_MINOR	0x31	/* Minor axis of touching ellipse */
+#define ABS_MT_WIDTH		0x32	/* Diameter of approaching circle */
+#define ABS_MT_WIDTH_MAJOR	0x32	/* Major axis of approaching ellipse */
+#define ABS_MT_WIDTH_MINOR	0x33	/* Minor axis of approaching ellipse */
+#define ABS_MT_ORIENTATION	0x34	/* Ellipse orientation */
+#define ABS_MT_POSITION_X	0x35	/* Center X ellipse position */
+#define ABS_MT_POSITION_Y	0x36	/* Center Y ellipse position */
+#define ABS_MT_TOOL_TYPE	0x37	/* Type of touching device */
+#define ABS_MT_BLOB_ID		0x38	/* Group a set of packets as a blob */
+
 #define ABS_MAX			0x3f
 #define ABS_CNT			(ABS_MAX+1)

@@ -742,6 +756,14 @@ struct input_absinfo {
 #define BUS_ATARI		0x1B

 /*
+ * MT_TOOL types
+ */
+#define MT_TOOL_FINGER		0
+#define MT_TOOL_PEN		1
+#define MT_TOOL_MAX		9
+#define MT_TOOL_CNT		(MT_TOOL_MAX + 1)
+
+/*
  * Values describing the status of a force-feedback effect
  */
 #define FF_STATUS_STOPPED	0x00
@@ -1310,6 +1332,11 @@ static inline void input_sync(struct input_dev *dev)
 	input_event(dev, EV_SYN, SYN_REPORT, 0);
 }

+static inline void input_mt_sync(struct input_dev *dev)
+{
+	input_event(dev, EV_SYN, SYN_MT_REPORT, 0);
+}
+
 void input_set_capability(struct input_dev *dev, unsigned int type, unsigned int code);

 static inline void input_set_abs_params(struct input_dev *dev, int axis, int min, int max, int fuzz, int flat)

Re: [PATCH 2/2] input: Add a detailed multi-touch finger data report protocol (rev2)

[Peter Hutterer]

On Sun, Dec 28, 2008 at 11:58:57PM +0100, Henrik Rydberg wrote:
> In order to utilize the full power of the new multi-touch devices, a
> way to report detailed finger data to user space is needed. This patch
> adds a multi-touch (MT) protocol which allows drivers to report details
> for an arbitrary number of fingers.
> 
> The driver sends a SYN_MT_REPORT event via the input_mt_sync() function
> when a complete finger has been reported.
> 
> In order to stay compatible with existing applications, the data
> reported in a finger packet must not be recognized as single-touch
> events. In addition, all finger data must bypass input filtering,
> since subsequent events of the same type refer to different fingers.
> 
> A set of ABS_MT events with the desired properties are defined. The
> events are divided into categories, to allow for partial implementation.
> The minimum set consists of ABS_MT_TOUCH_MAJOR, ABS_MT_POSITION_X and
> ABS_MT_POSITION_Y, which allows for multiple fingers to be tracked.
> If the device supports it, the ABS_MT_WIDTH_MAJOR may be used to provide
> the size of the approaching finger. Anisotropy and direction may be
> specified with ABS_MT_TOUCH_MINOR, ABS_MT_WIDTH_MINOR and
> ABS_MT_ORIENTATION. Devices with more granular information may specify
> general shapes as blobs, i.e., as a sequence of rectangular shapes
> grouped together by a ABS_MT_BLOB_ID. 

To clarify: the MT_TOUCH_MAJOR is to track a touchpoint over time, and the
BLOB_ID to compile a arbitrarily shaped touchpoint within the same event?

Would it make sense to allow specification of a blob as bit/bytemask? There
are a few devices that can do detection of multi-color non-rectangular
touchpoints, especially camera-based systems. Limiting these to a sequence of
rectangles means dropping information that may be useful for certain tasks
(e.g. fingerprint detection).
OTOH, a rectangle as bounding box with an accompanying (optional) bytemask can
pass this data on to prospective clients.

Cheers,
  Peter

Re: [PATCH 2/2] input: Add a detailed multi-touch finger data report protocol (rev2)

[Jim Gettys]

On Mon, 2009-01-05 at 13:57 +1000, Peter Hutterer wrote:
> On Sun, Dec 28, 2008 at 11:58:57PM +0100, Henrik Rydberg wrote:
> > In order to utilize the full power of the new multi-touch devices, a
> > way to report detailed finger data to user space is needed. This patch
> > adds a multi-touch (MT) protocol which allows drivers to report details
> > for an arbitrary number of fingers.
> > 
> > The driver sends a SYN_MT_REPORT event via the input_mt_sync() function
> > when a complete finger has been reported.
> > 
> > In order to stay compatible with existing applications, the data
> > reported in a finger packet must not be recognized as single-touch
> > events. In addition, all finger data must bypass input filtering,
> > since subsequent events of the same type refer to different fingers.
> > 
> > A set of ABS_MT events with the desired properties are defined. The
> > events are divided into categories, to allow for partial implementation.
> > The minimum set consists of ABS_MT_TOUCH_MAJOR, ABS_MT_POSITION_X and
> > ABS_MT_POSITION_Y, which allows for multiple fingers to be tracked.
> > If the device supports it, the ABS_MT_WIDTH_MAJOR may be used to provide
> > the size of the approaching finger. Anisotropy and direction may be
> > specified with ABS_MT_TOUCH_MINOR, ABS_MT_WIDTH_MINOR and
> > ABS_MT_ORIENTATION. Devices with more granular information may specify
> > general shapes as blobs, i.e., as a sequence of rectangular shapes
> > grouped together by a ABS_MT_BLOB_ID. 
> 
> To clarify: the MT_TOUCH_MAJOR is to track a touchpoint over time, and the
> BLOB_ID to compile a arbitrarily shaped touchpoint within the same event?
> 
> Would it make sense to allow specification of a blob as bit/bytemask? There
> are a few devices that can do detection of multi-color non-rectangular
> touchpoints, especially camera-based systems. Limiting these to a sequence of
> rectangles means dropping information that may be useful for certain tasks
> (e.g. fingerprint detection).
> OTOH, a rectangle as bounding box with an accompanying (optional) bytemask can
> pass this data on to prospective clients.

Urk.  I'd forgotten entirely about color.  Some people are certainly
using tokens on surfaces, distinguishable either by shape or color. But
maybe that is better left outside of this until we have a more concrete
non-research system to grok before defining a kernel interface; many of
these systems may be doing advanced image processing in user space
processes and talking directly to X with the result rather than
via /dev/input....  Or do you feel you have good enough experience with
the MERL table and other systems to take a stab at it now?
                               - Jim

-- 
Jim Gettys <jg@laptop.org>
One Laptop Per Child

Re: [PATCH 2/2] input: Add a detailed multi-touch finger data report protocol (rev2)

[Henrik Rydberg]

Peter Hutterer wrote:
[...]
> 
> To clarify: the MT_TOUCH_MAJOR is to track a touchpoint over time, and the
> BLOB_ID to compile a arbitrarily shaped touchpoint within the same event?

MT_TOUCH_MAJOR is the major size of the touching object, or "pressure" in
case the size of the approaching object (MT_WIDTH_MAJOR) is not available
to compare the size to. The touch and position packets are sent to the
multitouch X driver, where the actual finger tracking takes place. Thanks
for the chance to clarify this.


> Would it make sense to allow specification of a blob as bit/bytemask? There
> are a few devices that can do detection of multi-color non-rectangular
> touchpoints, especially camera-based systems. Limiting these to a sequence of
> rectangles means dropping information that may be useful for certain tasks
> (e.g. fingerprint detection).
> OTOH, a rectangle as bounding box with an accompanying (optional) bytemask can
> pass this data on to prospective clients.

Very interesting idea, although I believe it is beyond the scope of the
current patch.

Re: [PATCH 2/2] input: Add a detailed multi-touch finger data report protocol (rev2)

[Jim Gettys]

There is a remaining question: how/whether to implement this interface
in USB HID.....

On page 9 of the document you'll find at:
http://www.microsoft.com/whdc/device/input/DigitizerDrvs_touch.mspx

Microsoft is claiming in that document of December 8 they are proposing
the interfaces in the document to the USB IF organization, and that
hardware vendors for Windows 7 should follow that spec.

Jiri, Greg:

1) Does anyone know if Microsoft has actually made the proposal to the
USB IF?  Is there anyone in the community watching/commenting on USB IF
proposals?

2) do you want generic USB HID patches or do you want to, for now,
confine these to USB device specific drivers as much as possible?
                          - Jim

-- 
Jim Gettys <jg@laptop.org>
One Laptop Per Child

Re: [PATCH 2/2] input: Add a detailed multi-touch finger data report protocol (rev2)

[Greg KH]

On Tue, Jan 06, 2009 at 01:32:49PM -0500, Jim Gettys wrote:
> There is a remaining question: how/whether to implement this interface
> in USB HID.....
> 
> On page 9 of the document you'll find at:
> http://www.microsoft.com/whdc/device/input/DigitizerDrvs_touch.mspx
> 
> Microsoft is claiming in that document of December 8 they are proposing
> the interfaces in the document to the USB IF organization, and that
> hardware vendors for Windows 7 should follow that spec.
> 
> Jiri, Greg:
> 
> 1) Does anyone know if Microsoft has actually made the proposal to the
> USB IF?  Is there anyone in the community watching/commenting on USB IF
> proposals?

Yes, I'm watching them, but nothing is public at this time (if it was
currently under review privately within the USB-IF, I wouldn't be
allowed to tell anyone, sorry.)

> 2) do you want generic USB HID patches or do you want to, for now,
> confine these to USB device specific drivers as much as possible?

As more devices should be showing up with this kind of support, it would
seem to make sense to add it to the HID core, right?

thanks,

greg k-h

Re: [PATCH 2/2] input: Add a detailed multi-touch finger data report protocol (rev2)

[Jim Gettys]

On Tue, 2009-01-06 at 14:21 -0800, Greg KH wrote:

> 
> > 2) do you want generic USB HID patches or do you want to, for now,
> > confine these to USB device specific drivers as much as possible?
> 
> As more devices should be showing up with this kind of support, it would
> seem to make sense to add it to the HID core, right?
> 

Yes, the HID core makes sense to me if we had more than one (partially)
implemented example in captivity and/or more than the unconfirmed pious
claims of one Microsoft document.  Right now, it is less than clear to
me which way to jump.

I'll try to see what may be involved....
                          - Jim

-- 
Jim Gettys <jg@laptop.org>
One Laptop Per Child

Re: [PATCH 2/2] input: Add a detailed multi-touch finger data report protocol (rev2)

[Jiri Kosina]

On Tue, 6 Jan 2009, Jim Gettys wrote:

> 1) Does anyone know if Microsoft has actually made the proposal to the
> USB IF?  Is there anyone in the community watching/commenting on USB IF
> proposals?

This would be question for Greg.

> 2) do you want generic USB HID patches or do you want to, for now, 
> confine these to USB device specific drivers as much as possible?

We already have a lot of drivers for special HID devices that emit usages 
that have not been defined by HID standard. These drivers are independent 
on the underlying transport (USB, Bluetooth). Therefore all you ideally 
need to do is to add just another driver which hooks itself to HID bus and 
performs all that is needed in addition to what generic HID code does. 
There are lot of such drivers in drivers/hid/hid-*.c which you can use as 
examples. Please let me know if you have any further questions.

-- 
Jiri Kosina
SUSE Labs