[RFP] iio: Support of gesture sensor as a standard IIO sensor

[RFP] iio: Support of gesture sensor as a standard IIO sensor

[Pandruvada, Srinivas]

Hi Jonathan and Everyone,

The Intel Integrated sensor hub (ISH) on some platforms have ability to
identify gestures. This is available on some Android platforms. We want
to bring in this capability to other Linux flavors also. 

This algorithm for gesture recognition executes in ISH as this is a
computation intensive and need to be running even when the main
platform CPU goes to sleep.

We already have ISH support available in upstream Linux for a while.
The standard sensors in ISH like Accelerometer, Gyro, ALS etc already
exposed via Linux IIO. Several Laptops and 2-in-1s already using some
features like screen rotation, auto back light adjustment etc. We can
also add gestures. For example
-device wake up from deep power save state once device is picked up
-User can send device to sleep by simple hand gestures
-Some games can utilize gestures to be more user friendly	

The proposal here is to expose gesture sensor via standard IIO
interface.

A typical gesture recognition process involves:

Raw Data -> Preprocessing -> Feature Extraction -> Template match ->
Matched activity

This processing is done in ISH. Intel provides a content sensing SDK
[1] to develop applications. There are already some standard gestures
patterns already in built in the ISH for Android devices. The SDK
documentation provides details about them. Also there are user defined
gestures which essentially are templates for matching and needs to be
downloaded to ISH.

From an IIO sensor point of view A Gesture sensor:
Outputs
	A pre defined activity type
		WAKE
		TILT
		GLANCE
		PICK_UP
		 &
		 more

	A user defined activity type as "string"

Inputs
	A raw binary cdev interface to download templates/patterns


I want to gather more opinions before submitting a RFC patch.

References
1. https://software.intel.com/en-us/documentation/context-sensing-sdk-f
or-android-developer-guide
2. https://software.intel.com/en-us/documentation/using-gesture-recogni
tion-as-differentiation-feature-on-android

Thanks,
Srinivas

Re: [RFP] iio: Support of gesture sensor as a standard IIO sensor

[Alan Cox]

> >From an IIO sensor point of view A Gesture sensor:  
> Outputs
> 	A pre defined activity type
> 		WAKE
> 		TILT
> 		GLANCE
> 		PICK_UP
> 		 &
> 		 more
> 
> 	A user defined activity type as "string"
> 
> Inputs
> 	A raw binary cdev interface to download templates/patterns
> 
> 
> I want to gather more opinions before submitting a RFC patch.

The only question I have is should it appear under IIO or should it be an
input event interface. It feels to me more like an input device in that in
this case while it's not keys or joystick it is still 'please do X'. That
might also make it much easier (in the non-Android space in particular)
to bind these activities to actions in things like web browsers.

(+ linux-input)

Alan

Re: [RFP] iio: Support of gesture sensor as a standard IIO sensor

[Jonathan Cameron]

On Sat, 6 Jan 2018 00:20:24 +0000
Alan Cox <gnomes@lxorguk.ukuu.org.uk> wrote:

> > >From an IIO sensor point of view A Gesture sensor:    
> > Outputs
> > 	A pre defined activity type
> > 		WAKE
> > 		TILT
> > 		GLANCE
> > 		PICK_UP
> > 		 &
> > 		 more
> > 
> > 	A user defined activity type as "string"
> > 
> > Inputs
> > 	A raw binary cdev interface to download templates/patterns
> > 
> > 
> > I want to gather more opinions before submitting a RFC patch.  
> 
> The only question I have is should it appear under IIO or should it be an
> input event interface. It feels to me more like an input device in that in
> this case while it's not keys or joystick it is still 'please do X'. That
> might also make it much easier (in the non-Android space in particular)
> to bind these activities to actions in things like web browsers.
> 
I agree that this may well be an option for many of the gestures specifically
metioned (flicks etc and glyphs).

However, there are other obvious uses of this technology such as step detection
or activity classification (running, walking sitting) that so far have fallen
in the scope of IIO as they aren't really things you expect the device to
perform an an action in response to.  Another one of those messy corners
that fall through the gaps!

The drivers/iio/accel/mma9553.c does activity detection, but that isn't
really 'events' in the same way as we have here...

So right answer might be a hybrid of an underlying flexible IIO device
and an input front end for when it makes sense.

We probably need to get the in kernel use of IIO events sorted.  Non event
stuff has been sorted for years, but this last corner was never of enough
interest to anyone to actually implement it (it's fairly straight forward
to do).

> (+ linux-input)
> 
> Alan

Re: [RFP] iio: Support of gesture sensor as a standard IIO sensor

[Pandruvada, Srinivas]

On Sat, 2018-01-06 at 13:12 +0000, Jonathan Cameron wrote:
> On Sat, 6 Jan 2018 00:20:24 +0000
> Alan Cox <gnomes@lxorguk.ukuu.org.uk> wrote:
> 
> > 
> > > 
> > > > 
> > > > From an IIO sensor point of view A Gesture sensor:    
> > > Outputs
> > > 	A pre defined activity type
> > > 		WAKE
> > > 		TILT
> > > 		GLANCE
> > > 		PICK_UP
> > > 		 &
> > > 		 more
> > > 
> > > 	A user defined activity type as "string"
> > > 
> > > Inputs
> > > 	A raw binary cdev interface to download templates/patterns
> > > 
> > > 
> > > I want to gather more opinions before submitting a RFC patch.  
> > The only question I have is should it appear under IIO or should it
> > be an
> > input event interface. It feels to me more like an input device in
> > that in
> > this case while it's not keys or joystick it is still 'please do
> > X'. That
> > might also make it much easier (in the non-Android space in
> > particular)
> > to bind these activities to actions in things like web browsers.
> > 
> I agree that this may well be an option for many of the gestures
> specifically
> metioned (flicks etc and glyphs).
> 
> However, there are other obvious uses of this technology such as step
> detection
> or activity classification (running, walking sitting) that so far
> have fallen
> in the scope of IIO as they aren't really things you expect the
> device to
> perform an an action in response to.  Another one of those messy
> corners
> that fall through the gaps!
> 
> The drivers/iio/accel/mma9553.c does activity detection, but that
> isn't
> really 'events' in the same way as we have here...
> 
> So right answer might be a hybrid of an underlying flexible IIO
> device
> and an input front end for when it makes sense.

What about we can also send an input event, if the event is one of the
input event type which is defined in uapi/linux/input-event-codes.h?


> 
> We probably need to get the in kernel use of IIO events sorted.  Non
> event
> stuff has been sorted for years, but this last corner was never of
> enough
> interest to anyone to actually implement it (it's fairly straight
> forward
> to do).
Currently in IIO event is a u64 value in Fifo. But here we need some
user defined events also. Since this ABI may already may be in use so
can't change u64 to a structure with data type definition and
associated data. But we can define additional predefined ids for custom
events (Similar to HID sensor usage spec added CUSTOM usage ids).

Thanks,
Srinivas

> 
> > 
> > (+ linux-input)
> > 
> > Alan

Re: [RFP] iio: Support of gesture sensor as a standard IIO sensor

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 662 bytes --]

Hi!

> From an IIO sensor point of view A Gesture sensor:
> Outputs
> 	A pre defined activity type
> 		WAKE
> 		TILT
> 		GLANCE
> 		PICK_UP
> 		 &
> 		 more
> 
> 	A user defined activity type as "string"

Pre-defined activities are easy.

But what about user-defined activities? We'd really like common
interface across different hardware... 

> Inputs
> 	A raw binary cdev interface to download templates/patterns

....and "raw binary" will not work across different hardware :-(.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [RFP] iio: Support of gesture sensor as a standard IIO sensor

[Jonathan Cameron]

On Thu, 18 Jan 2018 23:40:26 +0100
Pavel Machek <pavel@ucw.cz> wrote:

> Hi!
> 
> > From an IIO sensor point of view A Gesture sensor:
> > Outputs
> > 	A pre defined activity type
> > 		WAKE
> > 		TILT
> > 		GLANCE
> > 		PICK_UP
> > 		 &
> > 		 more
> > 
> > 	A user defined activity type as "string"  
> 
> Pre-defined activities are easy.
> 
> But what about user-defined activities? We'd really like common
> interface across different hardware... 

Nasty to handle indeed.   It may be the best we can do initially
at least is user_definedN or similar.  No way of constraining
users from uploading something really odd that we can't define
an interface for (hopping whilst holding a phone in
their teeth?)
> 
> > Inputs
> > 	A raw binary cdev interface to download templates/patterns  
> 
> ....and "raw binary" will not work across different hardware :-(.

Sadly there probably isn't much we can do about the format being custom.
Best bet would ultimately be if there was at least a standard tool
to generate the files for different devices...

> 
> 									Pavel

Re: [RFP] iio: Support of gesture sensor as a standard IIO sensor

[Pandruvada, Srinivas]

On Sat, 2018-01-20 at 15:19 +0000, Jonathan Cameron wrote:
> On Thu, 18 Jan 2018 23:40:26 +0100
> Pavel Machek <pavel@ucw.cz> wrote:
> 
> > 
> > Hi!
> > 
> > > 
> > > From an IIO sensor point of view A Gesture sensor:
> > > Outputs
> > > 	A pre defined activity type
> > > 		WAKE
> > > 		TILT
> > > 		GLANCE
> > > 		PICK_UP
> > > 		 &
> > > 		 more
> > > 
> > > 	A user defined activity type as "string"  
> > Pre-defined activities are easy.
> > 
> > But what about user-defined activities? We'd really like common
> > interface across different hardware... 
> Nasty to handle indeed.   It may be the best we can do initially
> at least is user_definedN or similar.  No way of constraining
> users from uploading something really odd that we can't define
> an interface for (hopping whilst holding a phone in
> their teeth?)
That's what I think also. We pre define few USER defined activities
Ids.

> > 
> > 
> > > 
> > > Inputs
> > > 	A raw binary cdev interface to download templates/patterns  
> > ....and "raw binary" will not work across different hardware :-(.
> Sadly there probably isn't much we can do about the format being
> custom.
> Best bet would ultimately be if there was at least a standard tool
> to generate the files for different devices...
> 
Correct. This will depend on  the firmware expected format . I think
ultimately it will be standardized.

Thanks,
Srinivas

> > 
> > 
> > 									
> > Pavel

Re: [RFP] iio: Support of gesture sensor as a standard IIO sensor

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 1914 bytes --]

On Sat 2018-01-20 20:31:23, Pandruvada, Srinivas wrote:
> On Sat, 2018-01-20 at 15:19 +0000, Jonathan Cameron wrote:
> > On Thu, 18 Jan 2018 23:40:26 +0100
> > Pavel Machek <pavel@ucw.cz> wrote:
> > 
> > > 
> > > Hi!
> > > 
> > > > 
> > > > From an IIO sensor point of view A Gesture sensor:
> > > > Outputs
> > > > 	A pre defined activity type
> > > > 		WAKE
> > > > 		TILT
> > > > 		GLANCE
> > > > 		PICK_UP
> > > > 		 &
> > > > 		 more
> > > > 
> > > > 	A user defined activity type as "string"  
> > > Pre-defined activities are easy.
> > > 
> > > But what about user-defined activities? We'd really like common
> > > interface across different hardware... 
> > Nasty to handle indeed.   It may be the best we can do initially
> > at least is user_definedN or similar.  No way of constraining
> > users from uploading something really odd that we can't define
> > an interface for (hopping whilst holding a phone in
> > their teeth?)
> That's what I think also. We pre define few USER defined activities
> Ids.

Ok, so this is really device for human interaction -- input device.

> > > > Inputs
> > > > 	A raw binary cdev interface to download templates/patterns  
> > > ....and "raw binary" will not work across different hardware :-(.
> > Sadly there probably isn't much we can do about the format being
> > custom.
> > Best bet would ultimately be if there was at least a standard tool
> > to generate the files for different devices...
> > 
> Correct. This will depend on  the firmware expected format . I think
> ultimately it will be standardized.

Is it fair to say that "expected" format is some kind of bytecode for
CPU in the gesture sensor?

We have similar problem with smart LED devices...

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [RFP] iio: Support of gesture sensor as a standard IIO sensor

[Pandruvada, Srinivas]

On Sat, 2018-01-27 at 23:09 +0100, Pavel Machek wrote:
> On Sat 2018-01-20 20:31:23, Pandruvada, Srinivas wrote:
> > 
> > On Sat, 2018-01-20 at 15:19 +0000, Jonathan Cameron wrote:
> > > 
> > > On Thu, 18 Jan 2018 23:40:26 +0100
> > > Pavel Machek <pavel@ucw.cz> wrote:
> > > 
> > > > 
> > > > 
> > > > Hi!
> > > > 
> > > > > 
> > > > > 
> > > > > From an IIO sensor point of view A Gesture sensor:
> > > > > Outputs
> > > > > 	A pre defined activity type
> > > > > 		WAKE
> > > > > 		TILT
> > > > > 		GLANCE
> > > > > 		PICK_UP
> > > > > 		 &
> > > > > 		 more
> > > > > 
> > > > > 	A user defined activity type as "string"  
> > > > Pre-defined activities are easy.
> > > > 
> > > > But what about user-defined activities? We'd really like common
> > > > interface across different hardware... 
> > > Nasty to handle indeed.   It may be the best we can do initially
> > > at least is user_definedN or similar.  No way of constraining
> > > users from uploading something really odd that we can't define
> > > an interface for (hopping whilst holding a phone in
> > > their teeth?)
> > That's what I think also. We pre define few USER defined activities
> > Ids.
> Ok, so this is really device for human interaction -- input device.
> 
> > 
> > > 
> > > > 
> > > > > 
> > > > > Inputs
> > > > > 	A raw binary cdev interface to download
> > > > > templates/patterns  
> > > > ....and "raw binary" will not work across different hardware :-
> > > > (.
> > > Sadly there probably isn't much we can do about the format being
> > > custom.
> > > Best bet would ultimately be if there was at least a standard
> > > tool
> > > to generate the files for different devices...
> > > 
> > Correct. This will depend on  the firmware expected format . I
> > think
> > ultimately it will be standardized.
> Is it fair to say that "expected" format is some kind of bytecode for
> CPU in the gesture sensor?
Not necessarily a byte code to execute on the CPU in the gesture
sensor, but a format the code running on that CPU understands the way
gestures are formatted.
But it can be byte code for some manufacturers, so idea is just pass to
the firmware and let it process it an opaque format.

Thanks,
Srinivas

> 
> We have similar problem with smart LED devices...
> 
> 									
> Pavel

Re: [RFP] iio: Support of gesture sensor as a standard IIO sensor

[Pavel Machek]

[-- Attachment #1: Type: text/plain, Size: 839 bytes --]

hi!

> > > Correct. This will depend on  the firmware expected format . I
> > > think
> > > ultimately it will be standardized.
> > Is it fair to say that "expected" format is some kind of bytecode for
> > CPU in the gesture sensor?
> Not necessarily a byte code to execute on the CPU in the gesture
> sensor, but a format the code running on that CPU understands the way
> gestures are formatted.
> But it can be byte code for some manufacturers, so idea is just pass to
> the firmware and let it process it an opaque format.

It would be really good to figure out common format for different
sensors. Otherwise kernel is not providing proper hardware
abstraction.

									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

Re: [RFP] iio: Support of gesture sensor as a standard IIO sensor

[Jonathan Cameron]

On Sun, 28 Jan 2018 09:40:03 +0100
Pavel Machek <pavel@ucw.cz> wrote:

> hi!
> 
> > > > Correct. This will depend on  the firmware expected format . I
> > > > think
> > > > ultimately it will be standardized.  
> > > Is it fair to say that "expected" format is some kind of bytecode for
> > > CPU in the gesture sensor?  
> > Not necessarily a byte code to execute on the CPU in the gesture
> > sensor, but a format the code running on that CPU understands the way
> > gestures are formatted.
> > But it can be byte code for some manufacturers, so idea is just pass to
> > the firmware and let it process it an opaque format.  
> 
> It would be really good to figure out common format for different
> sensors. Otherwise kernel is not providing proper hardware
> abstraction.
> 
> 									Pavel

A common format, if it is byte code, would require in kernel compilers
for the various sensors.  Not going to happen.

Common format for everything other than the 'load magic firmware' part
might be doable. As in we can describe what the magic firmware is
doing...

Jonathan