[Intel-gfx] Plans for i915 GuC Submission with regards to IPTS/ME

[Intel-gfx] Plans for i915 GuC Submission with regards to IPTS/ME

[Maximilian Luz]

Hello,

I am working together with a small team of volunteers on a project that
aims to make Linux usable on Microsoft Surface devices (linux-surface).
The touchscreens of these devices use Intel Precise Touch & Stylus
(IPTS) technology, which makes use of the GPU (through GuC submission)
to process touch input events. Since IPTS is not supported by upstream
Linux, the linux-surface project relies on kernel patches and
out-of-tree drivers to make it work properly under Linux.

The driver we are currently using was written by an Intel employee back
in 2016, and has since been somewhat updated and maintained by
linux-surface.  However, due to how the IPTS hardware works, the driver
has to use GuC submission to setup the connections between the GPU and
the Intel ME / touch controller. This is made possible by patching an
IPTS-specific API into the i915 driver that allows the IPTS driver to
issue commands to the GuC.

Now, starting with Linux 5.3, a bunch of changes were done to the i915
driver that made it impossible for us to use the IPTS driver like we did
before.  First, GuC submission got disabled, and recently there was a
list of patches sent to this mailing list that removes the GuC APIs that
we have been using entirely. We have a rough understanding about how the
hardware works, but for the most part, the connection between the
various parts of IPTS (ME, GuC) is a giant black-box for us.

Currently the way that IPTS and the driver work is like this:

- the IPTS driver allocates a GuC client a doorbell, and a bunch of
   touch input / output buffers, and passes the addresses to the ME

- the driver loads a vendor specific firmware file and uploads it to the
   GPU

- the ME fills up the touch input buffer, and rings the GuC doorbell

- the firmware that was uploaded to the GPU processes the data from the
   input buffer into HID events, and saves them into the output buffers

- the IPTS driver polls for changes in those buffers and relays the
   contents into the kernels HID subsystem

Is there any chance that you have an idea about if it will be possible
to update the IPTS driver that is out there to work with the new GuC API
that is coming into i915, and how we might have to go about that?

As you see, the most part of this happens in the firmware, and the
driver just sets up the messaging framework around it. This makes it
hard for us to think of a possible solution without having a deep
understanding and insight into the hardware.

Another problem is that, except for the very latest generation of
Surface devices that was released some weeks ago, all devices that use
IPTS are either Skylake or Kabylake. While researching about this, we
found some comments, stating that any form of GuC submission might only
be supported on Icelake in the future. A patch
(https://patchwork.freedesktop.org/patch/335793/) on the other hand
reads like it would be re-enabled on all devices. Can you clarify if GuC
is intended to be re-enabled on all (Gen9+) devices?

We hope you can provide us with any help in getting this updated to
support future Linux releases. The Surface devices are pretty good
hardware-wise, and it would be great to continue to use them under Linux
with a working touchscreen.

Regards,
Maximilian Luz

---

Various links that might provide more insight into the IPTS side:

- Original IPTS implementation:
     https://github.com/ipts-linux-org/ipts-linux-new

- Updated version of IPTS, with support* for linux 5.3:
     https://github.com/qzed/linux-surface-kernel/tree/v5.3-surface-devel/drivers/misc/ipts

   * Support for 5.3 is currently done by porting the i915 driver from 5.2 to
     the newer kernel, since it still supports GuC submission. It is an _awful_
     hack but none of us had time to really dive into this issue back then, and
     this seemed like the easiest solution.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] Plans for i915 GuC Submission with regards to IPTS/ME

[Daniele Ceraolo Spurio]

Hi,

I can't comment on the ITPS side since I have never looked at that side 
of things, but I can give you an overview of why we turned off GuC 
submission and what are the short/medium term plans for it.

TL;DR: The GuC submission interface is changed enough that the code you 
have is no longer applicable. We're now focused on implementing support 
for the new interface to re-enable guc submission (gen12+), which is a 
prerequisite for what you need. IMO any ITPS support on the current tree 
will have to be postponed until then.

The disabling of GuC submission was done because recent binaries (v30+) 
introduced significant non-backward compatible changes in the interface; 
given that GuC submission was still not an "official" feature and that 
even more intrusive interface changes are coming with the upcoming GuC 
v40+, we decided to just disable the feature for now and wait for all 
the changes to land on the GuC side before adding support back in. The 
plan is to re-enable support from either gen11 or gen12, so the gen9 
platform will not have it, at least initially. It shouldn't bee too hard 
to add it in though since the great majority of the code is shared on 
both the GuC and the i915 side, so I wouldn't exclude us adding it in if 
there is demand for it.

The v40 firmware includes an almost complete rework of the submission 
interface, which is why, in preparation, we removed support from the 
driver for the legacy structures that are no longer used; we're also not 
going to use HW doorbells anymore and that's why those have been removed 
as well. I've had a look at the code in your github tree and most of the 
things you use are either gone from the interface or have been 
significantly updated, so I don't think there is an easy way to just 
port the patch to the new blobs and a significant rewrite is probably 
going to be required. Re-enabling GuC submission on the new interface, 
which is our current focus on the i915/GuC integration side, is a 
prerequisite for that, so IMO unfortunately you'll have to wait until 
the new interface support lands before any ITPS changes can be considered.

Daniele

On 12/15/19 3:12 AM, Maximilian Luz wrote:
> Hello,
> 
> I am working together with a small team of volunteers on a project that
> aims to make Linux usable on Microsoft Surface devices (linux-surface).
> The touchscreens of these devices use Intel Precise Touch & Stylus
> (IPTS) technology, which makes use of the GPU (through GuC submission)
> to process touch input events. Since IPTS is not supported by upstream
> Linux, the linux-surface project relies on kernel patches and
> out-of-tree drivers to make it work properly under Linux.
> 
> The driver we are currently using was written by an Intel employee back
> in 2016, and has since been somewhat updated and maintained by
> linux-surface.  However, due to how the IPTS hardware works, the driver
> has to use GuC submission to setup the connections between the GPU and
> the Intel ME / touch controller. This is made possible by patching an
> IPTS-specific API into the i915 driver that allows the IPTS driver to
> issue commands to the GuC.
> 
> Now, starting with Linux 5.3, a bunch of changes were done to the i915
> driver that made it impossible for us to use the IPTS driver like we did
> before.  First, GuC submission got disabled, and recently there was a
> list of patches sent to this mailing list that removes the GuC APIs that
> we have been using entirely. We have a rough understanding about how the
> hardware works, but for the most part, the connection between the
> various parts of IPTS (ME, GuC) is a giant black-box for us.
> 
> Currently the way that IPTS and the driver work is like this:
> 
> - the IPTS driver allocates a GuC client a doorbell, and a bunch of
>    touch input / output buffers, and passes the addresses to the ME
> 
> - the driver loads a vendor specific firmware file and uploads it to the
>    GPU
> 
> - the ME fills up the touch input buffer, and rings the GuC doorbell
> 
> - the firmware that was uploaded to the GPU processes the data from the
>    input buffer into HID events, and saves them into the output buffers
> 
> - the IPTS driver polls for changes in those buffers and relays the
>    contents into the kernels HID subsystem
> 
> Is there any chance that you have an idea about if it will be possible
> to update the IPTS driver that is out there to work with the new GuC API
> that is coming into i915, and how we might have to go about that?
> 
> As you see, the most part of this happens in the firmware, and the
> driver just sets up the messaging framework around it. This makes it
> hard for us to think of a possible solution without having a deep
> understanding and insight into the hardware.
> 
> Another problem is that, except for the very latest generation of
> Surface devices that was released some weeks ago, all devices that use
> IPTS are either Skylake or Kabylake. While researching about this, we
> found some comments, stating that any form of GuC submission might only
> be supported on Icelake in the future. A patch
> (https://patchwork.freedesktop.org/patch/335793/) on the other hand
> reads like it would be re-enabled on all devices. Can you clarify if GuC
> is intended to be re-enabled on all (Gen9+) devices?
> 
> We hope you can provide us with any help in getting this updated to
> support future Linux releases. The Surface devices are pretty good
> hardware-wise, and it would be great to continue to use them under Linux
> with a working touchscreen.
> 
> Regards,
> Maximilian Luz
> 
> ---
> 
> Various links that might provide more insight into the IPTS side:
> 
> - Original IPTS implementation:
>      https://github.com/ipts-linux-org/ipts-linux-new
> 
> - Updated version of IPTS, with support* for linux 5.3:
>      
> https://github.com/qzed/linux-surface-kernel/tree/v5.3-surface-devel/drivers/misc/ipts 
> 
> 
>    * Support for 5.3 is currently done by porting the i915 driver from 
> 5.2 to
>      the newer kernel, since it still supports GuC submission. It is an 
> _awful_
>      hack but none of us had time to really dive into this issue back 
> then, and
>      this seemed like the easiest solution.
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] Plans for i915 GuC Submission with regards to IPTS/ME

[Daniele Ceraolo Spurio]

On 12/18/19 10:01 AM, Maximilian Luz wrote:
> Hi,
> 
> On 12/16/19 11:40 PM, Daniele Ceraolo Spurio wrote:
>> Hi,
>>
>> I can't comment on the ITPS side since I have never looked at that
>> side of things, but I can give you an overview of why we turned off
>> GuC submission and what are the short/medium term plans for it.
> 
> By any chance, do you have any idea if it's possible/who we could ask to
> get some more information or some sort of documentation about the
> IPTS/ME/touch-controller interfaces? This could maybe allow us to find
> some sort of work-around not involving GuC.
> 

Unfortunately I have no idea about the docs in this area since it is 
outside the graphics core.

>> TL;DR: The GuC submission interface is changed enough that the code
>> you have is no longer applicable. We're now focused on implementing
>> support for the new interface to re-enable guc submission (gen12+),
>> which is a prerequisite for what you need. IMO any ITPS support on the
>> current tree will have to be postponed until then.
>>
>> The disabling of GuC submission was done because recent binaries
>> (v30+) introduced significant non-backward compatible changes in the
>> interface; given that GuC submission was still not an "official"
>> feature and that even more intrusive interface changes are coming with
>> the upcoming GuC v40+, we decided to just disable the feature for now
>> and wait for all the changes to land on the GuC side before adding
>> support back in. The plan is to re-enable support from either gen11 or
>> gen12, so the gen9 platform will not have it, at least initially. It
>> shouldn't bee too hard to add it in though since the great majority of
>> the code is shared on both the GuC and the i915 side, so I wouldn't
>> exclude us adding it in if there is demand for it.
> 
> Thank you, this is quite helpful. As all devices that I know of using
> IPTS are gen9, GuC support for that generation would be great to have.
> In the mean-time however, I think we will have to try and figure out if
> we can find a work-around, maybe bypassing GuC somehow.

If you only care about gen9, a possible solution would be to 
forward-port just the old GuC submission (instead of the whole i915) 
from a know working kernel. It shouldn't be too bad since the GuC code 
is relatively self-contained, but given the speed at which our 
submission code evolves there might be issue in the interactions between 
the old GuC code and the other parts of the submission flow.

> 
> Just out of curiosity: Are the firmware-changes also done on Windows or
> is this purely Linux specific? Would be interesting to know if they have
> the same problem.
> 

No idea about the details of what goes on on Windows. The firmware is 
OS-agnostic, but they might be using a different version compared to us, 
especially on older platforms.

Daniele

>> The v40 firmware includes an almost complete rework of the submission
>> interface, which is why, in preparation, we removed support from the
>> driver for the legacy structures that are no longer used; we're also
>> not going to use HW doorbells anymore and that's why those have been
>> removed as well. I've had a look at the code in your github tree and
>> most of the things you use are either gone from the interface or have
>> been significantly updated, so I don't think there is an easy way to
>> just port the patch to the new blobs and a significant rewrite is
>> probably going to be required. Re-enabling GuC submission on the new
>> interface, which is our current focus on the i915/GuC integration
>> side, is a prerequisite for that, so IMO unfortunately you'll have to
>> wait until the new interface support lands before any ITPS changes can
>> be considered.
> 
> I suspected that we'll likely have to do a substantial re-write of the
> driver, but it's nice to have confirmation for that so we can at least
> plan a bit ahead.
> 
> Thank you again,
> Maximilian
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Re: [Intel-gfx] Plans for i915 GuC Submission with regards to IPTS/ME

[Maximilian Luz]

Hi,

On 12/23/19 7:55 PM, Daniele Ceraolo Spurio wrote:
> If you only care about gen9, a possible solution would be to
> forward-port just the old GuC submission (instead of the whole i915)
> from a know working kernel. It shouldn't be too bad since the GuC code
> is relatively self-contained, but given the speed at which our
> submission code evolves there might be issue in the interactions
> between the old GuC code and the other parts of the submission flow.

Right, we may have to look at that. We're currently looking into a
non-GuC workaround as there doesn't seem to be any OpenCL kernels for
the Surface Pro 7/Surface Laptop 3 devices (that's another issue...),
but we will keep that in mind for the older devices.

> No idea about the details of what goes on on Windows. The firmware is
> OS-agnostic, but they might be using a different version compared to
> us, especially on older platforms.

Yeah, I suspect they're using older firmware on the older devices. Just
wanted to know if there's any special stuff on Windows. Given that the
newer devices (Pro 7/Laptop 3) don't seem to have any IPTS firmware at
all (OpenCL kernels etc.) could suggest that they might use a different
implementation (maybe non-GuC/no GPU processing), altough the IPTS
driver still seems to be in use.

Thank you for sharing your insights, this has really helped us.

Regards,
Maximilian
_______________________________________________
Intel-gfx mailing list
Intel-gfx@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/intel-gfx