[RFC] LE: Low Latency GATT Write-Sign-Cmd

[RFC] LE: Low Latency GATT Write-Sign-Cmd

[Brian Gix]

Hi All,

One of the useful features of Low Energy is the ability to rapidly 
establish a connection, send a responseless Signed Write Command, and 
tear the connection down.  The obvious use case for this would be a 
standard remote control, where extremely low latency is expected.

The difference between this use case, and something like one of the 
Health profiles, is that a Health device (Heart rate Monitor, or 
Thermometer) is that where the Health device would have a persistent 
connection with high latency (less frequent Beacons), the Remote Control 
use case would rarely be connected, but when it is it will be as low 
latency as possible.

Currently we do not support Write Signed Commands at all, although I 
hope to have that addressed in the not too distant future.  However the 
nature of it's usage model is not very well established here 
(BlueZ/Linux) so I would like to float a few ideas.


This is what should happen when the user hits a button on his remote 
control (or in his remote control linux APP):

1. Request LE Connection with Low Latency Parameters
2. Compose & sign the GATT Write Cmd (in parallel with Connect Req)
3. When Successful Connect Cmplt Evt occurs, send the Write Cmd Pkt
4. WAIT for the Num Completed Pkts Evt (VERY IMPORTANT)
5. As soon as you have it, Disconnect

This use case is only valid for LE (GATT connections must be Secure on 
BR/EDR, and therefore will never be as fast, and will never in fact even 
need to be signed).  If done properly, the entire sequence can be 
completed in the 10's of milliseconds, from button press to Disconnect.

This calls for a different thread of operation from current GATT 
operation for a few reasons:

1. Standard GATT connections will have different Connect Parameters

2. Standard GATT connection establishment requires on a round trip from
User space--> Kernel --> Baseband and back before the ACL packet is even 
written to the Socket in User Space.

3. The signing crypto code is in the Kernel, and must be applied to each 
signed write command in a way that suggests some LE-GATT-Write-Cmd 
knowledge must exist in the kernel, as opposed to being a simple ACL pipe.

4. There is no current mechanism for the Kernel to inform the User space 
code that the ACL packet has finished transmission, and that it is OK to 
disconnect.

5. We already special case the handling of the LE-L2Cap CIDs for 
Signaling and SMP.


So I would like to suggest the following straw-man:

A new LE only MGMT command, which accepts as parameters the Remote 
Address, a length and a buffer.  The buffer would be reasonable sized (I 
would recommend 27 bytes, which is the default LE MTU+HDR). The byte 
stream itself could either be composed by the User space GATT code 
(L2CAP header, Attribute Handle, Write Data) or they could be structure 
fields in the MGMT packet. On the Kernel side, the MGMT code would 
shepherd the packet through the connection, signing, and send process. 
If the connection to the remote device was already established, the 
existing connection would be used, the signing would only be done if the 
link was not encrypted, and the connection would be left up. When the 
Num-Completed-Pkts evt arrives, send the MGMT cmd_status response.

I think this would be way cleaner than trying to wedge this use case 
into our current GATT usage model.


Thoughts?



-- 
Brian Gix
bgix@codeaurora.org
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

Re: [RFC] LE: Low Latency GATT Write-Sign-Cmd

[Marcel Holtmann]

Hi Brian,

> One of the useful features of Low Energy is the ability to rapidly 
> establish a connection, send a responseless Signed Write Command, and 
> tear the connection down.  The obvious use case for this would be a 
> standard remote control, where extremely low latency is expected.
> 
> The difference between this use case, and something like one of the 
> Health profiles, is that a Health device (Heart rate Monitor, or 
> Thermometer) is that where the Health device would have a persistent 
> connection with high latency (less frequent Beacons), the Remote Control 
> use case would rarely be connected, but when it is it will be as low 
> latency as possible.
> 
> Currently we do not support Write Signed Commands at all, although I 
> hope to have that addressed in the not too distant future.  However the 
> nature of it's usage model is not very well established here 
> (BlueZ/Linux) so I would like to float a few ideas.
> 
> 
> This is what should happen when the user hits a button on his remote 
> control (or in his remote control linux APP):
> 
> 1. Request LE Connection with Low Latency Parameters
> 2. Compose & sign the GATT Write Cmd (in parallel with Connect Req)
> 3. When Successful Connect Cmplt Evt occurs, send the Write Cmd Pkt
> 4. WAIT for the Num Completed Pkts Evt (VERY IMPORTANT)
> 5. As soon as you have it, Disconnect
> 
> This use case is only valid for LE (GATT connections must be Secure on 
> BR/EDR, and therefore will never be as fast, and will never in fact even 
> need to be signed).  If done properly, the entire sequence can be 
> completed in the 10's of milliseconds, from button press to Disconnect.
> 
> This calls for a different thread of operation from current GATT 
> operation for a few reasons:
> 
> 1. Standard GATT connections will have different Connect Parameters
> 
> 2. Standard GATT connection establishment requires on a round trip from
> User space--> Kernel --> Baseband and back before the ACL packet is even 
> written to the Socket in User Space.
> 
> 3. The signing crypto code is in the Kernel, and must be applied to each 
> signed write command in a way that suggests some LE-GATT-Write-Cmd 
> knowledge must exist in the kernel, as opposed to being a simple ACL pipe.

the kernel crypto can be easily used from userspace. See PF_ALG.

So the real question is what's the price of a context switch for us?

> 4. There is no current mechanism for the Kernel to inform the User space 
> code that the ACL packet has finished transmission, and that it is OK to 
> disconnect.
> 
> 5. We already special case the handling of the LE-L2Cap CIDs for 
> Signaling and SMP.
> 
> 
> So I would like to suggest the following straw-man:
> 
> A new LE only MGMT command, which accepts as parameters the Remote 
> Address, a length and a buffer.  The buffer would be reasonable sized (I 
> would recommend 27 bytes, which is the default LE MTU+HDR). The byte 
> stream itself could either be composed by the User space GATT code 
> (L2CAP header, Attribute Handle, Write Data) or they could be structure 
> fields in the MGMT packet. On the Kernel side, the MGMT code would 
> shepherd the packet through the connection, signing, and send process. 
> If the connection to the remote device was already established, the 
> existing connection would be used, the signing would only be done if the 
> link was not encrypted, and the connection would be left up. When the 
> Num-Completed-Pkts evt arrives, send the MGMT cmd_status response.
> 
> I think this would be way cleaner than trying to wedge this use case 
> into our current GATT usage model.

I think one of the most important questions that we have to ask
ourselves at some point is if we wanna put ATT into the kernel.

The potential candidate that forces us to think about this is HID over
Low Energy. However I like to see numbers on how the context switches
with keeping ATT in userspace will effect our latency.

Regards

Marcel

Re: [RFC] LE: Low Latency GATT Write-Sign-Cmd

[Brian Gix]

Hi Marcel,

On 11/18/2011 11:08 AM, Marcel Holtmann wrote:
> the kernel crypto can be easily used from userspace. See PF_ALG.
>
> So the real question is what's the price of a context switch for us?
>

The timing to hit the earliest slot possible is what most concerns me. 
Although as Master we control the initial connection parameters, we 
really want the ACL packet ready to go as soon as we know the connection 
is up.

There is also the question about depending on positive verification from 
the Baseband that the packet has been sent (Num Cmplt Pkts Evt) before 
issuing the Disconnect HCI command.  An early problem that I had with 
gatttool (before the interactive enhancement) was that even if the ACL 
pkt had been properly sent over the HCI link, the Disconnect cmd was 
preempting it, and the remote device would never receive the ACL pkt.

[....]
>> I think this would be way cleaner than trying to wedge this use case
>> into our current GATT usage model.
>
> I think one of the most important questions that we have to ask
> ourselves at some point is if we wanna put ATT into the kernel.
>
> The potential candidate that forces us to think about this is HID over
> Low Energy. However I like to see numbers on how the context switches
> with keeping ATT in userspace will effect our latency.


I would support migrating ATT to the kernel.  In fact another issue I 
have dealt with is ensuring during Pairing, that we are able to at least 
respond with minimal Error responses if someone tries to read our name, 
and there is no GATT client socket to user space to respond.

-- 
Brian Gix
bgix@codeaurora.org
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

Re: [RFC] LE: Low Latency GATT Write-Sign-Cmd

[Johan Hedberg]

Hi Brian,

On Fri, Nov 18, 2011, Brian Gix wrote:
> I would support migrating ATT to the kernel.

Same here.

> In fact another issue I have dealt with is ensuring during Pairing,
> that we are able to at least respond with minimal Error responses if
> someone tries to read our name, and there is no GATT client socket to
> user space to respond.

This issue was already encountered during the last UPF and INdT has a
fix for it. For whatever reason that fix doesn't seem to have showed up
here on the mailing list for upstreaming. In general the capability of
doing ATT signaling should be available as soon as we have a connection
handle, and it should be independent of any ongoing SMP procedure. Only
if the characteristic accessed requires higher security than is
currently provided by the LE link should the ATT procedures fail with an
error. In the use case you describe, since the Name characteristic
doesn't have any special security requirements a request for it should
be always responded to with a positive reply regardless of the current
level of security or if there's an ongoing SMP operation.

Johan

Re: [RFC] LE: Low Latency GATT Write-Sign-Cmd

[Brian Gix]

Hi Johan,

On 11/18/2011 12:12 PM, Johan Hedberg wrote:
> Hi Brian,
>
> On Fri, Nov 18, 2011, Brian Gix wrote:
>> I would support migrating ATT to the kernel.
>
> Same here.
>
>> In fact another issue I have dealt with is ensuring during Pairing,
>> that we are able to at least respond with minimal Error responses if
>> someone tries to read our name, and there is no GATT client socket to
>> user space to respond.
>
> This issue was already encountered during the last UPF and INdT has a
> fix for it. For whatever reason that fix doesn't seem to have showed up
> here on the mailing list for upstreaming. In general the capability of
> doing ATT signaling should be available as soon as we have a connection
> handle, and it should be independent of any ongoing SMP procedure. Only
> if the characteristic accessed requires higher security than is
> currently provided by the LE link should the ATT procedures fail with an
> error. In the use case you describe, since the Name characteristic
> doesn't have any special security requirements a request for it should
> be always responded to with a positive reply regardless of the current
> level of security or if there's an ongoing SMP operation.


I have the solution to return an error that I could upstream if needed. 
  It worked sufficiently well to get past the UPF testing scenario's 
where it was an issue, but as you point out, it should never return an 
error in the first place.  However, as a stop-gap temporary solution, it 
works quite well.

-- 
Brian Gix
bgix@codeaurora.org
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

Re: [RFC] LE: Low Latency GATT Write-Sign-Cmd

[Anderson Lizardo]

Hi Marcel,

On Fri, Nov 18, 2011 at 3:08 PM, Marcel Holtmann <marcel@holtmann.org> wrote:
> I think one of the most important questions that we have to ask
> ourselves at some point is if we wanna put ATT into the kernel.
>
> The potential candidate that forces us to think about this is HID over
> Low Energy. However I like to see numbers on how the context switches
> with keeping ATT in userspace will effect our latency.

I still fail to see how ATT handling in kernel would reduce context
switches. A GATT operation is composed of one or more (possibly many,
see e.g. discovery procedures) ATT PDUs.

Unless you are proposing GATT on kernel as well?

Regards,
-- 
Anderson Lizardo
Instituto Nokia de Tecnologia - INdT
Manaus - Brazil

Re: [RFC] LE: Low Latency GATT Write-Sign-Cmd

[Brian Gix]

Hi Anderson,

On 11/18/2011 12:36 PM, Anderson Lizardo wrote:
> Hi Marcel,
>
> On Fri, Nov 18, 2011 at 3:08 PM, Marcel Holtmann<marcel@holtmann.org>  wrote:
>> I think one of the most important questions that we have to ask
>> ourselves at some point is if we wanna put ATT into the kernel.
>>
>> The potential candidate that forces us to think about this is HID over
>> Low Energy. However I like to see numbers on how the context switches
>> with keeping ATT in userspace will effect our latency.
>
> I still fail to see how ATT handling in kernel would reduce context
> switches. A GATT operation is composed of one or more (possibly many,
> see e.g. discovery procedures) ATT PDUs.
>
> Unless you are proposing GATT on kernel as well?

I don't think having GATT in the kernel is necessarily the logical 
conclusion to an ATT-to-kernel migration.

Remember that the focus of LE is *not* the discovery process.  Discovery 
(of Devices, Services and Characteristics) is important, but what makes 
LE "Low Energy" is what happens *after* pairing and discovery have been 
completed, which ideally would happen One Time Only.

The LE profiles and services are being spec'd so that they can be highly 
efficient over the lifetime of the Low Energy device.  The Linux side of 
the equation will rarely be an actual Low Energy device:  It will just 
know how to talk to an LE device in such a way as to make that 
Button-cell battery driven device last a couple years if possible.

For that reason MOST day-to-day communication between a Linux based 
"Central" device will in fact be single-ATT procedures.  If it takes 
more energy and context switches to do discovery using many compound 
procedures, then we pay that price, because it should only happens Once.

HID devices in particular, will almost certainly NOT use compound GATT 
procedures to communicate after the pairing and discovery point in time 
has past.


-- 
Brian Gix
bgix@codeaurora.org
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

Re: [RFC] LE: Low Latency GATT Write-Sign-Cmd

[Anderson Lizardo]

Hi Brian,

On Fri, Nov 18, 2011 at 4:48 PM, Brian Gix <bgix@codeaurora.org> wrote:
> Hi Anderson,
>
> On 11/18/2011 12:36 PM, Anderson Lizardo wrote:
>>
>> Hi Marcel,
>>
>> On Fri, Nov 18, 2011 at 3:08 PM, Marcel Holtmann<marcel@holtmann.org>
>>  wrote:
>>>
>>> I think one of the most important questions that we have to ask
>>> ourselves at some point is if we wanna put ATT into the kernel.
>>>
>>> The potential candidate that forces us to think about this is HID over
>>> Low Energy. However I like to see numbers on how the context switches
>>> with keeping ATT in userspace will effect our latency.
>>
>> I still fail to see how ATT handling in kernel would reduce context
>> switches. A GATT operation is composed of one or more (possibly many,
>> see e.g. discovery procedures) ATT PDUs.
>>
>> Unless you are proposing GATT on kernel as well?
>
> I don't think having GATT in the kernel is necessarily the logical
> conclusion to an ATT-to-kernel migration.

I was actually referring only to the context switch topic. You still
need to send the ATT request parameters from userspace, which is not
different from building the PDU on userspace (vs. building on kernel).

Unless I'm not understanding what Marcel meant with "ATT on kernel"
(which is most probably the case).

> Remember that the focus of LE is *not* the discovery process.  Discovery (of
> Devices, Services and Characteristics) is important, but what makes LE "Low
> Energy" is what happens *after* pairing and discovery have been completed,
> which ideally would happen One Time Only.

The discovery procedure was only an example of multiple PDU operation.
Take e.g. "read long". If only ATT is on kernel, it still need
multiple context switches for the multiple PDUs involved for the GATT
operation.

> The LE profiles and services are being spec'd so that they can be highly
> efficient over the lifetime of the Low Energy device.  The Linux side of the
> equation will rarely be an actual Low Energy device:  It will just know how
> to talk to an LE device in such a way as to make that Button-cell battery
> driven device last a couple years if possible.
>
> For that reason MOST day-to-day communication between a Linux based
> "Central" device will in fact be single-ATT procedures.  If it takes more
> energy and context switches to do discovery using many compound procedures,
> then we pay that price, because it should only happens Once.

... and for this reason I don't see much gain in reducing context
switches if *only* ATT is on kernel. (note: I'm not defending GATT on
kernel).

> HID devices in particular, will almost certainly NOT use compound GATT
> procedures to communicate after the pairing and discovery point in time has
> past.

Yes, and also I don't see why LE HID is an example of "many context
switches". For sure, you will need a bunch of them for setting up the
HID device (discovery + pairing), but after that it will "talk" HID
with the kernel input subsystem (If I understood the basics of this
profile).

Regards,
-- 
Anderson Lizardo
Instituto Nokia de Tecnologia - INdT
Manaus - Brazil

Re: [RFC] LE: Low Latency GATT Write-Sign-Cmd

[Brian Gix]

Hi Anderson,

On 11/18/2011 12:58 PM, Anderson Lizardo wrote:
> On Fri, Nov 18, 2011 at 4:48 PM, Brian Gix<bgix@codeaurora.org>  wrote:
>> On 11/18/2011 12:36 PM, Anderson Lizardo wrote:
>>> On Fri, Nov 18, 2011 at 3:08 PM, Marcel Holtmann<marcel@holtmann.org>
>>>   wrote:
>>>>
>>>> I think one of the most important questions that we have to ask
>>>> ourselves at some point is if we wanna put ATT into the kernel.
>>>>
>>>> The potential candidate that forces us to think about this is HID over
>>>> Low Energy. However I like to see numbers on how the context switches
>>>> with keeping ATT in userspace will effect our latency.
>>>
>>> I still fail to see how ATT handling in kernel would reduce context
>>> switches. A GATT operation is composed of one or more (possibly many,
>>> see e.g. discovery procedures) ATT PDUs.
>>>
>>> Unless you are proposing GATT on kernel as well?
>>
>> I don't think having GATT in the kernel is necessarily the logical
>> conclusion to an ATT-to-kernel migration.
>
> I was actually referring only to the context switch topic. You still
> need to send the ATT request parameters from userspace, which is not
> different from building the PDU on userspace (vs. building on kernel).
>
> Unless I'm not understanding what Marcel meant with "ATT on kernel"
> (which is most probably the case).
>
>> Remember that the focus of LE is *not* the discovery process.  Discovery (of
>> Devices, Services and Characteristics) is important, but what makes LE "Low
>> Energy" is what happens *after* pairing and discovery have been completed,
>> which ideally would happen One Time Only.
>
> The discovery procedure was only an example of multiple PDU operation.
> Take e.g. "read long". If only ATT is on kernel, it still need
> multiple context switches for the multiple PDUs involved for the GATT
> operation.
>
>> The LE profiles and services are being spec'd so that they can be highly
>> efficient over the lifetime of the Low Energy device.  The Linux side of the
>> equation will rarely be an actual Low Energy device:  It will just know how
>> to talk to an LE device in such a way as to make that Button-cell battery
>> driven device last a couple years if possible.
>>
>> For that reason MOST day-to-day communication between a Linux based
>> "Central" device will in fact be single-ATT procedures.  If it takes more
>> energy and context switches to do discovery using many compound procedures,
>> then we pay that price, because it should only happens Once.
>
> ... and for this reason I don't see much gain in reducing context
> switches if *only* ATT is on kernel. (note: I'm not defending GATT on
> kernel).
>
>> HID devices in particular, will almost certainly NOT use compound GATT
>> procedures to communicate after the pairing and discovery point in time has
>> past.
>
> Yes, and also I don't see why LE HID is an example of "many context
> switches". For sure, you will need a bunch of them for setting up the
> HID device (discovery + pairing), but after that it will "talk" HID
> with the kernel input subsystem (If I understood the basics of this
> profile).

I don't know about everyone else, but most of the context switching that 
I would like to see avoided is for single ATT-procedures that:
Connect-DoSomething-Disconnect

That is the case for any Write-Signed-Command type of operations.

Perhaps HID is different, although I think the idea for HID is to not 
require the round trip of:

Baseband --> Kernel --> UserSpace --> Kernel --> HIDQueue

Would the Context switching make significant difference?  I suspect that 
on some platforms it would be hardly noticeable, but I would rather 
design something that we *know* requires low latency, to assume it is 
operating in an environment that might be hostile to latency concerns.

I personally think a fully feature rich GATT is too heavy to include in 
the kernel, but that perhaps a specialized HID server (recognized by a 
kernel resident ATT driver as being in the proper ATT Handle range) 
might make some sense.

-- 
Brian Gix
bgix@codeaurora.org
Employee of Qualcomm Innovation Center, Inc.
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum

Re: [RFC] LE: Low Latency GATT Write-Sign-Cmd

[Vinicius Costa Gomes]

Hi Johan,

On 22:12 Fri 18 Nov, Johan Hedberg wrote:
> Hi Brian,
> 
> On Fri, Nov 18, 2011, Brian Gix wrote:
> > I would support migrating ATT to the kernel.
> 
> Same here.
> 
> > In fact another issue I have dealt with is ensuring during Pairing,
> > that we are able to at least respond with minimal Error responses if
> > someone tries to read our name, and there is no GATT client socket to
> > user space to respond.
> 
> This issue was already encountered during the last UPF and INdT has a
> fix for it. For whatever reason that fix doesn't seem to have showed up
> here on the mailing list for upstreaming. 

For that to work, and have a matching userspace and kernel, it needs proper 
support for pairing, i.e. "re-using" a LE link that was used for SMP for 
GATT requests. Those patches are being tested right now, they should
appear on the list soon.

> In general the capability of
> doing ATT signaling should be available as soon as we have a connection
> handle, and it should be independent of any ongoing SMP procedure. Only
> if the characteristic accessed requires higher security than is
> currently provided by the LE link should the ATT procedures fail with an
> error. In the use case you describe, since the Name characteristic
> doesn't have any special security requirements a request for it should
> be always responded to with a positive reply regardless of the current
> level of security or if there's an ongoing SMP operation.
> 
> Johan
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bluetooth" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Cheers,
-- 
Vinicius