[RFC] Convert storage to use per-remote device directories

[RFC] Convert storage to use per-remote device directories

[Frederic Danis]

[-- Attachment #1: Type: text/plain, Size: 1795 bytes --]

Hi everyone,

Here is my proposal for new storage directory structure using ini-file 
format.

Each adapter directory (/var/lib/bluetooth/<adapter address>/) will 
contain a config file for the local adapter and one directory per remote 
device.
The adapter config file just need to be converted to ini-file format 
with only 1 group called [adapter].

Each of remote device directories' name will be based on remote device 
address and address type (address#type).
This directory will contain a config file with remote device infos and a 
linkkey file.
Remote device config file will include a [device] group with general 
device infos (name, alias, profiles or services list, ...), and groups 
named by profile uuid (or service uuid) with related infos.

So the directory structure should be:
    /var/lib/bluetooth/<adapter address>/
        ./config
        ./<remote device address#type>/
            ./config
            ./linkkey
        ./<remote device address#type>/
            ./config
            ./linkkey
        ...

I attached sample of adapter and device config files.

-- 
Frederic Danis                            Open Source Technology Center
frederic.danis@intel.com                              Intel Corporation


---------------------------------------------------------------------
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

[-- Attachment #2: adapter-config-sample.txt --]
[-- Type: text/plain, Size: 93 bytes --]

[adapter]
name=desktop-0
class=0x780011
pairable=yes
onmode=discoverable
mode=discoverable



[-- Attachment #3: device-config-sample.txt --]
[-- Type: text/plain, Size: 777 bytes --]

[device]
name=MyPhone
alias=Fred's phone
class=0x180204
device_id=FFFF 0000 0000 0000
eir=040D040218
manufacturer=15
lmp_version=2
lmp_subversion=777
features=FFFE0D0008080000
lastseen=2012-09-26 11:19:40 GMT
lastused=2012-09-26 11:43:42 GMT
trusted=yes
profiles=00001101-0000-1000-8000-00805f9b34fb;00001103-0000-1000-8000-00805f9b34fb

[00001101-0000-1000-8000-00805f9b34fb]
handle=10001
record=35470900000A000100010900013503191101090004350C350319010035051900030802090005350319100209000935083506191101090100090100250C53657269616C20506F727400

[00001103-0000-1000-8000-00805f9b34fb]
handle=10002
record=35530900000A000100020900013503191103090004350C35031901003505190003080309000535031910020900093508350619110309010009010025134469616C2D7570204E6574776F726B696E67000903052800

Re: [RFC] Convert storage to use per-remote device directories

[Johan Hedberg]

Hi Frederic,

On Thu, Sep 27, 2012, Frederic Danis wrote:
> Here is my proposal for new storage directory structure using
> ini-file format.
> 
> Each adapter directory (/var/lib/bluetooth/<adapter address>/) will
> contain a config file for the local adapter and one directory per
> remote device.
> The adapter config file just need to be converted to ini-file format
> with only 1 group called [adapter].
> 
> Each of remote device directories' name will be based on remote
> device address and address type (address#type).
> This directory will contain a config file with remote device infos
> and a linkkey file.
> Remote device config file will include a [device] group with general
> device infos (name, alias, profiles or services list, ...), and
> groups named by profile uuid (or service uuid) with related infos.
> 
> So the directory structure should be:
>    /var/lib/bluetooth/<adapter address>/
>        ./config
>        ./<remote device address#type>/
>            ./config
>            ./linkkey
>        ./<remote device address#type>/
>            ./config
>            ./linkkey
>        ...

So far this all looks good, though maybe we should follow the convention
of having .conf suffixes like with our other INI files. Or maybe that's
not needed?

> I attached sample of adapter and device config files.

> [adapter]
> name=desktop-0
> class=0x780011
> pairable=yes
> onmode=discoverable
> mode=discoverable

I hope you've looked at our existing INI files like audio.conf and
main.conf. You should have noticed that we use CamelCase for the section
and variable names, so at least that needs fixing.

> [device]
> name=MyPhone
> alias=Fred's phone
> class=0x180204
> device_id=FFFF 0000 0000 0000
> eir=040D040218
> manufacturer=15
> lmp_version=2
> lmp_subversion=777
> features=FFFE0D0008080000
> lastseen=2012-09-26 11:19:40 GMT
> lastused=2012-09-26 11:43:42 GMT
> trusted=yes
> profiles=00001101-0000-1000-8000-00805f9b34fb;00001103-0000-1000-8000-00805f9b34fb
> 
> [00001101-0000-1000-8000-00805f9b34fb]
> handle=10001
> record=35470900000A000100010900013503191101090004350C350319010035051900030802090005350319100209000935083506191101090100090100250C53657269616C20506F727400
> 
> [00001103-0000-1000-8000-00805f9b34fb]
> handle=10002
> record=35530900000A000100020900013503191103090004350C35031901003505190003080309000535031910020900093508350619110309010009010025134469616C2D7570204E6574776F726B696E67000903052800

This looks ok too, except for the lack of CamelCase naming. One of the
most interesting files I was waiting to see is the linkkey one since in
the existing storage we've crammed lots of separate variables into the
same entry. Could you send a proposal for this too in your next
revision? Since we also need storage for LTK's maybe it'd make sense to
have a single "keys" file with [LinkKey] and [LongTermKey] sections? Or
do you think those should be separate files?

Also, it seems you've left out all LE-specific information from this
initial proposal, like the conversion of the existing primaries file.

Johan

Re: [RFC] Convert storage to use per-remote device directories

[Frederic Danis]

Hello Johan,

On 28/09/2012 11:37, Johan Hedberg wrote:
<snip>
>> So the directory structure should be:
>>     /var/lib/bluetooth/<adapter address>/
>>         ./config
>>         ./<remote device address#type>/
>>             ./config
>>             ./linkkey
>>         ./<remote device address#type>/
>>             ./config
>>             ./linkkey
>>         ...
>
> So far this all looks good, though maybe we should follow the convention
> of having .conf suffixes like with our other INI files. Or maybe that's
> not needed?

OK, I will rename adapter config file to adapter.conf, and device config 
file to device.conf.

>> I attached sample of adapter and device config files.
>
>> [adapter]
>> name=desktop-0
>> class=0x780011
>> pairable=yes
>> onmode=discoverable
>> mode=discoverable
>
> I hope you've looked at our existing INI files like audio.conf and
> main.conf. You should have noticed that we use CamelCase for the section
> and variable names, so at least that needs fixing.

OK, I will do this.

>> [device]
>> name=MyPhone
>> alias=Fred's phone
>> class=0x180204
>> device_id=FFFF 0000 0000 0000
>> eir=040D040218
>> manufacturer=15
>> lmp_version=2
>> lmp_subversion=777
>> features=FFFE0D0008080000
>> lastseen=2012-09-26 11:19:40 GMT
>> lastused=2012-09-26 11:43:42 GMT
>> trusted=yes
>> profiles=00001101-0000-1000-8000-00805f9b34fb;00001103-0000-1000-8000-00805f9b34fb
>>
>> [00001101-0000-1000-8000-00805f9b34fb]
>> handle=10001
>> record=35470900000A000100010900013503191101090004350C350319010035051900030802090005350319100209000935083506191101090100090100250C53657269616C20506F727400
>>
>> [00001103-0000-1000-8000-00805f9b34fb]
>> handle=10002
>> record=35530900000A000100020900013503191103090004350C35031901003505190003080309000535031910020900093508350619110309010009010025134469616C2D7570204E6574776F726B696E67000903052800
>
> This looks ok too, except for the lack of CamelCase naming. One of the
> most interesting files I was waiting to see is the linkkey one since in
> the existing storage we've crammed lots of separate variables into the
> same entry. Could you send a proposal for this too in your next
> revision? Since we also need storage for LTK's maybe it'd make sense to
> have a single "keys" file with [LinkKey] and [LongTermKey] sections? Or
> do you think those should be separate files?

Current linkkeys file is only accessible by root, this is why I kept it 
separated.
Should LTKs be world readable or only accessible by root ?

Keys file should be named keys.conf and looks like:

[LinkKey]
Key=9EF4BDFA68C5438A176DF42ACD59816C
Type=0
Length=4

[LongTermKey]
Key=
Authenticated=
EncSize=
EDiv=
Rand=

> Also, it seems you've left out all LE-specific information from this
> initial proposal, like the conversion of the existing primaries file.

Currently I do not have LE devices, so I am not sure of what should be 
LE-specific infos.

I need to dig into it before I send a new proposal.

Fred

-- 
Frederic Danis                            Open Source Technology Center
frederic.danis@intel.com                              Intel Corporation

Re: [RFC] Convert storage to use per-remote device directories

[Anderson Lizardo]

Hi Frederic,

On Thu, Sep 27, 2012 at 5:59 AM, Frederic Danis
<frederic.danis@intel.com> wrote:
> Hi everyone,
>
> Here is my proposal for new storage directory structure using ini-file
> format.

Do you know if the INI parser in glib allows for multiple groups with
same name? In LE we can have services with same UUID (not sure if this
is valid for BR/EDR also), and they are differentiated by other means
(e.g. a "Description" descriptor with different value, besides the
different handle).

Regarding LE, we have two kinds of storage needs: server profiles, and
client profiles.

For server profiles, we need to store the attribute database.
Currently we don't do this, but we need to implement it, or have
ServiceChanged characteristic notifying that the service ranges have
changed every time BlueZ restarts (which adds overhead as service
discovery is triggered on the client side). I suggest we store the
attribute database into a "attribute_db.conf" file containing the
attribute name/value/uuid triples. Additionally, we also need to store
the Client Characteristic Configuration descriptor values (which are
specific to each bonded device), currently stored into a "ccc" file.
We could have one group just for them into attribute_db.conf.

For client profiles, I think the storage needs are specific to each
profile. From memory, only the generic GATT client and the GAP plugin
use storage currently (other profiles eventually need to use storage
to avoid full service discovery every time a bonded device
re-connects). The generic GATT client stores service/characteristics
handles/uuids ("primaries" and "characteristics" files), and the GAP
plugin stores appearance data ("appearances" file).

Johan, what do you think?

Regards,
-- 
Anderson Lizardo
Instituto Nokia de Tecnologia - INdT
Manaus - Brazil

Re: [RFC] Convert storage to use per-remote device directories

[Frederic Danis]

Hello Anderson,

On 28/09/2012 12:56, Anderson Lizardo wrote:
> Hi Frederic,
>
> On Thu, Sep 27, 2012 at 5:59 AM, Frederic Danis
> <frederic.danis@intel.com> wrote:
>> Hi everyone,
>>
>> Here is my proposal for new storage directory structure using ini-file
>> format.
>
> Do you know if the INI parser in glib allows for multiple groups with
> same name?

Following glib documentation:
"groups in key files may contain the same key multiple times; the last 
entry wins. Key files may also contain multiple groups with the same 
name; they are merged together."

So, I do not think this meet our needs.

> In LE we can have services with same UUID (not sure if this
> is valid for BR/EDR also), and they are differentiated by other means
> (e.g. a "Description" descriptor with different value, besides the
> different handle).

For BDEDR, it is easy to retrieve info related to a profile as profile's 
UUIDs are unique.

Are "Description" descriptors for a profile well-know or implementation 
dependent ?

Do you think we can use [<UUID>,<Description>] as group name ?

> Regarding LE, we have two kinds of storage needs: server profiles, and
> client profiles.
>
> For server profiles, we need to store the attribute database.

Is it possible to save it as we did for SDP records (record entry in 
profile group) ?

> Currently we don't do this, but we need to implement it, or have
> ServiceChanged characteristic notifying that the service ranges have
> changed every time BlueZ restarts (which adds overhead as service
> discovery is triggered on the client side). I suggest we store the
> attribute database into a "attribute_db.conf" file containing the
> attribute name/value/uuid triples.

Isn't it possible to save it in [<uuid>] group of device.conf file ?

> Additionally, we also need to store
> the Client Characteristic Configuration descriptor values (which are
> specific to each bonded device), currently stored into a "ccc" file.
> We could have one group just for them into attribute_db.conf.
>
> For client profiles, I think the storage needs are specific to each
> profile. From memory, only the generic GATT client and the GAP plugin
> use storage currently (other profiles eventually need to use storage
> to avoid full service discovery every time a bonded device
> re-connects). The generic GATT client stores service/characteristics
> handles/uuids ("primaries" and "characteristics" files), and the GAP
> plugin stores appearance data ("appearances" file).

My first thought for LE data in device.conf is:
   [<Primary UUID>#<Description>]
   Handle=
   Characteristic=
   Attribute=
   CCC=

I think we can easily add other key entries when needed.

Regards

Fred

-- 
Frederic Danis                            Open Source Technology Center
frederic.danis@intel.com                              Intel Corporation

Re: [RFC] Convert storage to use per-remote device directories

[Anderson Lizardo]

Hi Frederic,

On Fri, Sep 28, 2012 at 9:12 AM, Frederic Danis
<frederic.danis@linux.intel.com> wrote:
> Following glib documentation:
> "groups in key files may contain the same key multiple times; the last entry
> wins. Key files may also contain multiple groups with the same name; they
> are merged together."
>
> So, I do not think this meet our needs.

I agree.

>> In LE we can have services with same UUID (not sure if this
>> is valid for BR/EDR also), and they are differentiated by other means
>> (e.g. a "Description" descriptor with different value, besides the
>> different handle).
>
> For BDEDR, it is easy to retrieve info related to a profile as profile's
> UUIDs are unique.
>
> Are "Description" descriptors for a profile well-know or implementation
> dependent ?
>
> Do you think we can use [<UUID>,<Description>] as group name ?

The Description descriptor is profile specific (some profiles don't
have description, but on the other hand are unique in the database).
Unfortunately, I think it is not possible to use them on the group
name.

>> Regarding LE, we have two kinds of storage needs: server profiles, and
>> client profiles.
>>
>> For server profiles, we need to store the attribute database.
>
>
> Is it possible to save it as we did for SDP records (record entry in profile
> group) ?

It seems the SDP records have the UUID as group on the INI file. This
is not feasible for GATT/ATT because the UUID is not a unique key, but
the attribute handle is.

Are you sure that for SDP each record has its own UUID (i.e. no SDP
record shares same UUID)?

>> Currently we don't do this, but we need to implement it, or have
>> ServiceChanged characteristic notifying that the service ranges have
>> changed every time BlueZ restarts (which adds overhead as service
>> discovery is triggered on the client side). I suggest we store the
>> attribute database into a "attribute_db.conf" file containing the
>> attribute name/value/uuid triples.
>
>
> Isn't it possible to save it in [<uuid>] group of device.conf file ?

The attribute database needs to be stored on the adapter side (the
services exposed to remote devices over GATT are group of attributes).

> My first thought for LE data in device.conf is:
>   [<Primary UUID>#<Description>]
>   Handle=
>   Characteristic=
>   Attribute=
>   CCC=

As mentioned above, the "#Description" part is profile specific and
thus not generic enough for the group name (IMHO). It is actually a
high level information (a Characteristic Descriptor, which is just a
special attribute with a specific UUID.

If that helps, this is how I could imagine the attribute database (see
Core Spec page 1950 for an example database where I based the info
below from):

/var/lib/bluetooth/<adapter address>/attribute_db.conf

[0x0001]
UUID=00002800-0000-1000-8000-00805f9b34fb
Value=0018

[0x0004]
UUID=00002803-0000-1000-8000-00805f9b34fb
Value=020600002A

[0x0006]
UUID=00002a00-0000-1000-8000-00805f9b34fb
Value=4578616D706C6520446576696365

[0x0010]
UUID=00002800-0000-1000-8000-00805f9b34fb
Value=0118

...

With this format, it is easy to "serialize" and "de-serialize" the
attribute database for the adapter.

I still haven't thought for the client side stuff, but they definitely
be on the device directory.

Hope that helps,
-- 
Anderson Lizardo
Instituto Nokia de Tecnologia - INdT
Manaus - Brazil

Re: [RFC] Convert storage to use per-remote device directories

[Johan Hedberg]

Hi Lizardo,

On Fri, Sep 28, 2012, Anderson Lizardo wrote:
> > Is it possible to save it as we did for SDP records (record entry in profile
> > group) ?
> 
> It seems the SDP records have the UUID as group on the INI file. This
> is not feasible for GATT/ATT because the UUID is not a unique key, but
> the attribute handle is.
> 
> Are you sure that for SDP each record has its own UUID (i.e. no SDP
> record shares same UUID)?

There's no such restriction for BR/EDR. As with ATT the only unique
identifier is the SDP record handle, however (at least with SDP) I think
this is only guaranteed to have the same value for a single connection ,
i.e. in theory subsequent connections could have the same record behind
a different handle.

Johan

Re: [RFC] Convert storage to use per-remote device directories

[Marcel Holtmann]

Hi Fred,

> Here is my proposal for new storage directory structure using ini-file 
> format.
> 
> Each adapter directory (/var/lib/bluetooth/<adapter address>/) will 
> contain a config file for the local adapter and one directory per remote 
> device.
> The adapter config file just need to be converted to ini-file format 
> with only 1 group called [adapter].
> 
> Each of remote device directories' name will be based on remote device 
> address and address type (address#type).
> This directory will contain a config file with remote device infos and a 
> linkkey file.
> Remote device config file will include a [device] group with general 
> device infos (name, alias, profiles or services list, ...), and groups 
> named by profile uuid (or service uuid) with related infos.
> 
> So the directory structure should be:
>     /var/lib/bluetooth/<adapter address>/
>         ./config
>         ./<remote device address#type>/
>             ./config
>             ./linkkey
>         ./<remote device address#type>/
>             ./config
>             ./linkkey
>         ...

why do we care about the address type here? Can we actually have a
different link key for BR/EDR and for LE? Right now it is really only
one choice on how to use that remote device. If it is dual-mode, then we
use BR/EDR and if it is single-mode we use LE.

Regards

Marcel

Re: [RFC] Convert storage to use per-remote device directories

[Johan Hedberg]

Hi Marcel,

On Fri, Sep 28, 2012, Marcel Holtmann wrote:
> > Here is my proposal for new storage directory structure using ini-file 
> > format.
> > 
> > Each adapter directory (/var/lib/bluetooth/<adapter address>/) will 
> > contain a config file for the local adapter and one directory per remote 
> > device.
> > The adapter config file just need to be converted to ini-file format 
> > with only 1 group called [adapter].
> > 
> > Each of remote device directories' name will be based on remote device 
> > address and address type (address#type).
> > This directory will contain a config file with remote device infos and a 
> > linkkey file.
> > Remote device config file will include a [device] group with general 
> > device infos (name, alias, profiles or services list, ...), and groups 
> > named by profile uuid (or service uuid) with related infos.
> > 
> > So the directory structure should be:
> >     /var/lib/bluetooth/<adapter address>/
> >         ./config
> >         ./<remote device address#type>/
> >             ./config
> >             ./linkkey
> >         ./<remote device address#type>/
> >             ./config
> >             ./linkkey
> >         ...
> 
> why do we care about the address type here? Can we actually have a
> different link key for BR/EDR and for LE? Right now it is really only
> one choice on how to use that remote device. If it is dual-mode, then we
> use BR/EDR and if it is single-mode we use LE.

I think the idea here was to avoid potential (though unlikely)
collisions of a static random address of device A and a public address
of device B. However, thinking about it, is such a collision actually
possible considering that the two most significant bits of a static
random address must be 1. I'd imagine that the Core spec. writers would
have tried to make sure that this will not clash with any OUI's.

In any case we do at least need to have the address type info somewhere
in the device-specific directory so bluetoothd knows how to connect to
the device.

Johan