Feedback on BlueZ DBus API

Feedback on BlueZ DBus API

[Barry Byford]

Hello,


Over the last couple of years I=E2=80=99ve been doing some work with access=
ing
BlueZ from the Python programming language. More recently I=E2=80=99ve been
trying to create a library to lower the barrier to getting started
with Bluetooth on Linux. On that library we have recently done a
retrospective of where we are.

(TL;DR https://github.com/ukBaz/python-bluezero/issues/126)

The discussion spilled over to twitter where there was a suggestion
that some of our concerns should be shared back to the BlueZ project.
https://twitter.com/sandeepmistry/status/891836962952314880


I am not sure how relevant our feedback is to the broader user base of
the BlueZ project. However, it is true that we can=E2=80=99t complain that =
our
needs aren=E2=80=99t being addressed by BlueZ if we don=E2=80=99t give feed=
back.


I also accept that part of the issue here is with me as I don=E2=80=99t com=
e
from a background where I know about the low level details of
Bluetooth and the C language. I am coming at this from the direction
of learning enough to rapidly prototype Python applications that
contain some Bluetooth functionality.


Context:

In my volunteering role as a STEM ambassador I have been a mentor on a
few school projects for engineering challenges where Bluetooth
functionality would have been helpful.

These school projects typically use hardware like Raspberry Pi=E2=80=99s, B=
BC
micro:bits and tablets/phones where Bluetooth hardware is readily
available.

My goal is to be able to create workshops that can be run at schools,
STEM fairs and community maker events that help people get started
exploring what is possible with Bluetooth and Python.


Building From Source:

My first bit of feedback is that things with BlueZ are a lot better
now than they were a few short years ago. However this is the first
challenge as it takes a long time for updates to appear in operating
systems.

The default Raspberry Pi OS is still shipping with BlueZ 5.23 which
isn=E2=80=99t much use for BLE. Even when Raspbian moves to being a Debian
Stretch based release, it will be BlueZ 5.43. This means that much of
the DBus API will be behind the experimental flag and will be missing
some of the great recent updates such as AcquireWrite, AcquireNotify
and set name in scan response.

While building from source is something I=E2=80=99ve become comfortable wit=
h,
getting organisers of workshop machines and users to do this is often
too large a barrier to entry.


Python DBus Library:

As I=E2=80=99ve written about before on this list, there is not an obvious
choice when it comes to a Python DBus library. I think the BlueZ
=E2=80=98test=E2=80=99 Python examples use the dbus-python library. There s=
eems to be
concerns about this Python library. Pydbus seems to be a recommended
library although there seems to be several holes in it for use with
BlueZ. The owner of the library appears to have moved on to another
project so updates (or merging of pull requests) seems to be sporadic.


Beacons:

Again another topic that has been discussed on this mailing list
before. There seems to have been a number of people that want to have
a Linux SBC continuously passively scan for beacons. This is something
that is not supported by the BlueZ DBus API. I believe the concern is
with flooding the bus. Could something similar to AcquireNotify be
done to solve this?


Testing:

It would be good to be able to test the Python apps without real
hardware. It would be good if BlueZ could provide some way of
mock/fake/stub so that unit testing could be easily done.

The library python-dbusmock showed some promise however the bluez5.py
template needs some work.


Our New Direction - Python Only Solution:

Because of the issues raised above, investigation has started on using
only Python standard libraries to find a solution for our library.
This will allow our library to move faster than OS updates allow as it
will be easy to install directly from PyPI with fewer dependencies.

The current investigation is looking to by-pass bluetoothd by using
HCI. What we have so far can be seen at:
https://github.com/TheBubbleworks/python-hcipy


Regards,

Barry

Re: Feedback on BlueZ DBus API

[Luiz Augusto von Dentz]

Hi Barry,

On Thu, Aug 10, 2017 at 8:07 PM, Barry Byford <31baz66@gmail.com> wrote:
> Hello,
>
>
> Over the last couple of years I=E2=80=99ve been doing some work with acce=
ssing
> BlueZ from the Python programming language. More recently I=E2=80=99ve be=
en
> trying to create a library to lower the barrier to getting started
> with Bluetooth on Linux. On that library we have recently done a
> retrospective of where we are.
>
> (TL;DR https://github.com/ukBaz/python-bluezero/issues/126)
>
> The discussion spilled over to twitter where there was a suggestion
> that some of our concerns should be shared back to the BlueZ project.
> https://twitter.com/sandeepmistry/status/891836962952314880
>
>
> I am not sure how relevant our feedback is to the broader user base of
> the BlueZ project. However, it is true that we can=E2=80=99t complain tha=
t our
> needs aren=E2=80=99t being addressed by BlueZ if we don=E2=80=99t give fe=
edback.

That is really a good start.

>
> I also accept that part of the issue here is with me as I don=E2=80=99t c=
ome
> from a background where I know about the low level details of
> Bluetooth and the C language. I am coming at this from the direction
> of learning enough to rapidly prototype Python applications that
> contain some Bluetooth functionality.

That shouldn't prevent you to give us feedback on our APIs, which btw
I have been updating recently.

>
> Context:
>
> In my volunteering role as a STEM ambassador I have been a mentor on a
> few school projects for engineering challenges where Bluetooth
> functionality would have been helpful.
>
> These school projects typically use hardware like Raspberry Pi=E2=80=99s,=
 BBC
> micro:bits and tablets/phones where Bluetooth hardware is readily
> available.
>
> My goal is to be able to create workshops that can be run at schools,
> STEM fairs and community maker events that help people get started
> exploring what is possible with Bluetooth and Python.
>
>
> Building From Source:
>
> My first bit of feedback is that things with BlueZ are a lot better
> now than they were a few short years ago. However this is the first
> challenge as it takes a long time for updates to appear in operating
> systems.
>
> The default Raspberry Pi OS is still shipping with BlueZ 5.23 which
> isn=E2=80=99t much use for BLE. Even when Raspbian moves to being a Debia=
n
> Stretch based release, it will be BlueZ 5.43. This means that much of
> the DBus API will be behind the experimental flag and will be missing
> some of the great recent updates such as AcquireWrite, AcquireNotify
> and set name in scan response.
>
> While building from source is something I=E2=80=99ve become comfortable w=
ith,
> getting organisers of workshop machines and users to do this is often
> too large a barrier to entry.

Can't you build a package using the latest changes? That would be
beneficial to BlueZ as we get more exposure to different user.

>
> Python DBus Library:
>
> As I=E2=80=99ve written about before on this list, there is not an obviou=
s
> choice when it comes to a Python DBus library. I think the BlueZ
> =E2=80=98test=E2=80=99 Python examples use the dbus-python library. There=
 seems to be
> concerns about this Python library. Pydbus seems to be a recommended
> library although there seems to be several holes in it for use with
> BlueZ. The owner of the library appears to have moved on to another
> project so updates (or merging of pull requests) seems to be sporadic.
>

Im not familiar with the differences between the D-Bus bindings, I
guess we started doing the tests in whatever binding existed at that
point and we never bothered changing it. Anyway you could in theory
have your own binding if that really becomes the bottleneck.

>
> Beacons:
>
> Again another topic that has been discussed on this mailing list
> before. There seems to have been a number of people that want to have
> a Linux SBC continuously passively scan for beacons. This is something
> that is not supported by the BlueZ DBus API. I believe the concern is
> with flooding the bus. Could something similar to AcquireNotify be
> done to solve this?

I guess you are suggesting disabling the duplicated filtering, this is
something I wish we discuss more in the list since we now also have
mesh that uses the advertising as protocol transport. Depending on the
system it might be alright, but in Linux distribution it might consume
too much power.

>
> Testing:
>
> It would be good to be able to test the Python apps without real
> hardware. It would be good if BlueZ could provide some way of
> mock/fake/stub so that unit testing could be easily done.
>
> The library python-dbusmock showed some promise however the bluez5.py
> template needs some work.

BlueZ/Linux support emulated controllers for a while, have a look at:

https://git.kernel.org/pub/scm/bluetooth/bluez.git/tree/emulator

We should probably add some documentation for it since it may not be
very clear how to use btvirt, etc.

>
> Our New Direction - Python Only Solution:
>
> Because of the issues raised above, investigation has started on using
> only Python standard libraries to find a solution for our library.
> This will allow our library to move faster than OS updates allow as it
> will be easy to install directly from PyPI with fewer dependencies.
>
> The current investigation is looking to by-pass bluetoothd by using
> HCI. What we have so far can be seen at:
> https://github.com/TheBubbleworks/python-hcipy

I guess that would also break any application and tools using BlueZ
APIs in the system, but you probably already know that, not to mention
having to have root access to use raw sockets, or making all python
scripts to run with cap_net_raw:

sudo setcap cap_net_raw+eip $(eval readlink -f `which python`)
sudo setcap cap_net_raw+eip $(eval readlink -f `which python3`)

I can give a lot more examples where this will lead to, but Im not
sure the communities you are referring to will be able to listen. Far
too many times Ive seen people disregard BlueZ to do its own thing,
which end up in many different partial solutions like bleno,
pygattlib... and there probably many many more to come, none is really
a complete a BLE stack:

https://github.com/sandeepmistry/bleno/issues/110

At least with bleno Ive tried to make it use the D-Bus APIs but there
was no interest, it seems it preferable to disable all other Bluetooth
functionality, like pairing, authorization, etc, just to have GATT and
Advertisement implemented natively, where only one simple process can
access them, without ensuring this solution can pass a Bluetooth
qualification.

That said I know BlueZ is not perfect, we took way too long to make
the GATT APIs stable and that possible cause this many solutions
around BLE, but if instead of creating their own solution this effort
was put into getting BlueZ GATT APIs tested we wouldn't be talking
about it now and the Advertising API would probably have been adopted
as well.

Re: Feedback on BlueZ DBus API

[Barry Byford]

Hi Luiz,

Thanks for the prompt reply.

On 11 August 2017 at 11:57, Luiz Augusto von Dentz <luiz.dentz@gmail.com> w=
rote:
> Hi Barry,
>
> On Thu, Aug 10, 2017 at 8:07 PM, Barry Byford <31baz66@gmail.com> wrote:
>> Hello,
>>
>>
>> Over the last couple of years I=E2=80=99ve been doing some work with acc=
essing
>> BlueZ from the Python programming language. More recently I=E2=80=99ve b=
een
>> trying to create a library to lower the barrier to getting started
>> with Bluetooth on Linux. On that library we have recently done a
>> retrospective of where we are.
>>
>> (TL;DR https://github.com/ukBaz/python-bluezero/issues/126)
>>
>> The discussion spilled over to twitter where there was a suggestion
>> that some of our concerns should be shared back to the BlueZ project.
>> https://twitter.com/sandeepmistry/status/891836962952314880
>>
>>
>> I am not sure how relevant our feedback is to the broader user base of
>> the BlueZ project. However, it is true that we can=E2=80=99t complain th=
at our
>> needs aren=E2=80=99t being addressed by BlueZ if we don=E2=80=99t give f=
eedback.
>
> That is really a good start.
>
>>
>> I also accept that part of the issue here is with me as I don=E2=80=99t =
come
>> from a background where I know about the low level details of
>> Bluetooth and the C language. I am coming at this from the direction
>> of learning enough to rapidly prototype Python applications that
>> contain some Bluetooth functionality.
>
> That shouldn't prevent you to give us feedback on our APIs, which btw
> I have been updating recently.

I can see the updates coming and they all look like they are going int
the right direction. Maybe my feedback could be summarised as barriers
to accessing this great new stuff.

>>
>> Context:
>>
>> In my volunteering role as a STEM ambassador I have been a mentor on a
>> few school projects for engineering challenges where Bluetooth
>> functionality would have been helpful.
>>
>> These school projects typically use hardware like Raspberry Pi=E2=80=99s=
, BBC
>> micro:bits and tablets/phones where Bluetooth hardware is readily
>> available.
>>
>> My goal is to be able to create workshops that can be run at schools,
>> STEM fairs and community maker events that help people get started
>> exploring what is possible with Bluetooth and Python.
>>
>>
>> Building From Source:
>>
>> My first bit of feedback is that things with BlueZ are a lot better
>> now than they were a few short years ago. However this is the first
>> challenge as it takes a long time for updates to appear in operating
>> systems.
>>
>> The default Raspberry Pi OS is still shipping with BlueZ 5.23 which
>> isn=E2=80=99t much use for BLE. Even when Raspbian moves to being a Debi=
an
>> Stretch based release, it will be BlueZ 5.43. This means that much of
>> the DBus API will be behind the experimental flag and will be missing
>> some of the great recent updates such as AcquireWrite, AcquireNotify
>> and set name in scan response.
>>
>> While building from source is something I=E2=80=99ve become comfortable =
with,
>> getting organisers of workshop machines and users to do this is often
>> too large a barrier to entry.
>
> Can't you build a package using the latest changes? That would be
> beneficial to BlueZ as we get more exposure to different user.

Building a Debian package is something I've looked into as it is good
idea. However, I always have to patch my builds to get them working on
the Raspberry Pi 3 and along with a few other concerns I am not sure
that I have enough confidence to share it widely.
Is building the Debian package something that could be done as part of
BlueZ releases?

>
>>
>> Python DBus Library:
>>
>> As I=E2=80=99ve written about before on this list, there is not an obvio=
us
>> choice when it comes to a Python DBus library. I think the BlueZ
>> =E2=80=98test=E2=80=99 Python examples use the dbus-python library. Ther=
e seems to be
>> concerns about this Python library. Pydbus seems to be a recommended
>> library although there seems to be several holes in it for use with
>> BlueZ. The owner of the library appears to have moved on to another
>> project so updates (or merging of pull requests) seems to be sporadic.
>>
>
> Im not familiar with the differences between the D-Bus bindings, I
> guess we started doing the tests in whatever binding existed at that
> point and we never bothered changing it. Anyway you could in theory
> have your own binding if that really becomes the bottleneck.

Not having a good Python DBus library is a barrier to using BlueZ.
Creating my own is probably beyond me in a reasonable timeframe.
For your reference dbus-python is listed under Obsolete libraries on
https://www.freedesktop.org/wiki/Software/DBusBindings/

I am happy to help improve the bindings but too quickly it goes off
into the world of C and so again I am probably not best placed to make
updates.

>
>>
>> Beacons:
>>
>> Again another topic that has been discussed on this mailing list
>> before. There seems to have been a number of people that want to have
>> a Linux SBC continuously passively scan for beacons. This is something
>> that is not supported by the BlueZ DBus API. I believe the concern is
>> with flooding the bus. Could something similar to AcquireNotify be
>> done to solve this?
>
> I guess you are suggesting disabling the duplicated filtering, this is
> something I wish we discuss more in the list since we now also have
> mesh that uses the advertising as protocol transport. Depending on the
> system it might be alright, but in Linux distribution it might consume
> too much power.

Not sure I understand the 'too much power' comment in the context I
want to use this.

I also thought the issue was more than turning off the duplicates
filter. I thought the issues was if you were constantly scanning for
beacons then there was an issue with delays going through DBus. That
is why I thought that the notifications for new beacon updates could
go through a socket.

>
>>
>> Testing:
>>
>> It would be good to be able to test the Python apps without real
>> hardware. It would be good if BlueZ could provide some way of
>> mock/fake/stub so that unit testing could be easily done.
>>
>> The library python-dbusmock showed some promise however the bluez5.py
>> template needs some work.
>
> BlueZ/Linux support emulated controllers for a while, have a look at:
>
> https://git.kernel.org/pub/scm/bluetooth/bluez.git/tree/emulator
>
> We should probably add some documentation for it since it may not be
> very clear how to use btvirt, etc.

Yes, some documentation would be greatly appreciated as this sounds promisi=
ng.

>>
>> Our New Direction - Python Only Solution:
>>
>> Because of the issues raised above, investigation has started on using
>> only Python standard libraries to find a solution for our library.
>> This will allow our library to move faster than OS updates allow as it
>> will be easy to install directly from PyPI with fewer dependencies.
>>
>> The current investigation is looking to by-pass bluetoothd by using
>> HCI. What we have so far can be seen at:
>> https://github.com/TheBubbleworks/python-hcipy
>
> I guess that would also break any application and tools using BlueZ
> APIs in the system, but you probably already know that, not to mention
> having to have root access to use raw sockets, or making all python
> scripts to run with cap_net_raw:
>
> sudo setcap cap_net_raw+eip $(eval readlink -f `which python`)
> sudo setcap cap_net_raw+eip $(eval readlink -f `which python3`)
>
> I can give a lot more examples where this will lead to, but Im not
> sure the communities you are referring to will be able to listen. Far
> too many times Ive seen people disregard BlueZ to do its own thing,
> which end up in many different partial solutions like bleno,
> pygattlib... and there probably many many more to come, none is really
> a complete a BLE stack:
>
> https://github.com/sandeepmistry/bleno/issues/110
>
> At least with bleno Ive tried to make it use the D-Bus APIs but there
> was no interest, it seems it preferable to disable all other Bluetooth
> functionality, like pairing, authorization, etc, just to have GATT and
> Advertisement implemented natively, where only one simple process can
> access them, without ensuring this solution can pass a Bluetooth
> qualification.
>
> That said I know BlueZ is not perfect, we took way too long to make
> the GATT APIs stable and that possible cause this many solutions
> around BLE, but if instead of creating their own solution this effort
> was put into getting BlueZ GATT APIs tested we wouldn't be talking
> about it now and the Advertising API would probably have been adopted
> as well.

Again, I agree that the BlueZ DBus API is getting better all the time
and is definitely going in the right direction.
The central role with pydbus is working well. When the next release of
Raspbian based on Debian Stretch is available this should mike it
easily accessible to all.
However doing applications for beacon, scanner and peripheral continue
to be painful.

I want to help make the BlueZ DBus interface better I just don't know
how. For me the barrier to working on BlueZ or pydbus directly is too
high. I think this is why people go off and create their own solutions
as it appears 'easier' or 'quicker for them in the short term.

I am happy to take suggestions of how I can help.

Thanks,
Barry

Re: Feedback on BlueZ DBus API

[Travis Griggs]

> On Aug 11, 2017, at 6:14 AM, Barry Byford <31baz66@gmail.com> wrote:
>=20
> Hi Luiz,
>=20
> Thanks for the prompt reply.
>=20
> On 11 August 2017 at 11:57, Luiz Augusto von Dentz =
<luiz.dentz@gmail.com> wrote:
>> Hi Barry,
>>=20
>> On Thu, Aug 10, 2017 at 8:07 PM, Barry Byford <31baz66@gmail.com> =
wrote:
>>> Hello,
>>>=20
>>>=20
>>> Over the last couple of years I=E2=80=99ve been doing some work with =
accessing
>>> BlueZ from the Python programming language. More recently I=E2=80=99ve=
 been
>>> trying to create a library to lower the barrier to getting started
>>> with Bluetooth on Linux. On that library we have recently done a
>>> retrospective of where we are.
>>>=20
>>> (TL;DR https://github.com/ukBaz/python-bluezero/issues/126)
>>>=20
>>> The discussion spilled over to twitter where there was a suggestion
>>> that some of our concerns should be shared back to the BlueZ =
project.
>>> https://twitter.com/sandeepmistry/status/891836962952314880
>>>=20
>>>=20
>>> I am not sure how relevant our feedback is to the broader user base =
of
>>> the BlueZ project. However, it is true that we can=E2=80=99t =
complain that our
>>> needs aren=E2=80=99t being addressed by BlueZ if we don=E2=80=99t =
give feedback.
>>=20
>> That is really a good start.
>>=20
>>>=20
>>> I also accept that part of the issue here is with me as I don=E2=80=99=
t come
>>> from a background where I know about the low level details of
>>> Bluetooth and the C language. I am coming at this from the direction
>>> of learning enough to rapidly prototype Python applications that
>>> contain some Bluetooth functionality.
>>=20
>> That shouldn't prevent you to give us feedback on our APIs, which btw
>> I have been updating recently.
>=20
> I can see the updates coming and they all look like they are going int
> the right direction. Maybe my feedback could be summarised as barriers
> to accessing this great new stuff.
>=20
>>>=20
>>> Context:
>>>=20
>>> In my volunteering role as a STEM ambassador I have been a mentor on =
a
>>> few school projects for engineering challenges where Bluetooth
>>> functionality would have been helpful.
>>>=20
>>> These school projects typically use hardware like Raspberry Pi=E2=80=99=
s, BBC
>>> micro:bits and tablets/phones where Bluetooth hardware is readily
>>> available.
>>>=20
>>> My goal is to be able to create workshops that can be run at =
schools,
>>> STEM fairs and community maker events that help people get started
>>> exploring what is possible with Bluetooth and Python.
>>>=20
>>>=20
>>> Building =46rom Source:
>>>=20
>>> My first bit of feedback is that things with BlueZ are a lot better
>>> now than they were a few short years ago. However this is the first
>>> challenge as it takes a long time for updates to appear in operating
>>> systems.
>>>=20
>>> The default Raspberry Pi OS is still shipping with BlueZ 5.23 which
>>> isn=E2=80=99t much use for BLE. Even when Raspbian moves to being a =
Debian
>>> Stretch based release, it will be BlueZ 5.43. This means that much =
of
>>> the DBus API will be behind the experimental flag and will be =
missing
>>> some of the great recent updates such as AcquireWrite, AcquireNotify
>>> and set name in scan response.
>>>=20
>>> While building from source is something I=E2=80=99ve become =
comfortable with,
>>> getting organisers of workshop machines and users to do this is =
often
>>> too large a barrier to entry.
>>=20
>> Can't you build a package using the latest changes? That would be
>> beneficial to BlueZ as we get more exposure to different user.
>=20
> Building a Debian package is something I've looked into as it is good
> idea. However, I always have to patch my builds to get them working on
> the Raspberry Pi 3 and along with a few other concerns I am not sure
> that I have enough confidence to share it widely.
> Is building the Debian package something that could be done as part of
> BlueZ releases?
>=20
>>=20
>>>=20
>>> Python DBus Library:
>>>=20
>>> As I=E2=80=99ve written about before on this list, there is not an =
obvious
>>> choice when it comes to a Python DBus library. I think the BlueZ
>>> =E2=80=98test=E2=80=99 Python examples use the dbus-python library. =
There seems to be
>>> concerns about this Python library. Pydbus seems to be a recommended
>>> library although there seems to be several holes in it for use with
>>> BlueZ. The owner of the library appears to have moved on to another
>>> project so updates (or merging of pull requests) seems to be =
sporadic.
>>>=20
>>=20
>> Im not familiar with the differences between the D-Bus bindings, I
>> guess we started doing the tests in whatever binding existed at that
>> point and we never bothered changing it. Anyway you could in theory
>> have your own binding if that really becomes the bottleneck.
>=20
> Not having a good Python DBus library is a barrier to using BlueZ.
> Creating my own is probably beyond me in a reasonable timeframe.
> For your reference dbus-python is listed under Obsolete libraries on
> https://www.freedesktop.org/wiki/Software/DBusBindings/
>=20
> I am happy to help improve the bindings but too quickly it goes off
> into the world of C and so again I am probably not best placed to make
> updates.
>=20
>>=20
>>>=20
>>> Beacons:
>>>=20
>>> Again another topic that has been discussed on this mailing list
>>> before. There seems to have been a number of people that want to =
have
>>> a Linux SBC continuously passively scan for beacons. This is =
something
>>> that is not supported by the BlueZ DBus API. I believe the concern =
is
>>> with flooding the bus. Could something similar to AcquireNotify be
>>> done to solve this?
>>=20
>> I guess you are suggesting disabling the duplicated filtering, this =
is
>> something I wish we discuss more in the list since we now also have
>> mesh that uses the advertising as protocol transport. Depending on =
the
>> system it might be alright, but in Linux distribution it might =
consume
>> too much power.
>=20
> Not sure I understand the 'too much power' comment in the context I
> want to use this.
>=20
> I also thought the issue was more than turning off the duplicates
> filter. I thought the issues was if you were constantly scanning for
> beacons then there was an issue with delays going through DBus. That
> is why I thought that the notifications for new beacon updates could
> go through a socket.
>=20
>>=20
>>>=20
>>> Testing:
>>>=20
>>> It would be good to be able to test the Python apps without real
>>> hardware. It would be good if BlueZ could provide some way of
>>> mock/fake/stub so that unit testing could be easily done.
>>>=20
>>> The library python-dbusmock showed some promise however the =
bluez5.py
>>> template needs some work.
>>=20
>> BlueZ/Linux support emulated controllers for a while, have a look at:
>>=20
>> https://git.kernel.org/pub/scm/bluetooth/bluez.git/tree/emulator
>>=20
>> We should probably add some documentation for it since it may not be
>> very clear how to use btvirt, etc.
>=20
> Yes, some documentation would be greatly appreciated as this sounds =
promising.
>=20
>>>=20
>>> Our New Direction - Python Only Solution:
>>>=20
>>> Because of the issues raised above, investigation has started on =
using
>>> only Python standard libraries to find a solution for our library.
>>> This will allow our library to move faster than OS updates allow as =
it
>>> will be easy to install directly from PyPI with fewer dependencies.
>>>=20
>>> The current investigation is looking to by-pass bluetoothd by using
>>> HCI. What we have so far can be seen at:
>>> https://github.com/TheBubbleworks/python-hcipy
>>=20
>> I guess that would also break any application and tools using BlueZ
>> APIs in the system, but you probably already know that, not to =
mention
>> having to have root access to use raw sockets, or making all python
>> scripts to run with cap_net_raw:
>>=20
>> sudo setcap cap_net_raw+eip $(eval readlink -f `which python`)
>> sudo setcap cap_net_raw+eip $(eval readlink -f `which python3`)
>>=20
>> I can give a lot more examples where this will lead to, but Im not
>> sure the communities you are referring to will be able to listen. Far
>> too many times Ive seen people disregard BlueZ to do its own thing,
>> which end up in many different partial solutions like bleno,
>> pygattlib... and there probably many many more to come, none is =
really
>> a complete a BLE stack:
>>=20
>> https://github.com/sandeepmistry/bleno/issues/110
>>=20
>> At least with bleno Ive tried to make it use the D-Bus APIs but there
>> was no interest, it seems it preferable to disable all other =
Bluetooth
>> functionality, like pairing, authorization, etc, just to have GATT =
and
>> Advertisement implemented natively, where only one simple process can
>> access them, without ensuring this solution can pass a Bluetooth
>> qualification.
>>=20
>> That said I know BlueZ is not perfect, we took way too long to make
>> the GATT APIs stable and that possible cause this many solutions
>> around BLE, but if instead of creating their own solution this effort
>> was put into getting BlueZ GATT APIs tested we wouldn't be talking
>> about it now and the Advertising API would probably have been adopted
>> as well.
>=20
> Again, I agree that the BlueZ DBus API is getting better all the time
> and is definitely going in the right direction.
> The central role with pydbus is working well. When the next release of
> Raspbian based on Debian Stretch is available this should mike it
> easily accessible to all.
> However doing applications for beacon, scanner and peripheral continue
> to be painful.
>=20
> I want to help make the BlueZ DBus interface better I just don't know
> how. For me the barrier to working on BlueZ or pydbus directly is too
> high. I think this is why people go off and create their own solutions
> as it appears 'easier' or 'quicker for them in the short term.
>=20
> I am happy to take suggestions of how I can help.
>=20

(text only resend in case last one didn=E2=80=99t really make it =
through)

I=E2=80=99m =E2=80=9Ccatching up=E2=80=9D on my bluez mailing list =
reading=E2=80=A6 I recognize that I=E2=80=99m 3 months late here. But =
I=E2=80=99d just like to chime in belatedly with this discussion. I =
agree with Barry wholeheartedly. And I agree with Luis too. I think at =
the end of the day Barry, you=E2=80=99re going to regret trying to go =
your own route. I *assume* the motivation for using dbus (instead of a =
classical C lib approach) was in part to be more language agnostic.

We have a working prototype of an irrigation controller that acts as BLE =
peripheral running on a small SBC, but it is probably the most difficult =
part of the stack to understand and deal with. I share Barry=E2=80=99s =
appreciation for making something possible, and parallel frustration =
with figuring out how to do it. The obsolescence of the pydbus package =
is worrying for me as well.

Maybe, as I catch up farther, these concerns have been better solved in =
the last 3 months.

Re: Feedback on BlueZ DBus API

[Luiz Augusto von Dentz]

Hi Travis,

On Thu, Nov 2, 2017 at 11:37 PM, Travis Griggs <travisgriggs@gmail.com> wro=
te:
>
>
>> On Aug 11, 2017, at 6:14 AM, Barry Byford <31baz66@gmail.com> wrote:
>>
>> Hi Luiz,
>>
>> Thanks for the prompt reply.
>>
>> On 11 August 2017 at 11:57, Luiz Augusto von Dentz <luiz.dentz@gmail.com=
> wrote:
>>> Hi Barry,
>>>
>>> On Thu, Aug 10, 2017 at 8:07 PM, Barry Byford <31baz66@gmail.com> wrote=
:
>>>> Hello,
>>>>
>>>>
>>>> Over the last couple of years I=E2=80=99ve been doing some work with a=
ccessing
>>>> BlueZ from the Python programming language. More recently I=E2=80=99ve=
 been
>>>> trying to create a library to lower the barrier to getting started
>>>> with Bluetooth on Linux. On that library we have recently done a
>>>> retrospective of where we are.
>>>>
>>>> (TL;DR https://github.com/ukBaz/python-bluezero/issues/126)
>>>>
>>>> The discussion spilled over to twitter where there was a suggestion
>>>> that some of our concerns should be shared back to the BlueZ project.
>>>> https://twitter.com/sandeepmistry/status/891836962952314880
>>>>
>>>>
>>>> I am not sure how relevant our feedback is to the broader user base of
>>>> the BlueZ project. However, it is true that we can=E2=80=99t complain =
that our
>>>> needs aren=E2=80=99t being addressed by BlueZ if we don=E2=80=99t give=
 feedback.
>>>
>>> That is really a good start.
>>>
>>>>
>>>> I also accept that part of the issue here is with me as I don=E2=80=99=
t come
>>>> from a background where I know about the low level details of
>>>> Bluetooth and the C language. I am coming at this from the direction
>>>> of learning enough to rapidly prototype Python applications that
>>>> contain some Bluetooth functionality.
>>>
>>> That shouldn't prevent you to give us feedback on our APIs, which btw
>>> I have been updating recently.
>>
>> I can see the updates coming and they all look like they are going int
>> the right direction. Maybe my feedback could be summarised as barriers
>> to accessing this great new stuff.
>>
>>>>
>>>> Context:
>>>>
>>>> In my volunteering role as a STEM ambassador I have been a mentor on a
>>>> few school projects for engineering challenges where Bluetooth
>>>> functionality would have been helpful.
>>>>
>>>> These school projects typically use hardware like Raspberry Pi=E2=80=
=99s, BBC
>>>> micro:bits and tablets/phones where Bluetooth hardware is readily
>>>> available.
>>>>
>>>> My goal is to be able to create workshops that can be run at schools,
>>>> STEM fairs and community maker events that help people get started
>>>> exploring what is possible with Bluetooth and Python.
>>>>
>>>>
>>>> Building From Source:
>>>>
>>>> My first bit of feedback is that things with BlueZ are a lot better
>>>> now than they were a few short years ago. However this is the first
>>>> challenge as it takes a long time for updates to appear in operating
>>>> systems.
>>>>
>>>> The default Raspberry Pi OS is still shipping with BlueZ 5.23 which
>>>> isn=E2=80=99t much use for BLE. Even when Raspbian moves to being a De=
bian
>>>> Stretch based release, it will be BlueZ 5.43. This means that much of
>>>> the DBus API will be behind the experimental flag and will be missing
>>>> some of the great recent updates such as AcquireWrite, AcquireNotify
>>>> and set name in scan response.
>>>>
>>>> While building from source is something I=E2=80=99ve become comfortabl=
e with,
>>>> getting organisers of workshop machines and users to do this is often
>>>> too large a barrier to entry.
>>>
>>> Can't you build a package using the latest changes? That would be
>>> beneficial to BlueZ as we get more exposure to different user.
>>
>> Building a Debian package is something I've looked into as it is good
>> idea. However, I always have to patch my builds to get them working on
>> the Raspberry Pi 3 and along with a few other concerns I am not sure
>> that I have enough confidence to share it widely.
>> Is building the Debian package something that could be done as part of
>> BlueZ releases?
>>
>>>
>>>>
>>>> Python DBus Library:
>>>>
>>>> As I=E2=80=99ve written about before on this list, there is not an obv=
ious
>>>> choice when it comes to a Python DBus library. I think the BlueZ
>>>> =E2=80=98test=E2=80=99 Python examples use the dbus-python library. Th=
ere seems to be
>>>> concerns about this Python library. Pydbus seems to be a recommended
>>>> library although there seems to be several holes in it for use with
>>>> BlueZ. The owner of the library appears to have moved on to another
>>>> project so updates (or merging of pull requests) seems to be sporadic.
>>>>
>>>
>>> Im not familiar with the differences between the D-Bus bindings, I
>>> guess we started doing the tests in whatever binding existed at that
>>> point and we never bothered changing it. Anyway you could in theory
>>> have your own binding if that really becomes the bottleneck.
>>
>> Not having a good Python DBus library is a barrier to using BlueZ.
>> Creating my own is probably beyond me in a reasonable timeframe.
>> For your reference dbus-python is listed under Obsolete libraries on
>> https://www.freedesktop.org/wiki/Software/DBusBindings/
>>
>> I am happy to help improve the bindings but too quickly it goes off
>> into the world of C and so again I am probably not best placed to make
>> updates.
>>
>>>
>>>>
>>>> Beacons:
>>>>
>>>> Again another topic that has been discussed on this mailing list
>>>> before. There seems to have been a number of people that want to have
>>>> a Linux SBC continuously passively scan for beacons. This is something
>>>> that is not supported by the BlueZ DBus API. I believe the concern is
>>>> with flooding the bus. Could something similar to AcquireNotify be
>>>> done to solve this?
>>>
>>> I guess you are suggesting disabling the duplicated filtering, this is
>>> something I wish we discuss more in the list since we now also have
>>> mesh that uses the advertising as protocol transport. Depending on the
>>> system it might be alright, but in Linux distribution it might consume
>>> too much power.
>>
>> Not sure I understand the 'too much power' comment in the context I
>> want to use this.
>>
>> I also thought the issue was more than turning off the duplicates
>> filter. I thought the issues was if you were constantly scanning for
>> beacons then there was an issue with delays going through DBus. That
>> is why I thought that the notifications for new beacon updates could
>> go through a socket.
>>
>>>
>>>>
>>>> Testing:
>>>>
>>>> It would be good to be able to test the Python apps without real
>>>> hardware. It would be good if BlueZ could provide some way of
>>>> mock/fake/stub so that unit testing could be easily done.
>>>>
>>>> The library python-dbusmock showed some promise however the bluez5.py
>>>> template needs some work.
>>>
>>> BlueZ/Linux support emulated controllers for a while, have a look at:
>>>
>>> https://git.kernel.org/pub/scm/bluetooth/bluez.git/tree/emulator
>>>
>>> We should probably add some documentation for it since it may not be
>>> very clear how to use btvirt, etc.
>>
>> Yes, some documentation would be greatly appreciated as this sounds prom=
ising.
>>
>>>>
>>>> Our New Direction - Python Only Solution:
>>>>
>>>> Because of the issues raised above, investigation has started on using
>>>> only Python standard libraries to find a solution for our library.
>>>> This will allow our library to move faster than OS updates allow as it
>>>> will be easy to install directly from PyPI with fewer dependencies.
>>>>
>>>> The current investigation is looking to by-pass bluetoothd by using
>>>> HCI. What we have so far can be seen at:
>>>> https://github.com/TheBubbleworks/python-hcipy
>>>
>>> I guess that would also break any application and tools using BlueZ
>>> APIs in the system, but you probably already know that, not to mention
>>> having to have root access to use raw sockets, or making all python
>>> scripts to run with cap_net_raw:
>>>
>>> sudo setcap cap_net_raw+eip $(eval readlink -f `which python`)
>>> sudo setcap cap_net_raw+eip $(eval readlink -f `which python3`)
>>>
>>> I can give a lot more examples where this will lead to, but Im not
>>> sure the communities you are referring to will be able to listen. Far
>>> too many times Ive seen people disregard BlueZ to do its own thing,
>>> which end up in many different partial solutions like bleno,
>>> pygattlib... and there probably many many more to come, none is really
>>> a complete a BLE stack:
>>>
>>> https://github.com/sandeepmistry/bleno/issues/110
>>>
>>> At least with bleno Ive tried to make it use the D-Bus APIs but there
>>> was no interest, it seems it preferable to disable all other Bluetooth
>>> functionality, like pairing, authorization, etc, just to have GATT and
>>> Advertisement implemented natively, where only one simple process can
>>> access them, without ensuring this solution can pass a Bluetooth
>>> qualification.
>>>
>>> That said I know BlueZ is not perfect, we took way too long to make
>>> the GATT APIs stable and that possible cause this many solutions
>>> around BLE, but if instead of creating their own solution this effort
>>> was put into getting BlueZ GATT APIs tested we wouldn't be talking
>>> about it now and the Advertising API would probably have been adopted
>>> as well.
>>
>> Again, I agree that the BlueZ DBus API is getting better all the time
>> and is definitely going in the right direction.
>> The central role with pydbus is working well. When the next release of
>> Raspbian based on Debian Stretch is available this should mike it
>> easily accessible to all.
>> However doing applications for beacon, scanner and peripheral continue
>> to be painful.
>>
>> I want to help make the BlueZ DBus interface better I just don't know
>> how. For me the barrier to working on BlueZ or pydbus directly is too
>> high. I think this is why people go off and create their own solutions
>> as it appears 'easier' or 'quicker for them in the short term.
>>
>> I am happy to take suggestions of how I can help.
>>
>
> (text only resend in case last one didn=E2=80=99t really make it through)
>
> I=E2=80=99m =E2=80=9Ccatching up=E2=80=9D on my bluez mailing list readin=
g=E2=80=A6 I recognize that I=E2=80=99m 3 months late here. But I=E2=80=99d=
 just like to chime in belatedly with this discussion. I agree with Barry w=
holeheartedly. And I agree with Luis too. I think at the end of the day Bar=
ry, you=E2=80=99re going to regret trying to go your own route. I *assume* =
the motivation for using dbus (instead of a classical C lib approach) was i=
n part to be more language agnostic.

Btw you can check what is happening with Qt which reinforces my view
why it is not a good idea to start rewriting parts of the stack at
application level, not only it creates problems later but I fill
people just don't give a fair chance to BlueZ's APIs and start writing
their own stack at first chance they have, most of the time using our
very own kernel interfaces.

In the meantime we had guys from nest/google just working with
upstream to get their usecases covered, which is all upstream by now,
and we have updated the adverting APIs and command line tools to make
use of them.

> We have a working prototype of an irrigation controller that acts as BLE =
peripheral running on a small SBC, but it is probably the most difficult pa=
rt of the stack to understand and deal with. I share Barry=E2=80=99s apprec=
iation for making something possible, and parallel frustration with figurin=
g out how to do it. The obsolescence of the pydbus package is worrying for =
me as well.

pydbus is a different problem though, while having to maintain a D-Bus
binding can be a problem I guess it still a lot simpler than
maintaining a Bluetooth stack.

> Maybe, as I catch up farther, these concerns have been better solved in t=
he last 3 months.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-bluetooth=
" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



--=20
Luiz Augusto von Dentz

Re: Feedback on BlueZ DBus API

[Barry Byford]

Hi Luiz,

On 3 November 2017 at 09:33, Luiz Augusto von Dentz
<luiz.dentz@gmail.com> wrote:
> Btw you can check what is happening with Qt which reinforces my view
> why it is not a good idea to start rewriting parts of the stack at
> application level, not only it creates problems later but I fill
> people just don't give a fair chance to BlueZ's APIs and start writing
> their own stack at first chance they have, most of the time using our
> very own kernel interfaces.

I first came here about three years ago looking for a way to scan for
beacon and process their information.
My absolute preference is to use the highest level of API possible.
I am totally in favour in getting things fixed as far upstream as possible.
So I think we are all agreeing on this point.

> In the meantime we had guys from nest/google just working with
> upstream to get their usecases covered, which is all upstream by now,
> and we have updated the adverting APIs and command line tools to make
> use of them.

Those with the skill level required to upstream is always going to be
a subset of those people that want to access Bluetooth from Python.
My gut feeling is that 'skill level' is one reason why people end up
fixing things at the 'wrong' level.
Another is that they can turn things around quicker by doing it in the
user space.
>From Python the two choices of API are DBus or raw HCI sockets. It is
not possible to access the mgmt API.
The problem seems to be that the current Python socket module doesn't
let you get at it. The relevant code appears to be in makesockaddr()
in CPython's socketmodule.c -  see
https://github.com/python/cpython/blob/master/Modules/socketmodule.c#L1276.
- it only ever seems to set the dev and bdaddr members, leaving the
hci_channel field set to 0 (HCI_CHANNEL_RAW).


> pydbus is a different problem though, while having to maintain a D-Bus
> binding can be a problem I guess it still a lot simpler than
> maintaining a Bluetooth stack.

I am going to put this in the same bucket as the previous point, the
skill and knowledge required to submitting fixes to D-Bus binding is
possibly off at a tangent from what many people coming to look at
Bluetooth might have.

I want to be constructive and to help move BlueZ forward in making it
more accessible for people from Python.
Upstreaming fixes has too big a barrier to entry for me and so I don't
know how to help move my issues forward.
I want to be able to build a scanner for beacons, I don't see how I
can do that without accessing the raw HCI sockets from Python.


Human nature is such that people normally take the path of least resistance.
I don't think there is a clear, strong and obvious way to implement
BLE in Python right now.
This seems to be reflected by others judging from the variety of ways
people have tried to do it:
https://github.com/ukBaz/python-bluezero/wiki
As you correctly identify, maintaining a Bluetooth stack is a lot of
work and so none of them really mature in to being really strong and
usable solutions.

People's comment to me is that there is not a good reliable source of
information/documentation which will help them get started when they
want to use BlueZ. As a result there is a lot of (outdated?)
misinformation that people refer to. It would be great if the BlueZ
project could be the source of great documentation when people want to
get started.
An example is the fact that the HCI* tools have been deprecated. I try
to spread the word about not using things like hcitool however there
is no documentation I can easily refer people to on the
correct/current tools to do these things.
It would be great if the reach of BlueZ could influence getting things
fixed in the infrastructure such a D-Bus binding and Python accessing
the mgmt API to make the BlueZ experience better.
Another example is it would be great if btvirt could be better
documented somewhere.
Yunhan Wang did a post to this group of how to reproduce an issue
using bluetoothctl. He used bluetoothctl to create a central and
peripheral on the same machine using btvirt. This seems like it would
be the good basis of an introduction to using bluetoothctl.

Regards,
Barry