[RFC 0/3] core/adapter: Add disabling duplicate device filtering from d-bus

[RFC 0/3] core/adapter: Add disabling duplicate device filtering from d-bus

[Don Zickus]

Recent discussions on the bluez mailing list revealed it was not easy
to disable duplicate device filtering from the d-bus interface.

As a result, if I wanted to monitor LE devices entering and leaving the
adapters range (using RSSI data), it was difficult.

This patchset is a dirty hack to make this work. The first patch enables
it on the kernel side, while the other two patches enable it from the bluez
side.

I understand there are concerns about flooding the d-bus interface when
enabling this.  I tried to write a throttling mechanism using the mainloop
as my timer, but soon realized you can only have 1 device RSSI event
per loop, so that wasn't going to work.  Open to suggestions if still a
concern.

Posted as an RFC just to generate discussion.  I expect I missed a lot of
little details here, but wanted to post my proof of concept to see if this
is something to work with.

This patchset includes both the kernel and bluez patches.  I understand this is
not recommended for normal practice.  But I thought for an RFC, it is nice to
keep things together for now.



kernel:

Don Zickus (1):
  bluetooth: Add managment ability to disable duplicate device fitlering

 include/net/bluetooth/hci_core.h | 1 +
 include/net/bluetooth/mgmt.h     | 1 +
 net/bluetooth/hci_request.c      | 2 +-
 net/bluetooth/mgmt.c             | 5 +++++
 4 files changed, 8 insertions(+), 1 deletion(-)


bluez:

Don Zickus (2):
  core/adapter: Add le_duplicates flag to management interface
  core/adapter: Hook le_duplicates into d-bus interface

 client/main.c       |  5 +++++
 doc/adapter-api.txt |  1 +
 lib/mgmt.h          |  1 +
 src/adapter.c       | 37 +++++++++++++++++++++++++++++++------
 4 files changed, 38 insertions(+), 6 deletions(-)

-- 
1.8.3.1

[RFC 1/3] bluetooth: Add managment ability to disable duplicate device fitlering

[Don Zickus]

There are use cases where userspace wants to disable duplicate device
filtering for LE scans on the adapater.  One example is real time update
of RSSI data.

This patch adds hooks into the management interface to allow that.

Currently only used on 'active_scan'.

Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 include/net/bluetooth/hci_core.h | 1 +
 include/net/bluetooth/mgmt.h     | 1 +
 net/bluetooth/hci_request.c      | 2 +-
 net/bluetooth/mgmt.c             | 5 +++++
 4 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/include/net/bluetooth/hci_core.h b/include/net/bluetooth/hci_core.h
index 554671c..fd34707 100644
--- a/include/net/bluetooth/hci_core.h
+++ b/include/net/bluetooth/hci_core.h
@@ -79,6 +79,7 @@ struct discovery_state {
 	bool			report_invalid_rssi;
 	bool			result_filtering;
 	bool			limited;
+	bool			filter_dups;
 	s8			rssi;
 	u16			uuid_count;
 	u8			(*uuids)[16];
diff --git a/include/net/bluetooth/mgmt.h b/include/net/bluetooth/mgmt.h
index 72a456b..6ffd2d8 100644
--- a/include/net/bluetooth/mgmt.h
+++ b/include/net/bluetooth/mgmt.h
@@ -503,6 +503,7 @@ struct mgmt_cp_start_service_discovery {
 	__s8 rssi;
 	__le16 uuid_count;
 	__u8 uuids[0][16];
+	__u8 le_duplicates;
 } __packed;
 #define MGMT_START_SERVICE_DISCOVERY_SIZE 4
 
diff --git a/net/bluetooth/hci_request.c b/net/bluetooth/hci_request.c
index 1015d9c..8b5ee64 100644
--- a/net/bluetooth/hci_request.c
+++ b/net/bluetooth/hci_request.c
@@ -2038,7 +2038,7 @@ static int active_scan(struct hci_request *req, unsigned long opt)
 
 	memset(&enable_cp, 0, sizeof(enable_cp));
 	enable_cp.enable = LE_SCAN_ENABLE;
-	enable_cp.filter_dup = LE_SCAN_FILTER_DUP_ENABLE;
+	enable_cp.filter_dup = hdev->discovery.filter_dups;
 
 	hci_req_add(req, HCI_OP_LE_SET_SCAN_ENABLE, sizeof(enable_cp),
 		    &enable_cp);
diff --git a/net/bluetooth/mgmt.c b/net/bluetooth/mgmt.c
index 1fba2a0..b07aa7f 100644
--- a/net/bluetooth/mgmt.c
+++ b/net/bluetooth/mgmt.c
@@ -3578,6 +3578,7 @@ static int start_service_discovery(struct sock *sk, struct hci_dev *hdev,
 	struct mgmt_pending_cmd *cmd;
 	const u16 max_uuid_count = ((U16_MAX - sizeof(*cp)) / 16);
 	u16 uuid_count, expected_len;
+	bool filter_dups = LE_SCAN_FILTER_DUP_ENABLE;
 	u8 status;
 	int err;
 
@@ -3631,6 +3632,9 @@ static int start_service_discovery(struct sock *sk, struct hci_dev *hdev,
 		goto failed;
 	}
 
+	if (cp->le_duplicates)
+		filter_dups = LE_SCAN_FILTER_DUP_DISABLE;
+
 	cmd = mgmt_pending_add(sk, MGMT_OP_START_SERVICE_DISCOVERY,
 			       hdev, data, len);
 	if (!cmd) {
@@ -3649,6 +3653,7 @@ static int start_service_discovery(struct sock *sk, struct hci_dev *hdev,
 	hdev->discovery.type = cp->type;
 	hdev->discovery.rssi = cp->rssi;
 	hdev->discovery.uuid_count = uuid_count;
+	hdev->discovery.filter_dups = filter_dups;
 
 	if (uuid_count > 0) {
 		hdev->discovery.uuids = kmemdup(cp->uuids, uuid_count * 16,
-- 
1.8.4

[RFC 2/3] core/adapter: Add le_duplicates flag to management interface

[Don Zickus]

Allow the ability to disable duplicate filtering on the adapater
from the management interface.

This patch just adds the appropriate hooks to communicate with the
kernel interface.  The following patches will utilize it.

The reason for doing something like this is to allow dynamic LE
scan data to be constantly updated. For example, RSSI changes can
now be seen frequently.

Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 lib/mgmt.h    |  1 +
 src/adapter.c | 15 +++++++++++----
 2 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/lib/mgmt.h b/lib/mgmt.h
index 798a05e..d9d6cc0 100644
--- a/lib/mgmt.h
+++ b/lib/mgmt.h
@@ -452,6 +452,7 @@ struct mgmt_cp_start_service_discovery {
 	int8_t rssi;
 	uint16_t uuid_count;
 	uint8_t uuids[0][16];
+	uint8_t le_duplicates;
 } __packed;
 
 #define MGMT_OP_READ_LOCAL_OOB_EXT_DATA	0x003B
diff --git a/src/adapter.c b/src/adapter.c
index d951307..ddf0d11 100644
--- a/src/adapter.c
+++ b/src/adapter.c
@@ -155,6 +155,7 @@ struct discovery_filter {
 	uint16_t pathloss;
 	int16_t rssi;
 	GSList *uuids;
+	bool le_duplicates;
 };
 
 struct watch_client {
@@ -1529,8 +1530,9 @@ static gboolean start_discovery_timeout(gpointer user_data)
 	/* Filtered discovery is required */
 	sd_cp = adapter->current_discovery_filter;
 
-	DBG("sending MGMT_OP_START_SERVICE_DISCOVERY %d, %d, %d",
-				sd_cp->rssi, sd_cp->type, sd_cp->uuid_count);
+	DBG("sending MGMT_OP_START_SERVICE_DISCOVERY %d, %d, %d %d %d",
+				sd_cp->rssi, sd_cp->type, sd_cp->uuid_count,
+				sd_cp->le_duplicates);
 
 	mgmt_send(adapter->mgmt, MGMT_OP_START_SERVICE_DISCOVERY,
 		  adapter->dev_id, sizeof(*sd_cp) + sd_cp->uuid_count * 16,
@@ -1772,7 +1774,8 @@ static void extract_unique_uuids(gpointer data, gpointer user_data)
  * Returns 1 if there was no filtered scan, 0 otherwise.
  */
 static int merge_discovery_filters(struct btd_adapter *adapter, int *rssi,
-					uint8_t *transport, GSList **uuids)
+					uint8_t *transport, GSList **uuids,
+					bool *le_duplicates)
 {
 	GSList *l;
 	bool empty_uuid = false;
@@ -1791,6 +1794,7 @@ static int merge_discovery_filters(struct btd_adapter *adapter, int *rssi,
 		has_filtered_discovery = true;
 
 		*transport |= item->type;
+		*le_duplicates |= item->le_duplicates;
 
 		/*
 		 * Rule for merging rssi and pathloss into rssi field of kernel
@@ -1877,10 +1881,12 @@ static int discovery_filter_to_mgmt_cp(struct btd_adapter *adapter,
 	int rssi = DISTANCE_VAL_INVALID;
 	int uuid_count;
 	uint8_t discovery_type = 0;
+	bool le_duplicates = 0;
 
 	DBG("");
 
-	if (merge_discovery_filters(adapter, &rssi, &discovery_type, &uuids)) {
+	if (merge_discovery_filters(adapter, &rssi, &discovery_type, &uuids,
+				    &le_duplicates)) {
 		/* There are only regular scans, run just regular scan. */
 		*cp_ptr = NULL;
 		return 0;
@@ -1898,6 +1904,7 @@ static int discovery_filter_to_mgmt_cp(struct btd_adapter *adapter,
 	cp->type = discovery_type;
 	cp->rssi = rssi;
 	cp->uuid_count = uuid_count;
+	cp->le_duplicates = le_duplicates;
 	populate_mgmt_filter_uuids(cp->uuids, uuids);
 
 	g_slist_free(uuids);
-- 
1.8.3.1

[RFC 3/3] core/adapter: Hook le_duplicates into d-bus interface

[Don Zickus]

The previous patch provided the management hooks to disable duplicate
device filtering on the adapter.  This patch allows the d-bus
interface to use it with another field in SetDiscoveryFilter()

The new field is:

bool LE_Duplicates

The adapter-api.txt doc has been updated to reflect this.

Signed-off-by: Don Zickus <dzickus@redhat.com>
---
 client/main.c       |  5 +++++
 doc/adapter-api.txt |  1 +
 src/adapter.c       | 22 ++++++++++++++++++++--
 3 files changed, 26 insertions(+), 2 deletions(-)

diff --git a/client/main.c b/client/main.c
index e1198a8..d74d068 100644
--- a/client/main.c
+++ b/client/main.c
@@ -1153,6 +1153,7 @@ struct set_discovery_filter_args {
 	dbus_int16_t pathloss;
 	char **uuids;
 	size_t uuids_len;
+	bool le_duplicates;
 };
 
 static void set_discovery_filter_setup(DBusMessageIter *iter, void *user_data)
@@ -1180,6 +1181,10 @@ static void set_discovery_filter_setup(DBusMessageIter *iter, void *user_data)
 		dict_append_entry(&dict, "Transport", DBUS_TYPE_STRING,
 						&args->transport);
 
+	if (args->le_duplicates)
+		dict_append_entry(&dict, "LE_Duplicates", DBUS_TYPE_BOOLEAN,
+						&args->le_duplicates);
+
 	dbus_message_iter_close_container(iter, &dict);
 }
 
diff --git a/doc/adapter-api.txt b/doc/adapter-api.txt
index 08de6cd..dc09752 100644
--- a/doc/adapter-api.txt
+++ b/doc/adapter-api.txt
@@ -58,6 +58,7 @@ Methods		void StartDiscovery()
 			int16	      RSSI	: RSSI threshold value
 			uint16        Pathloss	: Pathloss threshold value
 			string        Transport	: type of scan to run
+			bool          LE_Duplicates : Disable LE Duplicate filtering
 
 			When a remote device is found that advertises any UUID
 			from UUIDs, it will be reported if:
diff --git a/src/adapter.c b/src/adapter.c
index ddf0d11..1e961a1 100644
--- a/src/adapter.c
+++ b/src/adapter.c
@@ -2220,6 +2220,16 @@ static bool parse_transport(DBusMessageIter *value, uint8_t *transport)
 	return true;
 }
 
+static bool parse_le_duplicates(DBusMessageIter *value, bool *le_duplicates)
+{
+	if (dbus_message_iter_get_arg_type(value) != DBUS_TYPE_BOOLEAN)
+		return false;
+
+	dbus_message_iter_get_basic(value, le_duplicates);
+
+	return true;
+}
+
 static bool parse_discovery_filter_entry(char *key, DBusMessageIter *value,
 						struct discovery_filter *filter)
 {
@@ -2235,6 +2245,9 @@ static bool parse_discovery_filter_entry(char *key, DBusMessageIter *value,
 	if (!strcmp("Transport", key))
 		return parse_transport(value, &filter->type);
 
+	if (!strcmp("LE_Duplicates", key))
+		return parse_le_duplicates(value, &filter->le_duplicates);
+
 	DBG("Unknown key parameter: %s!\n", key);
 	return false;
 }
@@ -2260,6 +2273,7 @@ static bool parse_discovery_filter_dict(struct btd_adapter *adapter,
 	(*filter)->pathloss = DISTANCE_VAL_INVALID;
 	(*filter)->rssi = DISTANCE_VAL_INVALID;
 	(*filter)->type = get_scan_type(adapter);
+	(*filter)->le_duplicates = false;
 
 	dbus_message_iter_init(msg, &iter);
 	if (dbus_message_iter_get_arg_type(&iter) != DBUS_TYPE_ARRAY ||
@@ -2304,8 +2318,12 @@ static bool parse_discovery_filter_dict(struct btd_adapter *adapter,
 	    (*filter)->rssi != DISTANCE_VAL_INVALID)
 		goto invalid_args;
 
-	DBG("filtered discovery params: transport: %d rssi: %d pathloss: %d",
-	    (*filter)->type, (*filter)->rssi, (*filter)->pathloss);
+	/* only allow LE duplicates for type LE */
+	if ((*filter)->le_duplicates && (*filter)->type != SCAN_TYPE_LE)
+		goto invalid_args;
+
+	DBG("filtered discovery params: transport: %d rssi: %d pathloss: %d le_duplicates: %d",
+	    (*filter)->type, (*filter)->rssi, (*filter)->pathloss, (*filter)->le_duplicates);
 
 	return true;
 
-- 
1.8.3.1

Re: [RFC 0/3] core/adapter: Add disabling duplicate device filtering from d-bus

[Luiz Augusto von Dentz]

Hi Don,

On Tue, Dec 6, 2016 at 11:39 PM, Don Zickus <dzickus@redhat.com> wrote:
> Recent discussions on the bluez mailing list revealed it was not easy
> to disable duplicate device filtering from the d-bus interface.
>
> As a result, if I wanted to monitor LE devices entering and leaving the
> adapters range (using RSSI data), it was difficult.
>
> This patchset is a dirty hack to make this work. The first patch enables
> it on the kernel side, while the other two patches enable it from the bluez
> side.
>
> I understand there are concerns about flooding the d-bus interface when
> enabling this.  I tried to write a throttling mechanism using the mainloop
> as my timer, but soon realized you can only have 1 device RSSI event
> per loop, so that wasn't going to work.  Open to suggestions if still a
> concern.
>
> Posted as an RFC just to generate discussion.  I expect I missed a lot of
> little details here, but wanted to post my proof of concept to see if this
> is something to work with.
>
> This patchset includes both the kernel and bluez patches.  I understand this is
> not recommended for normal practice.  But I thought for an RFC, it is nice to
> keep things together for now.

I would avoid adding a new MGMT command and instead disabled
duplicated filtering if RSSI filtering is set since anyway RSSI
filtering needs to disable duplicates in order to do any RSSI
filtering reliable. So this would mean that if the user wants to see
to get duplicate filtering it needs to set a RSSI which should rate
limit as we would use a threshold.

>
>
> kernel:
>
> Don Zickus (1):
>   bluetooth: Add managment ability to disable duplicate device fitlering
>
>  include/net/bluetooth/hci_core.h | 1 +
>  include/net/bluetooth/mgmt.h     | 1 +
>  net/bluetooth/hci_request.c      | 2 +-
>  net/bluetooth/mgmt.c             | 5 +++++
>  4 files changed, 8 insertions(+), 1 deletion(-)
>
>
> bluez:
>
> Don Zickus (2):
>   core/adapter: Add le_duplicates flag to management interface
>   core/adapter: Hook le_duplicates into d-bus interface
>
>  client/main.c       |  5 +++++
>  doc/adapter-api.txt |  1 +
>  lib/mgmt.h          |  1 +
>  src/adapter.c       | 37 +++++++++++++++++++++++++++++++------
>  4 files changed, 38 insertions(+), 6 deletions(-)
>
> --
> 1.8.3.1
>



-- 
Luiz Augusto von Dentz

Re: [RFC 0/3] core/adapter: Add disabling duplicate device filtering from d-bus

[Don Zickus]

On Wed, Dec 07, 2016 at 12:21:58PM +0200, Luiz Augusto von Dentz wrote:
> Hi Don,
> 
> On Tue, Dec 6, 2016 at 11:39 PM, Don Zickus <dzickus@redhat.com> wrote:
> > Recent discussions on the bluez mailing list revealed it was not easy
> > to disable duplicate device filtering from the d-bus interface.
> >
> > As a result, if I wanted to monitor LE devices entering and leaving the
> > adapters range (using RSSI data), it was difficult.
> >
> > This patchset is a dirty hack to make this work. The first patch enables
> > it on the kernel side, while the other two patches enable it from the bluez
> > side.
> >
> > I understand there are concerns about flooding the d-bus interface when
> > enabling this.  I tried to write a throttling mechanism using the mainloop
> > as my timer, but soon realized you can only have 1 device RSSI event
> > per loop, so that wasn't going to work.  Open to suggestions if still a
> > concern.
> >
> > Posted as an RFC just to generate discussion.  I expect I missed a lot of
> > little details here, but wanted to post my proof of concept to see if this
> > is something to work with.
> >
> > This patchset includes both the kernel and bluez patches.  I understand this is
> > not recommended for normal practice.  But I thought for an RFC, it is nice to
> > keep things together for now.
> 
> I would avoid adding a new MGMT command and instead disabled
> duplicated filtering if RSSI filtering is set since anyway RSSI
> filtering needs to disable duplicates in order to do any RSSI
> filtering reliable. So this would mean that if the user wants to see
> to get duplicate filtering it needs to set a RSSI which should rate
> limit as we would use a threshold.

Hi Luiz,

Ok, fair enough.  I then simplified it down to a small kernel patch that
seems to work if I set an RSSI threshold with SetDiscoveryFilter.

I only do this on an active_scan.  Not sure if I should do it for the
passive scans too?

If this looks ok, I will resubmit properly.

Cheers,
Don


diff --git a/include/net/bluetooth/hci_core.h b/include/net/bluetooth/hci_core.h
index 554671c..fd34707 100644
--- a/include/net/bluetooth/hci_core.h
+++ b/include/net/bluetooth/hci_core.h
@@ -79,6 +79,7 @@ struct discovery_state {
 	bool			report_invalid_rssi;
 	bool			result_filtering;
 	bool			limited;
+	bool			filter_dups;
 	s8			rssi;
 	u16			uuid_count;
 	u8			(*uuids)[16];
diff --git a/net/bluetooth/hci_request.c b/net/bluetooth/hci_request.c
index 1015d9c..8b5ee64 100644
--- a/net/bluetooth/hci_request.c
+++ b/net/bluetooth/hci_request.c
@@ -2038,7 +2038,7 @@ static int active_scan(struct hci_request *req, unsigned long opt)
 
 	memset(&enable_cp, 0, sizeof(enable_cp));
 	enable_cp.enable = LE_SCAN_ENABLE;
-	enable_cp.filter_dup = LE_SCAN_FILTER_DUP_ENABLE;
+	enable_cp.filter_dup = hdev->discovery.filter_dups;
 
 	hci_req_add(req, HCI_OP_LE_SET_SCAN_ENABLE, sizeof(enable_cp),
 		    &enable_cp);
diff --git a/net/bluetooth/mgmt.c b/net/bluetooth/mgmt.c
index 1fba2a0..0ac6e6f 100644
--- a/net/bluetooth/mgmt.c
+++ b/net/bluetooth/mgmt.c
@@ -3578,6 +3578,7 @@ static int start_service_discovery(struct sock *sk, struct hci_dev *hdev,
 	struct mgmt_pending_cmd *cmd;
 	const u16 max_uuid_count = ((U16_MAX - sizeof(*cp)) / 16);
 	u16 uuid_count, expected_len;
+	bool filter_dups = LE_SCAN_FILTER_DUP_ENABLE;
 	u8 status;
 	int err;
 
@@ -3631,6 +3632,9 @@ static int start_service_discovery(struct sock *sk, struct hci_dev *hdev,
 		goto failed;
 	}
 
+	if (cp->rssi)
+		filter_dups = LE_SCAN_FILTER_DUP_DISABLE;
+
 	cmd = mgmt_pending_add(sk, MGMT_OP_START_SERVICE_DISCOVERY,
 			       hdev, data, len);
 	if (!cmd) {
@@ -3649,6 +3653,7 @@ static int start_service_discovery(struct sock *sk, struct hci_dev *hdev,
 	hdev->discovery.type = cp->type;
 	hdev->discovery.rssi = cp->rssi;
 	hdev->discovery.uuid_count = uuid_count;
+	hdev->discovery.filter_dups = filter_dups;
 
 	if (uuid_count > 0) {
 		hdev->discovery.uuids = kmemdup(cp->uuids, uuid_count * 16,

Re: [RFC 0/3] core/adapter: Add disabling duplicate device filtering from d-bus

[Marcel Holtmann]

Hi Don,

>>> Recent discussions on the bluez mailing list revealed it was not easy
>>> to disable duplicate device filtering from the d-bus interface.
>>> 
>>> As a result, if I wanted to monitor LE devices entering and leaving the
>>> adapters range (using RSSI data), it was difficult.
>>> 
>>> This patchset is a dirty hack to make this work. The first patch enables
>>> it on the kernel side, while the other two patches enable it from the bluez
>>> side.
>>> 
>>> I understand there are concerns about flooding the d-bus interface when
>>> enabling this.  I tried to write a throttling mechanism using the mainloop
>>> as my timer, but soon realized you can only have 1 device RSSI event
>>> per loop, so that wasn't going to work.  Open to suggestions if still a
>>> concern.
>>> 
>>> Posted as an RFC just to generate discussion.  I expect I missed a lot of
>>> little details here, but wanted to post my proof of concept to see if this
>>> is something to work with.
>>> 
>>> This patchset includes both the kernel and bluez patches.  I understand this is
>>> not recommended for normal practice.  But I thought for an RFC, it is nice to
>>> keep things together for now.
>> 
>> I would avoid adding a new MGMT command and instead disabled
>> duplicated filtering if RSSI filtering is set since anyway RSSI
>> filtering needs to disable duplicates in order to do any RSSI
>> filtering reliable. So this would mean that if the user wants to see
>> to get duplicate filtering it needs to set a RSSI which should rate
>> limit as we would use a threshold.
> 
> Hi Luiz,
> 
> Ok, fair enough.  I then simplified it down to a small kernel patch that
> seems to work if I set an RSSI threshold with SetDiscoveryFilter.
> 
> I only do this on an active_scan.  Not sure if I should do it for the
> passive scans too?
> 
> If this looks ok, I will resubmit properly.

I am not sure this is the best idea. So right now the kernel restarts scanning to get new RSSI values when we have a quirk set that strict scan filtering is performed by the controller (strict means address only).

That is for discovery. Something started by the user and something that does not last over long times. Discovery is using active scanning which takes way more TX/RX time on air than passive scanning. It is also known to be causing more problems then it solves when run constantly. The reason is that it will interrupt CONNECT_REQ since SCAN_REQ are more likely to win based on their shorter packet size.

So the main question is if you are after a certain set of devices (based on their address) and want to monitor their RSSI or are you after any device and want to match based on advertising data?

The first one could be dealt with by using the whitelist, the second one is nasty from a power consumption point of view. Any long term operation there would be causing major headaches.

Keep in mind that besides the kernel waking up (and USB transport), you also end up waking userspace all the time to process the information and send out over D-Bus. Which means you might going to wakeup even more processes.

Regards

Marcel

Re: [RFC 0/3] core/adapter: Add disabling duplicate device filtering from d-bus

[Northfield Stuart]

[-- Attachment #1: Type: text/plain, Size: 5628 bytes --]


> On 8 Dec 2016, at 06:18, Marcel Holtmann <marcel@holtmann.org> wrote:
> 
> Hi Don,
> 
>>>> Recent discussions on the bluez mailing list revealed it was not easy
>>>> to disable duplicate device filtering from the d-bus interface.
>>>> 
>>>> As a result, if I wanted to monitor LE devices entering and leaving the
>>>> adapters range (using RSSI data), it was difficult.
>>>> 
>>>> This patchset is a dirty hack to make this work. The first patch enables
>>>> it on the kernel side, while the other two patches enable it from the bluez
>>>> side.
>>>> 
>>>> I understand there are concerns about flooding the d-bus interface when
>>>> enabling this.  I tried to write a throttling mechanism using the mainloop
>>>> as my timer, but soon realized you can only have 1 device RSSI event
>>>> per loop, so that wasn't going to work.  Open to suggestions if still a
>>>> concern.
>>>> 
>>>> Posted as an RFC just to generate discussion.  I expect I missed a lot of
>>>> little details here, but wanted to post my proof of concept to see if this
>>>> is something to work with.
>>>> 
>>>> This patchset includes both the kernel and bluez patches.  I understand this is
>>>> not recommended for normal practice.  But I thought for an RFC, it is nice to
>>>> keep things together for now.
>>> 
>>> I would avoid adding a new MGMT command and instead disabled
>>> duplicated filtering if RSSI filtering is set since anyway RSSI
>>> filtering needs to disable duplicates in order to do any RSSI
>>> filtering reliable. So this would mean that if the user wants to see
>>> to get duplicate filtering it needs to set a RSSI which should rate
>>> limit as we would use a threshold.
>> 
>> Hi Luiz,
>> 
>> Ok, fair enough.  I then simplified it down to a small kernel patch that
>> seems to work if I set an RSSI threshold with SetDiscoveryFilter.
>> 
>> I only do this on an active_scan.  Not sure if I should do it for the
>> passive scans too?
>> 
>> If this looks ok, I will resubmit properly.
> 
> I am not sure this is the best idea. So right now the kernel restarts scanning to get new RSSI values when we have a quirk set that strict scan filtering is performed by the controller (strict means address only).
> 
> That is for discovery. Something started by the user and something that does not last over long times. Discovery is using active scanning which takes way more TX/RX time on air than passive scanning. It is also known to be causing more problems then it solves when run constantly. The reason is that it will interrupt CONNECT_REQ since SCAN_REQ are more likely to win based on their shorter packet size.
> 
> So the main question is if you are after a certain set of devices (based on their address) and want to monitor their RSSI or are you after any device and want to match based on advertising data?
> 
> The first one could be dealt with by using the whitelist, the second one is nasty from a power consumption point of view. Any long term operation there would be causing major headaches.
> 
> Keep in mind that besides the kernel waking up (and USB transport), you also end up waking userspace all the time to process the information and send out over D-Bus. Which means you might going to wakeup even more processes.

Can I just leap in here, as this is extremely relevant to the project I am working on. Sometimes you need to step back from what the ‘average user’ needs, and consider whether should allow certain behaviours to be enabled, even if they are not the default. What is ‘nasty’ to you in the big picture, is perfectly normal for me in a dedicated use case.

Our application transfers data between devices (and there could be well over a thousand within range at a single time) and the central system using purely the MSD in the advertisements. Our linux ‘gateway’ boxes need to have continuous reception of advertisements, and we don’t care about the power consumption or processing implications - that is their dedicated job. We are currently unable to migrate our code from some older gatttool derived code to the modern APIs because it simply isn’t possible to work in the manner we need. (Note that our gateways use multiple BLE dongles, with at least one dedicated to receiving beacons while the others are used to connect to devices for archived data retrieval.)

Please give some consideration to the fact that not everybody is using bluetooth for simple end-user devices, and whilst these are clearly the dominant use case, being able to engage esoteric modes of operation allows those of us who need to to consciously engage them as necessary for bespoke scenarios/applications. BLE is not just a desktop/peripheral technology :)

Hope you don’t mind the interjection…

Regards

Stu
--
Stuart Northfield
+44 (0) 1223 566728 (Direct), +44 (0) 1223 566727 (Fax)
Metanate Limited. Registered in England No 4046086 at:
Lincoln House, Station Court, Great Shelford, Cambridge CB22 5NE, UK
www.metanate.com (Consultancy) www.schemus.com (Data synchronisation)

This e-mail and all attachments it may contain is confidential and
intended solely for the use of the individual to whom it is addressed.
Any views or opinions presented are those of the author and do not
necessarily represent those of Metanate Ltd.  If you are not the
intended recipient, be advised that you have received this e-mail in
error and that any use, dissemination, printing, forwarding or copying
of this e-mail is strictly prohibited.  Please contact the sender if
you have received this e-mail in error.







[-- Attachment #2: smime.p7s --]
[-- Type: application/pkcs7-signature, Size: 1637 bytes --]

Re: [RFC 0/3] core/adapter: Add disabling duplicate device filtering from d-bus

[Don Zickus]

(please remember to cc me as I am not subscribed to the list)

On Thu, Dec 08, 2016 at 12:29:18PM +0000, Northfield Stuart wrote:
> 
> > On 8 Dec 2016, at 06:18, Marcel Holtmann <marcel@holtmann.org> wrote:
> > 
> > Hi Don,
> > 
> >>>> Recent discussions on the bluez mailing list revealed it was not easy
> >>>> to disable duplicate device filtering from the d-bus interface.
> >>>> 
> >>>> As a result, if I wanted to monitor LE devices entering and leaving the
> >>>> adapters range (using RSSI data), it was difficult.
> >>>> 
> >>>> This patchset is a dirty hack to make this work. The first patch enables
> >>>> it on the kernel side, while the other two patches enable it from the bluez
> >>>> side.
> >>>> 
> >>>> I understand there are concerns about flooding the d-bus interface when
> >>>> enabling this.  I tried to write a throttling mechanism using the mainloop
> >>>> as my timer, but soon realized you can only have 1 device RSSI event
> >>>> per loop, so that wasn't going to work.  Open to suggestions if still a
> >>>> concern.
> >>>> 
> >>>> Posted as an RFC just to generate discussion.  I expect I missed a lot of
> >>>> little details here, but wanted to post my proof of concept to see if this
> >>>> is something to work with.
> >>>> 
> >>>> This patchset includes both the kernel and bluez patches.  I understand this is
> >>>> not recommended for normal practice.  But I thought for an RFC, it is nice to
> >>>> keep things together for now.
> >>> 
> >>> I would avoid adding a new MGMT command and instead disabled
> >>> duplicated filtering if RSSI filtering is set since anyway RSSI
> >>> filtering needs to disable duplicates in order to do any RSSI
> >>> filtering reliable. So this would mean that if the user wants to see
> >>> to get duplicate filtering it needs to set a RSSI which should rate
> >>> limit as we would use a threshold.
> >> 
> >> Hi Luiz,
> >> 
> >> Ok, fair enough.  I then simplified it down to a small kernel patch that
> >> seems to work if I set an RSSI threshold with SetDiscoveryFilter.
> >> 
> >> I only do this on an active_scan.  Not sure if I should do it for the
> >> passive scans too?
> >> 
> >> If this looks ok, I will resubmit properly.
> > 
> > I am not sure this is the best idea. So right now the kernel restarts scanning to get new RSSI values when we have a quirk set that strict scan filtering is performed by the controller (strict means address only).
> > 
> > That is for discovery. Something started by the user and something that does not last over long times. Discovery is using active scanning which takes way more TX/RX time on air than passive scanning. It is also known to be causing more problems then it solves when run constantly. The reason is that it will interrupt CONNECT_REQ since SCAN_REQ are more likely to win based on their shorter packet size.
> > 
> > So the main question is if you are after a certain set of devices (based on their address) and want to monitor their RSSI or are you after any device and want to match based on advertising data?
> > 
> > The first one could be dealt with by using the whitelist, the second one is nasty from a power consumption point of view. Any long term operation there would be causing major headaches.
> > 
> > Keep in mind that besides the kernel waking up (and USB transport), you also end up waking userspace all the time to process the information and send out over D-Bus. Which means you might going to wakeup even more processes.
> 
> Can I just leap in here, as this is extremely relevant to the project I am working on. Sometimes you need to step back from what the ‘average user’ needs, and consider whether should allow certain behaviours to be enabled, even if they are not the default. What is ‘nasty’ to you in the big picture, is perfectly normal for me in a dedicated use case.
> 
> Our application transfers data between devices (and there could be well over a thousand within range at a single time) and the central system using purely the MSD in the advertisements. Our linux ‘gateway’ boxes need to have continuous reception of advertisements, and we don’t care about the power consumption or processing implications - that is their dedicated job. We are currently unable to migrate our code from some older gatttool derived code to the modern APIs because it simply isn’t possible to work in the manner we need. (Note that our gateways use multiple BLE dongles, with at least one dedicated to receiving beacons while the others are used to connect to devices for archived data retrieval.)

Hi,

I want to echo Stu's application.  The idea is to use bluetooth to
constantly scan the area for devices coming and going as a way to track
them (for IoT purposes).  Scanning for advertisement data seems to make
things easier for bluetooth tags to transmit for tracking as we don't have
to make connections are really gather any data other than address and maybe
RSSI.

I understand the power implications but the machine is plugged into the wall
and acts more like a server, so it is expected to consume lots of power.
It's the beacons that I would like to consume as little power as possible
(trying to figure out a good update frequency).

I really don't care how the scanning is done (active or passive), just as
long as I can detect devices in the area within a reasonable time slice.


I am happy to code up any suggestions, just looking for guidance. :-)

Cheers,
Don

Re: [RFC 0/3] core/adapter: Add disabling duplicate device filtering from d-bus

[Marcel Holtmann]

Hi Don,

>>>>>> Recent discussions on the bluez mailing list revealed it was not easy
>>>>>> to disable duplicate device filtering from the d-bus interface.
>>>>>> 
>>>>>> As a result, if I wanted to monitor LE devices entering and leaving the
>>>>>> adapters range (using RSSI data), it was difficult.
>>>>>> 
>>>>>> This patchset is a dirty hack to make this work. The first patch enables
>>>>>> it on the kernel side, while the other two patches enable it from the bluez
>>>>>> side.
>>>>>> 
>>>>>> I understand there are concerns about flooding the d-bus interface when
>>>>>> enabling this.  I tried to write a throttling mechanism using the mainloop
>>>>>> as my timer, but soon realized you can only have 1 device RSSI event
>>>>>> per loop, so that wasn't going to work.  Open to suggestions if still a
>>>>>> concern.
>>>>>> 
>>>>>> Posted as an RFC just to generate discussion.  I expect I missed a lot of
>>>>>> little details here, but wanted to post my proof of concept to see if this
>>>>>> is something to work with.
>>>>>> 
>>>>>> This patchset includes both the kernel and bluez patches.  I understand this is
>>>>>> not recommended for normal practice.  But I thought for an RFC, it is nice to
>>>>>> keep things together for now.
>>>>> 
>>>>> I would avoid adding a new MGMT command and instead disabled
>>>>> duplicated filtering if RSSI filtering is set since anyway RSSI
>>>>> filtering needs to disable duplicates in order to do any RSSI
>>>>> filtering reliable. So this would mean that if the user wants to see
>>>>> to get duplicate filtering it needs to set a RSSI which should rate
>>>>> limit as we would use a threshold.
>>>> 
>>>> Hi Luiz,
>>>> 
>>>> Ok, fair enough.  I then simplified it down to a small kernel patch that
>>>> seems to work if I set an RSSI threshold with SetDiscoveryFilter.
>>>> 
>>>> I only do this on an active_scan.  Not sure if I should do it for the
>>>> passive scans too?
>>>> 
>>>> If this looks ok, I will resubmit properly.
>>> 
>>> I am not sure this is the best idea. So right now the kernel restarts scanning to get new RSSI values when we have a quirk set that strict scan filtering is performed by the controller (strict means address only).
>>> 
>>> That is for discovery. Something started by the user and something that does not last over long times. Discovery is using active scanning which takes way more TX/RX time on air than passive scanning. It is also known to be causing more problems then it solves when run constantly. The reason is that it will interrupt CONNECT_REQ since SCAN_REQ are more likely to win based on their shorter packet size.
>>> 
>>> So the main question is if you are after a certain set of devices (based on their address) and want to monitor their RSSI or are you after any device and want to match based on advertising data?
>>> 
>>> The first one could be dealt with by using the whitelist, the second one is nasty from a power consumption point of view. Any long term operation there would be causing major headaches.
>>> 
>>> Keep in mind that besides the kernel waking up (and USB transport), you also end up waking userspace all the time to process the information and send out over D-Bus. Which means you might going to wakeup even more processes.
>> 
>> Can I just leap in here, as this is extremely relevant to the project I am working on. Sometimes you need to step back from what the ‘average user’ needs, and consider whether should allow certain behaviours to be enabled, even if they are not the default. What is ‘nasty’ to you in the big picture, is perfectly normal for me in a dedicated use case.
>> 
>> Our application transfers data between devices (and there could be well over a thousand within range at a single time) and the central system using purely the MSD in the advertisements. Our linux ‘gateway’ boxes need to have continuous reception of advertisements, and we don’t care about the power consumption or processing implications - that is their dedicated job. We are currently unable to migrate our code from some older gatttool derived code to the modern APIs because it simply isn’t possible to work in the manner we need. (Note that our gateways use multiple BLE dongles, with at least one dedicated to receiving beacons while the others are used to connect to devices for archived data retrieval.)
> 
> Hi,
> 
> I want to echo Stu's application.  The idea is to use bluetooth to
> constantly scan the area for devices coming and going as a way to track
> them (for IoT purposes).  Scanning for advertisement data seems to make
> things easier for bluetooth tags to transmit for tracking as we don't have
> to make connections are really gather any data other than address and maybe
> RSSI.
> 
> I understand the power implications but the machine is plugged into the wall
> and acts more like a server, so it is expected to consume lots of power.
> It's the beacons that I would like to consume as little power as possible
> (trying to figure out a good update frequency).
> 
> I really don't care how the scanning is done (active or passive), just as
> long as I can detect devices in the area within a reasonable time slice.

actually I care about active scan mainly since that causes disturbance for everybody around you. As I said, that is a big deal if you go for active scanning (in our case discovery).

That said, if people want such an interface, then we have to build one. However it needs to be a separate and dedicated interface. It also needs to be flexible enough that when some sort of filtering exists, it can be offloaded into the controller. I think the difference between scanning for discovery and this is that you are expecting it to run all the time.

> I am happy to code up any suggestions, just looking for guidance. :-)

What we need is an extra set management APIs that allow for telling the kernel what you are looking for and how long. The filtered discovery is nice, but I doubt it is what you are looking for. The only real question to answer is what is the refresh time expected of advertising data and RSSI changes. New devices are found quickly since that can be done by duplicate filtering, but if you want to monitor the RSSI or look for changed advertising data, we need to do things differently.

So before anybody starts coding, I think we need to figure out what we want and define the mgmt API to do this. The coding part will be relatively simple in the end.

Regards

Marcel

Re: [RFC 0/3] core/adapter: Add disabling duplicate device filtering from d-bus

[Don Zickus]

(I just subscribed to the bluetooth mailing list, but I don't have Marcel's
reply, so I am copying-n-pasting it here for a reply.  Sorry about the
ugliness of it)


Hi Marcel,

(copy-n-pasted response)

>> 
>> Can I just leap in here, as this is extremely relevant to the project I
>> am working on. Sometimes you need to step back from what the ‘average
>> user’ needs, and consider whether should allow certain behaviours to be
>> enabled, even if they are not the default. What is ‘nasty’ to you in
>> the big picture, is perfectly normal for me in a dedicated use case.
>> 
>> Our application transfers data between devices (and there could be well
>> over a thousand within range at a single time) and the central system
>> using purely the MSD in the advertisements. Our linux ‘gateway’ boxes
>> need to have continuous reception of advertisements, and we don’t care
>> about the power consumption or processing implications - that is their
>> dedicated job. We are currently unable to migrate our code from some
>> older gatttool derived code to the modern APIs because it simply isn’t
>> possible to work in the manner we need. (Note that our gateways use
>> multiple BLE dongles, with at least one dedicated to receiving beacons
>> while the others are used to connect to devices for archived data
>> retrieval.)
> 
> Hi,
> 
> I want to echo Stu's application.  The idea is to use bluetooth to
> constantly scan the area for devices coming and going as a way to track
> them (for IoT purposes).  Scanning for advertisement data seems to make
> things easier for bluetooth tags to transmit for tracking as we don't have
> to make connections are really gather any data other than address and
> maybe
> RSSI.
> 
> I understand the power implications but the machine is plugged into the
> wall
> and acts more like a server, so it is expected to consume lots of power.
> It's the beacons that I would like to consume as little power as possible
> (trying to figure out a good update frequency).
> 
> I really don't care how the scanning is done (active or passive), just as
> long as I can detect devices in the area within a reasonable time slice.

actually I care about active scan mainly since that causes disturbance for
everybody around you. As I said, that is a big deal if you go for active
scanning (in our case discovery).

That said, if people want such an interface, then we have to build one.
However it needs to be a separate and dedicated interface. It also needs to
be flexible enough that when some sort of filtering exists, it can be
offloaded into the controller. I think the difference between scanning for
discovery and this is that you are expecting it to run all the time.

> I am happy to code up any suggestions, just looking for guidance. :-)

What we need is an extra set management APIs that allow for telling the
kernel what you are looking for and how long. The filtered discovery is
nice, but I doubt it is what you are looking for. The only real question to
answer is what is the refresh time expected of advertising data and RSSI
changes. New devices are found quickly since that can be done by duplicate
filtering, but if you want to monitor the RSSI or look for changed
advertising data, we need to do things differently.

So before anybody starts coding, I think we need to figure out what we want
and define the mgmt API to do this. The coding part will be relatively
simple in the end.

Regards

Marcel

---------my response----

I am not entirely sure how to design it or provide the exact answers you are
looking for, but here is my initial thoughts if that helps.

- I wasn't looking for a 100% sampling, far less is fine as long as I can
  determine every second?? or so, if the device is still sending anything
  (adv.?)

- iBeacons supposedly broadcast every 100ms 3 packets?, so sampling 1/second
  for 100-200ms long is a good start?  that should be able to pick up one
  round.

- I really don't need to 'add a device' to any tables as this table will
  grow to be very huge unless we have rules to 'remove a device' rather
  quickly.

- so instead of 'discovering a new device' and registering a callback on
  that device's changing properties, just a generic interface to provide
  updated info as it becomes available?

- ability to tweak the above scan settings (frequency/duration)?

Just my initial thoughts.  Is there more specific info you need?

I am not sure if the above meets Stu's or Brennan's needs.

Cheers,
Don

Re: [RFC 0/3] core/adapter: Add disabling duplicate device filtering from d-bus

[Brennan Ashton]

On Thu, 2016-12-08 at 12:29 +0000, Northfield Stuart wrote:
> > 
> > On 8 Dec 2016, at 06:18, Marcel Holtmann <marcel@holtmann.org>
> > wrote:
> > 
> > Hi Don,
> > 
> > > 
> > > > 
> > > > > 
> > > > > Recent discussions on the bluez mailing list revealed it was
> > > > > not easy
> > > > > to disable duplicate device filtering from the d-bus
> > > > > interface.
> > > > > 
> > > > > As a result, if I wanted to monitor LE devices entering and
> > > > > leaving the
> > > > > adapters range (using RSSI data), it was difficult.
> > > > > 
> > > > > This patchset is a dirty hack to make this work. The first
> > > > > patch enables
> > > > > it on the kernel side, while the other two patches enable it
> > > > > from the bluez
> > > > > side.
> > > > > 
> > > > > I understand there are concerns about flooding the d-bus
> > > > > interface when
> > > > > enabling this.  I tried to write a throttling mechanism using
> > > > > the mainloop
> > > > > as my timer, but soon realized you can only have 1 device
> > > > > RSSI event
> > > > > per loop, so that wasn't going to work.  Open to suggestions
> > > > > if still a
> > > > > concern.
> > > > > 
> > > > > Posted as an RFC just to generate discussion.  I expect I
> > > > > missed a lot of
> > > > > little details here, but wanted to post my proof of concept
> > > > > to see if this
> > > > > is something to work with.
> > > > > 
> > > > > This patchset includes both the kernel and bluez patches.  I
> > > > > understand this is
> > > > > not recommended for normal practice.  But I thought for an
> > > > > RFC, it is nice to
> > > > > keep things together for now.
> > > > 
> > > > I would avoid adding a new MGMT command and instead disabled
> > > > duplicated filtering if RSSI filtering is set since anyway RSSI
> > > > filtering needs to disable duplicates in order to do any RSSI
> > > > filtering reliable. So this would mean that if the user wants
> > > > to see
> > > > to get duplicate filtering it needs to set a RSSI which should
> > > > rate
> > > > limit as we would use a threshold.
> > > 
> > > Hi Luiz,
> > > 
> > > Ok, fair enough.  I then simplified it down to a small kernel
> > > patch that
> > > seems to work if I set an RSSI threshold with SetDiscoveryFilter.
> > > 
> > > I only do this on an active_scan.  Not sure if I should do it for
> > > the
> > > passive scans too?
> > > 
> > > If this looks ok, I will resubmit properly.
> > 
> > I am not sure this is the best idea. So right now the kernel
> > restarts scanning to get new RSSI values when we have a quirk set
> > that strict scan filtering is performed by the controller (strict
> > means address only).
> > 
> > That is for discovery. Something started by the user and something
> > that does not last over long times. Discovery is using active
> > scanning which takes way more TX/RX time on air than passive
> > scanning. It is also known to be causing more problems then it
> > solves when run constantly. The reason is that it will interrupt
> > CONNECT_REQ since SCAN_REQ are more likely to win based on their
> > shorter packet size.
> > 
> > So the main question is if you are after a certain set of devices
> > (based on their address) and want to monitor their RSSI or are you
> > after any device and want to match based on advertising data?
> > 
> > The first one could be dealt with by using the whitelist, the
> > second one is nasty from a power consumption point of view. Any
> > long term operation there would be causing major headaches.
> > 
> > Keep in mind that besides the kernel waking up (and USB transport),
> > you also end up waking userspace all the time to process the
> > information and send out over D-Bus. Which means you might going to
> > wakeup even more processes.
> 
> Can I just leap in here, as this is extremely relevant to the project
> I am working on. Sometimes you need to step back from what the
> ‘average user’ needs, and consider whether should allow certain
> behaviours to be enabled, even if they are not the default. What is
> ‘nasty’ to you in the big picture, is perfectly normal for me in a
> dedicated use case.
> 
> Our application transfers data between devices (and there could be
> well over a thousand within range at a single time) and the central
> system using purely the MSD in the advertisements. Our linux
> ‘gateway’ boxes need to have continuous reception of advertisements,
> and we don’t care about the power consumption or processing
> implications - that is their dedicated job. We are currently unable
> to migrate our code from some older gatttool derived code to the
> modern APIs because it simply isn’t possible to work in the manner we
> need. (Note that our gateways use multiple BLE dongles, with at least
> one dedicated to receiving beacons while the others are used to
> connect to devices for archived data retrieval.)
> 
> Please give some consideration to the fact that not everybody is
> using bluetooth for simple end-user devices, and whilst these are
> clearly the dominant use case, being able to engage esoteric modes of
> operation allows those of us who need to to consciously engage them
> as necessary for bespoke scenarios/applications. BLE is not just a
> desktop/peripheral technology :)
> 
> Hope you don’t mind the interjection…
> 
> Regards
> 
> Stu

I want to second this as well.  I have a device that always scans and
processes all advertisement packets with manufacture specific data.  I
also constantly am updating the advertisement packets.  Power really is
not a worry here.  In the current design I have I switch between modes
where I use the management interface to perform GATT connections and
then fall back to raw HCI to process the packets.

--Brennan