[Bluez-devel] Handling a Hardware Error event on iPAQ h5550

[Bluez-devel] Handling a Hardware Error event on iPAQ h5550

[Catalin Drula]

Hi,

I was trying to debug the Bluetooth subsystem on an HP iPAQ h5550 running
Linux:

root@h3900:~# hciconfig -a
hci0:   Type: UART
        BD Address: 08:00:17:1A:EB:76 ACL MTU: 339:4 SCO MTU: 60:9
        UP RUNNING PSCAN ISCAN
        RX bytes:-2117082264 acl:19904935 sco:0 events:4940395 errors:0
        TX bytes:386017560 acl:9879838 sco:0 commands:430 errors:0
        Features: 0xff 0x3b 0x05 0x00 0x00 0x00 0x00 0x00
        Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3
        Link policy:
        Link mode: SLAVE ACCEPT
        Name: 'POCKET_PC'
        Class: 0x000000
        Service Classes: Unspecified
        Device Class: Miscellaneous,
        HCI Ver: 1.1 (0x1) HCI Rev: 0x180 LMP Ver: 1.1 (0x1) LMP Subver: 0x180
        Manufacturer: RTX Telecom A/S (21)

The problem I am facing on this machine is that it seems that the UART
communication is sometimes unreliable. What happens is that while sending
data, there sometimes seems to be corruption between the host and the
controller. For example, doing an l2test where the iPAQ is sending to a
PC, every once in a while on the receiving side there are data or sequence
or size mismatches. After a while, the corrupted bytes happen to be in the
H4 header, and the controller assumes it has lost synchronization with the
host, thus sending a "Hardware Error" event. It then expects a "Reset"
command to resynchronize.

Since a lot of digging has not found the root cause of these errors in the
UART communication, I wanted to hack the Bluez stack to better handle when
these loss of synchronization events occur (I am not refering to patching
the main Bluez stack, but rather make a hack for our own purposes here,
and perhaps share it with other iPAQ h5550 users).

It is easy enough to add the sending of a "Reset" command after a
"Hardware Error" event is received, but I get the impression that there's
more needed, such as tearing down the established connections somehow. The
point is to get a "clean" stack to start with after such an event (and of
course, propagate the error up the stack so that the application gets a
"broken connection"). Any advice on how hard it is to do that, and what
needs to be done?

Currently, what happens is that after sending "Reset", we resynchronize
with the controller, but the previous connections still show:

root@h3900:~# hcitool con
Connections:
        < ACL 00:10:DC:57:C9:79 handle 1 state 8 lm MASTER
        > ACL 08:00:17:1A:E6:02 handle 2 state 8 lm SLAVE

Thanks in advance for your help,

Catalin



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Bluez-devel mailing list
Bluez-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bluez-devel

Re: [Bluez-devel] Handling a Hardware Error event on iPAQ h5550

[Peter Wippich]

Hi Catalin,

I'm not that familiar with the architecture of BlueZ but with the
experiences for our own stack I can tell you this is not trivial. The main
problem here is that the Bluetooth Stack is not stateless. One possible
solution is to feed the reset done event to the higher layers (l2cap). But
than L2cap must be modified to correctly shut down all connections. It
would be easier if the HCI layer tracks the state of all ACL and SCO
connections (Marcel ??). Than a reset done event can be handled like a
unexpected link loss on all active connections and the higher layers do
not need any modifications (hopefully).

Anyway, I'm pretty sure there is no way to solve this with a simple hack.

It may be much simpler trying to find the cause of the data corruption.
Do you have hardware handshake enabled ??
Have you tried different baud rates ??
Do you get any errors from the UART driver (like FiFo overruns) ??
Have you tried to modify the UART FiFo settings ??

Good luck,

Peter

On Thu, 17 Mar 2005, Catalin Drula wrote:

> Hi,
>
> I was trying to debug the Bluetooth subsystem on an HP iPAQ h5550 running
> Linux:
>
> root@h3900:~# hciconfig -a
> hci0:   Type: UART
>         BD Address: 08:00:17:1A:EB:76 ACL MTU: 339:4 SCO MTU: 60:9
>         UP RUNNING PSCAN ISCAN
>         RX bytes:-2117082264 acl:19904935 sco:0 events:4940395 errors:0
>         TX bytes:386017560 acl:9879838 sco:0 commands:430 errors:0
>         Features: 0xff 0x3b 0x05 0x00 0x00 0x00 0x00 0x00
>         Packet type: DM1 DM3 DM5 DH1 DH3 DH5 HV1 HV2 HV3
>         Link policy:
>         Link mode: SLAVE ACCEPT
>         Name: 'POCKET_PC'
>         Class: 0x000000
>         Service Classes: Unspecified
>         Device Class: Miscellaneous,
>         HCI Ver: 1.1 (0x1) HCI Rev: 0x180 LMP Ver: 1.1 (0x1) LMP Subver: 0x180
>         Manufacturer: RTX Telecom A/S (21)
>
> The problem I am facing on this machine is that it seems that the UART
> communication is sometimes unreliable. What happens is that while sending
> data, there sometimes seems to be corruption between the host and the
> controller. For example, doing an l2test where the iPAQ is sending to a
> PC, every once in a while on the receiving side there are data or sequence
> or size mismatches. After a while, the corrupted bytes happen to be in the
> H4 header, and the controller assumes it has lost synchronization with the
> host, thus sending a "Hardware Error" event. It then expects a "Reset"
> command to resynchronize.
>
> Since a lot of digging has not found the root cause of these errors in the
> UART communication, I wanted to hack the Bluez stack to better handle when
> these loss of synchronization events occur (I am not refering to patching
> the main Bluez stack, but rather make a hack for our own purposes here,
> and perhaps share it with other iPAQ h5550 users).
>
> It is easy enough to add the sending of a "Reset" command after a
> "Hardware Error" event is received, but I get the impression that there's
> more needed, such as tearing down the established connections somehow. The
> point is to get a "clean" stack to start with after such an event (and of
> course, propagate the error up the stack so that the application gets a
> "broken connection"). Any advice on how hard it is to do that, and what
> needs to be done?
>
> Currently, what happens is that after sending "Reset", we resynchronize
> with the controller, but the previous connections still show:
>
> root@h3900:~# hcitool con
> Connections:
>         < ACL 00:10:DC:57:C9:79 handle 1 state 8 lm MASTER
>         > ACL 08:00:17:1A:E6:02 handle 2 state 8 lm SLAVE
>
> Thanks in advance for your help,
>
> Catalin
>
>
>
> -------------------------------------------------------
> SF email is sponsored by - The IT Product Guide
> Read honest & candid reviews on hundreds of IT Products from real users.
> Discover which products truly live up to the hype. Start reading now.
> http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
> _______________________________________________
> Bluez-devel mailing list
> Bluez-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/bluez-devel
>

|	Peter Wippich 			Voice: +49 30 46776411		|
|	G&W Instruments GmbH		fax:   +49 30 46776419		|
|	Gustav-Meyer-Allee 25, Geb. 12	Email: pewi@gw-instruments.de	|
|	D-13355 Berlin  / Germany					|




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Bluez-devel mailing list
Bluez-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bluez-devel

Re: [Bluez-devel] Handling a Hardware Error event on iPAQ h5550

[Marcel Holtmann]

Hi Peter,

> I'm not that familiar with the architecture of BlueZ but with the
> experiences for our own stack I can tell you this is not trivial. The main
> problem here is that the Bluetooth Stack is not stateless. One possible
> solution is to feed the reset done event to the higher layers (l2cap). But
> than L2cap must be modified to correctly shut down all connections. It
> would be easier if the HCI layer tracks the state of all ACL and SCO
> connections (Marcel ??). Than a reset done event can be handled like a
> unexpected link loss on all active connections and the higher layers do
> not need any modifications (hopefully).

we can do this and actually calling "hcitool cc" does this already, but
it waits indeed for a timeout of the HCI_Disconnect.

> It may be much simpler trying to find the cause of the data corruption.

This is really the best thing to do, because H:4 is a stupid protocol if
you run it over hardware. For stack internal use it is perfect.

Hopefully 3-Wire UART will come at some time and make the H:4 obsolete.

Regards

Marcel




-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Bluez-devel mailing list
Bluez-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bluez-devel

[Bluez-devel] Re: Handling a Hardware Error event on iPAQ h5550

[Catalin Drula]

Peter Wippich <pewi <at> gw-instruments.de> writes:
 
> It may be much simpler trying to find the cause of the data corruption.
> Do you have hardware handshake enabled ??
> Have you tried different baud rates ??
> Do you get any errors from the UART driver (like FiFo overruns) ??
> Have you tried to modify the UART FiFo settings ??

Well, I'm over 5 weeks into finding the cause of the data corruption. 
At first there were indeed overruns when receving. It turns out that 
on the processor of this device (Intel XScale PXA255) there's a HWUART 
(a UART with HW flow control capabilities) connected to the same lines 
as the "default" UART for communicating with the BT controller 
(this default UART is called BTUART). Changing to the HWUART solved the 
overruns problem. 
 
The problems now occur on the sending direction (host to controller). 
There is hardware flow control although the controller never deasserts 
the RTS (never tells the host to stop sending). There are no FIFO 
overruns, the FIFO does not overfill or anything like that. 

The speed of communication and the parameter appear to be hard coded in 
the BT module as 921600, 8N1 so there's not much to do there.

In fact even with an artificial slow-down of the sending stream (down to 
a few K/sec) these corrupted bytes still show. I could not correlate them
with anything else even after weeks of testing and instrumenting the 
kernel. There's really not much else that I can think of that could be 
wrong on the host side. I am unsure if the same problem occurs under 
Windows CE on this device, but apparently it does not. 

Catalin









-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
Bluez-devel mailing list
Bluez-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bluez-devel

[Bluez-devel] Re: Handling a Hardware Error event on iPAQ h5550

[Catalin Drula]

Hi Marcel,

Marcel Holtmann <marcel <at> holtmann.org> writes:
 
> Hi Peter,
> 
> > I'm not that familiar with the architecture of BlueZ but with the
> > experiences for our own stack I can tell you this is not trivial. The main
> > problem here is that the Bluetooth Stack is not stateless. One possible
> > solution is to feed the reset done event to the higher layers (l2cap). But
> > than L2cap must be modified to correctly shut down all connections. It
> > would be easier if the HCI layer tracks the state of all ACL and SCO
> > connections (Marcel ??). Than a reset done event can be handled like a
> > unexpected link loss on all active connections and the higher layers do
> > not need any modifications (hopefully).
> 
> we can do this and actually calling "hcitool cc" does this already, but
> it waits indeed for a timeout of the HCI_Disconnect.

Could you give a few more details on this one, please? What I want to do is what
Pieter suggested: have the Hardware Error event trigger an unexpected link loss
on all active connections. 

Thanks,

Catalin



-------------------------------------------------------
This SF.net email is sponsored by Microsoft Mobile & Embedded DevCon 2005
Attend MEDC 2005 May 9-12 in Vegas. Learn more about the latest Windows
Embedded(r) & Windows Mobile(tm) platforms, applications & content.  Register
by 3/29 & save $300 http://ads.osdn.com/?ad_id=6883&alloc_id=15149&op=click
_______________________________________________
Bluez-devel mailing list
Bluez-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bluez-devel

Re: [Bluez-devel] Re: Handling a Hardware Error event on iPAQ h5550

[Marcel Holtmann]

Hi Catalin,

> > > I'm not that familiar with the architecture of BlueZ but with the
> > > experiences for our own stack I can tell you this is not trivial. The main
> > > problem here is that the Bluetooth Stack is not stateless. One possible
> > > solution is to feed the reset done event to the higher layers (l2cap). But
> > > than L2cap must be modified to correctly shut down all connections. It
> > > would be easier if the HCI layer tracks the state of all ACL and SCO
> > > connections (Marcel ??). Than a reset done event can be handled like a
> > > unexpected link loss on all active connections and the higher layers do
> > > not need any modifications (hopefully).
> > 
> > we can do this and actually calling "hcitool cc" does this already, but
> > it waits indeed for a timeout of the HCI_Disconnect.
> 
> Could you give a few more details on this one, please? What I want to do is what
> Pieter suggested: have the Hardware Error event trigger an unexpected link loss
> on all active connections. 

if you wanna do this, then this should be done directly in the Bluetooth
core module in the kernel. Look at hci_event.c and add the handling for
the hardware error event to trigger the appropriate actions.

Regards

Marcel




-------------------------------------------------------
This SF.net email is sponsored by Microsoft Mobile & Embedded DevCon 2005
Attend MEDC 2005 May 9-12 in Vegas. Learn more about the latest Windows
Embedded(r) & Windows Mobile(tm) platforms, applications & content.  Register
by 3/29 & save $300 http://ads.osdn.com/?ad_id=6883&alloc_id=15149&op=click
_______________________________________________
Bluez-devel mailing list
Bluez-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bluez-devel

[Bluez-devel] Re: Handling a Hardware Error event on iPAQ h5550

[Catalin Drula]

Hi Marcel,

Marcel Holtmann <marcel <at> holtmann.org> writes:

> > > we can do this and actually calling "hcitool cc" does this already, but
> > > it waits indeed for a timeout of the HCI_Disconnect.
> > 
> > Could you give a few more details on this one, please? What I want to do is
> > what Pieter suggested: have the Hardware Error event trigger an unexpected
> > link loss on all active connections. 
> 
> if you wanna do this, then this should be done directly in the Bluetooth
> core module in the kernel. Look at hci_event.c and add the handling for
> the hardware error event to trigger the appropriate actions.
 
I know that it has to be done in hci_event.c; what I was trying to figure out
was what the necessary actions are. As far as I've been able to tell the
function hci_conn_hash_flush in hci_conn.c seems to do most of the work. Is that
it or is there something else to do? Should we tasklet_kill the RX and TX tasks?

A somewhat unrelated question, but I could not help noticing this: in
hci_conn.c, in function hci_conn_del there is this:

if (conn->type == SCO_LINK) {
                struct hci_conn *acl = conn->link;
                if (acl) {
                        acl->link = NULL;
                        hci_conn_put(acl);
                }
        } else {
                struct hci_conn *sco = conn->link;
                if (sco)
                        sco->link = NULL;

                /* Unacked frames */
                hdev->acl_cnt += conn->sent;
        }

I don't claim to have a good understanding of this code, but it seems that on
the true branch ACL is handled and on the false branch, SCO is handled. So
shouldn't that be:

if (conn->type != SCO_LINK)

Thanks for your help,

Catalin



-------------------------------------------------------
This SF.net email is sponsored by Microsoft Mobile & Embedded DevCon 2005
Attend MEDC 2005 May 9-12 in Vegas. Learn more about the latest Windows
Embedded(r) & Windows Mobile(tm) platforms, applications & content.  Register
by 3/29 & save $300 http://ads.osdn.com/?ad_id=6883&alloc_id=15149&op=click
_______________________________________________
Bluez-devel mailing list
Bluez-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bluez-devel

Re: [Bluez-devel] Re: Handling a Hardware Error event on iPAQ h5550

[Marcel Holtmann]

Hi Catalin,

> > > Could you give a few more details on this one, please? What I want to do is
> > > what Pieter suggested: have the Hardware Error event trigger an unexpected
> > > link loss on all active connections. 
> > 
> > if you wanna do this, then this should be done directly in the Bluetooth
> > core module in the kernel. Look at hci_event.c and add the handling for
> > the hardware error event to trigger the appropriate actions.
>  
> I know that it has to be done in hci_event.c; what I was trying to figure out
> was what the necessary actions are. As far as I've been able to tell the
> function hci_conn_hash_flush in hci_conn.c seems to do most of the work. Is that
> it or is there something else to do? Should we tasklet_kill the RX and TX tasks?

I am not 100% sure, but the right actions seem to me:

	disable RX task
	disable TX task

	lock device
	flush connection hash
	unlock device

	call flush() function of the driver

	disable cmd task

	drop rx queue
	drop cmd queue
	drop raw queue

	drop last sent command

	enable RX task
	enable TX task
	enable cmd task

It is somekind of tricky to get this fully correct and I don't have any
device to really test it. However send in a patch when you got it done.

> A somewhat unrelated question, but I could not help noticing this: in
> hci_conn.c, in function hci_conn_del there is this:
> 
> if (conn->type == SCO_LINK) {
>                 struct hci_conn *acl = conn->link;
>                 if (acl) {
>                         acl->link = NULL;
>                         hci_conn_put(acl);
>                 }
>         } else {
>                 struct hci_conn *sco = conn->link;
>                 if (sco)
>                         sco->link = NULL;
> 
>                 /* Unacked frames */
>                 hdev->acl_cnt += conn->sent;
>         }
> 
> I don't claim to have a good understanding of this code, but it seems that on
> the true branch ACL is handled and on the false branch, SCO is handled. So
> shouldn't that be:
> 
> if (conn->type != SCO_LINK)

You are wrong here. We check for conn->link and that is the associated
SCO or ACL link.

Regards

Marcel




-------------------------------------------------------
This SF.net email is sponsored by Microsoft Mobile & Embedded DevCon 2005
Attend MEDC 2005 May 9-12 in Vegas. Learn more about the latest Windows
Embedded(r) & Windows Mobile(tm) platforms, applications & content.  Register
by 3/29 & save $300 http://ads.osdn.com/?ad_id=6883&alloc_id=15149&op=click
_______________________________________________
Bluez-devel mailing list
Bluez-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/bluez-devel