Re: peak_pci: TX Frame Loss

[Stephane Grosjean <s.grosjean@peak-system.com>]

Hi Andri,

Could you first give me the result of sudo lspci -d 1c: -vvv please?

Regards,

Stéphane

Le 18/11/2015 15:51, Andri Yngvason a écrit :
> Hi all,
>
> We've been experiencing frame loss on transmission in the peak_pci netdev
> driver.
>
> The frames are not reported as "dumped" by the netlink interface.
>
> We are running CANopen and this manifests sporadically as nodes dropping off the
> network due to failure to answer node guarding RTR and as SDO request timeouts.
>
> Example with can0 and can1 on the same bus where the CANopen master is on can0
> and can1 is set up to listen only:
> (1446688151.783844)  can0  701  [1] remote request <- node guarding request on can0
> (1446688151.784296)  can0  70A  [1] remote request <- another node guarding request
> (1446688151.784304)  can1  70A  [1] remote request <- only the latter is seen by can1
> (1446688151.784751)  can0  720  [1] remote request
> (1446688151.784763)  can1  720  [1] remote request
> (1446688151.785793)  can1  283  [8] 00 00 00 00 00 00 00 00
> (1446688151.785792)  can0  283  [8] 00 00 00 00 00 00 00 00
> (1446688151.786164)  can0  70A  [1] 85 <-- node guarding response
> (1446688151.786163)  can1  70A  [1] 85
> (1446688151.786641)  can1  720  [1] 85
> (1446688151.786641)  can0  720  [1] 85 <-- node guarding response
> (1446688151.787057)  can0  721  [1] remote request
> (1446688151.787063)  can1  721  [1] remote request
> (1446688151.787728)  can0  721  [1] 05
> (1446688151.787733)  can1  721  [1] 05
>
> Node 1 never responded because it never received the request.
>
> The node guarding requests are sent in bursts where lower ids appear before
> higher ids. A curious observation is that it's always the lowest id that drops
> out first. I.e. the first frame in a burst of frames is the one that's lost.
>
> Another interesting thing that we've found out is that if we turn off SMP on the
> system, the problem disappears. But obviously we don't want to disable SMP in a
> production system. ;)
> It helps to set the cpy affinity of all threads and processes that touch the CAN
> bus to a single core but sadly it doesn't eliminate the problem.
>
> Our systems are running on kernel version 3.14.3 with the rt patch. I tried
> running 4.1.12-rt13 but that did not eliminate the problem. We also tried
> running with the pcan netdev driver from peak which does in fact run without
> frame loss. Thus, this is probably an issue with either peak_pci or sja1000.
>
> I tried poking around in sja1000.c. I noticed that sja1000_start_xmit() is not
> guarded against trying to transmit when the tx buffer is occupied, so I added a
> check and a print-out:
> diff --git a/drivers/net/can/sja1000/sja1000.c b/drivers/net/can/sja1000/sja1000.c
> index 32bd7f4..adc49db 100644
> --- a/drivers/net/can/sja1000/sja1000.c
> +++ b/drivers/net/can/sja1000/sja1000.c
> @@ -292,6 +292,11 @@ static netdev_tx_t sja1000_start_xmit(struct sk_buff *skb,
>   
>          netif_stop_queue(dev);
>   
> +       if (!(priv->read_reg(priv, SJA1000_SR) & SR_TBS)) {
> +               netdev_err(dev, "BUG!, TX FIFO full when queue awake!\n");
> +               return NETDEV_TX_BUSY;
> +       }
> +
>          fi = dlc = cf->can_dlc;
>          id = cf->can_id;
>
> There was no error message in dmesg after frame loss, so that's not the problem.
>
> The CPU is an Intel i7-4700EQ and the CAN interface is a Peak PCIe dual channel.
>
> Does anyone have an idea what might be wrong? :)
>
> Best regards,
> Andri

--
PEAK-System Technik GmbH
Sitz der Gesellschaft Darmstadt
Handelsregister Darmstadt HRB 9183 
Geschaeftsfuehrung: Alexander Gach, Uwe Wilhelm
--