peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is occupied!

peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is occupied!

[Diana Burgess]

Hi,

I am seeing the following error:

peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is occupied!

I am running 4.13.10-rt3 kernel using the following PCIe card connect
to my board:

06:00.0 Network controller: PEAK-System Technik GmbH Device 0013 (rev 01)
    Subsystem: PEAK-System Technik GmbH Device 0014
    Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin A routed to IRQ 16
    Region 0: Memory at fb100000 (32-bit, non-prefetchable) [size=64K]
    Capabilities: <access denied>
    Kernel driver in use: peak_pciefd

This occurs when I am sending messages over the CAN bus with
and without the other device on the bus also sending.

I have not seen this issue with other can devices in similar setups
 (different kernel versions). It also only seems to occur on CAN1 which is
set to a higher baud rate then all the other buses.

I do have CONFIG PREEMPT RT turned on which might be exposing this
issue more often.

Has anyone seen this before? Does anyone know how to fix this or have a fix?

Thanks,
Diana Burgess

RE: peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is occupied!

[Stéphane Grosjean]

Hi Diana,

We've pushed a patch to fix a potential issue around the handling of the echo frames. It has been included in 4.15. Are you able to build and run this version of the Kernel?

Regards,

> -----Original Message-----
> From: Diana Burgess <diana@peloton-tech.com>
> Sent: vendredi 23 février 2018 02:23
> To: Stéphane Grosjean <s.grosjean@peak-system.com>; linux-
> can@vger.kernel.org; linux-rt-users@vger.kernel.org
> Cc: Philipp Schrader <philipp@peloton-tech.com>
> Subject: peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb
> is occupied!
>
> Hi,
>
> I am seeing the following error:
>
> peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is
> occupied!
>
> I am running 4.13.10-rt3 kernel using the following PCIe card connect to my
> board:
>
> 06:00.0 Network controller: PEAK-System Technik GmbH Device 0013 (rev 01)
>     Subsystem: PEAK-System Technik GmbH Device 0014
>     Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx-
>     Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
>     Latency: 0, Cache Line Size: 64 bytes
>     Interrupt: pin A routed to IRQ 16
>     Region 0: Memory at fb100000 (32-bit, non-prefetchable) [size=64K]
>     Capabilities: <access denied>
>     Kernel driver in use: peak_pciefd
>
> This occurs when I am sending messages over the CAN bus with and without
> the other device on the bus also sending.
>
> I have not seen this issue with other can devices in similar setups  (different
> kernel versions). It also only seems to occur on CAN1 which is set to a higher
> baud rate then all the other buses.
>
> I do have CONFIG PREEMPT RT turned on which might be exposing this issue
> more often.
>
> Has anyone seen this before? Does anyone know how to fix this or have a
> fix?
>
> Thanks,
> Diana Burgess

--
PEAK-System Technik GmbH
Sitz der Gesellschaft Darmstadt - HRB 9183
Geschaeftsfuehrung: Alexander Gach / Uwe Wilhelm
--

Re: peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is occupied!

[Sebastian Andrzej Siewior]

On 2018-02-23 08:46:36 [+0000], Stéphane Grosjean wrote:
> We've pushed a patch to fix a potential issue around the handling of the echo frames. It has been included in 4.15. Are you able to build and run this version of the Kernel?

top posting and long lines.
Could you please quote the patches which you assume should fix this
behaviour and whether or not they are tagged stable?

> Regards,

Sebastian

Re: peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is occupied!

[Marc Kleine-Budde]

[-- Attachment #1.1: Type: text/plain, Size: 820 bytes --]

On 02/26/2018 10:46 AM, Sebastian Andrzej Siewior wrote:
> On 2018-02-23 08:46:36 [+0000], Stéphane Grosjean wrote:
>> We've pushed a patch to fix a potential issue around the handling of the echo frames. It has been included in 4.15. Are you able to build and run this version of the Kernel?
> 
> top posting and long lines.
> Could you please quote the patches which you assume should fix this
> behaviour and whether or not they are tagged stable?

91785de6f94b can: peak/pcie_fd: fix potential bug in restarting tx queue

Marc

-- 
Pengutronix e.K.                  | Marc Kleine-Budde           |
Industrial Linux Solutions        | Phone: +49-231-2826-924     |
Vertretung West/Dortmund          | Fax:   +49-5121-206917-5555 |
Amtsgericht Hildesheim, HRA 2686  | http://www.pengutronix.de   |


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is occupied!

[Diana Burgess]

On Mon, Feb 26, 2018 at 2:34 AM, Marc Kleine-Budde <mkl@pengutronix.de> wrote:
> On 02/26/2018 10:46 AM, Sebastian Andrzej Siewior wrote:
>> On 2018-02-23 08:46:36 [+0000], Stéphane Grosjean wrote:
>>> We've pushed a patch to fix a potential issue around the handling of the echo frames. It has been included in 4.15. Are you able to build and run this version of the Kernel?
>>
>> top posting and long lines.
>> Could you please quote the patches which you assume should fix this
>> behaviour and whether or not they are tagged stable?
>
> 91785de6f94b can: peak/pcie_fd: fix potential bug in restarting tx queue
>
> Marc
>
> --
> Pengutronix e.K.                  | Marc Kleine-Budde           |
> Industrial Linux Solutions        | Phone: +49-231-2826-924     |
> Vertretung West/Dortmund          | Fax:   +49-5121-206917-5555 |
> Amtsgericht Hildesheim, HRA 2686  | http://www.pengutronix.de   |
>

That patch is part of my branch already so it does not fix this bug.

Also 4.15 is not part of rt-devel yet so I did not try to build and run that
version.

Thanks,

-- 
Diana Burgess

Re: peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is occupied!

[Sebastian Andrzej Siewior]

On 2018-02-26 11:55:11 [-0800], Diana Burgess wrote:
> That patch is part of my branch already so it does not fix this bug.
> 
> Also 4.15 is not part of rt-devel yet so I did not try to build and run that
> version.
about that. The tree you were using is not updated (is EOL) so I suggest
to upgrade to 4.14-RT. This may not fix your bug but you get a kernel
with stable/security updates.

You mentioned that you use -RT and you see this problem only with the
peak card. You never mentioned if this also happens often or rarely or
if this happens once you disable RT and you keep doing what you are
doing.

> 
> Thanks,
> 

Sebastian

Re: peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is occupied!

[Diana Burgess]

On Tue, Feb 27, 2018 at 12:58 AM, Sebastian Andrzej Siewior
<bigeasy@linutronix.de> wrote:
> On 2018-02-26 11:55:11 [-0800], Diana Burgess wrote:
>> That patch is part of my branch already so it does not fix this bug.
>>
>> Also 4.15 is not part of rt-devel yet so I did not try to build and run that
>> version.
> about that. The tree you were using is not updated (is EOL) so I suggest
> to upgrade to 4.14-RT. This may not fix your bug but you get a kernel
> with stable/security updates.
>
> You mentioned that you use -RT and you see this problem only with the
> peak card. You never mentioned if this also happens often or rarely or
> if this happens once you disable RT and you keep doing what you are
> doing.
>
>>
>> Thanks,
>>
>
> Sebastian

I've upgraded to 4.14-RT and tried with and without RT. Using RT I see the
bug approximately every 45s (though it is not consistent). This time between
bugs increases when not using RT, but does not go away.

-- 
Diana Burgess

RE: peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is occupied!

[Stéphane Grosjean]

Hi,

The "BUG! echo_skb is occupied" msg is printed when the network tx queue is restarted, while there is no room left in the echo_skb array. Note that this msg does not prevent any CAN frame to be written on the bus.

There's actually one place where the network tx queue is restarted, however the echo_skb array is filled:

(see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/can/peak_canfd/peak_pciefd_main.c#n353)

A driver version in which the below patch has been applied

         /* handle tx link interrupt (if any) */
         if (pciefd_irq_is_lnk(priv->irq_status)) {
                 unsigned long flags;

                 spin_lock_irqsave(&priv->tx_lock, flags);
                 priv->tx_pages_free++;
                 spin_unlock_irqrestore(&priv->tx_lock, flags);
-
-                /* wake producer up */
-               netif_wake_queue(priv->ucan.ndev);
         }

has run during the whole weekend (~60 h). The echo_skb array size has also been reduced to 8 slots (instead of 24), to increase the probability for the issue to occur; two "cangen -g 0 -i" were running on can0 and can1. Unfortunately, during this period, the "BUG! echo is occupied" msg has been still sporadically printed (4 times, 2 times for can0, 2 times for can1).

Still under investigation...

Regards,


> -----Original Message-----
> From: Diana Burgess <diana@peloton-tech.com>
> Sent: samedi 3 mars 2018 02:38
> To: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
> Cc: Marc Kleine-Budde <mkl@pengutronix.de>; Stéphane Grosjean
> <s.grosjean@peak-system.com>; Philipp Schrader <philipp@peloton-
> tech.com>; linux-can@vger.kernel.org; linux-rt-users@vger.kernel.org
> Subject: Re: peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG!
> echo_skb is occupied!
>
> On Tue, Feb 27, 2018 at 12:58 AM, Sebastian Andrzej Siewior
> <bigeasy@linutronix.de> wrote:
> > On 2018-02-26 11:55:11 [-0800], Diana Burgess wrote:
> >> That patch is part of my branch already so it does not fix this bug.
> >>
> >> Also 4.15 is not part of rt-devel yet so I did not try to build and
> >> run that version.
> > about that. The tree you were using is not updated (is EOL) so I
> > suggest to upgrade to 4.14-RT. This may not fix your bug but you get a
> > kernel with stable/security updates.
> >
> > You mentioned that you use -RT and you see this problem only with the
> > peak card. You never mentioned if this also happens often or rarely or
> > if this happens once you disable RT and you keep doing what you are
> > doing.
> >
> >>
> >> Thanks,
> >>
> >
> > Sebastian
>
> I've upgraded to 4.14-RT and tried with and without RT. Using RT I see the
> bug approximately every 45s (though it is not consistent). This time between
> bugs increases when not using RT, but does not go away.
>
> --
> Diana Burgess

--
PEAK-System Technik GmbH
Sitz der Gesellschaft Darmstadt - HRB 9183
Geschaeftsfuehrung: Alexander Gach / Uwe Wilhelm
--

Re: peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG! echo_skb is occupied!

[Diana Burgess]

On Mon, Mar 5, 2018 at 1:35 AM, Stéphane Grosjean
<s.grosjean@peak-system.com> wrote:
> Hi,
>
> The "BUG! echo_skb is occupied" msg is printed when the network tx queue is restarted, while there is no room left in the echo_skb array. Note that this msg does not prevent any CAN frame to be written on the bus.
>
> There's actually one place where the network tx queue is restarted, however the echo_skb array is filled:
>
> (see https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/net/can/peak_canfd/peak_pciefd_main.c#n353)
>
> A driver version in which the below patch has been applied
>
>          /* handle tx link interrupt (if any) */
>          if (pciefd_irq_is_lnk(priv->irq_status)) {
>                  unsigned long flags;
>
>                  spin_lock_irqsave(&priv->tx_lock, flags);
>                  priv->tx_pages_free++;
>                  spin_unlock_irqrestore(&priv->tx_lock, flags);
> -
> -                /* wake producer up */
> -               netif_wake_queue(priv->ucan.ndev);
>          }
>
> has run during the whole weekend (~60 h). The echo_skb array size has also been reduced to 8 slots (instead of 24), to increase the probability for the issue to occur; two "cangen -g 0 -i" were running on can0 and can1. Unfortunately, during this period, the "BUG! echo is occupied" msg has been still sporadically printed (4 times, 2 times for can0, 2 times for can1).
>
> Still under investigation...
>
> Regards,
>
>
>> -----Original Message-----
>> From: Diana Burgess <diana@peloton-tech.com>
>> Sent: samedi 3 mars 2018 02:38
>> To: Sebastian Andrzej Siewior <bigeasy@linutronix.de>
>> Cc: Marc Kleine-Budde <mkl@pengutronix.de>; Stéphane Grosjean
>> <s.grosjean@peak-system.com>; Philipp Schrader <philipp@peloton-
>> tech.com>; linux-can@vger.kernel.org; linux-rt-users@vger.kernel.org
>> Subject: Re: peak_pciefd 0000:06:00.0 can1: can_put_echo_skb: BUG!
>> echo_skb is occupied!
>>
>> On Tue, Feb 27, 2018 at 12:58 AM, Sebastian Andrzej Siewior
>> <bigeasy@linutronix.de> wrote:
>> > On 2018-02-26 11:55:11 [-0800], Diana Burgess wrote:
>> >> That patch is part of my branch already so it does not fix this bug.
>> >>
>> >> Also 4.15 is not part of rt-devel yet so I did not try to build and
>> >> run that version.
>> > about that. The tree you were using is not updated (is EOL) so I
>> > suggest to upgrade to 4.14-RT. This may not fix your bug but you get a
>> > kernel with stable/security updates.
>> >
>> > You mentioned that you use -RT and you see this problem only with the
>> > peak card. You never mentioned if this also happens often or rarely or
>> > if this happens once you disable RT and you keep doing what you are
>> > doing.
>> >
>> >>
>> >> Thanks,
>> >>
>> >
>> > Sebastian
>>
>> I've upgraded to 4.14-RT and tried with and without RT. Using RT I see the
>> bug approximately every 45s (though it is not consistent). This time between
>> bugs increases when not using RT, but does not go away.
>>
>> --
>> Diana Burgess
>
> --
> PEAK-System Technik GmbH
> Sitz der Gesellschaft Darmstadt - HRB 9183
> Geschaeftsfuehrung: Alexander Gach / Uwe Wilhelm
> --

Stephane Grosjean sent out the following two patches which fixed the bug:
   can: peak/pcie_fd: fix echo_skb is occupied! bug
   can: peak/pcie_fd: remove useless code when interface starts

-- 
Diana Burgess