Re: XDP_TX bug report on mlx4

[Brenden Blanco <bblanco@plumgrid.com>]

On Sun, Sep 18, 2016 at 07:59:43PM -0400, Brenden Blanco wrote:
> On Fri, Sep 16, 2016 at 09:24:43PM +0200, Jesper Dangaard Brouer wrote:
> > On Fri, 16 Sep 2016 12:17:27 -0700
> > Brenden Blanco <bblanco@plumgrid.com> wrote:
> > 
> > > On Fri, Sep 16, 2016 at 09:03:40PM +0200, Jesper Dangaard Brouer wrote:
> > > > Hi Brenden,
> > > > 
> > > > I've discovered a bug with XDP_TX recycling of pages in the mlx4 driver.
> > > > 
> > > > If I increase the number of RX and TX queues/channels via ethtool cmd:
> > > >  ethtool -L mlx4p1 rx 10 tx 10
> > > > 
> > > > Then when running the xdp2 program, which does XDP_TX, the kernel will
> > > > crash with page errors, because the page refcnt goes to zero or even
> > > > minus.  I've noticed pages delivered to mlx4_en_rx_recycle() can have
> > > > a page refcnt of zero, which is wrong, they should always have 1 (for
> > > > XDP).
> > > > 
> > > > Debugging it further, I find that this can happen when mlx4_en_rx_recycle()
> > > > is called from mlx4_en_recycle_tx_desc().  This is the TX cleanup function,
> > > > associated with TX ring queues used for XDP_TX only. No others than the
> > > > XDP_TX action should be able to place packets into these TX rings
> > > > which call mlx4_en_recycle_tx_desc().  
> > > 
> > > Sounds pretty straightforward, let me look into it.
> > 
> > Here is some debug info I instrumented my kernel with, and I've
> > attached my minicom output with a warning and a panic.
> 
> Thanks for the info.
> 
> I've also spent this weekend trying to debug (pretty easy to reproduce),
> but with no conclusive answer. I was investigating the sequence in
> mlx4_en_stop_port to see if rx might still be running through the
> function, on the theory that the xdp tx might race with
> mlx4_en_free_tx_buf. For instance, I tried moving the napi_synchronize
> loop to be just below the msleep(10). No improvement.
> 
> Unfortunately, I'm out of options, since my one test box has decided not
> to reboot itself automatically, and I'll be out of email communication
> (for 3 weeks) before anybody can physicially resuscitate it (tomorrow).

Sorry to leave this hanging for so long. I've just recently come back
from vacation and I think I now have a fix for this. I'll send a patch
to the list shortly.

> > 
> > Enable some driver debug printks via::
> >  ethtool -s mlx4p1 msglvl drv on
> > 
> > Debug normal situation::
> > 
> [...]