From mboxrd@z Thu Jan  1 00:00:00 1970
From: Kamraan Nasim <knasim-LfGgLkH2qHaTtA8H5PvdGFaTQe2KTcn/@public.gmane.org>
Subject: Re: Panic in rte MEMPOOL__mempool_check_cookies()
Date: Wed, 12 Nov 2014 21:50:50 -0500
Message-ID: <CAPrTskjQaUrk5JsBQyrxXTdFEf4kzqyVG5+XUbUdKeAZSuqAdQ@mail.gmail.com>
References: <CAPrTskijLw42woTkBvt_cYcaNKcyDsYVhj_zEpQ2n7DPMw1_Kg@mail.gmail.com>
 <20141107094828.GC4628@bricha3-MOBL3>
Mime-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Cc: dev-VfR2kkLFssw@public.gmane.org
To: Bruce Richardson <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org>
Return-path: <dev-bounces-VfR2kkLFssw@public.gmane.org>
In-Reply-To: <20141107094828.GC4628@bricha3-MOBL3>
List-Id: patches and discussions about DPDK <dev.dpdk.org>
List-Unsubscribe: <http://dpdk.org/ml/options/dev>,
 <mailto:dev-request-VfR2kkLFssw@public.gmane.org?subject=unsubscribe>
List-Archive: <http://dpdk.org/ml/archives/dev/>
List-Post: <mailto:dev-VfR2kkLFssw@public.gmane.org>
List-Help: <mailto:dev-request-VfR2kkLFssw@public.gmane.org?subject=help>
List-Subscribe: <http://dpdk.org/ml/listinfo/dev>,
 <mailto:dev-request-VfR2kkLFssw@public.gmane.org?subject=subscribe>
Errors-To: dev-bounces-VfR2kkLFssw@public.gmane.org
Sender: "dev" <dev-bounces-VfR2kkLFssw@public.gmane.org>

Hi Bruce,

The issue seems to be triggered by a combination of using the
CONFIG_RTE_LIBRTE_MEMPOOL_DEBUG  option and contention in the mempool(small
mempool size).

I disabled that and the header panics went away.

Thank you for your help!

--Kam

On Fri, Nov 7, 2014 at 4:48 AM, Bruce Richardson <bruce.richardson-ral2JQCrhuEAvxtiuMwx3w@public.gmane.org
> wrote:

> On Thu, Nov 06, 2014 at 04:28:18PM -0500, Kamraan Nasim wrote:
> > Greetings,
> >
> > I have been hitting this issue fairly consistently for the ixgbe driver
> >
> > MEMPOOL: obj=0x7ffeed1f5d00, mempool=0x7ffeecb69bc0,
> cookie=badbadbadadd2e55
> > PANIC in __mempool_check_cookies():
> > MEMPOOL: bad header cookie (get)
> >
> > It seems to be a corruption in the mempool bound to my ixgbe port. What I
> > have observed is that this ONLY happens if I initialize dpdk(i.e. start
> > dpdk application) AFTER traffic is already flowing in through the port.
> If
> > I initialize dpdk and bind BEFORE I start traffic then things seem to
> work
> > fine.
> >
> > Any clues on why this might be happening?
> >
> > A bit stumped, so would really appreciate all the help I can get on this
> > one.
> >
>
> Hi.
> Can you confirm if your application works ok with the mempool debug options
> disabled? The mempool debug options are not often used because they slow
> things
> down so it's possible the bug could be in the cookie checking itself.
>
> /Bruce
>
> > Thanks,
> > Kam
> >
> >
> > (bt for your reference)
> >
> > #2  0x0000000000408cc6 in __rte_panic (funcname=0x571100
> > "__mempool_check_cookies", format=
> >     0x568fb0 "MEMPOOL: bad header cookie (get)\n%.0s")
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_eal/linuxapp/eal/eal_debug.c:83
> > #3  0x00000000004af027 in __mempool_check_cookies (rxq=<value optimized
> > out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:357
> > #4  rte_mempool_get_bulk (rxq=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/x86_64-native-linuxapp-gcc/include/rte_mempool.h:1094
> > #5  ixgbe_rx_alloc_bufs (rxq=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:997
> > #6  0x00000000004afce9 in rx_recv_pkts (rx_queue=0x7ffeec8edbc0,
> > rx_pkts=0x900410,
> >     nb_pkts=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1074
> > #7  ixgbe_recv_pkts_bulk_alloc (rx_queue=0x7ffeec8edbc0,
> rx_pkts=0x900410,
> > nb_pkts=<value optimized out>)
> >     at
> >
> /b/knasim/bandwagon/sbn/src/share/dpdk/lib/librte_pmd_ixgbe/ixgbe_rxtx.c:1124
> > #8  0x0000000000520d36 in rte_eth_rx_burst (lp=0x900340, n_workers=14,
> > bsz_rd=<value optimized out>, bsz_wr=
> >     144, pos_lb=0 '\000') at /usr/lib/dpdk/include/rte_ethdev.h:2368
>