Re: Please help! AM35xx mm/slab.c BUG

[Tony Lindgren <tony@atomide.com>]

* CF Adad <cfadad@rocketmail.com> [120604 23:47]:
> All,
> 
> I'm **really** hoping someone out there can help us with this.
> 
> My team has been working with the AM3517 for several months now, and we seem to be plagued every so often by what we have termed the "slab bug".  In short, it looks something like the pasted bootlog below.  This has been an *incredibly* hard bug to figure out.  We have a couple of different AM3517-based platforms at our disposal, but the one we see the issue on almost exclusively is a custom, prototype baseboard designed around the TechNexion TAM3157.  Over the last several months, we have tried several versions of the Linux off the linux-omap tree, with loads of different configurations, and even different bootloader versions and combinations.  We've spent most of our time with a linux-omap snapshot that was a 3.2-rc6, and more recently a 3.4-rc6 from late a week or two back.  (Tomorrow I anticipate pulling the latest 3.5 now that I see it's out.)  In all cases, since we switched to 3.0+, we've seen these errors.
> 
> They are *very* inconsistent in when they occur, but they happen often enough to be very frustrating.  Consequently, our team has had an incredibly difficult time tracking what's causing them.  They seem to occur at random, perhaps on average once every handful of days.  We've messed with everything we can think of from tweaking kernel options (like enabling/disabling preemption), to disabling various drivers and userspace components, to reviewing every single line in any of our board files.  We have tried different versions and combinations of the OS and both bootloaders (x-loader & u-boot), and even went so far as to do a full analysis of the RAM timings in the EMIF4.  Unfortunately, nothing so far has worked.  The error occurs when operating off both the SD/MMC and the NAND devices, with or without the Ethernets (LAN9221 & EMAC) up and/or running, with or without PREEMPT, under heavy load and sometimes just idling, ...  There is simply nothing
>  consistent about it.  After probably 2 weeks without seeing one, I saw 3 today.
> 
> Though the error's occurence is inconistent, the error itself is.  It always throws an internal OOPs at the following section of code in mm/slab.c:
> ---
> /*
> * The slab was either on partial or free list so
> * there must be at least one object available for
> * allocation.
> */
> BUG_ON(slabp->inuse >= cachep->num);
> ---
> (It appears this was patched in eons ago: https://lkml.org/lkml/2007/2/19/20.  So it's nothing new.)

I can think of at least three issues causing errors like this:

1. Missing retention/off idle workarounds

   You can test this one by booting with nohlt cmdline option and
   seeing if that helps.

2. Broken memory

   I've seen at least one case of this where things would work
   fine if only half of the memory was in use and devices would
   oops at random point within a week. To test for this you can
   pass cmdline options to artifically partition the memory and
   leave out some chunks to see if that helps. Or boot with
   mem=xxxM set to half of the physical memory. And run your tests
   with SLAB_DEBUG set.

3. Software bugs

   My experience is that things are behaving very reliably regarding
   cache and highmem, so I would check #1 and #2 fist.

Regards,

Tony 
--
To unsubscribe from this list: send the line "unsubscribe linux-omap" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html