Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Andrew Morton]

(switched to email.  Please respond via emailed reply-to-all, not via the
bugzilla web interface).

On Thu, 10 May 2012 23:29:46 +0000 (UTC)
bugzilla-daemon@bugzilla.kernel.org wrote:

> https://bugzilla.kernel.org/show_bug.cgi?id=43227
> 
>            Summary: BUG: Bad page state in process wcg_gfam_6.11_i
>            Product: Memory Management
>            Version: 2.5
>     Kernel Version: 3.3.5
>           Platform: All
>         OS/Version: Linux
>               Tree: Mainline
>             Status: NEW
>           Severity: normal
>           Priority: P1
>          Component: Other
>         AssignedTo: akpm@linux-foundation.org
>         ReportedBy: sliedes@cc.hut.fi
>         Regression: No
> 
> 
> Created an attachment (id=73246)
>  --> (https://bugzilla.kernel.org/attachment.cgi?id=73246)
> Entire dmesg
> 
> Hi,
> 
> I noticed that a number of bad page state bugs have apparently been fixed in
> recent stable kernels. Unfortunately it seems there's still some such bug left
> in 3.3.5.
> 
> I just saw this bug for the first time after upgrading from (mainline) 3.3.2 to
> 3.3.5 (having read the other bug reports I suspect this is the significant
> change) and changing my root filesystem from ext4 to xfs, hence causing heavier
> exercise of the xfs code than before (but I've had a backup partition on xfs
> before with no issues).
> 
> This is the bug I'm seeing:
> 
> [67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
> [67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
> (null) index:0x7f1eb293b
> [67031.755792] page flags: 0x4000000000000014(referenced|dirty)

AFAICT we got this warning because the page allocator found a free page
with PG_referenced and PG_dirty set.

It would be a heck of a lot more useful if we'd been told about this
when the page was freed, not when it was reused!  Can anyone think of a
reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at
least)?

> ...
>
> [67031.755872] Pid: 5229, comm: wcg_gfam_6.11_i Tainted: G           O 3.3.5 #2
> [67031.755874] Call Trace:
> [67031.755880]  [<ffffffff813f6a33>] bad_page+0xcb/0xe0
> [67031.755884]  [<ffffffff810dfc4e>] get_page_from_freelist+0x54e/0x5f0
> [67031.755888]  [<ffffffff811e0b4e>] ? string.isra.4+0x3e/0xd0
> [67031.755891]  [<ffffffff810dfe05>] __alloc_pages_nodemask+0x115/0x810
> [67031.755895]  [<ffffffff81143a06>] ? seq_printf+0x56/0x90
> [67031.755898]  [<ffffffff8118570a>] ? do_task_stat+0x6ba/0xbf0
> [67031.755901]  [<ffffffff8110561b>] ? anon_vma_prepare+0xfb/0x170
> [67031.755904]  [<ffffffff810fb0c6>] handle_pte_fault+0x736/0x970
> [67031.755906]  [<ffffffff810fb668>] handle_mm_fault+0x1c8/0x2f0
> [67031.755909]  [<ffffffff81400cbe>] do_page_fault+0x16e/0x4f0
> [67031.755913]  [<ffffffff813fdbc5>] page_fault+0x25/0x30
> [67031.755916]  [<ffffffff811e2bfd>] ? copy_user_generic_string+0x2d/0x40
> [67031.755919]  [<ffffffff81143e08>] ? seq_read+0x2c8/0x390
> [67031.755922]  [<ffffffff811240b4>] vfs_read+0xa4/0x180
> [67031.755924]  [<ffffffff811241d5>] sys_read+0x45/0x90
> [67031.755927]  [<ffffffff81406a09>] ia32_do_call+0x13/0x13
> [67031.755929] Disabling lock debugging due to kernel taint
> 
> Before that there's a warning about "irq 17: nobody cared" (see the attached
> entire dmesg if you are interested); this is apparently an unrelated, known
> issue with Asus motherboards which has persisted for a long time without
> causing  bad page state bugs.
> 
> FWIW, I noticed that a number of the patches that may have something to do with
> this and that have gone in since v3.3.2 are KVM related. I have run a large
> number of KVM guests on this computer between the boot and this bug occuring,
> so this might (or might not) be KVM related.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Sami Liedes]

[-- Attachment #1: Type: text/plain, Size: 801 bytes --]

On Fri, May 11, 2012 at 12:59:21PM -0700, Andrew Morton wrote:
> > [67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
> > [67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
> > (null) index:0x7f1eb293b
> > [67031.755792] page flags: 0x4000000000000014(referenced|dirty)
> 
> AFAICT we got this warning because the page allocator found a free page
> with PG_referenced and PG_dirty set.
> 
> It would be a heck of a lot more useful if we'd been told about this
> when the page was freed, not when it was reused!  Can anyone think of a
> reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at
> least)?

Would it be useful if I tried to reproduce this with some debugging
options turned on, for example CONFIG_DEBUG_VM?

	Sami

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Andrew Morton]

On Fri, 11 May 2012 23:02:13 +0300
Sami Liedes <sami.liedes@iki.fi> wrote:

> On Fri, May 11, 2012 at 12:59:21PM -0700, Andrew Morton wrote:
> > > [67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
> > > [67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
> > > (null) index:0x7f1eb293b
> > > [67031.755792] page flags: 0x4000000000000014(referenced|dirty)
> > 
> > AFAICT we got this warning because the page allocator found a free page
> > with PG_referenced and PG_dirty set.
> > 
> > It would be a heck of a lot more useful if we'd been told about this
> > when the page was freed, not when it was reused!  Can anyone think of a
> > reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at
> > least)?
> 
> Would it be useful if I tried to reproduce this with some debugging
> options turned on, for example CONFIG_DEBUG_VM?
> 

Sure, thanks, that might turn something up. 
Documentation/SubmitChecklist recommends 

: 12: Has been tested with CONFIG_PREEMPT, CONFIG_DEBUG_PREEMPT,
:     CONFIG_DEBUG_SLAB, CONFIG_DEBUG_PAGEALLOC, CONFIG_DEBUG_MUTEXES,
:     CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_ATOMIC_SLEEP, CONFIG_PROVE_RCU
:     and CONFIG_DEBUG_OBJECTS_RCU_HEAD all simultaneously enabled.

although that list might be a bit out of date; it certainly should
include CONFIG_DEBUG_VM!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Dave Jones]

On Fri, May 11, 2012 at 01:32:34PM -0700, Andrew Morton wrote:
 > On Fri, 11 May 2012 23:02:13 +0300
 > Sami Liedes <sami.liedes@iki.fi> wrote:
 > 
 > > On Fri, May 11, 2012 at 12:59:21PM -0700, Andrew Morton wrote:
 > > > > [67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
 > > > > [67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
 > > > > (null) index:0x7f1eb293b
 > > > > [67031.755792] page flags: 0x4000000000000014(referenced|dirty)
 > > > 
 > > > AFAICT we got this warning because the page allocator found a free page
 > > > with PG_referenced and PG_dirty set.
 > > > 
 > > > It would be a heck of a lot more useful if we'd been told about this
 > > > when the page was freed, not when it was reused!  Can anyone think of a
 > > > reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at
 > > > least)?
 > > 
 > > Would it be useful if I tried to reproduce this with some debugging
 > > options turned on, for example CONFIG_DEBUG_VM?
 > > 
 > 
 > Sure, thanks, that might turn something up. 
 > Documentation/SubmitChecklist recommends 
 > 
 > : 12: Has been tested with CONFIG_PREEMPT, CONFIG_DEBUG_PREEMPT,
 > :     CONFIG_DEBUG_SLAB, CONFIG_DEBUG_PAGEALLOC, CONFIG_DEBUG_MUTEXES,
 > :     CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_ATOMIC_SLEEP, CONFIG_PROVE_RCU
 > :     and CONFIG_DEBUG_OBJECTS_RCU_HEAD all simultaneously enabled.
 > 
 > although that list might be a bit out of date; it certainly should
 > include CONFIG_DEBUG_VM!

FWIW, that Fedora report of this had DEBUG_VM enabled, and it doesn't
seem that there was any earlier oops/warn judging by the lack of tainting.

	Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Sami Liedes]

[-- Attachment #1: Type: text/plain, Size: 981 bytes --]

On Fri, May 11, 2012 at 01:32:34PM -0700, Andrew Morton wrote:
> Sure, thanks, that might turn something up. 
> Documentation/SubmitChecklist recommends 
> 
> : 12: Has been tested with CONFIG_PREEMPT, CONFIG_DEBUG_PREEMPT,
> :     CONFIG_DEBUG_SLAB, CONFIG_DEBUG_PAGEALLOC, CONFIG_DEBUG_MUTEXES,
> :     CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_ATOMIC_SLEEP, CONFIG_PROVE_RCU
> :     and CONFIG_DEBUG_OBJECTS_RCU_HEAD all simultaneously enabled.
> 
> although that list might be a bit out of date; it certainly should
> include CONFIG_DEBUG_VM!

I wonder if there's somewhere a recommended list of generally most
useful debug options that only have a moderate performance impact? I'd
be happy to use a set of useful debug flags that generally impacts
performance by, say, <10%, on the computers I use for my everyday work
to help catch bugs. But it's sometimes quite hard to assess the impact
of different Kernel hacking options from just the description...

	Sami

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Dave Jones]

On Sat, May 12, 2012 at 01:23:41AM +0300, Sami Liedes wrote:
 > On Fri, May 11, 2012 at 01:32:34PM -0700, Andrew Morton wrote:
 > > Sure, thanks, that might turn something up. 
 > > Documentation/SubmitChecklist recommends 
 > > 
 > > : 12: Has been tested with CONFIG_PREEMPT, CONFIG_DEBUG_PREEMPT,
 > > :     CONFIG_DEBUG_SLAB, CONFIG_DEBUG_PAGEALLOC, CONFIG_DEBUG_MUTEXES,
 > > :     CONFIG_DEBUG_SPINLOCK, CONFIG_DEBUG_ATOMIC_SLEEP, CONFIG_PROVE_RCU
 > > :     and CONFIG_DEBUG_OBJECTS_RCU_HEAD all simultaneously enabled.
 > > 
 > > although that list might be a bit out of date; it certainly should
 > > include CONFIG_DEBUG_VM!
 > 
 > I wonder if there's somewhere a recommended list of generally most
 > useful debug options that only have a moderate performance impact? I'd
 > be happy to use a set of useful debug flags that generally impacts
 > performance by, say, <10%, on the computers I use for my everyday work
 > to help catch bugs. But it's sometimes quite hard to assess the impact
 > of different Kernel hacking options from just the description...

One problem here is that acceptable performance is entirely subjective,
based on your individual workload.  For example, I nearly always run
the fedora 'debug' builds, which enable near every debug option, and
don't find it particularly painful. Other people find those same builds
completely intolerable. Especially for example users of shiny new desktops
that rely heavily on the drm. Seems that code is particularly miserable
to use when spinlock debugging/lockdep are enabled.

A few releases back, something changed which made debug builds even more
'heavyweight' for some users. Still haven't figured out what that might be.

	Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Dave Jones]

On Fri, May 11, 2012 at 12:59:21PM -0700, Andrew Morton wrote:
 
 > > https://bugzilla.kernel.org/show_bug.cgi?id=43227
 > > 
 > > [67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
 > > [67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
 > > (null) index:0x7f1eb293b
 > > [67031.755792] page flags: 0x4000000000000014(referenced|dirty)
 > 
 > AFAICT we got this warning because the page allocator found a free page
 > with PG_referenced and PG_dirty set.
 > 
 > It would be a heck of a lot more useful if we'd been told about this
 > when the page was freed, not when it was reused!  Can anyone think of a
 > reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at
 > least)?

We got a similar bug reported in Fedora against 3.3.4 yesterday.

https://bugzilla.redhat.com/show_bug.cgi?id=820603

:[162276.196730] BUG: Bad page state in process transmission-gt  pfn:1bddff
:[162276.196735] page:ffffea0006f77fc0 count:0 mapcount:0 mapping: (null) index:0x7f2fdb962
:[162276.196737] page flags: 0x40000000000014(referenced|dirty)

Different trace, but same result.

	Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Andrew Morton]

On Fri, 11 May 2012 16:06:15 -0400
Dave Jones <davej@redhat.com> wrote:

> On Fri, May 11, 2012 at 12:59:21PM -0700, Andrew Morton wrote:
>  
>  > > https://bugzilla.kernel.org/show_bug.cgi?id=43227
>  > > 
>  > > [67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
>  > > [67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
>  > > (null) index:0x7f1eb293b
>  > > [67031.755792] page flags: 0x4000000000000014(referenced|dirty)
>  > 
>  > AFAICT we got this warning because the page allocator found a free page
>  > with PG_referenced and PG_dirty set.
>  > 
>  > It would be a heck of a lot more useful if we'd been told about this
>  > when the page was freed, not when it was reused!  Can anyone think of a
>  > reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at
>  > least)?
> 
> We got a similar bug reported in Fedora against 3.3.4 yesterday.
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=820603
> 
> :[162276.196730] BUG: Bad page state in process transmission-gt  pfn:1bddff
> :[162276.196735] page:ffffea0006f77fc0 count:0 mapcount:0 mapping: (null) index:0x7f2fdb962
> :[162276.196737] page flags: 0x40000000000014(referenced|dirty)
> 
> Different trace, but same result.
> 

That narrows it down to a 3.3.2 -> 3.3.4 regression, perhaps.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Sami Liedes]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

On Fri, May 11, 2012 at 01:10:03PM -0700, Andrew Morton wrote:
> That narrows it down to a 3.3.2 -> 3.3.4 regression, perhaps.

I believe 3.3.2 -> 3.3.3 in that case; I also hit this on 3.3.3
yesterday after downgrading 3.3.5 -> 3.3.3, although in that case I
got a large number (total of 36) bad page state warnings within a
single second. I have been using 3.3.2 succesfully for quite a while
before I first saw this when I upgraded to 3.3.5.

	Sami

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Hugh Dickins]

On Fri, 11 May 2012, Andrew Morton wrote:
> > 
> > [67031.755786] BUG: Bad page state in process wcg_gfam_6.11_i  pfn:02519
> > [67031.755790] page:ffffea0000094640 count:0 mapcount:0 mapping:         
> > (null) index:0x7f1eb293b
> > [67031.755792] page flags: 0x4000000000000014(referenced|dirty)
> 
> AFAICT we got this warning because the page allocator found a free page
> with PG_referenced and PG_dirty set.
> 
> It would be a heck of a lot more useful if we'd been told about this
> when the page was freed, not when it was reused!  Can anyone think of a
> reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at
> least)?

Because those flags may validly be set when a page is freed (I do have an
old patch to change anon dirty handling to stop that, but it's not really
needed).  They are then immediately cleared, along with all other page
flags.  So if page allocation finds any page flags set, it happened while
the page was supposedly free.

The only thought I have on this report: what binutils was used to build
this kernel?  We had "Bad page" and isolate_lru_pages BUG reports at the
start of the month, and they were traced to buggy binutils 2.22.52.0.2

Hugh

Hugh

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Sami Liedes]

[-- Attachment #1: Type: text/plain, Size: 327 bytes --]

On Fri, May 11, 2012 at 02:30:42PM -0700, Hugh Dickins wrote:
> The only thought I have on this report: what binutils was used to build
> this kernel?  We had "Bad page" and isolate_lru_pages BUG reports at the
> start of the month, and they were traced to buggy binutils 2.22.52.0.2

Debian unstable's binutils 2.22-6.

	Sami

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: [Bug 43227] New: BUG: Bad page state in process wcg_gfam_6.11_i

[Dave Jones]

On Fri, May 11, 2012 at 02:30:42PM -0700, Hugh Dickins wrote:
 
 > > when the page was freed, not when it was reused!  Can anyone think of a
 > > reason why PAGE_FLAGS_CHECK_AT_FREE doesn't include these flags (at
 > > least)?
 > Because those flags may validly be set when a page is freed (I do have an
 > old patch to change anon dirty handling to stop that, but it's not really
 > needed).

It would be nice to be able to distinguish from things like random memory
scribbles setting those flags.

 > The only thought I have on this report: what binutils was used to build
 > this kernel?  We had "Bad page" and isolate_lru_pages BUG reports at the
 > start of the month, and they were traced to buggy binutils 2.22.52.0.2

The fedora reports we have are from binutils prior to that (2.21.53.0.1 is
the newest we have in F16 for eg).

	Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Fight unfair telecom internet charges in Canada: sign http://stopthemeter.ca/
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>