[BUG] Page table corruption from commit 9542ada803198e6eba29d3289abb39ea82047b92.

[BUG] Page table corruption from commit 9542ada803198e6eba29d3289abb39ea82047b92.

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi Suresh -

I'm running into the following dump on one of my development nodes. I've
bisected it down to commit 9542ada803198e6eba29d3289abb39ea82047b92. The
SUSE hwinfo tool can reproduce this readily, at least on my system. I'd
be happy to try any test fixes.

Here's from the bisection run, 2.6.27-rc6.

hwinfo: Corrupted page table at address 7f388590d000
PGD 1f88cf067 PUD 1f88ce067 PMD 1fb4e6067 PTE ffffffffffffffff
Bad pagetable: 000d [1] SMP 
CPU 3 
Modules linked in: ipv6 af_packet fuse loop dm_mod tg3 sg rtc_cmos shpchp i2c_amd756 i2c_amd8111 rtc_core sr_mod pcspkr ppdev parport_pc serio_raw i2c_core button rtc_lib k8temp amd_rng cdrom parport pci_hotplug floppy sd_mod ohci_hcd crc_t10dif usbcore edd ext3 mbcache jbd fan ata_generic pata_amd sata_sil libata scsi_mod thermal processor thermal_sys hwmon
Pid: 5869, comm: hwinfo Not tainted 2.6.27-rc6-vanilla #13
RIP: 0033:[<00007f38850dbd3e>]  [<00007f38850dbd3e>] 0x7f38850dbd3e
RSP: 002b:00007fff8d9a7fa8  EFLAGS: 00010202
RAX: 0000000000651820 RBX: 0000000000001000 RCX: fefefefefefefeff
RDX: 0000000000000001 RSI: 00007f388590d000 RDI: 0000000000651820
RBP: 000000000009d000 R08: ffffffffff000000 R09: 00007f38853b0a00
R10: ffffffffffffffff R11: 0000000000000000 R12: 00007f3885427fb6
R13: 0000000000609010 R14: 00007f388590d000 R15: 0000000000000005
FS:  00007f388598f740(0000) GS:ffff8801ffb54a40(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f388590d000 CR3: 00000000f5014000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process hwinfo (pid: 5869, threadinfo ffff8801fb5d4000, task ffff8801f895c4c0)

RIP  [<00007f38850dbd3e>] 0x7f38850dbd3e
 RSP <00007fff8d9a7fa8>
- ---[ end trace 7aaf64f58d755288 ]---

.. and with 2.6.29-rc2, but I'm seeing another warning first:

- ------------[ cut here ]------------
WARNING: at arch/x86/mm/pat.c:620 reserve_pfn_range+0x5b/0x26d()
Hardware name: To Be Filled By O.E.M.
Modules linked in: ipv6 af_packet fuse loop dm_mod tg3 sg ppdev i2c_amd8111 i2c_amd756 shpchp rtc_cmos sr_mod parport_pc rtc_core rtc_lib button cdrom pci_hotplug serio_raw pcspkr floppy parport i2c_core amd_rng k8temp sd_mod ohci_hcd crc_t10dif usbcore edd ext3 mbcache jbd fan ata_generic pata_amd sata_sil libata scsi_mod thermal processor thermal_sys hwmon
Pid: 5532, comm: hwinfo Not tainted 2.6.29-rc2-vanilla #16
Call Trace:
 [<ffffffff80242053>] warn_slowpath+0xb1/0xed
 [<ffffffff802b5ea2>] ? vm_unmap_aliases+0x17c/0x18e
 [<ffffffff8022ba43>] ? lookup_address+0x1dd/0x209
 [<ffffffff80264ad1>] ? smp_call_function_many+0x1f0/0x210
 [<ffffffff8022b84c>] ? __cpa_flush_range+0x0/0x1a
 [<ffffffff8022dd92>] reserve_pfn_range+0x5b/0x26d
 [<ffffffff8022dfef>] track_pfn_vma_new+0x4b/0xb7
 [<ffffffff802aad2e>] remap_pfn_range+0xb5/0x50e
 [<ffffffff8022b2ae>] ? ioremap_change_attr+0x18/0x28
 [<ffffffff8022e3b3>] ? phys_mem_access_prot_allowed+0x100/0x1a9
 [<ffffffff802c88a1>] ? __dentry_open+0x174/0x265
 [<ffffffff803cac46>] mmap_mem+0x75/0x9d
 [<ffffffff802afff8>] mmap_region+0x2a3/0x52a
 [<ffffffff802b058c>] do_mmap_pgoff+0x30d/0x370
 [<ffffffff804a284b>] ? __down_write_nested+0x45/0xe1
 [<ffffffff8021052c>] sys_mmap+0x92/0xce
 [<ffffffff8020c42a>] system_call_fastpath+0x16/0x1b
- ---[ end trace 504e1a061d49152d ]---
hwinfo: Corrupted page table at address 7f8f68c5f000
PGD 1fe474067 PUD 1fe970067 PMD 1f900d067 PTE fffffffffffffbff
Bad pagetable: 000d [#1] SMP 
last sysfs file: /sys/devices/pci0000:00/0000:00:0a.0/0000:02:09.1/class
CPU 3 
Modules linked in: ipv6 af_packet fuse loop dm_mod tg3 sg ppdev i2c_amd8111 i2c_amd756 shpchp rtc_cmos sr_mod parport_pc rtc_core rtc_lib button cdrom pci_hotplug serio_raw pcspkr floppy parport i2c_core amd_rng k8temp sd_mod ohci_hcd crc_t10dif usbcore edd ext3 mbcache jbd fan ata_generic pata_amd sata_sil libata scsi_mod thermal processor thermal_sys hwmon
Pid: 5532, comm: hwinfo Tainted: G        W  2.6.29-rc2-vanilla #16
RIP: 0033:[<00007f8f6842dd3e>]  [<00007f8f6842dd3e>] 0x7f8f6842dd3e
RSP: 002b:00007fff70cf82d8  EFLAGS: 00010202
RAX: 0000000000652280 RBX: 0000000000001000 RCX: fefefefefefefeff
RDX: 0000000000000001 RSI: 00007f8f68c5f000 RDI: 0000000000652280
RBP: 000000000009d000 R08: ffffffffff000000 R09: 00007f8f68702a00
R10: ffffffffffffffff R11: 0000000000000000 R12: 00007f8f68779fb6
R13: 0000000000609010 R14: 00007f8f68c5f000 R15: 0000000000000005
FS:  00007f8f68ce1740(0000) GS:ffff8801ff209b40(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00007f8f68c5f000 CR3: 00000001fdd19000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process hwinfo (pid: 5532, threadinfo ffff8801fe938000, task ffff8801f907a040)

RIP  [<00007f8f6842dd3e>] 0x7f8f6842dd3e
 RSP <00007fff70cf82d8>
- ---[ end trace 504e1a061d49152e ]---

- -Jeff

- -- 
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkl4DT0ACgkQLPWxlyuTD7LbpQCeK6aVOiHY4niprseJatUm27w5
OHsAnjbO9btEuudeeeqVWxsVfa3yiUzU
=MJvA
-----END PGP SIGNATURE-----

Re: [BUG] Page table corruption from commit 9542ada803198e6eba29d3289abb39ea82047b92.

[Ingo Molnar]

* Jeff Mahoney <jeffm@suse.com> wrote:

> 
> Hi Suresh -
> 
> I'm running into the following dump on one of my development nodes. I've 
> bisected it down to commit 9542ada803198e6eba29d3289abb39ea82047b92. The 
> SUSE hwinfo tool can reproduce this readily, at least on my system. I'd 
> be happy to try any test fixes.
> 
> Here's from the bisection run, 2.6.27-rc6.

Could you try the current -tip tree please?

  http://people.redhat.com/mingo/tip.git/README

There's a number of fixes in there that i think will fix your crash. In 
particular:

  9597134: x86: fix PTE corruption issue while mapping RAM using /dev/mem

	Ingo

Re: [BUG] Page table corruption from commit 9542ada803198e6eba29d3289abb39ea82047b92.

[Jeff Mahoney]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ingo Molnar wrote:
> * Jeff Mahoney <jeffm@suse.com> wrote:
> 
>> Hi Suresh -
>>
>> I'm running into the following dump on one of my development nodes. I've 
>> bisected it down to commit 9542ada803198e6eba29d3289abb39ea82047b92. The 
>> SUSE hwinfo tool can reproduce this readily, at least on my system. I'd 
>> be happy to try any test fixes.
>>
>> Here's from the bisection run, 2.6.27-rc6.
> 
> Could you try the current -tip tree please?
> 
>   http://people.redhat.com/mingo/tip.git/README
> 
> There's a number of fixes in there that i think will fix your crash. In 
> particular:
> 
>   9597134: x86: fix PTE corruption issue while mapping RAM using /dev/mem

Yep. That change fixed the corruption, but I still get the other
warning. It originates with mainline commit
5899329b19100c0b82dc78e9b21ed8b920c9ffb3. I get another warning at
the bisect site, but it seems secondary. It doesn't occur in tip, though.

- -Jeff

In tip:
- ------------[ cut here ]------------
WARNING: at arch/x86/mm/pat.c:637 reserve_pfn_range+0x5b/0x26d()
Hardware name: To Be Filled By O.E.M.
Modules linked in: ipv6 af_packet fuse loop dm_mod tg3 shpchp sg ppdev i2c_amd756 i2c_amd8111 rtc_cmos rtc_core parport_pc amd_rng serio_raw pcspkr button pci_hotplug rtc_lib parport sr_mod k8temp i2c_core cdrom floppy ohci_hcd sd_mod crc_t10dif usbcore edd ext3 mbcache jbd fan ata_generic pata_amd sata_sil libata scsi_mod thermal processor thermal_sys hwmon
Pid: 5841, comm: hwinfo Not tainted 2.6.29-rc2-tip-vanilla #19
Call Trace:
 [<ffffffff80241f6f>] warn_slowpath+0xb1/0xed
 [<ffffffff802b6bc4>] ? vm_unmap_aliases+0x175/0x185
 [<ffffffff8022be04>] ? change_page_attr_set_clr+0x13e/0x33a
 [<ffffffff804a3a40>] ? _spin_lock+0x17/0x1a
 [<ffffffff802b54c3>] ? __purge_vmap_area_lazy+0x173/0x183
 [<ffffffff8022d3dd>] reserve_pfn_range+0x5b/0x26d
 [<ffffffff8022d63a>] track_pfn_vma_new+0x4b/0xb8
 [<ffffffff802aba68>] remap_pfn_range+0xb5/0x50e
 [<ffffffff8022a8a7>] ? ioremap_change_attr+0x26/0x28
 [<ffffffff8022d9ff>] ? phys_mem_access_prot_allowed+0x100/0x1a9
 [<ffffffff802c9639>] ? __dentry_open+0x174/0x265
 [<ffffffff803cc136>] mmap_mem+0x75/0x9d
 [<ffffffff802b0d08>] mmap_region+0x2a3/0x52a
 [<ffffffff802b129c>] do_mmap_pgoff+0x30d/0x370
 [<ffffffff804a372e>] ? __down_write_nested+0x45/0xe1
 [<ffffffff8021057c>] sys_mmap+0x92/0xce
 [<ffffffff8020c42a>] system_call_fastpath+0x16/0x1b
- ---[ end trace e793dff9eb62b2bd ]---


At the bisect site:

- ------------[ cut here ]------------                                            
WARNING: at arch/x86/mm/pat.c:619 reserve_pfn_range+0x4c/0x231()                
Modules linked in: ipv6 af_packet fuse loop dm_mod ppdev sg tg3 i2c_amd8111 parport_pc sr_mod i2c_amd756 rtc_cmos shpchp amd_rng button rtc_core parport floppy i2c_core serio_raw cdrom pcspkr rtc_lib k8temp pci_hotplug ohci_hcd sd_mod crc_t10dif usbcore edd ext3 mbcache jbd fan ata_generic pata_amd sata_sil libata scsi_mod thermal processor thermal_sys hwmon                                        
Pid: 5642, comm: hwinfo Not tainted 2.6.28-rc8-vanilla #44                      
Call Trace:                                                                     
 [<ffffffff802425e6>] warn_on_slowpath+0x58/0x7d                                
 [<ffffffff8022c8c0>] ? change_page_attr_set_clr+0x136/0x32e                    
 [<ffffffff804916ed>] ? _spin_lock+0x17/0x1a                                    
 [<ffffffff802ae099>] ? __purge_vmap_area_lazy+0x171/0x181                      
 [<ffffffff8022b269>] ? pagerange_is_ram+0x31/0x5c                              
 [<ffffffff8022de48>] reserve_pfn_range+0x4c/0x231                              
 [<ffffffff8022e076>] track_pfn_vma_new+0x49/0xb3                               
 [<ffffffff802a4741>] remap_pfn_range+0xa4/0x4dc                                
 [<ffffffff8022cbe5>] ? _set_memory_wb+0x22/0x24                                
 [<ffffffff8022b3b8>] ? ioremap_change_attr+0x26/0x28                           
 [<ffffffff8022e4cb>] ? phys_mem_access_prot_allowed+0x100/0x1a9                
 [<ffffffff803c2804>] mmap_mem+0x75/0x9d                                        
 [<ffffffff802a988f>] mmap_region+0x297/0x506                                   
 [<ffffffff802a9e0b>] do_mmap_pgoff+0x30d/0x370                                 
 [<ffffffff804913e7>] ? __down_write_nested+0x45/0xdc                           
 [<ffffffff80211108>] sys_mmap+0x92/0xce                                        
 [<ffffffff8020c2ca>] system_call_fastpath+0x16/0x1b                            
- ---[ end trace a2d1666c2f766ad5 ]---                                            
- ------------[ cut here ]------------                                            
WARNING: at arch/x86/mm/pageattr.c:560 __change_page_attr_set_clr+0x10b/0xc0c() 
CPA: called for zero pte. vaddr = ffff8800f7ff0000 cpa->vaddr = ffff8800f7ff0000
Modules linked in: ipv6 af_packet fuse loop dm_mod ppdev sg tg3 i2c_amd8111 parport_pc sr_mod i2c_amd756 rtc_cmos shpchp amd_rng button rtc_core parport floppy i2c_core serio_raw cdrom pcspkr rtc_lib k8temp pci_hotplug ohci_hcd sd_mod crc_t10dif usbcore edd ext3 mbcache jbd fan ata_generic pata_amd sata_sil libata scsi_mod thermal processor thermal_sys hwmon                                        
Pid: 5645, comm: acpidump Tainted: G        W  2.6.28-rc8-vanilla #44
Call Trace:
 [<ffffffff80242567>] warn_slowpath+0x8c/0xb3
 [<ffffffff8029e248>] ? zone_statistics+0x62/0x67
 [<ffffffff8022bb1b>] ? lookup_address+0x1dd/0x209
 [<ffffffff8029e248>] ? zone_statistics+0x62/0x67
 [<ffffffff8029679b>] ? get_page_from_freelist+0x4e4/0x69e
 [<ffffffff8029e248>] ? zone_statistics+0x62/0x67
 [<ffffffff8029679b>] ? get_page_from_freelist+0x4e4/0x69e
 [<ffffffff8022bc89>] __change_page_attr_set_clr+0x10b/0xc0c
 [<ffffffff80290acb>] ? find_lock_page+0x1e/0x5c
 [<ffffffff8022bb1b>] ? lookup_address+0x1dd/0x209
 [<ffffffff802910e8>] ? filemap_fault+0x1c4/0x31b
 [<ffffffff80290aa9>] ? unlock_page+0x22/0x26
 [<ffffffff802a1013>] ? __do_fault+0x46e/0x4b4
 [<ffffffff804916ed>] ? _spin_lock+0x17/0x1a
 [<ffffffff802ae099>] ? __purge_vmap_area_lazy+0x171/0x181
 [<ffffffff802af79c>] ? vm_unmap_aliases+0x172/0x182
 [<ffffffff8022c8c0>] change_page_attr_set_clr+0x136/0x32e
 [<ffffffff8022b269>] ? pagerange_is_ram+0x31/0x5c
 [<ffffffff8022ccf0>] _set_memory_uc+0x22/0x24
 [<ffffffff8022b3aa>] ioremap_change_attr+0x18/0x28
 [<ffffffff8022df9b>] reserve_pfn_range+0x19f/0x231
 [<ffffffff8022e076>] track_pfn_vma_new+0x49/0xb3
 [<ffffffff802a4741>] remap_pfn_range+0xa4/0x4dc
 [<ffffffff804916ed>] ? _spin_lock+0x17/0x1a
 [<ffffffff8022dcdc>] ? reserve_memtype+0x3e1/0x501
 [<ffffffff8022e480>] ? phys_mem_access_prot_allowed+0xb5/0x1a9
 [<ffffffff803c2804>] mmap_mem+0x75/0x9d
 [<ffffffff802a988f>] mmap_region+0x297/0x506
 [<ffffffff802a9e0b>] do_mmap_pgoff+0x30d/0x370
 [<ffffffff804913e7>] ? __down_write_nested+0x45/0xdc
 [<ffffffff80211108>] sys_mmap+0x92/0xce
 [<ffffffff8020c2ca>] system_call_fastpath+0x16/0x1b
- ---[ end trace a2d1666c2f766ad5 ]---
acpidump:5645 reserve_pfn_range ioremap_change_attr failed uncached-minus for f7ff0000-f7ff1000

- -- 
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkl449YACgkQLPWxlyuTD7JZcwCeL9b6Rqvz5otheyDjLvFi/lPf
E78An20tdqum1+G80ZzTqs7baRDbyhTL
=J7ag
-----END PGP SIGNATURE-----

Re: [BUG] Page table corruption from commit 9542ada803198e6eba29d3289abb39ea82047b92.

[Suresh Siddha]

On Thu, Jan 22, 2009 at 01:23:34PM -0800, Jeff Mahoney wrote:
> Ingo Molnar wrote:
> > Could you try the current -tip tree please?
> > 
> >   http://people.redhat.com/mingo/tip.git/README
> > 
> > There's a number of fixes in there that i think will fix your crash. In 
> > particular:
> > 
> >   9597134: x86: fix PTE corruption issue while mapping RAM using /dev/mem
> 
> Yep. That change fixed the corruption, but I still get the other
> warning. It originates with mainline commit
> 5899329b19100c0b82dc78e9b21ed8b920c9ffb3.

> WARNING: at arch/x86/mm/pat.c:637 reserve_pfn_range+0x5b/0x26d()

We are aware of this warning issue. We will soon have a fix for this.
For now, this is benign (essentially mmap of /dev/mem is doing
the reserve_memtype() twice, one through phys_mem_access_prot_allowed()
and reserve_pfn_range(). We are planning to remove this double
reserve, which should fix the warning).


> I get another warning at
> the bisect site, but it seems secondary. It doesn't occur in tip, though.
> 
> WARNING: at arch/x86/mm/pageattr.c:560 __change_page_attr_set_clr+0x10b/0xc0c() 
> CPA: called for zero pte. vaddr = ffff8800f7ff0000 cpa->vaddr = ffff8800f7ff0000

This one is fixed in tip by commit a1e46212a410793d575718818e81ddc442a65283
	x86: fix page attribute corruption with cpa()

thanks,
suresh