x86-64 general protection fault in ioremap_nocache() possibly related to memory beyond 4GB

x86-64 general protection fault in ioremap_nocache() possibly related to memory beyond 4GB

[Steve Holland]

I'm having a memory/io mapping related problem with the 2.6 kernel
(as shipped with Fedora Core 2, tested with kernel-2.6.5-1.358
and kernel-2.6.6-1.435).

My system has a dual Opteron Tyan K8W with 4 GB of ram. Half of that
memory is mapped by the BIOS beyond 4GB, leaving plenty of address space
between 2GB and 4GB for pci devices. 

If I tell the kernel (via mem=0x80000000) to only use the first 2GB, all
of these problems go away. There were also no problems doing any of
these things under a previous installation 2.4 x86-64 kernel. 

Hence the problem seems to be related to the presence of memory beyond
the 4GB boundary. 

I have seen three different symptoms of this one problem. These all
appear independent of each other:
   1. Oops in ioremap_nocache() during modprobe of ohci-hcd.ko during
rc.sysinit.
   2. X locks up on startup (Radeon driver) with general protection
fault (even with dri, etc. disabled):

Jun 22 17:28:41 eponine kernel: general protection fault: 0000 [1] SMP 
Jun 22 17:28:41 eponine kernel: CPU 0 
Jun 22 17:28:41 eponine kernel: Modules linked in: ipv6 parport_pc lp parport autofs4 tg3 ipt_REJECT ipt_state ip_conntrack iptable_filter ip_tables dm_mod ext3 jbd sata_sil libata sd_mod scsi_mod
Jun 22 17:28:41 eponine kernel: Pid: 2658, comm: X Not tainted 2.6.6-1.435smp
Jun 22 17:28:41 eponine kernel: RIP: 0010:[<ffffffff8015f3c8>] <ffffffff8015f3c8>{remap_page_range+407}
Jun 22 17:28:41 eponine kernel: Call Trace:<ffffffff801f85bc>{mmap_mem+224} <ffffffff80161f3f>{do_mmap_pgoff+1186} 
Jun 22 17:28:41 eponine kernel:        <ffffffff80117849>{sys_mmap+135} <ffffffff801112c6>{system_call+126} 
Jun 22 17:28:41 eponine kernel: Code: 48 2b 82 f0 18 00 00 48 6b c0 38 48 03 82 e0 18 00 00 8b 00 

   3. A driver (with which I am very familiar with the internals) gets a general protection fault
during insmod. This driver starts by calling  pci_set_dma_mask(), then calls request_mem_region() for several smallish
(a few kb or less) chunks found with pci_resource_start() and pci_resource_len().  It then calls 
ioremap_nocache() on  one of these chunks, but ioremap_nocache() fails with a general protection fault:

Jun 22 17:37:04 eponine kernel: general protection fault: 0000 [1] SMP 
Jun 22 17:37:04 eponine kernel: CPU 0 
Jun 22 17:37:04 eponine kernel: Modules linked in: pci_das4020_12 ipv6 parport_pc lp parport autofs4 tg3 ipt_REJECT ipt_state ip_conntrack iptable_filter ip_tables dm_mod ext3 jbd sata_sil libata sd_mod scsi_mod
Jun 22 17:37:04 eponine kernel: Pid: 2655, comm: insmod Not tainted 2.6.6-1.435smp
Jun 22 17:37:04 eponine kernel: RIP: 0010:[<ffffffff80122e38>] <ffffffff80122e38>{ioremap_nocache+197}
Jun 22 17:37:04 eponine kernel: Call Trace:<ffffffffa01081c5>{:pci_das4020_12:das4020_init_one+453} 
Jun 22 17:37:04 eponine kernel:        <ffffffff801d5a73>{pci_device_probe_static+47} <ffffffff801d5aa7>{__pci_device_probe+30} 
Jun 22 17:37:04 eponine kernel:        <ffffffff801d5ae1>{pci_device_probe+37} <ffffffff801d5abc>{pci_device_probe+0} 
Jun 22 17:37:04 eponine kernel:        <ffffffff80218831>{bus_match+57} <ffffffff80218938>{driver_attach+68} 
Jun 22 17:37:04 eponine kernel:        <ffffffff80218b7d>{bus_add_driver+112} <ffffffff80176590>{exact_match+0} 
Jun 22 17:37:04 eponine kernel:        <ffffffff801d5cd2>{pci_register_driver+111} <ffffffffa011004f>{:pci_das4020_12:das4020_init+79} 
Jun 22 17:37:04 eponine kernel:        <ffffffff8014ad48>{sys_init_module+264} <ffffffff801112c6>{system_call+126} 
Jun 22 17:37:04 eponine kernel: Code: 48 8b 8f f0 18 00 00 76 10 48 b8 00 00 00 80 00 01 00 00 48 

As I mentioned above, the problems only occur when using the 2GB of memory above 4GB. 
Jun 22 20:05:40 eponine kernel: BIOS-provided physical RAM map:
Jun 22 20:05:40 eponine kernel:  BIOS-e820: 0000000000000000 - 000000000009fc00 (usable)
Jun 22 20:05:40 eponine kernel:  BIOS-e820: 000000000009fc00 - 00000000000a0000 (reserved)
Jun 22 20:05:40 eponine kernel:  BIOS-e820: 00000000000e0000 - 0000000000100000 (reserved)
Jun 22 20:05:40 eponine kernel:  BIOS-e820: 0000000000100000 - 0000000080000000 (usable)
Jun 22 20:05:40 eponine kernel:  BIOS-e820: 00000000ff7c0000 - 0000000100000000 (reserved)
Jun 22 20:05:40 eponine kernel:  BIOS-e820: 0000000100000000 - 0000000180000000 (usable)

All these things worked normally under the 2.4 kernel of Fedora Core 1

	Thanks,
	Steve

ess dee h 4 at cornell dot edu

Re: x86-64 general protection fault in ioremap_nocache() possibly related to memory beyond 4GB

[Andi Kleen]

Steve Holland <sdh4_no_spammers_throwaway_acct@cornell.edu> writes:

> I'm having a memory/io mapping related problem with the 2.6 kernel
> (as shipped with Fedora Core 2, tested with kernel-2.6.5-1.358
> and kernel-2.6.6-1.435).

Please try a 2.6.7 kernel.org kernel.

Also your oops messages don't contain registers for some reasons.
Please post full oops messages next time.

-Andi 

Re: x86-64 general protection fault in ioremap_nocache() possibly related to memory beyond 4GB

[Steve Holland]

The same problem
(http://www.uwsg.indiana.edu/hypermail/linux/kernel/0406.2/1919.html)
exists with a 2.6.7 kernel.org kernel. 

The full oops message for ohci_hcd on the vanilla 2.6.7:
Jun 24 13:21:00 eponine kernel: ohci_hcd: 2004 Feb 02 USB 1.1 'Open' Host Controller (OHCI) Driver (PCI)
Jun 24 13:21:00 eponine kernel: ohci_hcd: block sizes: ed 80 td 96
Jun 24 13:21:00 eponine kernel: Unable to handle kernel paging request at 00000000000018f0 RIP: 
Jun 24 13:21:00 eponine kernel: <ffffffff80125335>{ioremap_nocache+197}
Jun 24 13:21:00 eponine kernel: PML4 7eb3b067 PGD 2fa6067 PMD 0 
Jun 24 13:21:00 eponine kernel: Oops: 0000 [1] SMP 
Jun 24 13:21:00 eponine kernel: CPU 0 
Jun 24 13:21:00 eponine kernel: Modules linked in: ohci_hcd ext3 jbd
Jun 24 13:21:00 eponine kernel: Pid: 635, comm: modprobe Not tainted 2.6.7
Jun 24 13:21:00 eponine kernel: RIP: 0010:[<ffffffff80125335>] <ffffffff80125335>{ioremap_nocache+197}
Jun 24 13:21:00 eponine kernel: RSP: 0018:000001017ed2fde8  EFLAGS: 00010213
Jun 24 13:21:00 eponine kernel: RAX: 00000100ff3f6000 RBX: 00000000ff3f6000 RCX: 0000000000000019
Jun 24 13:21:00 eponine kernel: RDX: ffffffff7fffffff RSI: 0000010180000000 RDI: 0000000000000000
Jun 24 13:21:00 eponine kernel: RBP: 0000000000001000 R08: 00000000ff3f7000 R09: 000001017ed2fd3c
Jun 24 13:21:00 eponine kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffff000000d000
Jun 24 13:21:00 eponine kernel: R13: 00000000ff3f6000 R14: 0000000000000000 R15: 0000000000000000
Jun 24 13:21:00 eponine kernel: FS:  0000002a9557a4c0(0000) GS:ffffffff804abe80(0000) knlGS:0000000000000000
Jun 24 13:21:00 eponine kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jun 24 13:21:00 eponine kernel: CR2: 00000000000018f0 CR3: 0000000000101000 CR4: 00000000000006e0
Jun 24 13:21:00 eponine kernel: Process modprobe (pid: 635, threadinfo 000001017ed2e000, task 000001017ed5a9d0)
Jun 24 13:21:00 eponine kernel: Stack: ffffffffa00343c0 ffffffffa0034340 0000010101c8c000 ffffffff80289178 
Jun 24 13:21:00 eponine kernel:        0000010101c54990 0000000000001000 ffffffff00000008 ffffffffa00362c0 
Jun 24 13:21:00 eponine kernel:        0000010101c8c000 00000000ffffffed 
Jun 24 13:21:00 eponine kernel: Call Trace:<ffffffff80289178>{usb_hcd_pci_probe+280} <ffffffff801eda5d>{pci_device_probe_static+61} 
Jun 24 13:21:00 eponine kernel:        <ffffffff801edab9>{__pci_device_probe+41} <ffffffff801edb10>{pci_device_probe+48} 
Jun 24 13:21:00 eponine kernel:        <ffffffff80235257>{bus_match+71} <ffffffff80235376>{driver_attach+70} 
Jun 24 13:21:00 eponine kernel:        <ffffffff80235620>{bus_add_driver+128} <ffffffff801eddd9>{pci_register_driver+121} 
Jun 24 13:21:00 eponine kernel:        <ffffffffa0038051>{:ohci_hcd:ohci_hcd_pci_init+81} 
Jun 24 13:21:00 eponine kernel:        <ffffffff80151a7e>{sys_init_module+270} <ffffffff8011189a>{system_call+126} 
Jun 24 13:21:00 eponine kernel:        
Jun 24 13:21:00 eponine kernel: 
Jun 24 13:21:00 eponine kernel: Code: 48 8b 8f f0 18 00 00 76 2a 48 b8 00 00 00 80 00 01 00 00 48 
Jun 24 13:21:00 eponine kernel: RIP <ffffffff80125335>{ioremap_nocache+197} RSP <000001017ed2fde8>
Jun 24 13:21:00 eponine kernel: CR2: 00000000000018f0

Once again, removing memory beyond 4GB with a 'mem=' parameter eliminates the problem. 

With the 2.6.7 kernel X no longer causes an oops but instead locks itself up. 
My own driver still has the same problem as ohci_hcd in ioremap_nocache().

I tried disabling the hardware iommu with 'iommu=soft', and that did not have any effect. 
	Steve


Andi Kleen wrote: 
>Steve Holland <sdh4_no_spammers_throwaway_acct@xxxxxxxxxxx> writes:
>> I'm having a memory/io mapping related problem with the 2.6 kernel
>> (as shipped with Fedora Core 2, tested with kernel-2.6.5-1.358
>> and kernel-2.6.6-1.435).
>
>Please try a 2.6.7 kernel.org kernel.
>
>Also your oops messages don't contain registers for some reasons.
>Please post full oops messages next time.