Re: PCI memory allocation bug with CONFIG_HIGHMEM

[Andi Kleen <ak@muc.de>]

David Hinds <dhinds@sonic.net> writes:

> In arch/i386/kernel/setup.c we have:
>
> 	/* Tell the PCI layer not to allocate too close to the RAM area.. */
> 	low_mem_size = ((max_low_pfn << PAGE_SHIFT) + 0xfffff) & ~0xfffff;
> 	if (low_mem_size > pci_mem_start)
> 		pci_mem_start = low_mem_size;
>
> which is meant to round up pci_mem_start to the nearest 1 MB boundary
> past the top of physical RAM.  However this does not consider highmem.
> Should this just be using max_pfn rather than max_low_pfn?

max_pfn would get memory >4GB on highmem systems, which generally
doesn't work because many PCI devices only support 32bit addresses.

IMHO the only reliable way to get physical bus space for mappings
is to allocate some memory and map the mapping over that.
On x86-64 the allocation must be GFP_DMA, on i386 it can be GFP_KERNEL.

The problem is that BIOS commonly use physical address space without
marking it in the e820 map. For example the AGP aperture is normally
not marked in any way in the e820 map, but you definitely cannot reuse
its bus space. The old code assumed that there is a memory hole below
the highest memory address <4GB, but that can be not true on a system
with >3GB.

We unfortunately must assume on such systems that all holes in e820
space are already used by something. On a system with <3GB you 
are usually lucky because there is some space left, but even that
can break and e.g. conflict with reserved ACPI mappings. In theory
you could have a heuristic with something like "if E820_RAM is 
<2GB just allocate it after the highest E820_RAM map not conflicting
with other E820 mappings", but this would be quite hackish and 
may break on weird systems.

BTW drivers/char/mem.c makes the same broken assumption. It really
wants to default to uncached access for any holes, but default to
cached for real memory. Doing that also requires reliable hole detection,
which we don't have.

One approach I haven't checked is that the ACPI memory map may have
fixed the problem (no defined way to get a hole)

As long as you only have e820 I think there is no real alternative to
the "put io map over memory" technique.

-Andi