Re: DRAM and PCI devices at same physical address

[Matt Porter <mporter@kernel.crashing.org>]

On Mon, Jun 28, 2004 at 11:51:14AM -0400, Matt Sexton wrote:
> On Sat, 2004-06-26 at 23:26, Matt Porter wrote:
> > On Fri, Jun 25, 2004 at 05:23:00PM -0400, Matt Sexton wrote:
> > > I have a dual Xeon system with the Lindenhurst (E7710) chip set and 1 GB
> > > of memory.  In order to reserve a very large block of memory for a
> > > (user-space) device driver I am writing, I pass "mem=XX" to the kernel
> > > at boot time.  Unfortunately, /proc/pci shows two devices now appearing
> > > in the reserved upper memory range.  
> > 
> > <snip>
> >  
> > > The devices always appear right after the limit I specify on the kernel
> > > boot line.  If I specify "mem=512M", then the first device appears at
> > > 0x20000000.  If I specify nothing, then it appears at 0x40000000.  All
> > > other PCI devices show up at addresses of 0xDD000000 and above.
> > > 
> > > Is there any way to prevent these devices from showing up in the
> > > physical address range of my reserved memory?
> > 
> > You could try using reserve_bootmem() to reserve your driver memory.
> > 
> 
> But then I'd have to modify the kernel.  I'd rather just use a loadable
> module or user-space driver.

Yes, but this is the most reliable way to do your large allocation.
This reminds me that it would be handy to have a "bootmem=" cmdline
parameter.

Alternatively, set CONFIG_FORCE_MAX_ZONEORDER to an order that allows
your huge allocation using __get_free_pages().  However, there's no
guarantees it will succeed after things are fragmented.

> > > Should they be appearing there at all?  Does Linux make any guarantees
> > > when there is more physical memory than specified by "mem=" ?
> > 
> 
> The problem appears to be that the BIOS did not assign PCI addresses to
> the two devices.  Linux (2.6.3-4mdkenterprise) then did so, but it
> starts assigning at either 256MB, or the first 1MB aligned page after
> the end of DRAM, whichever is higher.  On my 1GB system with "mem=768M",
> this the region of my "reserved" DRAM.
> 
> So, using "mem=" to reserve DRAM and having Linux assign PCI addresses
> are not compatible.

Yes.  It's going to call pci_assign_resource() which uses
PCIBIOS_MIN_[IO|MEM] to determine the allowed range of resource
assignment for a given resource type.  On i386, PCIBIOS_MIN_MEM
is defined as pci_mem_start. In arch/i386/setup.c this is configured as:

    if (low_mem_size > pci_mem_start)
        pci_mem_start = low_mem_size;

So you can see that using "mem=" isn't compatible with the assignment
methodology.

-Matt