DRAM and PCI devices at same physical address

DRAM and PCI devices at same physical address

[Matt Sexton]

I have a dual Xeon system with the Lindenhurst (E7710) chip set and 1 GB
of memory.  In order to reserve a very large block of memory for a
(user-space) device driver I am writing, I pass "mem=XX" to the kernel
at boot time.  Unfortunately, /proc/pci shows two devices now appearing
in the reserved upper memory range.  

For instance, if I set "mem=768M", the following two entries appear in
/proc/pci:

  Bus  0, device   1, function  0:
    System peripheral: PCI device 8086:3594 (Intel Corp.) (rev 4).
      Non-prefetchable 32 bit memory at 0x30000000 [0x30000fff].

  Bus  0, device  31, function  1:
    IDE interface: Intel Corp. 82801EB Ultra ATA Storage Controller (rev
2).
      IRQ 18.
      I/O at 0x14a0 [0x14af].
      Non-prefetchable 32 bit memory at 0x30001000 [0x300013ff].


The devices always appear right after the limit I specify on the kernel
boot line.  If I specify "mem=512M", then the first device appears at
0x20000000.  If I specify nothing, then it appears at 0x40000000.  All
other PCI devices show up at addresses of 0xDD000000 and above.

Is there any way to prevent these devices from showing up in the
physical address range of my reserved memory?

Should they be appearing there at all?  Does Linux make any guarantees
when there is more physical memory than specified by "mem=" ?

Please CC me on any responses.

Thank you,
Matt Sexton
sexton@mc.com

Re: DRAM and PCI devices at same physical address

[Matt Porter]

On Fri, Jun 25, 2004 at 05:23:00PM -0400, Matt Sexton wrote:
> I have a dual Xeon system with the Lindenhurst (E7710) chip set and 1 GB
> of memory.  In order to reserve a very large block of memory for a
> (user-space) device driver I am writing, I pass "mem=XX" to the kernel
> at boot time.  Unfortunately, /proc/pci shows two devices now appearing
> in the reserved upper memory range.  

<snip>
 
> The devices always appear right after the limit I specify on the kernel
> boot line.  If I specify "mem=512M", then the first device appears at
> 0x20000000.  If I specify nothing, then it appears at 0x40000000.  All
> other PCI devices show up at addresses of 0xDD000000 and above.
> 
> Is there any way to prevent these devices from showing up in the
> physical address range of my reserved memory?

You could try using reserve_bootmem() to reserve your driver memory.

> Should they be appearing there at all?  Does Linux make any guarantees
> when there is more physical memory than specified by "mem=" ?

Depends on the arch, I don't know what ia32 does.
 
-Matt

Re: DRAM and PCI devices at same physical address

[Matt Sexton]

On Sat, 2004-06-26 at 23:26, Matt Porter wrote:
> On Fri, Jun 25, 2004 at 05:23:00PM -0400, Matt Sexton wrote:
> > I have a dual Xeon system with the Lindenhurst (E7710) chip set and 1 GB
> > of memory.  In order to reserve a very large block of memory for a
> > (user-space) device driver I am writing, I pass "mem=XX" to the kernel
> > at boot time.  Unfortunately, /proc/pci shows two devices now appearing
> > in the reserved upper memory range.  
> 
> <snip>
>  
> > The devices always appear right after the limit I specify on the kernel
> > boot line.  If I specify "mem=512M", then the first device appears at
> > 0x20000000.  If I specify nothing, then it appears at 0x40000000.  All
> > other PCI devices show up at addresses of 0xDD000000 and above.
> > 
> > Is there any way to prevent these devices from showing up in the
> > physical address range of my reserved memory?
> 
> You could try using reserve_bootmem() to reserve your driver memory.
> 

But then I'd have to modify the kernel.  I'd rather just use a loadable
module or user-space driver.

> > Should they be appearing there at all?  Does Linux make any guarantees
> > when there is more physical memory than specified by "mem=" ?
> 

The problem appears to be that the BIOS did not assign PCI addresses to
the two devices.  Linux (2.6.3-4mdkenterprise) then did so, but it
starts assigning at either 256MB, or the first 1MB aligned page after
the end of DRAM, whichever is higher.  On my 1GB system with "mem=768M",
this the region of my "reserved" DRAM.

So, using "mem=" to reserve DRAM and having Linux assign PCI addresses
are not compatible.

Matt

> Depends on the arch, I don't know what ia32 does.
>  
> -Matt

Re: DRAM and PCI devices at same physical address

[Matt Porter]

On Mon, Jun 28, 2004 at 11:51:14AM -0400, Matt Sexton wrote:
> On Sat, 2004-06-26 at 23:26, Matt Porter wrote:
> > On Fri, Jun 25, 2004 at 05:23:00PM -0400, Matt Sexton wrote:
> > > I have a dual Xeon system with the Lindenhurst (E7710) chip set and 1 GB
> > > of memory.  In order to reserve a very large block of memory for a
> > > (user-space) device driver I am writing, I pass "mem=XX" to the kernel
> > > at boot time.  Unfortunately, /proc/pci shows two devices now appearing
> > > in the reserved upper memory range.  
> > 
> > <snip>
> >  
> > > The devices always appear right after the limit I specify on the kernel
> > > boot line.  If I specify "mem=512M", then the first device appears at
> > > 0x20000000.  If I specify nothing, then it appears at 0x40000000.  All
> > > other PCI devices show up at addresses of 0xDD000000 and above.
> > > 
> > > Is there any way to prevent these devices from showing up in the
> > > physical address range of my reserved memory?
> > 
> > You could try using reserve_bootmem() to reserve your driver memory.
> > 
> 
> But then I'd have to modify the kernel.  I'd rather just use a loadable
> module or user-space driver.

Yes, but this is the most reliable way to do your large allocation.
This reminds me that it would be handy to have a "bootmem=" cmdline
parameter.

Alternatively, set CONFIG_FORCE_MAX_ZONEORDER to an order that allows
your huge allocation using __get_free_pages().  However, there's no
guarantees it will succeed after things are fragmented.

> > > Should they be appearing there at all?  Does Linux make any guarantees
> > > when there is more physical memory than specified by "mem=" ?
> > 
> 
> The problem appears to be that the BIOS did not assign PCI addresses to
> the two devices.  Linux (2.6.3-4mdkenterprise) then did so, but it
> starts assigning at either 256MB, or the first 1MB aligned page after
> the end of DRAM, whichever is higher.  On my 1GB system with "mem=768M",
> this the region of my "reserved" DRAM.
> 
> So, using "mem=" to reserve DRAM and having Linux assign PCI addresses
> are not compatible.

Yes.  It's going to call pci_assign_resource() which uses
PCIBIOS_MIN_[IO|MEM] to determine the allowed range of resource
assignment for a given resource type.  On i386, PCIBIOS_MIN_MEM
is defined as pci_mem_start. In arch/i386/setup.c this is configured as:

    if (low_mem_size > pci_mem_start)
        pci_mem_start = low_mem_size;

So you can see that using "mem=" isn't compatible with the assignment
methodology.

-Matt

Re: DRAM and PCI devices at same physical address

[Ross Biro]

On 25 Jun 2004 17:23:00 -0400, Matt Sexton <sexton@mc.com> wrote:

> 
> Should they be appearing there at all?  Does Linux make any guarantees
> when there is more physical memory than specified by "mem=" ?

You've told linux there is only 512M of physical memory and it
believes you, so it is using the available address space for memory
mapped i/o.  I know of no way to reserve memory on the command line.

Re: DRAM and PCI devices at same physical address

[Matt Sexton]

On Mon, 2004-06-28 at 15:27, Ross Biro wrote:
> On 25 Jun 2004 17:23:00 -0400, Matt Sexton <sexton@mc.com> wrote:
> 
> > 
> > Should they be appearing there at all?  Does Linux make any guarantees
> > when there is more physical memory than specified by "mem=" ?
> 
> You've told linux there is only 512M of physical memory and it
> believes you, so it is using the available address space for memory
> mapped i/o.  I know of no way to reserve memory on the command line.

Fair enough.  I was just following the advice of the "Linux Device
Drivers" book by Rubini & Corbet, where in the "Reserving High RAM
Addresses" section of Chapter 7, they describe exactly this method of
reserving memory for device drivers.

Matt