64bit PCI IORESOURCE_MEM bugs

64bit PCI IORESOURCE_MEM bugs

[Thomas Crowley]

I am running linux 2.6.12.2 with the memmap patch, on an Intel 64 bit 
machine with 16 Gigs of RAM.  I have written a device driver that DMAs to 14 
Gigs of RAM.

I am attempting to give 1 Gig of RAM to the OS and 1Gig of address space for 
PCI addressing and 14 Gigs for my DMA.  I have added the following to my 
grub command line:
memmap=exactmap memmap=640K@0 memmap=1024M@1M memmap=2000M$2048M 
memmap=12000M$4048M

This does not seam to be working.  I think I have run into some bugs in the 
kernel unless I am mistaken how physical RAM addressing and the bus 
addressing work.

1) in arch/x86_64/kernel/e820.c    the e820_reserve_resources function the 
line if (e820.map[i].addr + e820.map[i].size > 0x100000000ULL) makes it so 
any region that
     starts below the 4Gig mark but ends above 4Gig mark is ignored.  So I 
have been forced to reserve the 15 Gigs using two different memmap calls one 
that reserves memory below 4 Gigs and one that reserves the memory above 4 
Gigs.

2) I am unable to get all of my PCI devices to move there addressing from 
the default values to the hole I have given it.  I am getting several errors 
of the form: "PCI: Cannot allocate resource region 8 of bridge 
0000:00:07.0".  Some of my devices are moving to the proper location.
Neither the code in drivers/pci/probe.c pci_read_bases which gets the 
default addresses for the resources or /arch/i386/pci/i386.c 
pcibios_allocate_bus_resources check to see if the address that are being 
used are reserved.  They just attempt to allocate the memory and fail.  It 
seams like a check should be made in one of these functions and if the 
resources are reserved then the addresses should be changed. (Note: 64 bit 
uses the i386 pci code that is why I am pointing out potential errors in the 
i386 code)

I also noticed the reserve kernel param can only take ints so large 
addresses can not be reserved

Thank you,
Tom

Re: 64bit PCI IORESOURCE_MEM bugs

[Andi Kleen]

"Thomas Crowley" <tbcrowley@hotmail.com> writes:
> 
> 1) in arch/x86_64/kernel/e820.c    the e820_reserve_resources function
> the line if (e820.map[i].addr + e820.map[i].size > 0x100000000ULL)
> makes it so any region that
>      starts below the 4Gig mark but ends above 4Gig mark is ignored.

That is already gone in 2.6.13rc1.

> 2) I am unable to get all of my PCI devices to move there addressing
> from the default values to the hole I have given it.  I am getting
> several errors of the form: "PCI: Cannot allocate resource region 8 of
> bridge 0000:00:07.0".  Some of my devices are moving to the proper
> location.
> Neither the code in drivers/pci/probe.c pci_read_bases which gets the
> default addresses for the resources or /arch/i386/pci/i386.c
> pcibios_allocate_bus_resources check to see if the address that are
> being used are reserved.  They just attempt to allocate the memory and
> fail.  It seams like a check should be made in one of these functions
> and if the resources are reserved then the addresses should be
> changed. (Note: 64 bit uses the i386 pci code that is why I am
> pointing out potential errors in the i386 code)

Yes, that code is quite bad and in need of an overhaul. There 
are other problems in there too. Also the default hole selection
algorithm is not very reliable and some BIOS tend to put 
hidden mappings in there. Most likely it needs more support from ACPI.

> I also noticed the reserve kernel param can only take ints so large
> addresses can not be reserved


What reserve kernel param? mem= certainly supports more than 4GB.

-Andi

Re: 64bit PCI IORESOURCE_MEM bugs

[Thomas Crowley]

Thanks for the info.  The reserve param I refer to is the "reserve" param 
from kernel-parameters.txt
"reserve=	[KNL,BUGS] Force the kernel to ignore some iomem area"

On my number 2.  I have been testing just giving 4 Gigs to the OS which ends 
up getting cut into by the iomem addressing and then just trying to use the 
remaining 12Gigs.  However I have been running into trouble when I address 
into the 7-8 Gig area.  Could that be one of the hidden maps you mentioned?

Thanks
Tom

>
>1) in arch/x86_64/kernel/e820.c    the e820_reserve_resources function
>the line if (e820.map[i].addr + e820.map[i].size > 0x100000000ULL)
>makes it so any region that
>      starts below the 4Gig mark but ends above 4Gig mark is ignored.

That is already gone in 2.6.13rc1.

>2) I am unable to get all of my PCI devices to move there addressing
>from the default values to the hole I have given it.  I am getting
>several errors of the form: "PCI: Cannot allocate resource region 8 of
>bridge 0000:00:07.0".  Some of my devices are moving to the proper
>location.
>Neither the code in drivers/pci/probe.c pci_read_bases which gets the
>default addresses for the resources or /arch/i386/pci/i386.c
>pcibios_allocate_bus_resources check to see if the address that are
>being used are reserved.  They just attempt to allocate the memory and
>fail.  It seams like a check should be made in one of these functions
>and if the resources are reserved then the addresses should be
>changed. (Note: 64 bit uses the i386 pci code that is why I am
>pointing out potential errors in the i386 code)

Yes, that code is quite bad and in need of an overhaul. There
are other problems in there too. Also the default hole selection
algorithm is not very reliable and some BIOS tend to put
hidden mappings in there. Most likely it needs more support from ACPI.

>I also noticed the reserve kernel param can only take ints so large
>addresses can not be reserved


What reserve kernel param? mem= certainly supports more than 4GB.

-Andi