Maximum ioremap size for ppc arch?

Maximum ioremap size for ppc arch?

[michael.firth]

I'm trying to get am MPC834x system running that has 256MBytes of NOR
flash connected.

The physmap flash driver is failing to ioremap() that amount of space,
while on a similar system with 128Mbytes of flash, there are no
problems.

Is this a known limitation of ioremap() on the ppc architecture, or
specifically the MPC834x family, and is there any (hopefully easy) way
to increase this limit?

Thanks

Michael

Re: Maximum ioremap size for ppc arch?

[Arnd Bergmann]

On Monday 03 December 2007, michael.firth@bt.com wrote:
> I'm trying to get am MPC834x system running that has 256MBytes of NOR
> flash connected.
> 
> The physmap flash driver is failing to ioremap() that amount of space,
> while on a similar system with 128Mbytes of flash, there are no
> problems.
> 
> Is this a known limitation of ioremap() on the ppc architecture, or
> specifically the MPC834x family, and is there any (hopefully easy) way
> to increase this limit?

The amount of ioremap space is very limited on 32bit systems in general,
since most people are more interested in having a large physical address
space without highmem. Look at the definition of VMALLOC_START and
VMALLOC_END if you want to change the settings on your system.

	Arnd <><

Re: Maximum ioremap size for ppc arch?

[Matt Porter]

On Mon, Dec 03, 2007 at 09:22:06AM -0000, michael.firth@bt.com wrote:
> I'm trying to get am MPC834x system running that has 256MBytes of NOR
> flash connected.
> 
> The physmap flash driver is failing to ioremap() that amount of space,
> while on a similar system with 128Mbytes of flash, there are no
> problems.
> 
> Is this a known limitation of ioremap() on the ppc architecture, or
> specifically the MPC834x family, and is there any (hopefully easy) way
> to increase this limit?

The answer is "it depends". It depends on the amount of system memory
you have. By default, your system memory is mapped at 0xc0000000, leaving
not enough space for vmalloc allocations to grab 256MB for the
ioremap (and avoid the fixed virtual mapping in the high virtual
address area).

See the "Advanced setup" menu. Normally, you can set "Set custom kernel
base address" to 0xa0000000 safely. That will give you an additional
256MB of vmalloc space. On arch/powerpc, you'll also have to set
"Size of user task space" to 0x80000000 or 0xa0000000.

-Matt

Re: Maximum ioremap size for ppc arch?

[David Hawkins]

Hi Matt,

> On Mon, Dec 03, 2007 at 09:22:06AM -0000, michael.firth@bt.com wrote:
>> I'm trying to get am MPC834x system running that has 256MBytes of NOR
>> flash connected.
>>
>> The physmap flash driver is failing to ioremap() that amount of space,
>> while on a similar system with 128Mbytes of flash, there are no
>> problems.
>>
>> Is this a known limitation of ioremap() on the ppc architecture, or
>> specifically the MPC834x family, and is there any (hopefully easy) way
>> to increase this limit?
> 
> The answer is "it depends". It depends on the amount of system memory
> you have. By default, your system memory is mapped at 0xc0000000, leaving
> not enough space for vmalloc allocations to grab 256MB for the
> ioremap (and avoid the fixed virtual mapping in the high virtual
> address area).
> 
> See the "Advanced setup" menu. Normally, you can set "Set custom kernel
> base address" to 0xa0000000 safely. That will give you an additional
> 256MB of vmalloc space. On arch/powerpc, you'll also have to set
> "Size of user task space" to 0x80000000 or 0xa0000000.

Could you comment on a similar problem I had/have.

I have a CPU with 1GB memory, and I use about 20 cPCI boards that
I give 8MB windows in PCI space. When I was trying to load my
custom driver with these boards, it would give me ioremap failures.
On a CPU that had 512MB of memory it worked fine. My 'temporary hack'
(which is still in place) for the 1GB CPUs was to add mem=512M (or 
whatever it is) to the kernel command line. That was a good
enough fix at the time :)

I have figured I was running out of page table entries or something
like that and was going to investigate one of these days ...

However, perhaps it was that I was running out of address space.
But 0xC0000000 is at 3GB, I can't see that I would be triggering
an address space issue:

1GB = 0x40000000
20 x 8MB = 160MB

But, I figured I'd ask anyway :)

Thanks,
Dave

PS. The CPUs in this case are x86 based, while the PCI boards use
PLX-9054 bridges. I'm building new peripheral boards with MPC8349EAs
so this problem is going to rear its ugly head again soon, when
I work on the drivers for the new peripheral boards.

Re: Maximum ioremap size for ppc arch?

[Matt Porter]

On Mon, Dec 03, 2007 at 09:50:21AM -0800, David Hawkins wrote:
> Could you comment on a similar problem I had/have.
> 
> I have a CPU with 1GB memory, and I use about 20 cPCI boards that
> I give 8MB windows in PCI space. When I was trying to load my
> custom driver with these boards, it would give me ioremap failures.
> On a CPU that had 512MB of memory it worked fine. My 'temporary hack'
> (which is still in place) for the 1GB CPUs was to add mem=512M (or 
> whatever it is) to the kernel command line. That was a good
> enough fix at the time :)
> 
> I have figured I was running out of page table entries or something
> like that and was going to investigate one of these days ...
> 
> However, perhaps it was that I was running out of address space.
> But 0xC0000000 is at 3GB, I can't see that I would be triggering
> an address space issue:
> 
> 1GB = 0x40000000
> 20 x 8MB = 160MB
> 
> But, I figured I'd ask anyway :)

Yes, same thing. There's N ways to fix it. But I see you're talking
x86.

> PS. The CPUs in this case are x86 based, while the PCI boards use
> PLX-9054 bridges. I'm building new peripheral boards with MPC8349EAs
> so this problem is going to rear its ugly head again soon, when
> I work on the drivers for the new peripheral boards.

You should be able to do something similar on x86 but the details
are TBD. I would probably try to limit low memory to 512MB in the
x86 case.

-Matt

Re: Maximum ioremap size for ppc arch?

[David Hawkins]

Hi Matt,

> Yes, same thing. There's N ways to fix it. But I see you're talking
> x86.
> 
>> PS. The CPUs in this case are x86 based, while the PCI boards use
>> PLX-9054 bridges. I'm building new peripheral boards with MPC8349EAs
>> so this problem is going to rear its ugly head again soon, when
>> I work on the drivers for the new peripheral boards.
> 
> You should be able to do something similar on x86 but the details
> are TBD. I would probably try to limit low memory to 512MB in the
> x86 case.

Yeah, that works for me. The CPUs were ordered with 512MB,
but arrived with 1GB. It was thought a windfall at the time ...
but not-so after the driver weirdness :)

Thanks for the info. Now at least if I want to understand the
problem, I know where to look.

Cheers,
Dave

RE: Maximum ioremap size for ppc arch?

[michael.firth]

> -----Original Message-----
> From: Matt Porter [mailto:mporter@kernel.crashing.org]=20
> Sent: 03 December 2007 15:30
> To: Firth,MJC,Michael,DMM R
> Cc: linuxppc-embedded@ozlabs.org
> Subject: Re: Maximum ioremap size for ppc arch?
>=20
> On Mon, Dec 03, 2007 at 09:22:06AM -0000, michael.firth@bt.com wrote:
> > I'm trying to get am MPC834x system running that has=20
> 256MBytes of NOR=20
> > flash connected.
> >=20
> > The physmap flash driver is failing to ioremap() that=20
> amount of space,=20
> > while on a similar system with 128Mbytes of flash, there are no=20
> > problems.
> >=20
> > Is this a known limitation of ioremap() on the ppc architecture, or=20
> > specifically the MPC834x family, and is there any=20
> (hopefully easy) way=20
> > to increase this limit?
>=20
> The answer is "it depends". It depends on the amount of=20
> system memory you have. By default, your system memory is=20
> mapped at 0xc0000000, leaving not enough space for vmalloc=20
> allocations to grab 256MB for the ioremap (and avoid the=20
> fixed virtual mapping in the high virtual address area).
>=20
> See the "Advanced setup" menu. Normally, you can set "Set=20
> custom kernel base address" to 0xa0000000 safely. That will=20
> give you an additional 256MB of vmalloc space. On=20
> arch/powerpc, you'll also have to set "Size of user task=20
> space" to 0x80000000 or 0xa0000000.
>=20
> -Matt
>=20

I've solved my problem for now, but I'm not sure I'm that happy that
it's a scalable solution.

I tried moving the kernel base address to 0x80000000, but the system
became very unstable - in particular, though it detected the flash
partitions, as soon as I tried to write to them the system spontaneously
rebooted - not even a kernel panic, just a straight reboot.

As I'm using arch/ppc, it seems that the default user task space is
0x80000000 which shouldn't have conflicted with this.

It seemed that the bottom of the vmalloc space is defined as 'start of
kernel + amount of physical RAM', which in the case of our board becomes
'0xc0000000 + 0x10000000', as there is 256MB of RAM present. The top of
vmalloc space was being limited by the CPU registers, mapped by IMMRBAR.
This was configured to 0xe0000000, which left only 256MB of vmalloc
space.

I've got things working by moving IMMRBAR up to 0xeff00000, which gives
nearly 256MB more vmalloc space.

My main queries are:
1) Why did changing the kernel base address to 0x80000000 make the
system unstable? Would 0xa0000000 as suggested not have caused this
problem?
2) Currently IMMRBAR has the same physical and virtual address. Does
this need to be the case? If this is a restriction, it seems to mean
that the top 256MB of the virtual address space becomes unusable.
3) Why the kernel is designed to run at 0xc0000000? This seems to leave
only 1GB of addressing space for all the physically addressable memory
(RAM + ioremapped + registers), while reserving 3GB of space for user
processes. The 3GB is presumably mostly unusable on a system without a
large amount of swap, as the 1GB limit on memory will prevent much more
than that being available for user space.

Thanks for the assistance so far, the pointer to the definition of
VMALLOC_START and VMALLOC_END gave me the hook in I needed to work out
where the limitation was coming from. I would suggest that its also
worth changing at the error message that's generated when the vmalloc
space is exhausted:

"allocation failed: out of vmalloc space - use vmalloc=3D<size> to
increase size."

>From my understanding of the code so far, it seems that the 'vmalloc'
command line option is only available on the x86 architecture.

What is the process for getting a concensus on this, and getting the
message changed?

Thanks again

Michael

Re: Maximum ioremap size for ppc arch?

[Scott Wood]

michael.firth@bt.com wrote:
> My main queries are: 1) Why did changing the kernel base address to
> 0x80000000 make the system unstable?

Because there's a bug somewhere. :-)

> 2) Currently IMMRBAR has the same physical and virtual address. Does 
> this need to be the case?

No, and it is not done that way in arch/powerpc.

> 3) Why the kernel is designed to run at 0xc0000000?

My guess is because it's a number that Linus pulled out of thin air back
when a gig of RAM was unimaginably large. :-P

> This seems to leave only 1GB of addressing space for all the
> physically addressable memory (RAM + ioremapped + registers), while
> reserving 3GB of space for user processes. The 3GB is presumably
> mostly unusable on a system without a large amount of swap, as the
> 1GB limit on memory will prevent much more than that being available
> for user space.

Well, it's also useful for sparse mappings, but I agree that the 3/1 
split is probably suboptimal for most workloads.

-Scott

Re: Maximum ioremap size for ppc arch?

[Scott Wood]

Scott Wood wrote:
> michael.firth@bt.com wrote:
>> This seems to leave only 1GB of addressing space for all the
>> physically addressable memory (RAM + ioremapped + registers), while
>> reserving 3GB of space for user processes. The 3GB is presumably
>> mostly unusable on a system without a large amount of swap, as the
>> 1GB limit on memory will prevent much more than that being available
>> for user space.
> 
> Well, it's also useful for sparse mappings, but I agree that the 3/1 
> split is probably suboptimal for most workloads.

Oh, and the 3G user is also useful for accessing highmem, of course.

-Scott