CONFIG_PS3_USE_LPAR_ADDR dramically increases boot time

CONFIG_PS3_USE_LPAR_ADDR dramically increases boot time

[Jimi Xenidis]

The following change set:
   http://git.kernel.org/?p=linux/kernel/git/paulus/ 
powerpc.git;a=commit;h=261efc3f178c8c5b55d76208aee1f39ce247f723

setting CONFIG_PS3_USE_LPAR_ADDR (which is the default for PS3)  
changes increases MAX_PHYSADDR_BITS  from 44 to 47 which makes the  
array 8 time bigger and takes way longer for sparse_init() to run on  
simulator.

Is this really the right approach?
Cell is actually 42 so could not the special address space be 43 or 44?

-JX

Re: CONFIG_PS3_USE_LPAR_ADDR dramically increases boot time

[Arnd Bergmann]

On Monday 18 June 2007, Jimi Xenidis wrote:
>=20
> The following change set:
> =A0 =A0http://git.kernel.org/?p=3Dlinux/kernel/git/paulus/=20
> powerpc.git;a=3Dcommit;h=3D261efc3f178c8c5b55d76208aee1f39ce247f723
>=20
> setting CONFIG_PS3_USE_LPAR_ADDR (which is the default for PS3) =A0
> changes increases MAX_PHYSADDR_BITS =A0from 44 to 47 which makes the =A0
> array 8 time bigger and takes way longer for sparse_init() to run on =A0
> simulator.
>=20
> Is this really the right approach?
> Cell is actually 42 so could not the special address space be 43 or 44?

Unfortunately, the hypervisor chooses the addresses where things get
mapped, and since they are guest-real, there is no hard limit where
the hypervisor puts them, like the 42 bit limit on bare metal.

I guess this will improve if we get virt_mem_map support on powerpc,
but I'm not sure if that's currently being worked on.

	Arnd <><

Re: CONFIG_PS3_USE_LPAR_ADDR dramically increases boot time

[Jimi Xenidis]

On Jun 18, 2007, at 5:14 PM, Arnd Bergmann wrote:

> On Monday 18 June 2007, Jimi Xenidis wrote:
>>
>> The following change set:
>>    http://git.kernel.org/?p=linux/kernel/git/paulus/
>> powerpc.git;a=commit;h=261efc3f178c8c5b55d76208aee1f39ce247f723
>>
>> setting CONFIG_PS3_USE_LPAR_ADDR (which is the default for PS3)
>> changes increases MAX_PHYSADDR_BITS  from 44 to 47 which makes the
>> array 8 time bigger and takes way longer for sparse_init() to run on
>> simulator.
>>
>> Is this really the right approach?
>> Cell is actually 42 so could not the special address space be 43  
>> or 44?
>
> Unfortunately, the hypervisor chooses the addresses where things get
> mapped, and since they are guest-real, there is no hard limit where
> the hypervisor puts them, like the 42 bit limit on bare metal.
>
> I guess this will improve if we get virt_mem_map support on powerpc,
> but I'm not sure if that's currently being worked on.

I feel a little nit-picky worrying about slowing down boot, I'm sure  
it negligible on HW, and complaining about 2K vs 16K memsection. But  
it is important to point out that this effects all PPC64 targets if a  
distro decides to include the PS3 config in their PPC64 image.

-JX

Re: CONFIG_PS3_USE_LPAR_ADDR dramically increases boot time

[Geoff Levand]

Hi Jimi.

Jimi Xenidis wrote:
> On Jun 18, 2007, at 5:14 PM, Arnd Bergmann wrote:
> 
>> On Monday 18 June 2007, Jimi Xenidis wrote:
>>>
>>> The following change set:
>>>    http://git.kernel.org/?p=linux/kernel/git/paulus/
>>> powerpc.git;a=commit;h=261efc3f178c8c5b55d76208aee1f39ce247f723
>>>
>>> setting CONFIG_PS3_USE_LPAR_ADDR (which is the default for PS3)
>>> changes increases MAX_PHYSADDR_BITS  from 44 to 47 which makes the
>>> array 8 time bigger and takes way longer for sparse_init() to run on
>>> simulator.
>>>
>>> Is this really the right approach?
>>> Cell is actually 42 so could not the special address space be 43  
>>> or 44?
>>
>> Unfortunately, the hypervisor chooses the addresses where things get
>> mapped, and since they are guest-real, there is no hard limit where
>> the hypervisor puts them, like the 42 bit limit on bare metal.
>>
>> I guess this will improve if we get virt_mem_map support on powerpc,
>> but I'm not sure if that's currently being worked on.
> 
> I feel a little nit-picky worrying about slowing down boot, I'm sure  
> it negligible on HW, and complaining about 2K vs 16K memsection. But  
> it is important to point out that this effects all PPC64 targets if a  
> distro decides to include the PS3 config in their PPC64 image.


When CONFIG_PS3_USE_LPAR_ADDR=n a translation in the platform code
sets up the page table to map the region to low addresses (continuous
with the boot mem region), but that translation is not compatible
with the mapping scheme used by the spu regions, which use sparse mem.

Arnd mention that he planed to re-do the spu region management, and
I think with that we can use CONFIG_PS3_USE_LPAR_ADDR=n and the spufs
at the same time.  Arnd, what is the status?

-Geoff

Re: CONFIG_PS3_USE_LPAR_ADDR dramically increases boot time

[Arnd Bergmann]

On Tuesday 19 June 2007, Geoff Levand wrote:
> Arnd mention that he planed to re-do the spu region management, and
> I think with that we can use CONFIG_PS3_USE_LPAR_ADDR=3Dn and the spufs
> at the same time. =C2=A0Arnd, what is the status?

BenH already changed the spufs code to use the nopfn method instead of
nopage, so we no longer depend on sparsemem. Does that take care
of the problem?

	Arnd <><

Re: CONFIG_PS3_USE_LPAR_ADDR dramically increases boot time

[Geoff Levand]

Arnd Bergmann wrote:
> On Tuesday 19 June 2007, Geoff Levand wrote:
>> Arnd mention that he planed to re-do the spu region management, and
>> I think with that we can use CONFIG_PS3_USE_LPAR_ADDR=n and the spufs
>> at the same time.  Arnd, what is the status?
> 
> BenH already changed the spufs code to use the nopfn method instead of
> nopage, so we no longer depend on sparsemem. Does that take care
> of the problem?

I think that there is a good chance it will, but I'll need to look at
it, as it was some time ago I did that work.

-Geoff

Re: CONFIG_PS3_USE_LPAR_ADDR dramically increases boot time

[Jimi Xenidis]

On Jun 18, 2007, at 7:12 PM, Geoff Levand wrote:

> Hi Jimi.
>
> Jimi Xenidis wrote:
>> On Jun 18, 2007, at 5:14 PM, Arnd Bergmann wrote:
>>
>>> On Monday 18 June 2007, Jimi Xenidis wrote:
>>>>
>>>> The following change set:
>>>>    http://git.kernel.org/?p=linux/kernel/git/paulus/
>>>> powerpc.git;a=commit;h=261efc3f178c8c5b55d76208aee1f39ce247f723
>>>>
>>>> setting CONFIG_PS3_USE_LPAR_ADDR (which is the default for PS3)
>>>> changes increases MAX_PHYSADDR_BITS  from 44 to 47 which makes the
>>>> array 8 time bigger and takes way longer for sparse_init() to  
>>>> run on
>>>> simulator.
>>>>
>>>> Is this really the right approach?
>>>> Cell is actually 42 so could not the special address space be 43
>>>> or 44?
>>>
>>> Unfortunately, the hypervisor chooses the addresses where things get
>>> mapped, and since they are guest-real, there is no hard limit where
>>> the hypervisor puts them, like the 42 bit limit on bare metal.
>>>
>>> I guess this will improve if we get virt_mem_map support on powerpc,
>>> but I'm not sure if that's currently being worked on.
>>
>> I feel a little nit-picky worrying about slowing down boot, I'm sure
>> it negligible on HW, and complaining about 2K vs 16K memsection. But
>> it is important to point out that this effects all PPC64 targets if a
>> distro decides to include the PS3 config in their PPC64 image.
>
>
> When CONFIG_PS3_USE_LPAR_ADDR=n a translation in the platform code
> sets up the page table to map the region to low addresses (continuous
> with the boot mem region), but that translation is not compatible
> with the mapping scheme used by the spu regions, which use sparse mem.

hmm.. sparse mem is for the linear map.
Why is are SPE areas  in the linear map?

>
> Arnd mention that he planed to re-do the spu region management, and
> I think with that we can use CONFIG_PS3_USE_LPAR_ADDR=n and the spufs
> at the same time.  Arnd, what is the status?
>
> -Geoff
>

Re: CONFIG_PS3_USE_LPAR_ADDR dramically increases boot time

[Benjamin Herrenschmidt]

On Mon, 2007-06-18 at 20:13 -0400, Jimi Xenidis wrote:
> > When CONFIG_PS3_USE_LPAR_ADDR=n a translation in the platform code
> > sets up the page table to map the region to low addresses
> (continuous
> > with the boot mem region), but that translation is not compatible
> > with the mapping scheme used by the spu regions, which use sparse
> mem.
> 
> hmm.. sparse mem is for the linear map. 

SPE local store used to have special struct pages for it in order to
return things from nopage(). It was dodgy and racy. I killed that
recently, it was causing more problem than it was solving. So it should
be fine now. I'm now using nopfn() and inserting the PTEs directly from
the nopfn() hook.

So Geoff, you can remove the option for now or make it n by default.

Ben.

Re: CONFIG_PS3_USE_LPAR_ADDR dramically increases boot time

[Geoff Levand]

Benjamin Herrenschmidt wrote:
> On Mon, 2007-06-18 at 20:13 -0400, Jimi Xenidis wrote:
>> > When CONFIG_PS3_USE_LPAR_ADDR=n a translation in the platform code
>> > sets up the page table to map the region to low addresses
>> (continuous
>> > with the boot mem region), but that translation is not compatible
>> > with the mapping scheme used by the spu regions, which use sparse
>> mem.
>> 
>> hmm.. sparse mem is for the linear map. 
> 
> SPE local store used to have special struct pages for it in order to
> return things from nopage(). It was dodgy and racy. I killed that
> recently, it was causing more problem than it was solving. So it should
> be fine now. I'm now using nopfn() and inserting the PTEs directly from
> the nopfn() hook.
> 
> So Geoff, you can remove the option for now or make it n by default.

OK, I'll look into it after I return from OLS.

-Geoff