Extended Addressing Mode

Extended Addressing Mode

[Régis Odeyé]

Hello Everyone,
I'm looking for some information about the Extended Addressing Mode 
(XAEN bit of HID0 register) of PPC32 support in Linux.
I do not see anything in the main kernel tree but there may be some 
patches available ?
Any information will be appreciate.
Regards.

-- 
Régis ODEYE

Kontron Modular Computers SA
150, rue M. Berthelot / ZI Toulon Est / BP 244 / Fr 83078 TOULON Cedex 9
Phone: (33) 4 98 16 34 86           Fax: (33) 4 98 16 34 01
E-mail: regis.odeye@kontron.com  Web : www.kontron.com

Re: Extended Addressing Mode

[Kumar Gala]

On Oct 22, 2008, at 3:59 AM, R=E9gis Odey=E9 wrote:

> Hello Everyone,
> I'm looking for some information about the Extended Addressing Mode =20=

> (XAEN bit of HID0 register) of PPC32 support in Linux.
> I do not see anything in the main kernel tree but there may be some =20=

> patches available ?
> Any information will be appreciate.
> Regards.

There are patches from Becky Bruce that are going into 2.6.28.  What =20
are you needing >32-bit for?  There are some SW IO MMU changes that =20
are still pending to complete this work.

- k=

Re: Extended Addressing Mode

[Régis Odeyé]

Kumar Gala wrote:
>
> On Oct 22, 2008, at 3:59 AM, Régis Odeyé wrote:
>
>> Hello Everyone,
>> I'm looking for some information about the Extended Addressing Mode 
>> (XAEN bit of HID0 register) of PPC32 support in Linux.
>> I do not see anything in the main kernel tree but there may be some 
>> patches available ?
>> Any information will be appreciate.
>> Regards.
>
> There are patches from Becky Bruce that are going into 2.6.28.  What 
> are you needing >32-bit for?  There are some SW IO MMU changes that 
> are still pending to complete this work.
>
> - k
We are developing a board based on Freescale 8641D which can get 4GB of 
ram. So I need 4GB+IOs (~1GB) of physical addressing space.
My plan is to put a part of this ram above of 4GB to keep accesses to 
the IOs below the 4GB limit. It means non-contiguous ram addressing and 
XAEN features to be working.
Where can I glance through Becky patches ?

-- 
Régis ODEYE

Kontron Modular Computers SA
150, rue M. Berthelot / ZI Toulon Est / BP 244 / Fr 83078 TOULON Cedex 9
Phone: (33) 4 98 16 34 86           Fax: (33) 4 98 16 34 01
E-mail: regis.odeye@kontron.com  Web : www.kontron.com

Re: Extended Addressing Mode

[Kumar Gala]

On Oct 22, 2008, at 7:59 AM, R=E9gis Odey=E9 wrote:

> Kumar Gala wrote:
>>
>> On Oct 22, 2008, at 3:59 AM, R=E9gis Odey=E9 wrote:
>>
>>> Hello Everyone,
>>> I'm looking for some information about the Extended Addressing =20
>>> Mode (XAEN bit of HID0 register) of PPC32 support in Linux.
>>> I do not see anything in the main kernel tree but there may be =20
>>> some patches available ?
>>> Any information will be appreciate.
>>> Regards.
>>
>> There are patches from Becky Bruce that are going into 2.6.28.  =20
>> What are you needing >32-bit for?  There are some SW IO MMU changes =20=

>> that are still pending to complete this work.
>>
>> - k
> We are developing a board based on Freescale 8641D which can get 4GB =20=

> of ram. So I need 4GB+IOs (~1GB) of physical addressing space.
> My plan is to put a part of this ram above of 4GB to keep accesses =20
> to the IOs below the 4GB limit. It means non-contiguous ram =20
> addressing and XAEN features to be working.

So we have XAEN support in the tree.. however non-contiguous is =20
something you'll have to work on yourself.  Patches are welcome for this

>
> Where can I glance through Becky patches ?

This is the bulk:

=
http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linux-2.6.git;a=3Dcom=
mitdiff;h=3D4ee7084eb11e00eb02dc8435fd18273a61ffa9bf

- k=

Re: Extended Addressing Mode

[Régis Odeyé]

Kumar Gala wrote:
> So we have XAEN support in the tree.. however non-contiguous is 
> something you'll have to work on yourself.  Patches are welcome for this
OK. I will let you know about this.
>> Where can I glance through Becky patches ?
>
> This is the bulk:
>
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=4ee7084eb11e00eb02dc8435fd18273a61ffa9bf 
>
>
Thank you very much, Sir.

-- 
Régis ODEYE

Kontron Modular Computers SA
150, rue M. Berthelot / ZI Toulon Est / BP 244 / Fr 83078 TOULON Cedex 9
Phone: (33) 4 98 16 34 86           Fax: (33) 4 98 16 34 01
E-mail: regis.odeye@kontron.com  Web : www.kontron.com

Re: Extended Addressing Mode

[Matt Sealey]

Kumar Gala wrote:
> 
> On Oct 22, 2008, at 7:59 AM, Régis Odeyé wrote:
> 
>> of ram. So I need 4GB+IOs (~1GB) of physical addressing space.
>> My plan is to put a part of this ram above of 4GB to keep accesses to 
>> the IOs below the 4GB limit. It means non-contiguous ram addressing 
>> and XAEN features to be working.
> 
> So we have XAEN support in the tree.. however non-contiguous is 
> something you'll have to work on yourself.  Patches are welcome for this

So to confirm, XAEN support through Becky's patches does 
support the MPC8641D/e600 cores?

>> Where can I glance through Becky patches ?
> 
> This is the bulk:
> 
> http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=4ee7084eb11e00eb02dc8435fd18273a61ffa9bf 

I'd also be interested in any work done to enable 
non-contiguous memory areas. Reading the docs for the MPC8641D 
though I am not sure you can set up LAWs for it?

One thing I wanted to try was installing 4GB in a system and 
"overlapping" IO (since there is very little of it on a stock 
MPC8641DHPCN) in the top ~256MB-512MB, but I am fairly sure 
this is NOT supported because of the way the LAWs work, and 
also the alignment of the LAWs means it is not fine enough 
granularity to map between 2GB and 4GB into a window (you can 
have 2GB or 4GB but not some more arbitrary value?)

-- 
Matt Sealey <matt@genesi-usa.com>
Genesi, Manager, Developer Relations

Re: Extended Addressing Mode

[Kumar Gala]

On Oct 22, 2008, at 8:42 AM, Matt Sealey wrote:

>
>
> Kumar Gala wrote:
>> On Oct 22, 2008, at 7:59 AM, R=E9gis Odey=E9 wrote:
>>> of ram. So I need 4GB+IOs (~1GB) of physical addressing space.
>>> My plan is to put a part of this ram above of 4GB to keep accesses =20=

>>> to the IOs below the 4GB limit. It means non-contiguous ram =20
>>> addressing and XAEN features to be working.
>> So we have XAEN support in the tree.. however non-contiguous is =20
>> something you'll have to work on yourself.  Patches are welcome for =20=

>> this
>
> So to confirm, XAEN support through Becky's patches does support the =20=

> MPC8641D/e600 cores?

Yes, its the only part that has XAEN.

>>> Where can I glance through Becky patches ?
>> This is the bulk:
>> =
http://git.kernel.org/?p=3Dlinux/kernel/git/torvalds/linux-2.6.git;a=3Dcom=
mitdiff;h=3D4ee7084eb11e00eb02dc8435fd18273a61ffa9bf
>
> I'd also be interested in any work done to enable non-contiguous =20
> memory areas. Reading the docs for the MPC8641D though I am not sure =20=

> you can set up LAWs for it?

You can if you can configure your DDR to support it.

> One thing I wanted to try was installing 4GB in a system and =20
> "overlapping" IO (since there is very little of it on a stock =20
> MPC8641DHPCN) in the top ~256MB-512MB, but I am fairly sure this is =20=

> NOT supported because of the way the LAWs work, and also the =20
> alignment of the LAWs means it is not fine enough granularity to map =20=

> between 2GB and 4GB into a window (you can have 2GB or 4GB but not =20
> some more arbitrary value?)

You can overlap LAWs because they have priority encoding.  The lower =20
the LAW # the higher the priority.

Your bigger issue is if you can setup the DDR controller for the hole =20=

you want.

- k=

Re: Extended Addressing Mode

[Régis Odeyé]

Matt Sealey wrote:
>
> I'd also be interested in any work done to enable non-contiguous 
> memory areas. Reading the docs for the MPC8641D though I am not sure 
> you can set up LAWs for it?
>
> One thing I wanted to try was installing 4GB in a system and 
> "overlapping" IO (since there is very little of it on a stock 
> MPC8641DHPCN) in the top ~256MB-512MB, but I am fairly sure this is 
> NOT supported because of the way the LAWs work, and also the alignment 
> of the LAWs means it is not fine enough granularity to map between 2GB 
> and 4GB into a window (you can have 2GB or 4GB but not some more 
> arbitrary value?)
>
On my point of view, LAWs sizes are power of 2 and should be aligned to 
the window size. But there is 10 LAWs you can combine to achieve quite a 
lot of different mappings.
Regards.

-- 
Régis ODEYE

Kontron Modular Computers SA
150, rue M. Berthelot / ZI Toulon Est / BP 244 / Fr 83078 TOULON Cedex 9
Phone: (33) 4 98 16 34 86           Fax: (33) 4 98 16 34 01
E-mail: regis.odeye@kontron.com  Web : www.kontron.com

Re: Extended Addressing Mode

[Matt Sealey]

Kumar Gala wrote:
> 
> On Oct 22, 2008, at 8:42 AM, Matt Sealey wrote:
>> So to confirm, XAEN support through Becky's patches does support the 
>> MPC8641D/e600 cores?
> 
> Yes, its the only part that has XAEN.

Okay I saw a lot of e500/BookE support go past but nothing 
specific :)

>> NOT supported because of the way the LAWs work, and also the alignment 
>> of the LAWs means it is not fine enough granularity to map between 2GB 
>> and 4GB into a window (you can have 2GB or 4GB but not some more 
>> arbitrary value?)
> 
> You can overlap LAWs because they have priority encoding.  The lower the 
> LAW # the higher the priority.

Hmm :)

> Your bigger issue is if you can setup the DDR controller for the hole 
> you want.

Hmmmmm :)

Okay good to know. I'll go back to reading the docs.

-- 
Matt Sealey <matt@genesi-usa.com>
Genesi, Manager, Developer Relations

Re: Extended Addressing Mode

[Matt Sealey]

Kumar Gala wrote:
> 
> 
> Your bigger issue is if you can setup the DDR controller for the hole 
> you want.

I just remembered;;

~~
The CCSR window always takes precedence over all local access 
windows. However, the CCSR window must not overlap an LAW that 
maps to the DDR controller. Otherwise, undefined behavior occurs.
~~

So, it's not really possible to map 4GB of RAM in the lower 
32-bit area, without interacting badly with the CCSR. This 
means you're forced to select a 2GB LAW for DDR, then leave 
2GB free, then map the rest above.. using more than 2Gb 
therefore absolutely requires non-contiguous memory..?

-- 
Matt Sealey <matt@genesi-usa.com>
Genesi, Manager, Developer Relations

Re: Extended Addressing Mode

[Kumar Gala]

On Oct 22, 2008, at 9:19 AM, Matt Sealey wrote:

>
>
> Kumar Gala wrote:
>> On Oct 22, 2008, at 8:42 AM, Matt Sealey wrote:
>>> So to confirm, XAEN support through Becky's patches does support  
>>> the MPC8641D/e600 cores?
>> Yes, its the only part that has XAEN.
>
> Okay I saw a lot of e500/BookE support go past but nothing specific :)

The "XAEN" is specific to e600.  e500 has its own ability to do 36-bit  
physical (not called XAEN).

- k

Re: Extended Addressing Mode

[Kumar Gala]

On Oct 22, 2008, at 9:22 AM, Matt Sealey wrote:

>
>
> Kumar Gala wrote:
>> Your bigger issue is if you can setup the DDR controller for the  
>> hole you want.
>
> I just remembered;;
>
> ~~
> The CCSR window always takes precedence over all local access  
> windows. However, the CCSR window must not overlap an LAW that maps  
> to the DDR controller. Otherwise, undefined behavior occurs.
> ~~
>
> So, it's not really possible to map 4GB of RAM in the lower 32-bit  
> area, without interacting badly with the CCSR. This means you're  
> forced to select a 2GB LAW for DDR, then leave 2GB free, then map  
> the rest above.. using more than 2Gb therefore absolutely requires  
> non-contiguous memory..?

As I said, its all about your physical DDR layout.  If you have two  
DDR dimms each 2Gb you can do:

0..2G  - DDR DIMM A
2G..4G - IO
4G..6G - DDR DIMM B

- k

Re: Extended Addressing Mode

[Matt Sealey]

Kumar Gala wrote:
> 
> On Oct 22, 2008, at 9:19 AM, Matt Sealey wrote:
> 
>>
>>
>> Kumar Gala wrote:
>>> On Oct 22, 2008, at 8:42 AM, Matt Sealey wrote:
>>>> So to confirm, XAEN support through Becky's patches does support the 
>>>> MPC8641D/e600 cores?
>>> Yes, its the only part that has XAEN.
>>
>> Okay I saw a lot of e500/BookE support go past but nothing specific :)
> 
> The "XAEN" is specific to e600.  e500 has its own ability to do 36-bit 
> physical (not called XAEN).

You have to forgive me I was only half awake when I started :D

Yeah so I saw BookE and e500 stuff go past but nothing 
specific for e600/XAEN. I mailed Becky and got no response. 
I'm glad to know it's in there though, somewhere.

Just so it has been asked, do you know in a broad sense what 
it would take to add the non-contiguous memory mapping 
support? Doesn't ppc64 already have this?

-- 
Matt Sealey <matt@genesi-usa.com>
Genesi, Manager, Developer Relations

Re: Extended Addressing Mode

[Matt Sealey]

Kumar Gala wrote:
> 
> On Oct 22, 2008, at 9:22 AM, Matt Sealey wrote:
> 
>> ~~
>> The CCSR window always takes precedence over all local access windows. 
>> However, the CCSR window must not overlap an LAW that maps to the DDR 
>> controller. Otherwise, undefined behavior occurs.
>> ~~
>>
>> So, it's not really possible to map 4GB of RAM in the lower 32-bit 
>> area, without interacting badly with the CCSR. This means you're 
>> forced to select a 2GB LAW for DDR, then leave 2GB free, then map the 
>> rest above.. using more than 2Gb therefore absolutely requires 
>> non-contiguous memory..?
> 
> As I said, its all about your physical DDR layout.  If you have two DDR 
> dimms each 2Gb you can do:
> 
> 0..2G  - DDR DIMM A
> 2G..4G - IO
> 4G..6G - DDR DIMM B

I assume on the HPCN "DDR DIMM A" would be one or both of one 
set of DDR slots, and DDR DIMM B would be one or both o the 
other set (since there are 4 slots, two for each controller)?

Or are we talking about actual, physical DIMMs?

If we're talking about controllers, could you not do;

0..2GB DDR Controller 1 (partial)
2G..4GB IO
4GB..NGB DDR Controller 1 (the rest)
NGB-64GB DDR Controller 2 (or whatever)

Or do LAWs not cooperate when for the same target? I would 
assume if you set up the CSn_BNDS registers right you could 
get a real fine grained mapping of DDR controller to memory 
space in combination with the LAWs? It would then be actually 
possible (however disgusting this config would be) to have a 
2GB DIMM, 1GB DIMM on the first controller with two LAWs, and 
the appropriate chip selects, then 1GB IO space, then up to 
32GB (since 16GB DIMMs are about as high as it goes for DDR2) 
memory space mapped after that, with a single LAW?

Or more comfortably, pair up 2x 2GB DIMMs and simply ignore 
the last 1GB, and pair up 2x 16GB DIMMs?

-- 
Matt Sealey <matt@genesi-usa.com>
Genesi, Manager, Developer Relations

Re: Extended Addressing Mode

[Becky Bruce]

On Oct 22, 2008, at 1:11 PM, Matt Sealey wrote:

>
>
> Kumar Gala wrote:
>> On Oct 22, 2008, at 9:19 AM, Matt Sealey wrote:
>>>
>>>
>>> Kumar Gala wrote:
>>>> On Oct 22, 2008, at 8:42 AM, Matt Sealey wrote:
>>>>> So to confirm, XAEN support through Becky's patches does support  
>>>>> the MPC8641D/e600 cores?
>>>> Yes, its the only part that has XAEN.
>>>
>>> Okay I saw a lot of e500/BookE support go past but nothing  
>>> specific :)
>> The "XAEN" is specific to e600.  e500 has its own ability to do 36- 
>> bit physical (not called XAEN).
>
> You have to forgive me I was only half awake when I started :D
>
> Yeah so I saw BookE and e500 stuff go past but nothing specific for  
> e600/XAEN. I mailed Becky and got no response. I'm glad to know it's  
> in there though, somewhere.

Huh?  I have one mail from you, on Sep 1, to which I responded on Sep  
2.  Was there another mail I missed, or did you not see my response?   
Freescale's mail server is unreliable, and since I'm not in the habit  
of ignoring emails, feel free to nag me anytime if you send me  
something and don't get a response.

>
>
> Just so it has been asked, do you know in a broad sense what it  
> would take to add the non-contiguous memory mapping support? Doesn't  
> ppc64 already have this?


PPC64 has SPARSEMEM support, IIRC. It's non-trivial to add for 32-bit,  
although I haven't scoped it out in any detail.

Cheers,
B

Re: Extended Addressing Mode

[Benjamin Herrenschmidt]

On Wed, 2008-10-22 at 08:08 -0500, Kumar Gala wrote:
> > We are developing a board based on Freescale 8641D which can get
> 4GB  
> > of ram. So I need 4GB+IOs (~1GB) of physical addressing space.
> > My plan is to put a part of this ram above of 4GB to keep accesses  
> > to the IOs below the 4GB limit. It means non-contiguous ram  
> > addressing and XAEN features to be working.

Why would you put the IOs below the 4G point ? It's actually easier
to have the IOs up above, like a lot of 4xx do.

Cheers,
Ben.

Re: Extended Addressing Mode

[Matt Sealey]

Becky Bruce wrote:
> 
> On Oct 22, 2008, at 1:11 PM, Matt Sealey wrote:
> 
>> Yeah so I saw BookE and e500 stuff go past but nothing specific for 
>> e600/XAEN. I mailed Becky and got no response. I'm glad to know it's 
>> in there though, somewhere.
> 
> Huh?  I have one mail from you, on Sep 1, to which I responded on Sep 
> 2.  Was there another mail I missed, or did you not see my response?  
> Freescale's mail server is unreliable, and since I'm not in the habit of 
> ignoring emails, feel free to nag me anytime if you send me something 
> and don't get a response.

Between Freescale's mail server and the trouble we're having 
with Google right now (along with thousands of others), I am 
NOT surprised it got lost somewhere along the way.

I wasn't bitching :)

>> Just so it has been asked, do you know in a broad sense what it would 
>> take to add the non-contiguous memory mapping support? Doesn't ppc64 
>> already have this?
> 
> PPC64 has SPARSEMEM support, IIRC. It's non-trivial to add for 32-bit, 
> although I haven't scoped it out in any detail.

Okay, non-trivial was pretty much what I was looking for, I 
was just trying to weigh up how much effort is required..

If you actually scope it out in any detail or someone has some 
10% time and decides this would be an awesome project, give me 
a nudge? My MPC8641D is itching to actually do something 
besides build packages. Actually having 8GB of memory would 
REALLY help run SUSE Build Service :]

-- 
Matt Sealey <matt@genesi-usa.com>
Genesi, Manager, Developer Relations

Re: Extended Addressing Mode

[Matt Sealey]

Benjamin Herrenschmidt wrote:
> On Wed, 2008-10-22 at 08:08 -0500, Kumar Gala wrote:
>>> We are developing a board based on Freescale 8641D which can get
>> 4GB  
>>> of ram. So I need 4GB+IOs (~1GB) of physical addressing space.
>>> My plan is to put a part of this ram above of 4GB to keep accesses  
>>> to the IOs below the 4GB limit. It means non-contiguous ram  
>>> addressing and XAEN features to be working.
> 
> Why would you put the IOs below the 4G point ? It's actually easier
> to have the IOs up above, like a lot of 4xx do.

Because we're Genesi! And we have a firmware solution that 
kind of has to keep 32-bit pointers in the unlikely event that 
someone actually uses the client interface (besides yaboot!).

Having I/O in the 36-bit range could cause all sorts of 
explosions, and we're running real-mode so trapping and faking 
I/O accesses is REALLy difficult :}

-- 
Matt Sealey <matt@genesi-usa.com>
Genesi, Manager, Developer Relations

Re: Extended Addressing Mode

[Benjamin Herrenschmidt]

On Wed, 2008-10-22 at 17:21 -0500, Matt Sealey wrote:
> 
> Because we're Genesi! And we have a firmware solution that 
> kind of has to keep 32-bit pointers in the unlikely event that 
> someone actually uses the client interface (besides yaboot!).
> 
> Having I/O in the 36-bit range could cause all sorts of 
> explosions, and we're running real-mode so trapping and faking 
> I/O accesses is REALLy difficult :}

how so ? we have plethora of 32-bit firmwares including OF
implementations that can perfectly address 36 bit physical address
space. Nothing to do with having 32-bit pointers.

Cheers,
Ben.