fast_iob() vs PHYS_OFFSET

fast_iob() vs PHYS_OFFSET

[Måns Rullgård]

I'm a little confused over the interaction between fast_iob() and
non-zero PHYS_OFFSET.

The __fast_iob() macro performs a load from CKSEG1.  AFAICT, CKSEG1 is
always (on 32-bit systems) defined to 0xa0000000.  In case of a non-zero
PHYS_OFFSET, this address might not map to anything in particular, and
almost certainly not RAM.  There is a special case for IP28, but this is
not the only system with an unusual address map.

What am I missing?

-- 
Måns Rullgård
mans@mansr.com

Re: fast_iob() vs PHYS_OFFSET

[Måns Rullgård]

peter fuerst <post@pfrst.de> writes:

> Hello,
>
> i don't know where fast_iob() is currently defined (it used to be in
> system.h in 2.6 kernels) and so don't know its current definition. But

It's in asm/barrier.h, and looks like this for non-IP28:

#define __fast_iob()				\
	__asm__ __volatile__(			\
		".set	push\n\t"		\
		".set	noreorder\n\t"		\
		"lw	$0,%0\n\t"		\
		"nop\n\t"			\
		".set	pop"			\
		: /* no output */		\
		: "m" (*(int *)CKSEG1)		\
		: "memory")

#  define fast_iob()				\
	do {					\
		__sync();			\
		__fast_iob();			\
	} while (0)
# endif

> if PHYS_OFFSET appears in the definition, this may be plain wrong.
> (and any conjunction of PHYS_OFFSET with CKSEG1 would be bizarre).
>
> PHYS_OFFSET is (at least on IP28) just the physical address, where the
> RAM starts, and has no righteous place in address-calculations, MIPS's
> "unmapped" kernel-addresses (CKSEG[12], XKPHYS) are independent from
> soldering the RAM.

CKSEG[01] map to physical addresses starting from 0 bypassing the MMU.
If there is no RAM at address 0, I wouldn't expect reading from the
first word of CKSEG1 to work very well.

> On IP28, with PHYS_OFFSET 0x20000000, RAM consequently begins at the
> kernel-address(es) 0xXY00000020000000.  (Btw, you won't reach the RAM
> with CKSEG[12] here, XKPHYS is needed)

IP28 has a special definition of __fast_iob() that reads from
CKSEG1ADDR(0x1fa00004).  I have no idea what lives at that address, but
presumably something that responds to reads in a RAM-like fashion.

My concern is over systems like AR7.  It defines PHYS_OFFSET to
0x14000000 and UNCAC_BASE, correspondingly, to 0xb4000000.  According to
the linux-mips wiki, there is some on-chip RAM at physical address 0,
which explains why the __fast_iob() macro works there.

I'm dealing with another chip with non-zero PHYS_OFFSET which AFAIK does
not have anything mapped at physical address 0 (I don't have data sheets).
The Evil Vendor Tree has ifdefs all over the place, most of them bizarre
and wrong, and I'm trying to clean things up a bit.

> The spread of PHYS_OFFSET began several years ago with someone
> introducing '#define PAGE_OFFSET (CAC_BASE + PHYS_OFFSET)' instead of
> the former '#define PAGE_OFFSET CAC_BASE'.
> Perhaps just to allow retaining 'free_area_init(zones_size)' instead of
> the need to replace this by 'free_area_init_node(0, NODE_DATA(0),
> zones_size, PHYS_OFFSET, NULL)', since the "new" PAGE_OFFSET brings in
> PHYS_OFFSET through the backdoor.
> But, since kernel-addresses are (canonically ;-) calculated by adding or
> subtracting PAGE_OFFSET, the new definition had/has to be compensated
> for by introducing PHYS_OFFSET into macros in numerous places (maybe
> by some trial-and-error actions ;-)

That was more or less the impression I got while chasing the values
through the various macro definitions.

> kind regards
>
> peter
>
> On Fri, 28 Nov 2014, Måns Rullgård wrote:
>
>> Date: Fri, 28 Nov 2014 23:50:16 +0000
>> From: Måns Rullgård <mans@mansr.com>
>> To: linux-mips@linux-mips.org
>> Subject: fast_iob() vs PHYS_OFFSET
>>
>> I'm a little confused over the interaction between fast_iob() and
>> non-zero PHYS_OFFSET.
>>
>> The __fast_iob() macro performs a load from CKSEG1.  AFAICT, CKSEG1 is
>> always (on 32-bit systems) defined to 0xa0000000.  In case of a non-zero
>> PHYS_OFFSET, this address might not map to anything in particular, and
>> almost certainly not RAM.  There is a special case for IP28, but this is
>> not the only system with an unusual address map.
>>
>> What am I missing?
>>
>> -- 
>> Måns Rullgård
>> mans@mansr.com
>>
>>

-- 
Måns Rullgård
mans@mansr.com

Re: fast_iob() vs PHYS_OFFSET

[peter fuerst]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4482 bytes --]



Hello Måns,

On Sun, 30 Nov 2014, Måns Rullgård wrote:

> Date: Sun, 30 Nov 2014 12:20:17 +0000
> From: Måns Rullgård <mans@mansr.com>
> To: peter fuerst <post@pfrst.de>
> Cc: linux-mips@linux-mips.org
> Subject: Re: fast_iob() vs PHYS_OFFSET
>

>> Date: Sun, 30 Nov 2014 04:49:30 +0000 (UTC)
>> To: Måns Rullgård <mans@mansr.com>
>>
>> Sorry for the mistake below: should be CKSEG[01], not CKSEG[12].

> peter fuerst <post@pfrst.de> writes:
>
>> Hello,
>>
>> i don't know where fast_iob() is currently defined (it used to be in
>> system.h in 2.6 kernels) and so don't know its current definition. But
>
> It's in asm/barrier.h, and looks like this for non-IP28:
>
> #define __fast_iob()				\
...
> 		: "m" (*(int *)CKSEG1)		\
...
> # endif

this looks as it used to look since long.

>
>> if PHYS_OFFSET appears in the definition, this may be plain wrong.
>> (and any conjunction of PHYS_OFFSET with CKSEG1 would be bizarre).
>>
>> PHYS_OFFSET is (at least on IP28) just the physical address, where the
>> RAM starts, and has no righteous place in address-calculations, MIPS's
>> "unmapped" kernel-addresses (CKSEG[12], XKPHYS) are independent from
>> soldering the RAM.
>
> CKSEG[01] map to physical addresses starting from 0 bypassing the MMU.
> If there is no RAM at address 0, I wouldn't expect reading from the
> first word of CKSEG1 to work very well.
>
>> On IP28, with PHYS_OFFSET 0x20000000, RAM consequently begins at the
>> kernel-address(es) 0xXY00000020000000.  (Btw, you won't reach the RAM
>> with CKSEG[12] here, XKPHYS is needed)
>
> IP28 has a special definition of __fast_iob() that reads from
> CKSEG1ADDR(0x1fa00004).  I have no idea what lives at that address, but

IOC2, HPC3, ...

> presumably something that responds to reads in a RAM-like fashion.
>
> My concern is over systems like AR7.  It defines PHYS_OFFSET to
> 0x14000000 and UNCAC_BASE, correspondingly, to 0xb4000000.  According to

I'm confused. Shouldn't *CAC_BASE point to the begin of an address-space?

> the linux-mips wiki, there is some on-chip RAM at physical address 0,
> which explains why the __fast_iob() macro works there.
>
> I'm dealing with another chip with non-zero PHYS_OFFSET which AFAIK does
> not have anything mapped at physical address 0 (I don't have data sheets).
> The Evil Vendor Tree has ifdefs all over the place, most of them bizarre
> and wrong, and I'm trying to clean things up a bit.

There are mailing list members, that worked a lot on AR7, perhaps they
can help you. At least provide you with data sheets.

Besides AR7, (more or less) precise definitions of *CAC_BASE, PAGE_OFFSET,
PHYS_OFFSET, ... perhaps wouldn't hurt either.

>
>> The spread of PHYS_OFFSET began several years ago with someone
>> introducing '#define PAGE_OFFSET (CAC_BASE + PHYS_OFFSET)' instead of
>> the former '#define PAGE_OFFSET CAC_BASE'.
...thus opening Pandora's can.
>> Perhaps just to allow retaining 'free_area_init(zones_size)' instead of
>> the need to replace this by 'free_area_init_node(0, NODE_DATA(0),
>> zones_size, PHYS_OFFSET, NULL)', since the "new" PAGE_OFFSET brings in
>> PHYS_OFFSET through the backdoor.
>> But, since kernel-addresses are (canonically ;-) calculated by adding or
>> subtracting PAGE_OFFSET, the new definition had/has to be compensated
>> for by introducing PHYS_OFFSET into macros in numerous places (maybe
>> by some trial-and-error actions ;-)
>
> That was more or less the impression I got while chasing the values
> through the various macro definitions.
>
>> kind regards
>>
>> peter
>>
>> On Fri, 28 Nov 2014, Måns Rullgård wrote:
>>
>>> Date: Fri, 28 Nov 2014 23:50:16 +0000
>>> From: Måns Rullgård <mans@mansr.com>
>>> To: linux-mips@linux-mips.org
>>> Subject: fast_iob() vs PHYS_OFFSET
>>>
>>> I'm a little confused over the interaction between fast_iob() and
>>> non-zero PHYS_OFFSET.
>>>
>>> The __fast_iob() macro performs a load from CKSEG1.  AFAICT, CKSEG1 is
>>> always (on 32-bit systems) defined to 0xa0000000.  In case of a non-zero
>>> PHYS_OFFSET, this address might not map to anything in particular, and
>>> almost certainly not RAM.  There is a special case for IP28, but this is
>>> not the only system with an unusual address map.
>>>
>>> What am I missing?
>>>
>>> --
>>> Måns Rullgård
>>> mans@mansr.com
>>>
>>>
>
> -- 
> Måns Rullgård
> mans@mansr.com
>
>

best wishes

peter

Re: fast_iob() vs PHYS_OFFSET

[Måns Rullgård]

peter fuerst <post@pfrst.de> writes:

> Hello Måns,
>
> On Sun, 30 Nov 2014, Måns Rullgård wrote:
>
>> Date: Sun, 30 Nov 2014 12:20:17 +0000
>> From: Måns Rullgård <mans@mansr.com>
>> To: peter fuerst <post@pfrst.de>
>> Cc: linux-mips@linux-mips.org
>> Subject: Re: fast_iob() vs PHYS_OFFSET
>>
>
>>> Date: Sun, 30 Nov 2014 04:49:30 +0000 (UTC)
>>> To: Måns Rullgård <mans@mansr.com>
>>>
>>> Sorry for the mistake below: should be CKSEG[01], not CKSEG[12].
>
>> peter fuerst <post@pfrst.de> writes:
>>
>>> Hello,
>>>
>>> i don't know where fast_iob() is currently defined (it used to be in
>>> system.h in 2.6 kernels) and so don't know its current definition. But
>>
>> It's in asm/barrier.h, and looks like this for non-IP28:
>>
>> #define __fast_iob()				\
> ...
>> 		: "m" (*(int *)CKSEG1)		\
> ...
>> # endif
>
> this looks as it used to look since long.
>
>>
>>> if PHYS_OFFSET appears in the definition, this may be plain wrong.
>>> (and any conjunction of PHYS_OFFSET with CKSEG1 would be bizarre).
>>>
>>> PHYS_OFFSET is (at least on IP28) just the physical address, where the
>>> RAM starts, and has no righteous place in address-calculations, MIPS's
>>> "unmapped" kernel-addresses (CKSEG[12], XKPHYS) are independent from
>>> soldering the RAM.
>>
>> CKSEG[01] map to physical addresses starting from 0 bypassing the MMU.
>> If there is no RAM at address 0, I wouldn't expect reading from the
>> first word of CKSEG1 to work very well.
>>
>>> On IP28, with PHYS_OFFSET 0x20000000, RAM consequently begins at the
>>> kernel-address(es) 0xXY00000020000000.  (Btw, you won't reach the RAM
>>> with CKSEG[12] here, XKPHYS is needed)
>>
>> IP28 has a special definition of __fast_iob() that reads from
>> CKSEG1ADDR(0x1fa00004).  I have no idea what lives at that address, but
>
> IOC2, HPC3, ...
>
>> presumably something that responds to reads in a RAM-like fashion.
>>
>> My concern is over systems like AR7.  It defines PHYS_OFFSET to
>> 0x14000000 and UNCAC_BASE, correspondingly, to 0xb4000000.  According to
>
> I'm confused. Shouldn't *CAC_BASE point to the begin of an address-space?

I can't find any description of what all these macros are supposed to
be.  This makes it a bit difficult to reason about correctness.

That said, if they were supposed to duplicate [CX]SEG[01], there
wouldn't be much need for them in the first place.

>> the linux-mips wiki, there is some on-chip RAM at physical address 0,
>> which explains why the __fast_iob() macro works there.
>>
>> I'm dealing with another chip with non-zero PHYS_OFFSET which AFAIK does
>> not have anything mapped at physical address 0 (I don't have data sheets).
>> The Evil Vendor Tree has ifdefs all over the place, most of them bizarre
>> and wrong, and I'm trying to clean things up a bit.
>
> There are mailing list members, that worked a lot on AR7, perhaps they
> can help you. At least provide you with data sheets.

I'm not working on AR7.  That was just an example currently in-tree.

> Besides AR7, (more or less) precise definitions of *CAC_BASE, PAGE_OFFSET,
> PHYS_OFFSET, ... perhaps wouldn't hurt either.

Definitely.

-- 
Måns Rullgård
mans@mansr.com

Re: fast_iob() vs PHYS_OFFSET

[Thomas Bogendoerfer]

On Sun, Nov 30, 2014 at 12:20:17PM +0000, Måns Rullgård wrote:
> My concern is over systems like AR7.  It defines PHYS_OFFSET to
> 0x14000000 and UNCAC_BASE, correspondingly, to 0xb4000000.  According to
> the linux-mips wiki, there is some on-chip RAM at physical address 0,
> which explains why the __fast_iob() macro works there.

there is really on-chip RAM for AR7 at address 0. The AR7 CPU core
is only MIPS32r1, so it doesn't have the exception vector base register
and needs ram at physical address 0 for exception handlers (which all
older cores do). I can't check right now, but even IP28 should have
some memory mirrored there. The problem on IP28 is that accessing
memory uncached requires reprogramming the memory controller (which
then doesn't fit the concept of fast_iob()).

Thomas.

-- 
Crap can work. Given enough thrust pigs will fly, but it's not necessarily a
good idea.                                                [ RFC1925, 2.3 ]

Re: fast_iob() vs PHYS_OFFSET

[Måns Rullgård]

Thomas Bogendoerfer <tsbogend@alpha.franken.de> writes:

> On Sun, Nov 30, 2014 at 12:20:17PM +0000, Måns Rullgård wrote:
>> My concern is over systems like AR7.  It defines PHYS_OFFSET to
>> 0x14000000 and UNCAC_BASE, correspondingly, to 0xb4000000.  According to
>> the linux-mips wiki, there is some on-chip RAM at physical address 0,
>> which explains why the __fast_iob() macro works there.
>
> there is really on-chip RAM for AR7 at address 0. The AR7 CPU core
> is only MIPS32r1, so it doesn't have the exception vector base register
> and needs ram at physical address 0 for exception handlers (which all
> older cores do). I can't check right now, but even IP28 should have
> some memory mirrored there. The problem on IP28 is that accessing
> memory uncached requires reprogramming the memory controller (which
> then doesn't fit the concept of fast_iob()).

Well, that explains why it works on AR7.  It still doesn't tell me the
correct way to handle a 74k-based chip (MIPS32r2) with RAM starting at
physical address 0x04000000.

-- 
Måns Rullgård
mans@mansr.com

Re: fast_iob() vs PHYS_OFFSET

[Maciej W. Rozycki]

On Mon, 1 Dec 2014, Måns Rullgård wrote:

> > there is really on-chip RAM for AR7 at address 0. The AR7 CPU core
> > is only MIPS32r1, so it doesn't have the exception vector base register
> > and needs ram at physical address 0 for exception handlers (which all
> > older cores do). I can't check right now, but even IP28 should have
> > some memory mirrored there. The problem on IP28 is that accessing
> > memory uncached requires reprogramming the memory controller (which
> > then doesn't fit the concept of fast_iob()).
> 
> Well, that explains why it works on AR7.  It still doesn't tell me the
> correct way to handle a 74k-based chip (MIPS32r2) with RAM starting at
> physical address 0x04000000.

 I defined the thing long ago (circa 2000), before such configurations 
were of concern, under the assumption there must be readable storage that 
behaves like memory (i.e. safe to read, no side effects) at the base of 
KSEG0/KSEG1 because this is where the exception handlers reside.  The 
definition should have been updated by whoever implemented support for 
newer processors for which the assumption does not stand anymore.

 For MIPSr2 and newer architectures you may define a variant that reads 
from CP0.EBase, masks out the unwanted fields and ORs in the KSEG1 bits.  
The natural way to implement this would be in C code, probably directly in 
the input operand's expression, replacing CKSEG1 hardcoded there now, so 
that the compiler can optimise the address calculation sequence (e.g. it 
can keep it around in a register rather than recalculating it every time 
if the piece is invoked repeatedly).

 FWIW I plan to rewrite our I/O barriers in an effort to make them 
portable across all Linux targets so that cross-platform drivers can use 
them.  I think I outlined the plan sometime ago, on LKML or maybe netdev.  
Such low-level handlers like this will obviously stay, but names can 
change and definitions may be shuffled around.  Right now we have quite a 
mess across various platforms whose access ordering strength varies for 
I/O (all the world is not PCI!), it's only memory access ordering strength 
handling that we've got nice and clean once DEC Alpha entered the game.

  Maciej