building an elf64 R10k kernel

building an elf64 R10k kernel

[Charles Lepple]

Hey all,
Does anyone out there know any details on building elf64 objects? I was
all happy about seeing Andrew's Indigo2 patches, and decided that I
_had_ to try and make it work on an Indigo2 Impact 10000... Suffice it
to say that it isn't straightforward.

I first tried to get the machine to boot an elf32 kernel. That failed
miserably, since it doesn't recognize 32-bit ELF objects. Comparison of
sashARCS and sash64 indicated that I needed to use elf64. My first
inclination was to just try the same load address as the Indys and r4400
Indigo2s use. I can't remember the exact error message, but the PROM
basically panicked while loading the first word at 0x88002000 (or
wherever). So I tried using the 64 bit start address from sash64 (high
hex nibble is 0xa). Needless to say, the elf32 linker options conflicted
with that one, and the linker (from binutils-2.8.1 or so) seg faulted.

When I finally recompiled with -mips3 and with a revised linker script,
I noticed something curious. Instead of just truncating the 64-bit
addresses, it rounded them off to 0xffffffff. Any thoughts on this? Am I
approaching this wrong?

And does anyone have a semi-offical memory map for the R10k Indigo2s?

-- 
Charles Lepple
System Administrator, Virginia Tech EE Workstation Labs
clepple@ee.vt.edu || http://www.foo.tho.org/charles/

Re: building an elf64 R10k kernel

[Dave Olson]

Charles Lepple wrote: 
|  Hey all,
|  Does anyone out there know any details on building elf64 objects? I was
|  all happy about seeing Andrew's Indigo2 patches, and decided that I
|  _had_ to try and make it work on an Indigo2 Impact 10000... Suffice it
|  to say that it isn't straightforward.

For r10k Indigo2 to work, you will need to hack the compiler,
for various reasons (the way the r10k works, plus the fact that
indigo2 doesn't have i/o cache coherency, interact in some "interesting"
ways.  I would suggest not attempting this port, unless you have a *lot*
of spare time.

Dave Olson, Silicon Graphics
http://reality.sgi.com/olson   olson@sgi.com

Re: building an elf64 R10k kernel

[Dave Olson]

Charles Lepple wrote: 
|  Hey all,
|  Does anyone out there know any details on building elf64 objects? I was
|  all happy about seeing Andrew's Indigo2 patches, and decided that I
|  _had_ to try and make it work on an Indigo2 Impact 10000... Suffice it
|  to say that it isn't straightforward.

For r10k Indigo2 to work, you will need to hack the compiler,
for various reasons (the way the r10k works, plus the fact that
indigo2 doesn't have i/o cache coherency, interact in some "interesting"
ways.  I would suggest not attempting this port, unless you have a *lot*
of spare time.

Dave Olson, Silicon Graphics
http://reality.sgi.com/olson   olson@sgi.com

Re: building an elf64 R10k kernel

[Ralf Baechle]

On Mon, May 03, 1999 at 08:54:28PM -0700, Dave Olson wrote:

> Charles Lepple wrote: 
> |  Hey all,
> |  Does anyone out there know any details on building elf64 objects? I was
> |  all happy about seeing Andrew's Indigo2 patches, and decided that I
> |  _had_ to try and make it work on an Indigo2 Impact 10000... Suffice it
> |  to say that it isn't straightforward.
> 
> For r10k Indigo2 to work, you will need to hack the compiler,
> for various reasons (the way the r10k works, plus the fact that
> indigo2 doesn't have i/o cache coherency, interact in some "interesting"
> ways.  I would suggest not attempting this port, unless you have a *lot*
> of spare time.

Let me point out that SGI has invented an almost genious workaround for a
R10000 bug that only hits systems without I/O cache coherency, that is the
Indigo2 and O2.

When I first read the description of the bug and it's workaround I thought
I'd be halucinating ...

  Ralf

Re: building an elf64 R10k kernel

[William J. Earl]

Ralf Baechle writes:
 > On Mon, May 03, 1999 at 08:54:28PM -0700, Dave Olson wrote:
 > 
 > > Charles Lepple wrote: 
 > > |  Hey all,
 > > |  Does anyone out there know any details on building elf64 objects? I was
 > > |  all happy about seeing Andrew's Indigo2 patches, and decided that I
 > > |  _had_ to try and make it work on an Indigo2 Impact 10000... Suffice it
 > > |  to say that it isn't straightforward.
 > > 
 > > For r10k Indigo2 to work, you will need to hack the compiler,
 > > for various reasons (the way the r10k works, plus the fact that
 > > indigo2 doesn't have i/o cache coherency, interact in some "interesting"
 > > ways.  I would suggest not attempting this port, unless you have a *lot*
 > > of spare time.
 > 
 > Let me point out that SGI has invented an almost genious workaround for a
 > R10000 bug that only hits systems without I/O cache coherency, that is the
 > Indigo2 and O2.
...

     The R10000 "bug" is, in a sense, a feature, in that it improves
performance, and is harmless on machines with cache-coherent I/O.
Specifically, on a speculative store miss (a cache miss due to a
speculatively executed store instruction), the R10000 fetches the line
dirty-exclusive and marks it modified, in anticipation of the store.
If, however, the speculatively executed store never graduates (is
never committed), the line is left dirty, even though it has not been
modified.  If the line happens to be part of a buffer into which data
is being DMAed, a subsequent victim writeback of the dirty cache line
might overwrite good data from the DMA with the obsolete data in the
cache line.  This means that, one way or the other, a system with
non-cache-coherent I/O and an R10000 must avoid allowing the
processor to perform a speculative store miss with respect to memory
into which a DMA is taking place.

     Note that the Indigo2 and O2 have somewhat different workarounds.
The Indigo2 deals with the kernel side using a special compilation mode,
and the O2 deals with the kernel side using a special hardware feature
plus a generalization of the solution for the user mode part of the problem.
Both deal with the user mode by invalidating TLB entries for pages into
which data is being transferred via DMA, so that the processor cannot
resolve the virtual address, and hence cannot speculatively fetch
a cache line at that address, while the DMA is in progress.  The kernel
side is harder, since the TLB is not used for K0SEG and XKPHYS address
spaces, which is where things get complicated.

     I can provide the details to someone who is really interested
in working on this, but, as Dave Olson indicated, you don't want to
start on this unless you have a LOT of spare time.