FW: Linux on 857T?

FW: Linux on 857T?

[Kerl, John]

Sorry, forgot to cc the mailing list.

-----Original Message-----
From: Kerl, John
Sent: Tuesday, January 22, 2002 12:44 PM
To: 'Dan Malek'
Subject: RE: Linux on 857T?


Dan:

Thanks for the reply.

I wasn't very specific in my original message, for two
reasons:

1.	Due to some tool problems I do not have as clear a view
	of what's really going on as I would like;

2.	There are actually several different symptoms, all
	puzzling.

OK, if you want details, here they are.  Again, please be
aware that this information is sketchy (that's the reason I
was reluctant to say too much in public.)

---------------------------------------------------------

Disclaimers:

Until yesterday I did not have an ICE.  I had been told
that our visionICE, which works on our 860 FADS board,
wouldn't work on our 857T board since the visionICE isn't
new enough to know about new parts such at the 857T.  Also,
we have a Macraigor probe, but it too wouldn't work.  So,
most of what I know -- as of today -- comes from using
printk's and the logic analyzer.  And this is of course
incomplete information; if the I-cache is on, you won't see
all instruction fetches in the logic analyzer, etc.  Also,
you only get to see bus activity; using the logic analyzer,
you can't view registers etc.

Yesterday, a co-worker (they are all hardware people)
informed me that, oops, we needed to remove an rpack & add
a jumper and then, voila, we could use the debug port after
all.  So I've been using the Macraigor for about a day now,
which isn't long enough to say very much.  (The visionICE
still is not able to control the 857T; a shame since the
visionICE is a much nicer probe than the Macraigor.)

Having made those disclaimers:

I've been told that 857 silicon errata parallel 860 silicon
errata.  So, I was not certain whether or not to include
CPU6 support in my .config.  So I tried it both ways.

With or without CPU6 support: My debug monitor (with which
we've tested the bejeebers out of the SRAM & SDRAM [with
and without cache enabled], SMC1 and FEC before ever
attempting an OS port) runs fine.  Then I TFTP up
zvmlinux.initrd.  (I modified mbxboot, customizing it for
our purposes, e.g. I want the bd as well as the command
line to be passed in from the debug monitor.)  This
"secondary boot loader", as I call it (the modified
mbxboot) also runs fine, prints out status to the screen,
prints out copies of the bd struct and command line which
match what the debug monitor passed in, decompresses the
kernel and jumps to it.  No apparent problems with the
debug monitor or the secondary boot loader.

The kernel runs for a while and then seems to "stop" -- and
when I say that, what I mean is I stop seeing print
statements in the terminal window.

How far does it get?  The kernel might not print anything
to the console at all.  This is the earliest "hang" -- e.g.
the last thing I see on my terminal window is the secondary
boot loader saying "Now booting the kernel." Or, the kernel
might successfully find the initrd but "stop" somewhere in
spawning /linuxrc.  In no case does it ever stop before
entering the kernel, and in no case does it ever get as far
as printing a shell prompt.  (And always, the message
"Freeing unused kernel memory: 36k init" gets something
greeked on the "ni" in the "init".  E.g. maybe the "ni"
becomes a theta.  This is the one oddity which is
consistent.)

(Note:  One might cast doubt on the initrd.  However,
printk's inside do_execve(), the ELF loader, etc. convince
me that the kernel is finding /linuxrc, seeing that it starts
with "#! /bin/sh", then loading /bin/sh, seeing that /bin/sh
starts with 0x7f ELF, etc. etc.)

Now, I wanted to see *why* the kernel stopped printing
things to the screen, so first I inserted printk's
(starting with execve() from init() in init/main.c) hoping
to narrow in on "the" problem.  Well, as I added or removed
printk's, the "hang" would move around (!) as follows:
E.g. I put half a dozen printk's in function a, in which it
calls b, c, d, e, f and g.  Printk's stop appearing on the
terminal screen after a calls d, so I put another
half-dozen printk's inside function d.  Then, perhaps, when
I run that, I start getting the printk's for e and f, but
not g.  Etc.

One very, very odd thing is as follows:  I tried this
printk business (also tried inserting some LED statements,
for the same purpose) many times before I gave up on it.
And in many circumstances, I would include a printk or LED
immediately before a calls function b, and also first thing
in b.  Or, a printk/LED as the very last thing in b, before
returning from a, and also first thing in a on return from
b.

For example, something like:

void a(void)
{
	printk/LED #1
	b();
	printk/LED #4
	c();
	d();
	e();
	f();
	g();
}

void b(void)
{
	printk/LED #2
	x();
	y();
	z();
	printk/LED #3
}


My thinking was that the only stuff that executes between
pairs of printk/LEDs is saving/restoring context between
functions.  And lo and behold, very often, I'd see one
printk but not the other -- e.g. in the above pseudocode
snippet, I'd see printk/LED #1 but not #2, or #3 but not #4
(!).  And yet calling or returning from a function should
be the easiest thing in the world; after all, we do it all
the time.  Some other things that I saw were also related
to ITLB misses, so I am going out on a limb and hypothesizing
that there is an ITLB miss when a calls b, or when b returns
to a.  Certainly, function calls are one way in which one
loses locality of reference.

(Note:  My debug monitor is small (~256K) & uses some
Motorola-provided sample code to set up the MMU before
jumping into Linux.  I have about 21 DTLB entries and 11
ITLB entries, which is not enough to fill up either TLB,
since they have 32 entries each.  So, my debug-monitor code
never needs to handle an ITLB miss.  Also, the secondary
boot loader disables cache & MMU before jumping into the
kernel.  As well, I've tried having my debug monitor never
initialize the cache & MMU at all.)

Since printk's weren't helping me, I next turned to the
logic analyzer.

With CPU6 support compiled in:  At some point (early or
late) the kernel takes an instruction TLB miss (surely not
the first one -- I don't know what makes this ITLB miss
different from any other), but after a few dozen
instructions I see the same instruction on the bus forever
-- which is what I meant by the CPU "parking".  It may be
the case that the processor is going into debug mode; can't
tell from the logic analyzer.  (Certainly, it shouldn't be
*in* debug mode since I set my DER to 0!)  But the CPU is
clearly halted on the fifth or sixth of 10 consecutive lis
instructions.

Without CPU6 support compiled in:  The kernel again stops
printing stuff to the terminal window, but the logic
analyzer shows that the CPU keeps running.  Tracing back
from the LA to the source code is tedious, but from what I
could tell yesterday, it appeared to be in do_signal(),
collect_signal(), kmem_cache_free(), et al., going back
around to do_signal() again.

Debug option 1 is printk's, which is not too useful for
this particular problem as I mentioned; debug option 2 is
the logic analyzer, which gives one limited information as
I mentioned; debug option 3 is the Macraigor.  Well, when I
hook up the Macraigor (changing *no* code), I get different
results (!) -- a panic as follows:

Kernel command line at :1001e2e4
Relocated to:  00200000
Contents:
  	root=/dev/ram0 init=/linuxrc rw
zimage at:     02017000 0207b2d2
initrd at:     0207b2d2 0215979d
Relocated to:  03f21000 03fff4cb
Available RAM: 0207c000 03f21000

Uncompressing Linux ... done.
Now booting the kernel.
Linux version 2.4.4 (a021502@u483wklnx001) (gcc version 2.95.3 20010315
(release
)) #263 Mon Jan 21 10:02:26 MST 2002
On node 0 totalpages: 16384
zone(0): 16384 pages.
zone(1): 0 pages.
zone(2): 0 pages.
Kernel command line: root=/dev/ram0 init=/linuxrc rw
Decrementer Frequency: 4125000
Calibrating delay loop... 65.12 BogoMIPS
Memory: 62328k available (788k kernel code, 316k data, 36k init, 0k highmem)
Dentry-cache hash table entries: 8192 (order: 4, 65536 bytes)
Buffer-cache hash table entries: 4096 (order: 2, 16384 bytes)
Page-cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 4096 (order: 3, 32768 bytes)
POSIX conformance testing by UNIFIX
Linux NET4.0 for Linux 2.4
Based upon Swansea University Computer Society NET3.039
Starting kswapd v1.8
CPM UART driver version 0.03
ttyS0 on SMC1 at 0x0280, BRG1
block: queued sectors max/low 41352kB/13784kB, 128 slots per queue
RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize
RAMDISK: Compressed image found at block 0
Oops: kernel access of bad area, sig: 11
NIP: 00000400 XER: C000187F LR: 00000400 SP: C024B440 REGS: c024b390 TRAP:
0400
MSR: 08209032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11
TASK = c024a000[1] 'swapper' Last syscall: 120
last math 00000000 last altivec 00000000
GPR00: 00000400 C024B440 C024A000 FFFFFFFF 00000008 00000000 C0270BFC
00000000
GPR08: 00003C85 00003C85 000000FF C00EA4C8 00000800 10008F80 00000000
C024BD68
GPR16: 00000001 C024BD48 00007400 00000400 00000C00 00000000 00001000
C0268C00
GPR24: 00000100 00000003 00000000 00000000 00000000 00000400 C0265120
00000000
Call backtrace:
00000400
Kernel panic: Attempted to kill init!
Rebooting in 180 seconds..

I am still looking into this.  Still, the strange thing is
that only happens when the Macraigor is connected.  Maybe
that is due to theMacraigor itself; I don't know.

In the panic, the NIP is 0x400 -- making me think it's
happening on the branch *to* 0x400 (InstructionAccess in
head_8xx.S).  InstructionAccess, in turn, is branched to
from InstructionTLBError and InstructionTLBMiss.  I don't
know which one is the callee since the backtrace is short.

Ideas:

*	I know people have done Linux on 8xx's before, many a
	time.  And everything I've ever read has told me that
	most of the work in porting Linux is in the boot
	loader, passing parameters in correctly, etc.; besides
	customizing .config, the kernel code is already OK --
	if the kernel has already been ported to your processor.
	(In fact, I found porting the secondary boot loader to
	be pretty easy.)

*	I know people have done Linux on 860's; I also suspect
	that people have done it on 857's as well.  Honestly,
	although it would be nice if there were a silicon
	erratum, I don't expect there will be.

*	I've used three different source trees -- Denx 2.2.13
	and 2.4.4, and Lineo's 2.4.15-pre1.  & I am told that
	all three are reliable sources.  And all three trees
	have been used for 8xx designs.

*	I've used two different cross-tool suites -- one that
	I downloaded from gnu.org and compiled from scratch;
	the other that I got from Lineo.  I really don't think
	I'm dealing with a compiler bug ... .

*	I've done a very thorough memory test on our boards, and
	this is far from our first PowerPC design.  In fact, we
	think we're pretty good at doing PowerPCs.

	(In the last few years we've done an 823 board, an 8260
	and a 755.  In those previous cases, we wrote the debug
	monitor / hardware-validation code; then, our customer
	would contract with a third-party vendor to port VxWorks,
	OS/9, etc.  Doing an OS port here is a first for us;
	a business we want to get into since our customers'
	feedback regarding Wind River and Microware has been
	uniformly negative.)

	Every test I throw at our board, from inside my debug
	montior, passes.

	Now, what could my debug monitor be *not* testing? One,
	for simplicity, the debug monitor uses no interrupts;
	it polls all devices.  For example (hypothetically) an
	interrupt pin could be completely unconnected and my
	debug monitor wouldn't catch it.

	Second, for simplicity, the debug monitor is
	single-threaded and uses a 1-1 address translation
	mechanism -- the TLBs are populated at reset time, and
	are just left that way.  So, hypothetically, there is
	something there I could be missing too.  But I don't
	know what.

*	This is not only my first Linux port, but my first OS
	port.  I consider myself to be good at embedded
	programming and hardware verification, but it's
	entirely possible that all my problems are caused by my
	naivete when it comes to OSs.

	We do have a FADS 860 board, which I would have liked
	to have ported Linux to first, before trying our board.
	However, the FADS has only 4MB DRAM, and I could not
	find a larger.

So, I know the 8xx port of Linux is not broken, I don't
believe the compiler is broken, I don't believe the 857T is
broken, and I don't think our board is defective.  I can
only imagine that I as a software person have screwed
something up, or that one of my coworkers has made some
mistake in the board design.

So what, what, what ... the only things this leaves are the
way in which I'm initializing the processor -- which I
don't believe is rocket science, by the way, but maybe
there's some special register which one needs to set on the
857 that I just don't know about.  In particular, I got the
processor-reset code from www.mot.com and customized it to
my own purposes -- just as we always have done on previous
projects.  However, I could not find any 857 sample code at
Motorola, so I used their sample 860 reset code.  But as
you say, and as I've heard from my co-workers, there
shouldn't *be* big differences between the 860 and 857,
except for UTOPIA and FEC.

As I mentioned, what I've said in this message is sketchy
and only half-coherent; I apologize for being so vague.

-----Original Message-----
From: Dan Malek [mailto:dan@embeddededge.com]
Sent: Monday, January 21, 2002 10:14 PM
To: Kerl, John
Cc: 'linuxppc-embedded@lists.linuxppc.org'
Subject: Re: Linux on 857T?


Kerl, John wrote:

> Hello,
>
> I am attempting a Linux port to a custom 857T board.  I've tried
> several different kernel trees, with the same results:  unhandled
> ITLB misses parking the CPU.

What do you mean "unhandled ITLB misses"?

The 857T _should_ work like an 860T with Ethernet either on the SCC or
the FEC (or both).  Of course, the reason to choose this part is you get
both
parallel UTOPIA and FEC, which will require some software changes to enable
both of these functions.


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: FW: Linux on 857T?

[Dan Malek]

Kerl, John wrote:

> Sorry, forgot to cc the mailing list.

Set burst inhibit (BIH) in the memory controller ORx register for
your DRAM banks.  If that has no effect (but I suspect it will),
disable copyback cache (this is a configuration option).  If this
has no effect (but I suspect it will), modify the code in head_8xx.S
to globally disable (or actually not enable) the cache.  If you need
help with this last one, let me know.

None of the ICEs you listed are Linux friendly.....run without them.


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: FW: Linux on 857T?

[Kerl, John]

Dan:

Per your suggestions, I tried clearing BIH for OR1 (my
SDRAM is on CS1); no difference except the greek character
on "Freeing unused kernel:  36k i*t" is now an epsilon, not
a theta.

Copy-back cache was already disabled.

I started taking a look at head_8xx.S as you suggested, but
I had a realization, as follows (sorry for the length of
this message, but it does get interesting toward the end.)

We have three alpha boards which are in some working order.
Board A, let's call it, is the best one, and the one I've
been using for all the things I discussed in my recent
postings.  Boards B and C, on the other hand, haven't been
"working" for a few weeks, so I pretty much forgot about
them, as I will explain.

Some basic things about my debug monitor:

* One copy of my debug monitor is flashed in as reset code.
  It runs some generic processor-init code (modified from
  mot.com as I mentioned) then jumps into crt0 and thence
  into my C code.  Text is taken from flash; read/write
  data is in SRAM.  (We have 1MB SRAM and 64MB SDRAM.)

* Also, I have two more copies of my debug monitor.
  Neither one includes processor-init code: only crt0 & my
  C code.  One runs entirely from SRAM.  By this I mean
  that text, data, bss, and stack (I don't use a heap) are
  all in SRAM.  The other copy runs entirely from SDRAM.

* All three copies have my TFTP client code in them; I can
  be running one copy, then TFTP up another and jump into
  it.

* No interrupts are used; devices are polled.

* Every 10,000 poll cycles or so (which comes out to something
  like a few times a second, less when running from flash) I
  update the LED to provide a heartbeat.  This is an 8-bit LED
  *not* in any of ports A-D in the IMMR space, but rather a
  memory-mapped access to our CPLD, which among other duties
  takes a certain chip select & updates the 8 LEDs.

* I successfully use SMC1 all the time for my terminal
  (simple command-line stuff), and I successfully use the
  FEC all the time for TFTP'ing things to the board.

* I don't set up the MMU at all.

So: on all of the three boards, I run my debug monitor
from flash at power-up.  Then, I can TFTP up an SRAM-only
or SDRAM-only debug monitor, or zvmlinux.initrd.  On all
three boards, the kernel runs as far as "Freeing unused
kernel:  36k i*t"; all are the same in this regard.
However, while on board A (my good board) I can also TFTP
up an SRAM-only or SDRAM-only debug monitor, on boards B
and C I cannot.  The last thing the flash debug monitor
says is "Jumping to {address}"; the SRAM-only or SDRAM-only
copies never print anything out.

This is the point at which I forgot about these two boards
a few weeks ago.  However, more recently I've been making
more extensive use of the logic analyzer and the Macraigor
probe (which seems to work passably well for debugging my
debug monitor, since I don't use interrupts).  So I thought
I'd devote a little more time to *why* these two boards are
having the problems they are.

It turns out that the following happens:

* My flash debug monitor receives the TFTP'ed SRAM-only
  or SDRAM-only executable and writes it to the specified
  address range in SRAM or SDRAM.

* The flash debug monitor jumps into the start address for
  the SRAM-only or SDRAM-only executable.

* The crt0 code in SRAM or SDRAM starts running and
  zero-fills the bss, etc.

* The crt0 code jumps into my C code.

* One of the first things my C code does in main() is write
  LEDs to provide initialization status.  Then, once init
  code is done (e.g. setting up SMC & FEC BDs etc.), the
  LEDs are written periodically as described above.

* On board A, this works fine.  But on boards B and C,
  the probe reveals that I am taking an ITLB miss (!!!)
  while trying to return from the LED-write routine.
  The latter is very simple; it looks like this:

	#define CPLD_ADDR 0x38100002
	#define CPLD_REG  (*(volatile int16u *)CPLD_ADDR)

	void LED_dev_write(
		int32u value)
	{
	#ifndef NO_LED // In the build file
		CPLD_REG = ((~(int16u)value & 0xff) << 8);
	#endif
	}

  which just boils down to about a half-dozen instructions,
  ending in an sth and a blr.

* The probe shows instructions up to the sth, but not the blr.
  Instead, the next value for the PC, as I single-step in the
  probe, is 0x00001104 (which is the NIP for ITLB-miss exception).

Now, if I #define NO_LED in my build file, then the debug
monitor doesn't write LEDs -- and runs fine (!) on all
three boards.  And the only code change between those two
cases is whether or not I do that LED write via the CPLD.

There are two very weird things:

* On all three boards, currently the MMU and cache are
  never enabled by any copy of my debug monitor.  Only
  Linux enables them, when and if I run it. I don't see how
  I can *possibly* be taking an ITLB miss in my code when
  the MMU is not even enabled.  In fact, I can print out the
  MSR in my debug monitor & see that bits 24-25 are zero.

* Even if the TLBs are somehow used when the MMU is not
  enabled (I need to check the 857 manual), I see *no* way
  I should ever be getting any TLB miss.  The DTLB and ITLB
  are both supposedly 32 entries each; I am only loading up
  21 entries for data and 11 for instructions with a very
  simple 1-1 virtual-physical mapping, with 8MB page sizes.
  TLB entries should (in my understanding) *never* get
  pushed out, if I'm not putting new ones in.

In short, I was puzzled by problems which I was seeing in
the OS, but not in my debug monitor nor in the secondary
boot loader.  In fact, that statement is only true for 1 of
3 boards.  For the other two, my debug monitor is
apparently encountering some of the *same* problems as the
kernel is.  I could believe that one board has a cold
solder joint, etc. -- after all, these are low-volume alpha
boards (we made a total of 8, only three of which have been
modded up to where they will run).  But I don't believe
that two of them have the same assembly problem.  Maybe
there is something wrong in our CPLD code, or maybe in my
BR/ORs for the CPLD's CS, or maybe something else.

At this point the discussion seems no longer to pertain to
a Linux-PPC mailing list, so I think I'll stop.  Anyway, if
you were losing sleep awaiting a reply from me (hee hee),
please don't.  I think I have some questions for my
hardware guys.

Thanks very much for the help.


-----Original Message-----
From: Dan Malek [mailto:dan@embeddededge.com]
Sent: Tuesday, January 22, 2002 1:42 PM
To: Kerl, John
Cc: 'linuxppc-embedded@lists.linuxppc.org'
Subject: Re: FW: Linux on 857T?


Kerl, John wrote:

> Sorry, forgot to cc the mailing list.

Set burst inhibit (BIH) in the memory controller ORx register for
your DRAM banks.  If that has no effect (but I suspect it will),
disable copyback cache (this is a configuration option).  If this
has no effect (but I suspect it will), modify the code in head_8xx.S
to globally disable (or actually not enable) the cache.  If you need
help with this last one, let me know.

None of the ICEs you listed are Linux friendly.....run without them.


	-- Dan

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: FW: Linux on 857T?

[Jerry Van Baren]

Just one dumb question: do you have branches to self in ALL the unused
exceptions (since you don't appear to be using any in your monitor, that
would be all but the reset exception)?  Is it possible you are getting a
different exception and your CPU is executing uninitialized memory (opcode
0x00xxxxxx appears to be illegal -- opcodes 0xFFxxxxxx are floating point
instructions that tend to cause floating point exceptions).

Try filling all unused memory (or at least the exception vector area) with
branches to self.  If I read my S-record file correctly, this looks like
0x48000000.  Now if your CPU decides to jump somewhere unexpected, at least
you will stop it in its tracks at the (unexpected) destination and be able
to see where it jumped to.  The question remains "why" at that point, but
you will have one more piece of the puzzle, as well as a smoking gun to
beat up your hardware guys up (to mix my metaphors :-).

If unexpected jumps are occurring, you could also try filling memory with a
jump to a routine that toggles an output line.  If you get lucky and it
works, you can trigger your logic analyzer on the output line and get a
back trace of where you were before you jumped.

Good luck, sounds like flaky hardware.  That's always the worst thing to
debug :-(.

gvb


At 06:36 PM 1/22/2002 -0700, Kerl, John wrote:

[snip]

>We have three alpha boards which are in some working order.
>Board A, let's call it, is the best one, and the one I've
>been using for all the things I discussed in my recent
>postings.  Boards B and C, on the other hand, haven't been
>"working" for a few weeks, so I pretty much forgot about
>them, as I will explain.

[major snip]

>* On board A, this works fine.  But on boards B and C,
>   the probe reveals that I am taking an ITLB miss (!!!)
>   while trying to return from the LED-write routine.
>   The latter is very simple; it looks like this:
>
>         #define CPLD_ADDR 0x38100002
>         #define CPLD_REG  (*(volatile int16u *)CPLD_ADDR)
>
>         void LED_dev_write(
>                 int32u value)
>         {
>         #ifndef NO_LED // In the build file
>                 CPLD_REG = ((~(int16u)value & 0xff) << 8);
>         #endif
>         }
>
>   which just boils down to about a half-dozen instructions,
>   ending in an sth and a blr.
>
>* The probe shows instructions up to the sth, but not the blr.
>   Instead, the next value for the PC, as I single-step in the
>   probe, is 0x00001104 (which is the NIP for ITLB-miss exception).
>
>Now, if I #define NO_LED in my build file, then the debug
>monitor doesn't write LEDs -- and runs fine (!) on all
>three boards.  And the only code change between those two
>cases is whether or not I do that LED write via the CPLD.
>
>There are two very weird things:
>
>* On all three boards, currently the MMU and cache are
>   never enabled by any copy of my debug monitor.  Only
>   Linux enables them, when and if I run it. I don't see how
>   I can *possibly* be taking an ITLB miss in my code when
>   the MMU is not even enabled.  In fact, I can print out the
>   MSR in my debug monitor & see that bits 24-25 are zero.
>
>* Even if the TLBs are somehow used when the MMU is not
>   enabled (I need to check the 857 manual), I see *no* way
>   I should ever be getting any TLB miss.  The DTLB and ITLB
>   are both supposedly 32 entries each; I am only loading up
>   21 entries for data and 11 for instructions with a very
>   simple 1-1 virtual-physical mapping, with 8MB page sizes.
>   TLB entries should (in my understanding) *never* get
>   pushed out, if I'm not putting new ones in.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/