Re: SMP badness on 5kc

[Ralf Baechle <ralf@linux-mips.org>]

On Wed, Aug 03, 2005 at 10:16:27AM -0400, Joshua Wise wrote:

> I have been setting up a simulation of a multiprocessor 5kc board that my 
> company is developing using gxemul, and I've come across a few problems with 
> the MIPS SMP support.

The 5Kc has no cache coherency, so I don't see how it could be done other
than by offloading that into software - which performancewise I would
expect to be an unatractive solution.  Well, there's always the option of
disabling caches which is the easiest way but probably even slower.

> The first issue that I came across was that 5kc does not have an scache, and 
> hence r4k_blast_scache will be null, and hence local_r4k_flush_icache_range 
> crashes the system with a kernel NULL dereference. (Actually, that happens 
> early enough that it drops back into YAMON, without even giving me a kernel 
> panic.) Strangely enough, this only happens on SMP... My solution for this 
> was to put a switch statement like the one in local_r4k___flush_cache_all 
> into local_r4k_flush_icache_range around the if 
> (!cpu_icache_snoops_remote_store). This seemed to cause the crash to stop 
> happening.

That is acutally somewhat similar to another problem somebody found
recently for a uniprocessor configuration.  The whole second level cache
flushing thing only really is needed at all if a system has such a cache
at all, so something like:

  if (!cpu_icache_snoops_remote_store && cpu_has_sc) {
     [...]
  }

would be sensible.  Well, there's no cpu_has_sc but cpu_scache_line_size()
would serve the same purpose.

> My second issue, unresolved to date, is that we are calling 
> on_each_cpu(local_r4k___flush_cache_all) (called from r4k_flush_cache_all) 
> while interrupts are disabled. This doesn't happen while we are bringing up 
> CPUs 0 or 1 -- it seems to only happen when bringing up CPU2. This causes a 
> "badness" message, followed by either a lot of oopses, or a deadlock. Below 
> my sig is a boot log from serial, complete with debug messages written by the 
> Sirius Cybernetics Corporation. :)

Who were the first to be shot when the great revolution came, says the
Hitchhiker's Guide to The Galaxy ;-)

I recently modified the code to avoid such SMP cacheflushes during startup.
As it turned out only a local_* flush was needed which nicely dealt with
the warnings.

  Ralf