CFE problem: starting secondary CPU.

CFE problem: starting secondary CPU.

[Kaz Kylheku]

Anyone seen a problem like this? cfe_cpu_start() works fine on a
32 bit kernel, but not on 64.

I added a function to cfe/smp.c:

  static asmlinkage void dummy()
  {
    prom_printf("dummy called\n");
  }

This serves as the simplest possible "hello world". 

If I substitute this for the function passed to cfe_cpu_start(),
on a 32 bit kernel, the slave CPU calls the function and the
message is printed on the serial console.

On a 64 bit kernel, the function is never reached. The API call returns,
but the
secondary CPU never calls in.

The sign-extension of the address looks good. The function pointer looks
like
FFFFFFFF8XXXXXXX. This is cast to long before being assigned into the
right field
of the CFE request structure, where it is converted to 64 bit unsigned.

Inside CFE, the CPU is just looping around waiting for the address to
call with
a direct jump.

So it's a puzzling problem.

CFE problem: starting secondary CPU.

[Kaz Kylheku]

Anyone seen a problem like this? cfe_cpu_start() works fine on a
32 bit kernel, but not on 64.

I added a function to cfe/smp.c:

  static asmlinkage void dummy()
  {
    prom_printf("dummy called\n");
  }

This serves as the simplest possible "hello world". 

If I substitute this for the function passed to cfe_cpu_start(),
on a 32 bit kernel, the slave CPU calls the function and the
message is printed on the serial console.

On a 64 bit kernel, the function is never reached. The API call returns,
but the
secondary CPU never calls in.

The sign-extension of the address looks good. The function pointer looks
like
FFFFFFFF8XXXXXXX. This is cast to long before being assigned into the
right field
of the CFE request structure, where it is converted to 64 bit unsigned.

Inside CFE, the CPU is just looping around waiting for the address to
call with
a direct jump.

So it's a puzzling problem.

Re: CFE problem: starting secondary CPU.

[Jonathan Day]

I've seen the case where the second CPU did not start
on a Broadcom 1250 running a 64-bit kernel, but I
don't know if anyone has a good solution. I just
rigged the values in the Linux kernel so that it knows
about the second CPU. It's a godawful hack, but I
needed something quick at the time.

Personally, I am not a fan of CFE and would love to
know if there's a better way to bootstrap.

--- Kaz Kylheku <kaz@zeugmasystems.com> wrote:

> Anyone seen a problem like this? cfe_cpu_start()
> works fine on a
> 32 bit kernel, but not on 64.
> 
> I added a function to cfe/smp.c:
> 
>   static asmlinkage void dummy()
>   {
>     prom_printf("dummy called\n");
>   }
> 
> This serves as the simplest possible "hello world". 
> 
> If I substitute this for the function passed to
> cfe_cpu_start(),
> on a 32 bit kernel, the slave CPU calls the function
> and the
> message is printed on the serial console.
> 
> On a 64 bit kernel, the function is never reached.
> The API call returns,
> but the
> secondary CPU never calls in.
> 
> The sign-extension of the address looks good. The
> function pointer looks
> like
> FFFFFFFF8XXXXXXX. This is cast to long before being
> assigned into the
> right field
> of the CFE request structure, where it is converted
> to 64 bit unsigned.
> 
> Inside CFE, the CPU is just looping around waiting
> for the address to
> call with
> a direct jump.
> 
> So it's a puzzling problem.
> 
> 
> 
> 
> 


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: CFE problem: starting secondary CPU.

[Ralf Baechle]

On Fri, Oct 06, 2006 at 07:15:23PM -0700, Jonathan Day wrote:

> I've seen the case where the second CPU did not start
> on a Broadcom 1250 running a 64-bit kernel, but I
> don't know if anyone has a good solution. I just
> rigged the values in the Linux kernel so that it knows
> about the second CPU. It's a godawful hack, but I
> needed something quick at the time.
> 
> Personally, I am not a fan of CFE and would love to
> know if there's a better way to bootstrap.

Firmware is a stepchild and all implementations have in common that they're
hated by they're users.  And my grief is there are way to many different
firmwares for MIPS systems.

  Ralf

Re: CFE problem: starting secondary CPU.

[girish]

On Oct 7, 2006, at 7:53 PM, Ralf Baechle wrote:

> On Fri, Oct 06, 2006 at 07:15:23PM -0700, Jonathan Day wrote:
>
>> I've seen the case where the second CPU did not start
>> on a Broadcom 1250 running a 64-bit kernel, but I
>> don't know if anyone has a good solution. I just
>> rigged the values in the Linux kernel so that it knows
>> about the second CPU. It's a godawful hack, but I
>> needed something quick at the time.
>>
>> Personally, I am not a fan of CFE and would love to
>> know if there's a better way to bootstrap.
>
> Firmware is a stepchild and all implementations have in common that  
> they're
> hated by they're users.  And my grief is there are way to many  
> different
> firmwares for MIPS systems.
>
>   Ralf
>

would it be reasonable to choose couple of bootmonitors and support  
them under MIPS/Linux umbrella. even bootable linux would be a good  
choice.

Re: CFE problem: starting secondary CPU.

[Jonathan Day]

--- girish <girishvg@gmail.com> wrote:
> would it be reasonable to choose couple of
> bootmonitors and support  
> them under MIPS/Linux umbrella. even bootable linux
> would be a good  
> choice.

I can't see why not. For that matter, I can't imagine
it would be too hard to write the necessary flash
support to get LinuxBIOS working.

It does puzzle me, though, that the kernel seems to
get a lot of data from CFE rather than using the
probes it already has. Whether it's a built-in BIOS
(as on the PC), ACPI, CFE, or some other external
table of what is present, the one thing experience has
taught is that these tables cannot (and should not) be
trusted. Kernel discovery, although unsafe for some
hardware*, generally produces more accurate and
reliable results.

*Of course, that's a killer. There is an unfortunately
large amount of hardware in the world that cannot be
used safely in conjunction with probes. Other hardware
will respond incorrectly, screw up the machine, play
all your MP3's backwards, ...


__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: CFE problem: starting secondary CPU.

[Ralf Baechle]

On Sun, Oct 08, 2006 at 02:35:37PM -0700, Jonathan Day wrote:

> > would it be reasonable to choose couple of
> > bootmonitors and support  
> > them under MIPS/Linux umbrella. even bootable linux
> > would be a good  
> > choice.
> 
> I can't see why not. For that matter, I can't imagine
> it would be too hard to write the necessary flash
> support to get LinuxBIOS working.

The average firmware implementation contains alot of dark magic about
hardware initialization.  Producing a decent replacement is not trivial.

The current situation is that every vendor has a favorite firmware
implementation or sometimes even several depending on vintage or endianess.
Repeated problems over the years have taught me the only productive
way to live with most firmware implementations is to touch them as little
as possible since the rules for coexistence with the OS are usually very
weakly worded, inconsistent across platforms and the governing features
generally are bugs, incompatibilites and lack of features.  End of rant :-)

How x86 or Linux centric is LinuxBIOS?  Makers of Linux devices want to
support other operating systems as well.

  Ralf

Re: CFE problem: starting secondary CPU.

[Jonathan Day]

--- Ralf Baechle <ralf@linux-mips.org> wrote:
> The average firmware implementation contains alot of
> dark magic about
> hardware initialization.  Producing a decent
> replacement is not trivial.

Ok, does anyone here know if Voldemort is a good code
hacker? :)
 
(snip)
> How x86 or Linux centric is LinuxBIOS?  Makers of
> Linux devices want to
> support other operating systems as well.

LinuxBIOS is listed as supporting Linux (!), Plan 9
and Windows 2000. They don't support any of the *BSDs,
as there's apparently some BIOS calls in their
bootloaders and LinuxBIOS doesn't do BIOS calls. The
total intrusion into Linux is a 10-line hook and there
obviously can't even be that for Windows, so I think
most other OS' should be ok with it.

The supported motherboards include your usual x86
stuff (including Opterons, which I'm taking as meaning
it'll work on 64-bit systems) and PowerPC chips. The
FAQ also mentions the Alpha processor, but I didn't
see any examples amongst the listed vendors or
supported motherboards.

Jonathan

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

RE: CFE problem: starting secondary CPU.

[Mark E Mason]

Hello,

> -----Original Message-----
> From: linux-mips-bounce@linux-mips.org 
> [mailto:linux-mips-bounce@linux-mips.org] On Behalf Of Kaz Kylheku
> Sent: Friday, October 06, 2006 4:59 PM
> To: linux-mips@linux-mips.org
> Subject: CFE problem: starting secondary CPU.
> 
> Anyone seen a problem like this? cfe_cpu_start() works fine on a
> 32 bit kernel, but not on 64.

Which version of CFE are you using?  We'd seen something like this with
the 1480 eval boards, some specific versions of CFE, and 64-bit SMP
Linux.

Thx,
Mark

RE: CFE problem: starting secondary CPU.

[Kaz Kylheku]

Jonathan Day wrote:
> I've seen the case where the second CPU did not start
> on a Broadcom 1250 running a 64-bit kernel, but I
> don't know if anyone has a good solution. I just
> rigged the values in the Linux kernel so that it knows
> about the second CPU.

Do you have a patch for that?

> It's a godawful hack, but I
> needed something quick at the time.

I wrote another hack. I struggled with it for hours and
then finally got it to work. Whoo! 

The idea is based on the hypothesis that the 64 bit
kernel's environment somehow makes the CFE calls unreliable.

Under this hack, the CPUs are started early through CFE,
before most other initializations. They go into a holding function
which is almost identical to the one inside CFE, 
except that it's running kernel code.

Then later at the point where the CPU's are to be released
into the kernel, it's purely an in-kernel operation, no
longer involving any calls to CFE.



Remarks:



I have some CFE code which reads environment variables
using CFE. Under 64 bit, there is a mysterious oops in it.
The code looks good, and runs fine under 32.

What could it be? Maybe CFE doesn't like being called from
a task! Something about the 64 bit stack?

Note that code which starts the other CPU's is called from a
kernel thread which calls the init() function, not from
the mainline.

I will investigate further.

RE: CFE problem: starting secondary CPU.

[Kaz Kylheku]

Jonathan Day wrote:
> I've seen the case where the second CPU did not start
> on a Broadcom 1250 running a 64-bit kernel, but I
> don't know if anyone has a good solution. I just
> rigged the values in the Linux kernel so that it knows
> about the second CPU.

Do you have a patch for that?

> It's a godawful hack, but I
> needed something quick at the time.

I wrote another hack. I struggled with it for hours and
then finally got it to work. Whoo! 

The idea is based on the hypothesis that the 64 bit
kernel's environment somehow makes the CFE calls unreliable.

Under this hack, the CPUs are started early through CFE,
before most other initializations. They go into a holding function
which is almost identical to the one inside CFE, 
except that it's running kernel code.

Then later at the point where the CPU's are to be released
into the kernel, it's purely an in-kernel operation, no
longer involving any calls to CFE.



Remarks:



I have some CFE code which reads environment variables
using CFE. Under 64 bit, there is a mysterious oops in it.
The code looks good, and runs fine under 32.

What could it be? Maybe CFE doesn't like being called from
a task! Something about the 64 bit stack?

Note that code which starts the other CPU's is called from a
kernel thread which calls the init() function, not from
the mainline.

I will investigate further.

RE: CFE problem: starting secondary CPU.

[Kaz Kylheku]

Mark E Mason: 
> Kaz Kylheku:
> > 
> > Anyone seen a problem like this? cfe_cpu_start() works fine on a
> > 32 bit kernel, but not on 64.
> 
> Which version of CFE are you using?  We'd seen something like 
> this with
> the 1480 eval boards, some specific versions of CFE, and 64-bit SMP
> Linux.

I see in the release notes that something was fixed in 1.2.4 that could
prevent 64 bit SMP from booting: a bug that was preventing the upper
halves of registers from being saved. Is that what you are talking
about?

We are on 1.3.0.

Thanks.

RE: CFE problem: starting secondary CPU.

[Kaz Kylheku]

Mark E Mason: 
> Kaz Kylheku:
> > 
> > Anyone seen a problem like this? cfe_cpu_start() works fine on a
> > 32 bit kernel, but not on 64.
> 
> Which version of CFE are you using?  We'd seen something like 
> this with
> the 1480 eval boards, some specific versions of CFE, and 64-bit SMP
> Linux.

I see in the release notes that something was fixed in 1.2.4 that could
prevent 64 bit SMP from booting: a bug that was preventing the upper
halves of registers from being saved. Is that what you are talking
about?

We are on 1.3.0.

Thanks.

RE: CFE problem: starting secondary CPU.

[Mark E Mason]

Hello, 

> Mark E Mason: 
> > Kaz Kylheku:
> > > 
> > > Anyone seen a problem like this? cfe_cpu_start() works fine on a
> > > 32 bit kernel, but not on 64.
> > 
> > Which version of CFE are you using?  We'd seen something like 
> > this with
> > the 1480 eval boards, some specific versions of CFE, and 64-bit SMP
> > Linux.
> 
> I see in the release notes that something was fixed in 1.2.4 
> that could
> prevent 64 bit SMP from booting: a bug that was preventing the upper
> halves of registers from being saved. Is that what you are talking
> about?
> 
> We are on 1.3.0.

Yes, I was thinking of the one fixed in 1.2.4.

Definitely sounds like a [new] bug then.  Have you filed a bug report
with sibyte-software@broadcom.com?

Thx,
Mark

> 
> Thanks.
> 
> 
>