SMP on the MacBook

SMP on the MacBook

[Marco Gerards]

Hi,

For a while now I have been trying to debug an issue with Xen on the
MacBook, doing this is part of my Summer of Code project to get Xen
working properly on the Intel Mac.  In case some of you don't know,
the MacBook is an Intel Core Duo based laptop from Apple, which lacks
some legacy hardware.

Xen (testing) works perfectly for me when starting it with "nosmp",
however when not using it, it crashes while starting up the second
core.  There have been some bug report on this issue already.

Below I will explain the problem in more detail and with some
observations.  I will be very grateful if someone can come up with
ideas of how to fix this or give me some ideas that can lead to a
solution.  Not only Xen has this problem, but so has FreeBSD and
NetBSD, so a solution will be beneficial to these projects as well.

What happens is the following: The first core sends two IPI
messages(INIT and STARTUP) to start the second core.  When the second
core is started, it notifies the first core of this so it can continue
with the boot process.  However, the second core is not started by
these messages, the loop that waits for the second core times out,
continues and Xen crashes.

One strange thing is that Xen initializes the second core properly
when you press a key on the keyboard after the IPIs are sent.  I am
expecting this interrupt somehow wakes up the second core so it can
accept and process the IPIs.  Because it is a laptop the second core
is put in deep sleep mode to save power in case an OS does not use
this core at all.  But as I understand it, it should wake up when an
INIT IPI arrives.

The problems I am having with Xen can be reproduced with linux when
booting with "acpi=ht", which means the ACPI interpreter is not
started.

Hopefully someone has some ideas how to deal with this issue and how
to properly fix this.  If you want more information or want me to test
things, feel free to ask.

Thanks,
Marco

Re: SMP on the MacBook

[gimli]

[-- Attachment #1: Type: text/plain, Size: 2516 bytes --]

Hi.

I have xen running on my Macbook. The problem is cause by grub.
When i start the xen entry in grub with the key "return" xen
fails too boot. When i use the "Cursor key right" to start
the xen entry xen boots fine. I used the atached patch against
grub 0.97 for the Intel Macs.

cu

Edgar (gimli) Hucek

Marco Gerards schrieb:
> Hi,
> 
> For a while now I have been trying to debug an issue with Xen on the
> MacBook, doing this is part of my Summer of Code project to get Xen
> working properly on the Intel Mac.  In case some of you don't know,
> the MacBook is an Intel Core Duo based laptop from Apple, which lacks
> some legacy hardware.
> 
> Xen (testing) works perfectly for me when starting it with "nosmp",
> however when not using it, it crashes while starting up the second
> core.  There have been some bug report on this issue already.
> 
> Below I will explain the problem in more detail and with some
> observations.  I will be very grateful if someone can come up with
> ideas of how to fix this or give me some ideas that can lead to a
> solution.  Not only Xen has this problem, but so has FreeBSD and
> NetBSD, so a solution will be beneficial to these projects as well.
> 
> What happens is the following: The first core sends two IPI
> messages(INIT and STARTUP) to start the second core.  When the second
> core is started, it notifies the first core of this so it can continue
> with the boot process.  However, the second core is not started by
> these messages, the loop that waits for the second core times out,
> continues and Xen crashes.
> 
> One strange thing is that Xen initializes the second core properly
> when you press a key on the keyboard after the IPIs are sent.  I am
> expecting this interrupt somehow wakes up the second core so it can
> accept and process the IPIs.  Because it is a laptop the second core
> is put in deep sleep mode to save power in case an OS does not use
> this core at all.  But as I understand it, it should wake up when an
> INIT IPI arrives.
> 
> The problems I am having with Xen can be reproduced with linux when
> booting with "acpi=ht", which means the ACPI interpreter is not
> started.
> 
> Hopefully someone has some ideas how to deal with this issue and how
> to properly fix this.  If you want more information or want me to test
> things, feel free to ask.
> 
> Thanks,
> Marco
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 


[-- Attachment #2: intelmac_grub.diff --]
[-- Type: text/x-patch, Size: 1246 bytes --]

--- grub-0.97.orig/stage2/asm.S	2004-06-19 18:55:22.000000000 +0200
+++ grub-0.97/stage2/asm.S	2006-04-21 11:10:52.000000000 +0200
@@ -1651,7 +1651,29 @@
 	jnz	3f
 	ret
 
-3:	/* use keyboard controller */
+3:	/*
+	 * try to switch gateA20 using PORT92, the "Fast A20 and Init"
+	 * register
+	*/
+	mov $0x92, %dx
+	inb %dx, %al
+	/* skip the port92 code if it's unimplemented (read returns 0xff) */
+	cmpb $0xff, %al
+	jz 6f
+	
+	/* set or clear bit1, the ALT_A20_GATE bit */
+	movb 4(%esp), %ah
+	testb %ah, %ah
+	jz 4f
+	orb $2, %al
+	jmp 5f
+4:	and $0xfd, %al
+	
+	/* clear the INIT_NOW bit don't accidently reset the machine */
+5:	and $0xfe, %al
+	outb %al, %dx
+	
+6:	/* use keyboard controller */
 	pushl	%eax
 
 	call    gloop1
@@ -1661,9 +1683,12 @@
 
 gloopint1:
 	inb	$K_STATUS
+	cmpb	$0xff, %al
+	jz	gloopint1_done
 	andb	$K_IBUF_FUL, %al
 	jnz	gloopint1
 
+gloopint1_done:	
 	movb	$KB_OUTPUT_MASK, %al
 	cmpb	$0, 0x8(%esp)
 	jz	gdoit
@@ -1684,6 +1709,8 @@
 
 gloop1:
 	inb	$K_STATUS
+	cmpb	$0xff, %al
+	jz	gloop2ret
 	andb	$K_IBUF_FUL, %al
 	jnz	gloop1
 
@@ -1991,6 +2018,11 @@
 ENTRY(console_getkey)
 	push	%ebp
 
+wait_for_key:
+	call	EXT_C(console_checkkey)
+	incl	%eax
+	jz	wait_for_key
+	
 	call	EXT_C(prot_to_real)
 	.code16
 

[-- Attachment #3: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: SMP on the MacBook

[Marco Gerards]

gimli <gimli@dark-green.com> writes:

Hi Edgar,

What a small world...

> I have xen running on my Macbook. The problem is cause by grub.
> When i start the xen entry in grub with the key "return" xen
> fails too boot. When i use the "Cursor key right" to start
> the xen entry xen boots fine. I used the atached patch against
> grub 0.97 for the Intel Macs.

Amazing, that works for me!  I will have a look why this has some
influence on SMP.

The patch I have used for GRUB Legacy is a backport for enabling the
A20 gate from GRUB 2.  I have posted it on the bug-grub mailinglist
in case you are interested and will commit it soon.  Essentially it
does almost the same as the one you are using.

Thanks,
Marco

RE: SMP on the MacBook

[Ian Pratt]

> Xen (testing) works perfectly for me when starting it with "nosmp",
> however when not using it, it crashes while starting up the second
> core.  There have been some bug report on this issue already.

Marco,
Have you tried -unstable? Some potentially relevant files have been
updated.

Also, exactly which version of Linux boots OK? Does 2.6.16 e.g. from the
DemoCD?

Thanks,
Ian

 
> Below I will explain the problem in more detail and with some
> observations.  I will be very grateful if someone can come up with
> ideas of how to fix this or give me some ideas that can lead to a
> solution.  Not only Xen has this problem, but so has FreeBSD and
> NetBSD, so a solution will be beneficial to these projects as well.
> 
> What happens is the following: The first core sends two IPI
> messages(INIT and STARTUP) to start the second core.  When the second
> core is started, it notifies the first core of this so it can continue
> with the boot process.  However, the second core is not started by
> these messages, the loop that waits for the second core times out,
> continues and Xen crashes.
> 
> One strange thing is that Xen initializes the second core properly
> when you press a key on the keyboard after the IPIs are sent.  I am
> expecting this interrupt somehow wakes up the second core so it can
> accept and process the IPIs.  Because it is a laptop the second core
> is put in deep sleep mode to save power in case an OS does not use
> this core at all.  But as I understand it, it should wake up when an
> INIT IPI arrives.
> 
> The problems I am having with Xen can be reproduced with linux when
> booting with "acpi=ht", which means the ACPI interpreter is not
> started.
> 
> Hopefully someone has some ideas how to deal with this issue and how
> to properly fix this.  If you want more information or want me to test
> things, feel free to ask.
> 
> Thanks,
> Marco
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: SMP on the MacBook

[Marco Gerards]

"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> writes:

Hi Ian,

>> Xen (testing) works perfectly for me when starting it with "nosmp",
>> however when not using it, it crashes while starting up the second
>> core.  There have been some bug report on this issue already.
>
> Marco,
> Have you tried -unstable? Some potentially relevant files have been
> updated.

Yes, unstable has the same problem.  Would you encourage development
in unstable or testing?

> Also, exactly which version of Linux boots OK? Does 2.6.16 e.g. from the
> DemoCD?

I have not tested the DemoCD because it still contains an unpatched
GRUB.  Because of that it doesn't work with the Intel Mac.

But I have tried linux 2.6.16 (self compiled), which works perfectly.

Thanks,
Marco

RE: SMP on the MacBook

[Ian Pratt]

> > Marco,
> > Have you tried -unstable? Some potentially relevant files have been
> > updated.
> 
> Yes, unstable has the same problem.  Would you encourage development
> in unstable or testing?

I'd go for -unstable. In reality, the stability is usually rather good.

> > Also, exactly which version of Linux boots OK? Does 2.6.16 e.g. from
the
> > DemoCD?
> 
> I have not tested the DemoCD because it still contains an unpatched
> GRUB.  Because of that it doesn't work with the Intel Mac.
> 
> But I have tried linux 2.6.16 (self compiled), which works perfectly.

I might be worth trying the native kernel from the demo cd just to check
it's not a kernel config issue, but I admit it's a bit of a long shot.

Also, it might be good to test a similarly configed native 2.6.15/14/12
to see if any of them fail in the same way as Xen -- that would give us
a big clue.

Thanks,
Ian 

Re: SMP on the MacBook

[Marco Gerards]

"Ian Pratt" <m+Ian.Pratt@cl.cam.ac.uk> writes:

>> > Marco,
>> > Have you tried -unstable? Some potentially relevant files have been
>> > updated.
>> 
>> Yes, unstable has the same problem.  Would you encourage development
>> in unstable or testing?
>
> I'd go for -unstable. In reality, the stability is usually rather good.

Sure.

>> > Also, exactly which version of Linux boots OK? Does 2.6.16 e.g. from
> the
>> > DemoCD?
>> 
>> I have not tested the DemoCD because it still contains an unpatched
>> GRUB.  Because of that it doesn't work with the Intel Mac.
>> 
>> But I have tried linux 2.6.16 (self compiled), which works perfectly.
>
> I might be worth trying the native kernel from the demo cd just to check
> it's not a kernel config issue, but I admit it's a bit of a long shot.

The problem is in Xen itself, if that is what you mean.  I was a bit
tired yesterday evening and for some reason I was thinking you meant a
normal linux kernel. :)

> Also, it might be good to test a similarly configed native 2.6.15/14/12
> to see if any of them fail in the same way as Xen -- that would give us
> a big clue.

I have been trying this, back to Linux 2.6.8.  I will play a bit more
with this, although it really looks like this being an ACPI related
issue, for some reason ACPI is being able to wakeup the second core
properly.

--
Marco

RE: SMP on the MacBook

[Ian Pratt]

> >> I have not tested the DemoCD because it still contains an unpatched
> >> GRUB.  Because of that it doesn't work with the Intel Mac.
> >>
> >> But I have tried linux 2.6.16 (self compiled), which works
perfectly.
> >
> > I might be worth trying the native kernel from the demo cd just to
check
> > it's not a kernel config issue, but I admit it's a bit of a long
shot.
> 
> The problem is in Xen itself, if that is what you mean.  I was a bit
> tired yesterday evening and for some reason I was thinking you meant a
> normal linux kernel. :)

There is a native kernel on the demo CD as well as the xen/linux one.
They have very similar configs, so it's often interesting to compare the
behaviour of them.

> > Also, it might be good to test a similarly configed native
2.6.15/14/12
> > to see if any of them fail in the same way as Xen -- that would give
us
> > a big clue.
> 
> I have been trying this, back to Linux 2.6.8.  I will play a bit more
> with this, although it really looks like this being an ACPI related
> issue, for some reason ACPI is being able to wakeup the second core
> properly.

As you point out, things go wrong too early to be influenced by anything
in xen/linux. However, trying booting different versions of native linux
is useful as it will tell us when the issue was fixed in linux and hence
identify what we need to do.

Thanks,
Ian