Re: [PATCH][2.5] Honour dont_enable_local_apic flag

Re: [PATCH][2.5] Honour dont_enable_local_apic flag

[mikpe]

On Sat, 31 May 2003 23:30:10 -0400 (EDT), Zwane Mwaikambo wrote:
>--- linux-2.5/arch/i386/kernel/apic.c	30 May 2003 20:14:41 -0000	1.50
>+++ linux-2.5/arch/i386/kernel/apic.c	31 May 2003 05:53:34 -0000
>@@ -665,6 +665,7 @@ static int __init detect_init_APIC (void
> 	return 0;
> 
> no_apic:
>+	dont_enable_local_apic = 1;
> 	printk("No local APIC present or hardware disabled\n");
> 	return -1;
> }
>@@ -1127,6 +1128,9 @@ asmlinkage void smp_error_interrupt(void
>  */
> int __init APIC_init_uniprocessor (void)
> {
>+	if (dont_enable_local_apic)
>+		return -1;
>+
> 	if (!smp_found_config && !cpu_has_apic)
> 		return -1;

I didn't follow this thread closely, but:

1. dont_enable_local_apic was originally only intended for those
   systems (i.e., Dell laptops) where enabling the local APIC in
   HW (via APIC_BASE MSR) causes problems due to broken BIOSen.
   It was never intended as a "Kernel, please don't use the local
   APIC even though I configured it" option. It doesn't help you
   if your machine boots with the local APIC enabled, but the
   local APIC doesn't work for some reason.

2. The only way to reach no_apic is if we boot on a vendor/model
   combination that isn't known to have a local APIC that can be
   enabled in software. And the second patch hunk only makes a
   difference if the CPU boots with an enabled local APIC.

   So what vendor/model CPU is used in the failure case?

   (And if its local APIC is broken, cpu_has_apic should be cleared
   rather than setting the dont_enable flag. Post-boot code may
   access the local APIC if CONFIG_X86_LOCAL_APIC && cpu_has_apic.)

3. What is the exact failure? Hang or crash? Where? UP or SMP kernel?
   (If SMP kernel, does booting with "nosmp" help?)

/Mikael

Re: [PATCH][2.5] Honour dont_enable_local_apic flag

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1702 bytes --]

On Sun, 2003-06-01 at 07:23, mikpe@csd.uu.se wrote:
> 
> I didn't follow this thread closely, but:
> 
> 1. dont_enable_local_apic was originally only intended for those
>    systems (i.e., Dell laptops) where enabling the local APIC in
>    HW (via APIC_BASE MSR) causes problems due to broken BIOSen.
>    It was never intended as a "Kernel, please don't use the local
>    APIC even though I configured it" option. It doesn't help you
>    if your machine boots with the local APIC enabled, but the
>    local APIC doesn't work for some reason.

OK.  I really have no experience or knowledge about this stuff to argue
this point.

> 2. The only way to reach no_apic is if we boot on a vendor/model
>    combination that isn't known to have a local APIC that can be
>    enabled in software. And the second patch hunk only makes a
>    difference if the CPU boots with an enabled local APIC.
> 
>    So what vendor/model CPU is used in the failure case?

VMware 2.0.4.

When the kernel boots, it detects the local APIC, however when it comes
time to calibrate the APIC timer interrupts, the kernel "stops"/hangs.

>    (And if its local APIC is broken, cpu_has_apic should be cleared
>    rather than setting the dont_enable flag. Post-boot code may
>    access the local APIC if CONFIG_X86_LOCAL_APIC && cpu_has_apic.)
> 
> 3. What is the exact failure? Hang or crash?

Normal boot until here:

Using local APIC timer interrupts.
calibrating APIC timer ...
..... CPU clock speed is 1658.7651 MHz.
..... host bus clock speed is 0.0000 MHz.
cpu: 0, clocks: 0, slice: 0

>  Where? UP or SMP kernel?

UP.

b.

-- 
Brian J. Murrell <brian@interlinx.bc.ca>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH][2.5] Honour dont_enable_local_apic flag

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1024 bytes --]

On Sun, 2003-06-01 at 07:23, mikpe@csd.uu.se wrote:
> 
>    (And if its local APIC is broken, cpu_has_apic should be cleared
>    rather than setting the dont_enable flag. Post-boot code may
>    access the local APIC if CONFIG_X86_LOCAL_APIC && cpu_has_apic.)

So would you prefer something more along the lines of:

--- arch/i386/kernel/setup.c.orig       2003-04-26 10:34:35.000000000 -0400
+++ arch/i386/kernel/setup.c    2003-06-01 13:11:47.000000000 -0400
@@ -845,6 +845,10 @@
                 */
                else if (!memcmp(from, "highmem=", 8))
                        highmem_pages = memparse(from+8, &from) >> PAGE_SHIFT;
+               else if (!memcmp(from, "nolapic", 7)) {
+                               clear_bit(X86_FEATURE_APIC, &boot_cpu_data.x86_capability);
+                               set_bit(X86_FEATURE_APIC, &disabled_x86_caps);
+               }
 nextchar:
                c = *(from++);
                if (!c)

b.

-- 
Brian J. Murrell <brian@interlinx.bc.ca>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH][2.5] Honour dont_enable_local_apic flag

[mikpe]

On 01 Jun 2003 13:25:33 -0400, Brian J. Murrell wrote:
>So would you prefer something more along the lines of:
>
>--- arch/i386/kernel/setup.c.orig       2003-04-26 10:34:35.000000000 -0400
>+++ arch/i386/kernel/setup.c    2003-06-01 13:11:47.000000000 -0400
>@@ -845,6 +845,10 @@
>                 */
>                else if (!memcmp(from, "highmem=3D", 8))
>                        highmem_pages =3D memparse(from+8, &from) >> PAGE_S=
>HIFT;
>+               else if (!memcmp(from, "nolapic", 7)) {
>+                               clear_bit(X86_FEATURE_APIC, &boot_cpu_data.=
>x86_capability);
>+                               set_bit(X86_FEATURE_APIC, &disabled_x86_cap=
>s);
>+               }

Yes, but I meant using a __setup() or doing it in the no_apic case
in detect_init_APIC().

Re: [PATCH][2.5] Honour dont_enable_local_apic flag

[mikpe]

On 01 Jun 2003 12:26:56 -0400, Brian J. Murrell wrote:
>>    So what vendor/model CPU is used in the failure case?
>
>VMware 2.0.4.

Details, please. What does `cat /proc/cpuinfo` say?

My intention here is that we should be able to detect
this apparently broken "CPU" by its vendor/model and
clear cpu_has_apic for it.

Alternatively the no_apic label in detect_init_APIC()
could clear cpu_has_apic.

>> 3. What is the exact failure? Hang or crash?
>
>Normal boot until here:
>
>Using local APIC timer interrupts.
>calibrating APIC timer ...
>..... CPU clock speed is 1658.7651 MHz.
>..... host bus clock speed is 0.0000 MHz.
>cpu: 0, clocks: 0, slice: 0

Hmm, obviously a 2.4 kernel.
Looks like a hang in setup_APIC_timer(). My guess is that
the do loops in that procedure don't work if clocks==0
or the local APIC timer registers are frozen.

/Mikael

Re: [PATCH][2.5] Honour dont_enable_local_apic flag

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 2011 bytes --]

On Sun, 2003-06-01 at 19:08, mikpe@csd.uu.se wrote:
> 
> Details, please. What does `cat /proc/cpuinfo` say?

Oops.  Sorry.  Problem is that VMware just reports the underlying (host)
system's CPU, so it will vary from (real) machine to machine.  So for
instance, both my host system and my VMware virtual machine running on
it report:

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 6
model           : 4
model name      : AMD Athlon(tm) Processor
stepping        : 2
cpu MHz         : 796.626
cache size      : 256 KB
fdiv_bug        : no
hlt_bug         : no
f00f_bug        : no
coma_bug        : no
fpu             : yes
fpu_exception   : yes
cpuid level     : 1
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow
bogomips        : 1589.24

With the exception of the "apic" flag (and cpu MHz values and bogomips
values differ).  The above is from the host system.  The VMware virtual
system has the same flags minus the apic flag because the only way to
boot it such that I can get the /proc/cpuinfo is by using the nolapic
command arg in the patch I sent earlier.

> My intention here is that we should be able to detect
> this apparently broken "CPU" by its vendor/model and
> clear cpu_has_apic for it.

Not doable by CPU identification I don't think.

> Alternatively the no_apic label in detect_init_APIC()
> could clear cpu_has_apic.

Fair enough, but what would cause the code at the no_apic label to be
executed without any of the command line arg patches proposed?  We could
use Zwane's __setup() for nolapic and instead of doing:

if (dont_enable_local_apic)
	return -1;

in detect_init_APIC() we can do:

if (dont_enable_local_apic)
	goto no_apic;

> Hmm, obviously a 2.4 kernel.

Yes indeed.  This is for testing production systems, so I need stable
kernels.

b.

-- 
Brian J. Murrell <brian@interlinx.bc.ca>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH][2.5] Honour dont_enable_local_apic flag

[Zwane Mwaikambo]

I agree with doing the clear apic capability flag, Brian how does this 
fare? This patch alone should fix it.

Index: linux-2.5/arch/i386/kernel/apic.c
===================================================================
RCS file: /home/cvs/linux-2.5/arch/i386/kernel/apic.c,v
retrieving revision 1.54
diff -u -p -B -r1.54 apic.c
--- linux-2.5/arch/i386/kernel/apic.c	31 May 2003 19:01:05 -0000	1.54
+++ linux-2.5/arch/i386/kernel/apic.c	2 Jun 2003 03:50:31 -0000
@@ -609,7 +609,7 @@ static int __init detect_init_APIC (void
 
 	/* Disabled by DMI scan or kernel option? */
 	if (dont_enable_local_apic)
-		return -1;
+		goto no_apic;
 
 	/* Workaround for us being called before identify_cpu(). */
 	get_cpu_vendor(&boot_cpu_data);
@@ -665,6 +665,7 @@ static int __init detect_init_APIC (void
 	return 0;
 
 no_apic:
+	clear_bit(X86_FEATURE_APIC, boot_cpu_data.x86_capability);
 	printk("No local APIC present or hardware disabled\n");
 	return -1;
 }

-- 
function.linuxpower.ca

Re: [PATCH][2.5] Honour dont_enable_local_apic flag

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 1341 bytes --]

On Mon, 2003-06-02 at 00:50, Zwane Mwaikambo wrote:
> I agree with doing the clear apic capability flag,

Indeed.  I sure does seem to be the right way to go.

>  Brian how does this 
> fare? This patch alone should fix it.

It looks good and will try it out.  But before I do, should not:

	set_bit(X86_FEATURE_APIC, &disabled_x86_caps);

also be done?

> 
> Index: linux-2.5/arch/i386/kernel/apic.c
> ===================================================================
> RCS file: /home/cvs/linux-2.5/arch/i386/kernel/apic.c,v
> retrieving revision 1.54
> diff -u -p -B -r1.54 apic.c
> --- linux-2.5/arch/i386/kernel/apic.c	31 May 2003 19:01:05 -0000	1.54
> +++ linux-2.5/arch/i386/kernel/apic.c	2 Jun 2003 03:50:31 -0000
> @@ -609,7 +609,7 @@ static int __init detect_init_APIC (void
>  
>  	/* Disabled by DMI scan or kernel option? */
>  	if (dont_enable_local_apic)
> -		return -1;
> +		goto no_apic;
>  
>  	/* Workaround for us being called before identify_cpu(). */
>  	get_cpu_vendor(&boot_cpu_data);
> @@ -665,6 +665,7 @@ static int __init detect_init_APIC (void
>  	return 0;
>  
>  no_apic:
> +	clear_bit(X86_FEATURE_APIC, boot_cpu_data.x86_capability);
>  	printk("No local APIC present or hardware disabled\n");
>  	return -1;
>  }

b.

-- 
Brian J. Murrell <brian@interlinx.bc.ca>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: [PATCH][2.5] Honour dont_enable_local_apic flag

[mikpe]

Zwane Mwaikambo writes:
 > I agree with doing the clear apic capability flag, Brian how does this 
 > fare? This patch alone should fix it.
 > 
 > Index: linux-2.5/arch/i386/kernel/apic.c
 > ===================================================================
 > RCS file: /home/cvs/linux-2.5/arch/i386/kernel/apic.c,v
 > retrieving revision 1.54
 > diff -u -p -B -r1.54 apic.c
 > --- linux-2.5/arch/i386/kernel/apic.c	31 May 2003 19:01:05 -0000	1.54
 > +++ linux-2.5/arch/i386/kernel/apic.c	2 Jun 2003 03:50:31 -0000
 > @@ -609,7 +609,7 @@ static int __init detect_init_APIC (void
 >  
 >  	/* Disabled by DMI scan or kernel option? */
 >  	if (dont_enable_local_apic)
 > -		return -1;
 > +		goto no_apic;
 >  
 >  	/* Workaround for us being called before identify_cpu(). */
 >  	get_cpu_vendor(&boot_cpu_data);
 > @@ -665,6 +665,7 @@ static int __init detect_init_APIC (void
 >  	return 0;
 >  
 >  no_apic:
 > +	clear_bit(X86_FEATURE_APIC, boot_cpu_data.x86_capability);
 >  	printk("No local APIC present or hardware disabled\n");
 >  	return -1;
 >  }

Looks good to me. Add a __setup to set dont_enable_local_apic and this
should be sufficient for users of severely broken HW or emulations.

Re: [PATCH][2.5] Honour dont_enable_local_apic flag

[Zwane Mwaikambo]

On Mon, 2 Jun 2003 mikpe@csd.uu.se wrote:

> Looks good to me. Add a __setup to set dont_enable_local_apic and this
> should be sufficient for users of severely broken HW or emulations.

This should do it then;

Thanks,
	Zwane

Index: linux-2.5/Documentation/kernel-parameters.txt
===================================================================
RCS file: /home/cvs/linux-2.5/Documentation/kernel-parameters.txt,v
retrieving revision 1.36
diff -u -p -B -r1.36 kernel-parameters.txt
--- linux-2.5/Documentation/kernel-parameters.txt	2 Jun 2003 02:49:19 -0000	1.36
+++ linux-2.5/Documentation/kernel-parameters.txt	2 Jun 2003 15:20:46 -0000
@@ -624,6 +624,9 @@ running once the system is up.
 
 	nointroute	[IA-64]
 
+	nolapic		[IA-32, APIC]
+			Disable Local APIC.
+
 	nomce		[IA-32] Machine Check Exception
 
 	noresume	[SWSUSP] Disables resume and restore original swap space.
Index: linux-2.5/arch/i386/kernel/apic.c
===================================================================
RCS file: /home/cvs/linux-2.5/arch/i386/kernel/apic.c,v
retrieving revision 1.54
diff -u -p -B -r1.54 apic.c
--- linux-2.5/arch/i386/kernel/apic.c	31 May 2003 19:01:05 -0000	1.54
+++ linux-2.5/arch/i386/kernel/apic.c	2 Jun 2003 15:20:47 -0000
@@ -602,6 +602,14 @@ static void apic_pm_activate(void) { }
  */
 int dont_enable_local_apic __initdata = 0;
 
+static int __init nolapic_setup(char *str)
+{
+	dont_enable_local_apic = 1;
+	return 1;
+}
+
+__setup("nolapic", nolapic_setup);
+
 static int __init detect_init_APIC (void)
 {
 	u32 h, l, features;
@@ -609,7 +617,7 @@ static int __init detect_init_APIC (void
 
 	/* Disabled by DMI scan or kernel option? */
 	if (dont_enable_local_apic)
-		return -1;
+		goto no_apic;
 
 	/* Workaround for us being called before identify_cpu(). */
 	get_cpu_vendor(&boot_cpu_data);
@@ -665,6 +673,7 @@ static int __init detect_init_APIC (void
 	return 0;
 
 no_apic:
+	clear_bit(X86_FEATURE_APIC, boot_cpu_data.x86_capability);
 	printk("No local APIC present or hardware disabled\n");
 	return -1;
 }
-- 
function.linuxpower.ca

Re: [PATCH][2.5] Honour dont_enable_local_apic flag

[Brian J. Murrell]

[-- Attachment #1: Type: text/plain, Size: 218 bytes --]

On Mon, 2003-06-02 at 12:20, Zwane Mwaikambo wrote:
> 
> This should do it then;

For the record, this works perfect.

> Thanks,
> 	Zwane

Thanks Zwane.

b.

-- 
Brian J. Murrell <brian@interlinx.bc.ca>

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

[PATCH][2.5] Honour dont_enable_local_apic flag

[Zwane Mwaikambo]

This flag doesn't get honoured in all cases, we still try and frob the 
APIC in APIC_init_uniprocessor, Tested by Brian.

Alan can i also send you a patch to disable the local apic from the kernel 
command line? It would be dependent on this patch.

	Zwane

Index: linux-2.5/arch/i386/kernel/apic.c
===================================================================
RCS file: /home/cvs/linux-2.5/arch/i386/kernel/apic.c,v
retrieving revision 1.50
diff -u -p -B -r1.50 apic.c
--- linux-2.5/arch/i386/kernel/apic.c	30 May 2003 20:14:41 -0000	1.50
+++ linux-2.5/arch/i386/kernel/apic.c	31 May 2003 05:53:34 -0000
@@ -665,6 +665,7 @@ static int __init detect_init_APIC (void
 	return 0;
 
 no_apic:
+	dont_enable_local_apic = 1;
 	printk("No local APIC present or hardware disabled\n");
 	return -1;
 }
@@ -1127,6 +1128,9 @@ asmlinkage void smp_error_interrupt(void
  */
 int __init APIC_init_uniprocessor (void)
 {
+	if (dont_enable_local_apic)
+		return -1;
+
 	if (!smp_found_config && !cpu_has_apic)
 		return -1;
 

-- 
function.linuxpower.ca