[Opps] Invalid opcode

[Opps] Invalid opcode

[S.Çağlar Onur]

[-- Attachment #1: Type: text/plain, Size: 779 bytes --]

Hi;

2.6.18, 2.6.18.1 and 2.6.18.2 still panics randomly (it seems this is not 
related to smpreplacament bug solved in .2) in VmWare and Microsoft Virtual 
PC and in order to confirm this bug is not our distro specific i downloaded 
and tried latest OpenSuse also [1]  and [2] are screens captured by vmware 
but exact same panic occurs in Virtual PC as reported to us in [3]. I CC'ed 
previous threads receivers also.

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic.png
[2] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic.png
[3] http://bugs.pardus.org.tr/show_bug.cgi?id=3804

Cheers
-- 
S.Çağlar Onur <caglar@pardus.org.tr>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [Opps] Invalid opcode

[Andi Kleen]

On Sunday 05 November 2006 14:07, S.Çağlar Onur wrote:
> Hi;
> 
> 2.6.18, 2.6.18.1 and 2.6.18.2 still panics randomly (it seems this is not 
> related to smpreplacament bug solved in .2) 

How do you know this? 

And does it still happen in 2.6.19-rc4?

> in VmWare and Microsoft Virtual  
> PC and in order to confirm this bug is not our distro specific i downloaded 
> and tried latest OpenSuse also [1]  and [2] are screens captured by vmware 
> but exact same panic occurs in Virtual PC as reported to us in [3]. 

Always the same BUG()?

> I CC'ed  
> previous threads receivers also.
> 
> [1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic.png
> [2] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic.png

There is just some rolling Turkish text there.

-Andi

Re: [Opps] Invalid opcode

[S.Çağlar Onur]

[-- Attachment #1: Type: text/plain, Size: 992 bytes --]

05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı: 
> How do you know this?

Just guessing, if im not wrong panics occur after SMP alternative switching 
code done its job.

> And does it still happen in 2.6.19-rc4?

Will try

> > in VmWare and Microsoft Virtual
> > PC and in order to confirm this bug is not our distro specific i
> > downloaded and tried latest OpenSuse also [1]  and [2] are screens
> > captured by vmware but exact same panic occurs in Virtual PC as reported
> > to us in [3].
>
> Always the same BUG()?

Yes, same bug

> There is just some rolling Turkish text there.

Ah im sorry here is the correct links :(

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic_on_opensuse.png
[2] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic_on_pardus.png

Cheers
-- 
S.Çağlar Onur <caglar@pardus.org.tr>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [Opps] Invalid opcode

[Jan Engelhardt]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 311 bytes --]

>05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı: 
>> How do you know this?
>
>Just guessing, if im not wrong panics occur after SMP alternative switching 
>code done its job.

Possibly compiled a kernel with instructions your processor does not 
support? Come to think of cmov...



	-`J'
-- 

Re: [Opps] Invalid opcode

[S.Çağlar Onur]

[-- Attachment #1: Type: text/plain, Size: 674 bytes --]

05 Kas 2006 Paz 19:25 tarihinde, Jan Engelhardt şunları yazmıştı: 
> >05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı:
> >> How do you know this?
> >
> >Just guessing, if im not wrong panics occur after SMP alternative
> > switching code done its job.
>
> Possibly compiled a kernel with instructions your processor does not
> support? Come to think of cmov...

That machine is a Intel(R) Pentium(R) 4 CPU 3.00GHz so if im not wrong cmov is 
supported on that processor

-- 
S.Çağlar Onur <caglar@pardus.org.tr>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [Opps] Invalid opcode

[Andi Kleen]

On Sunday 05 November 2006 18:17, S.Çağlar Onur wrote:
> 05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı: 
> > How do you know this?
> 
> Just guessing, if im not wrong panics occur after SMP alternative switching 
> code done its job.

Can you test with "noreplacement" to make sure?

Anyways I suspect we're just getting back some variant of the old CPU setup race. 

Normally CPU booting in Linux follows a special "cpu hotplug" state machine,
but for historical reasons i386 only implements one state of  this. At one
point we had a similar bug (but not in the callback on CPU #0, but in
the timer on newly booted CPU). I don't see currently how it can happen
(but i haven't thought very deeply about it yet)

Probably your timing is just unlucky on those simulators.

Previously we avoided converting i386 cpu bootup fully to the new state 
machine because it is very fragile, but it's possible that there
is no other choice than to do it properly. Or maybe another kludge
is possible.

-Andi

Re: [Opps] Invalid opcode

[S.Çağlar Onur]

[-- Attachment #1: Type: text/plain, Size: 1334 bytes --]

Hi;

05 Kas 2006 Paz 20:57 tarihinde, Andi Kleen şunları yazmıştı: 
> Can you test with "noreplacement" to make sure?

I sorry for not to mention that, i tried noreplacement before reporting which 
is also ends up with same panic.

> Anyways I suspect we're just getting back some variant of the old CPU setup
> race.
>
> Normally CPU booting in Linux follows a special "cpu hotplug" state
> machine, but for historical reasons i386 only implements one state of 
> this. At one point we had a similar bug (but not in the callback on CPU #0,
> but in the timer on newly booted CPU). I don't see currently how it can
> happen (but i haven't thought very deeply about it yet)
>
> Probably your timing is just unlucky on those simulators.

Hmm, Novell bugzilla seems has similiar issues, 
https://bugzilla.novell.com/show_bug.cgi?id=204647 and its duplicated ones 
gaves same or similiar panic outputs.

> Previously we avoided converting i386 cpu bootup fully to the new state
> machine because it is very fragile, but it's possible that there
> is no other choice than to do it properly. Or maybe another kludge
> is possible.

Cheers
-- 
S.Çağlar Onur <caglar@pardus.org.tr>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [Opps] Invalid opcode

[Zachary Amsden]

S.Çağlar Onur wrote:
> Hmm, Novell bugzilla seems has similiar issues, 
> https://bugzilla.novell.com/show_bug.cgi?id=204647 and its duplicated ones 
> gaves same or similiar panic outputs.
>
>   
>> Previously we avoided converting i386 cpu bootup fully to the new state
>> machine because it is very fragile, but it's possible that there
>> is no other choice than to do it properly. Or maybe another kludge
>> is possible.
>>     

Yes, this is some kind of softirq race during init.

Re: [Opps] Invalid opcode

[Andi Kleen]

On Monday 06 November 2006 00:13, Zachary Amsden wrote:
> S.Çağlar Onur wrote:
> > Hmm, Novell bugzilla seems has similiar issues, 
> > https://bugzilla.novell.com/show_bug.cgi?id=204647 and its duplicated ones 
> > gaves same or similiar panic outputs.
> >
> >   
> >> Previously we avoided converting i386 cpu bootup fully to the new state
> >> machine because it is very fragile, but it's possible that there
> >> is no other choice than to do it properly. Or maybe another kludge
> >> is possible.
> >>     
> 
> Yes, this is some kind of softirq race during init.

Yes, the callbacks run at the wrong time. Unlike modern architectures
i386 doesn't do callback cpu boot callback repeat, but boot all cpus then callback.

But the strange thing is that the BP hits it. Normally the new CPU
hit it because it tried to run a timer interrupt before the callback
ran and initialized all the per CPU state (this happened often
when dual core CPUs were first introduced for some reason)

In this case it looks like the AP managed to queue a tasklet before 
the cpu up callback runs.

I suppose we'll either need to convert i386 really over to standard
cpu_up() or add some additional spinlocks to stop the APs with interrupts
off before the callbacks start to run.

-Andi

Fix for OpenSUSE kernel bug (was Re: [Opps] Invalid opcode)

[Zachary Amsden]

[-- Attachment #1: Type: text/plain, Size: 1416 bytes --]

S.Çağlar Onur wrote:
> 05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı: 
>   
>> How do you know this?
>>     
>
> Just guessing, if im not wrong panics occur after SMP alternative switching 
> code done its job.
>
>   
>> And does it still happen in 2.6.19-rc4?
>>     
>
> Will try
>
>   
>>> in VmWare and Microsoft Virtual
>>> PC and in order to confirm this bug is not our distro specific i
>>> downloaded and tried latest OpenSuse also [1]  and [2] are screens
>>> captured by vmware but exact same panic occurs in Virtual PC as reported
>>> to us in [3].
>>>       
>> Always the same BUG()?
>>     
>
> Yes, same bug
>
>   
>> There is just some rolling Turkish text there.
>>     
>
> Ah im sorry here is the correct links :(
>
> [1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic_on_opensuse.png
> [2] http://cekirdek.pardus.org.tr/~caglar/2.6.18/panic_on_pardus.png
>
> Cheers
>   

I'm proposing this as a fix for your bug. Having tasklets scheduled 
before softirqd gets to run might be somewhat backwards, but there is 
nothing I can find wrong about it from a correctness point of view. 
Better to boot the kernel even when compiled with bug checking on, I think.

This bug started becoming apparent in 2.6.18 because of some rework with 
the CPU hotplug code, but in theory, it exists at least all the way back 
to 2.6.10, which is as far as I looked backwards in time.

Zach

[-- Attachment #2: fix-softirq-race --]
[-- Type: text/plain, Size: 1619 bytes --]

It is possible to have tasklets get scheduled before softirqd has had
a chance to spawn on all CPUs.  This is totally harmless; after success
during action CPU_UP_PREPARE, action CPU_ONLINE will be called, which
immediately wakes softirqd on the appropriate CPU to process the already
pending tasklets.  So there is no danger of having a missed wakeup for
any tasklets that were already pending.

In particular, i386 is affected by this during startup, and is visible when
using a very large initrd; during the time it takes for the initrd to be
decompressed, a timer IRQ can come in and schedule RCU callbacks.  It is also
possible that resending of a hardware IRQ via a softirq triggers the same bug.

Because of different timing conditions, this shows up in all emulators
and virtual machines tested, including Xen, VMware, Virtual PC, and Qemu.
It is also possible to trigger on native hardware with a large enough initrd,
although I don't have a reliable case demonstrating that.

Signed-off-by: Zachary Amsden <zach@vmware.com>

Index: linux-2.6.18/kernel/softirq.c
===================================================================
--- linux-2.6.18.orig/kernel/softirq.c	2006-11-10 14:44:39.000000000 -0800
+++ linux-2.6.18/kernel/softirq.c	2006-11-29 22:19:36.000000000 -0800
@@ -574,8 +574,6 @@ static int __cpuinit cpu_callback(struct
 
 	switch (action) {
 	case CPU_UP_PREPARE:
-		BUG_ON(per_cpu(tasklet_vec, hotcpu).list);
-		BUG_ON(per_cpu(tasklet_hi_vec, hotcpu).list);
 		p = kthread_create(ksoftirqd, hcpu, "ksoftirqd/%d", hotcpu);
 		if (IS_ERR(p)) {
 			printk("ksoftirqd for %i failed\n", hotcpu);

Re: Fix for OpenSUSE kernel bug (was Re: [Opps] Invalid opcode)

[S.Çağlar Onur]

[-- Attachment #1: Type: text/plain, Size: 876 bytes --]

Hi;

30 Kas 2006 Per 08:44 tarihinde, Zachary Amsden şunları yazmıştı: 
> I'm proposing this as a fix for your bug. Having tasklets scheduled
> before softirqd gets to run might be somewhat backwards, but there is
> nothing I can find wrong about it from a correctness point of view.
> Better to boot the kernel even when compiled with bug checking on, I think.
>
> This bug started becoming apparent in 2.6.18 because of some rework with
> the CPU hotplug code, but in theory, it exists at least all the way back
> to 2.6.10, which is as far as I looked backwards in time.

I cannot reproduce that opps with 2.6.18.4 + your patch any longer, so at 
least works for me :), thanks

Cheers
-- 
S.Çağlar Onur <caglar@pardus.org.tr>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [Opps] Invalid opcode

[S.Çağlar Onur]

[-- Attachment #1: Type: text/plain, Size: 361 bytes --]

05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı: 
> And does it still happen in 2.6.19-rc4?

Sorry for delayed test result, i cannot reproduce this panic with 2.6.19-rc5

-- 
S.Çağlar Onur <caglar@pardus.org.tr>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: [Opps] Invalid opcode

[Andi Kleen]

On Sunday 12 November 2006 03:39, S.Çağlar Onur wrote:
> 05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı: 
> > And does it still happen in 2.6.19-rc4?
> 
> Sorry for delayed test result, i cannot reproduce this panic with 2.6.19-rc5

It's probably still there, just hopefully it won't be release critical
for .19 then.

At some point it has to be fixed properly by converting i386 to the 
new hotplug architecture i suppose.

-Andi

Re: [Opps] Invalid opcode

[Zachary Amsden]

S.Çağlar Onur wrote:
> 05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı: 
>   
>> And does it still happen in 2.6.19-rc4?
>>     
>
> Sorry for delayed test result, i cannot reproduce this panic with 2.6.19-rc5
>   

I would like to find the exact cause of the problem; I suspect, as does 
Andi, that it could just be dormant. You had problems still with 
2.6.18.latest, correct? If I can find the cause, I would like to get a 
fix into 2.6.18-stable if possible. I think you already sent me the 
reproducing kernel config, but I seem to have misplaced it. Could you 
resend? I should have some time to look at this early this week.

Thanks,

Zach

Re: [Opps] Invalid opcode

[Andi Kleen]

On Monday 13 November 2006 05:20, Zachary Amsden wrote:
> S.Çağlar Onur wrote:
> > 05 Kas 2006 Paz 18:40 tarihinde, Andi Kleen şunları yazmıştı: 
> >   
> >> And does it still happen in 2.6.19-rc4?
> >>     
> >
> > Sorry for delayed test result, i cannot reproduce this panic with 2.6.19-rc5
> >   
> 
> I would like to find the exact cause of the problem; 

It's all related to i386's abuse of the cpu hotplug state machine.
Eventually that needs to be fixed properly like it was on x86-64
(I didn't dare touch i386 back then because this code is so fragile on old
hardware) 

-Andi

Re: [Opps] Invalid opcode

[S.Çağlar Onur]

[-- Attachment #1: Type: text/plain, Size: 889 bytes --]

13 Kas 2006 Pts 06:20 tarihinde, Zachary Amsden şunları yazmıştı: 
> I would like to find the exact cause of the problem; I suspect, as does
> Andi, that it could just be dormant. You had problems still with
> 2.6.18.latest, correct? If I can find the cause, I would like to get a
> fix into 2.6.18-stable if possible. I think you already sent me the
> reproducing kernel config, but I seem to have misplaced it. Could you
> resend? I should have some time to look at this early this week.

Sorry for late reply, [1] is the kernel config i used and if you want i can 
provide ~5mb iso contains that kernel and its 26.19-rc5 version to test?

[1] http://cekirdek.pardus.org.tr/~caglar/2.6.18/config.2.6.18
-- 
S.Çağlar Onur <caglar@pardus.org.tr>
http://cekirdek.pardus.org.tr/~caglar/

Linux is like living in a teepee. No Windows, no Gates and an Apache in house!

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]