[Adeos-main] Problems with last adeos patches

[Adeos-main] Problems with last adeos patches

[Roberto Bucher]

Hi

I have some problems with the last patches for kernel 2.6.26.x and 2.6.27.x.

I found first the problem using the RTAI patch, but then I tryed the original 
adeos patch and the problem remains and I can reproduce it every time.

When I start scilab-gtk from a patched kernel, RTAI or Adeos, the system 
freezes completely.
This problem doesn't happen with a 2.6.24.x kernel or with a non patched 
2.6.27.x kernel.

Paolo Mantegazza and other RTAI users have the same problem, but Paolo for 
example is not able to reproduce it in the same matter. By him for example the 
problem happens sometimes after launching Acrobat Reader. On my system it 
happens sometimes when I try to download a file from the web too.

The probem persists with the last 2.6.27-x86-2.2-01.patch. I didn't try a 
2.6.25.x kernel, but it seems to be ok (after Paolo Mantegazza).

Any idea?

Thanks in advance

Roberto

-- 
University of Applied Sciences of Southern Switzerland
Dept. Innovative Technologies
CH-6928 Lugano-Manno
http://www.dti.supsi.ch/~bucher

Re: [Adeos-main] Problems with last adeos patches

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 1181 bytes --]

Roberto Bucher wrote:
> Hi
> 
> I have some problems with the last patches for kernel 2.6.26.x and 2.6.27.x.
> 
> I found first the problem using the RTAI patch, but then I tryed the original 
> adeos patch and the problem remains and I can reproduce it every time.
> 
> When I start scilab-gtk from a patched kernel, RTAI or Adeos, the system 
> freezes completely.
> This problem doesn't happen with a 2.6.24.x kernel or with a non patched 
> 2.6.27.x kernel.
> 
> Paolo Mantegazza and other RTAI users have the same problem, but Paolo for 
> example is not able to reproduce it in the same matter. By him for example the 
> problem happens sometimes after launching Acrobat Reader. On my system it 
> happens sometimes when I try to download a file from the web too.
> 
> The probem persists with the last 2.6.27-x86-2.2-01.patch. I didn't try a 
> 2.6.25.x kernel, but it seems to be ok (after Paolo Mantegazza).
> 
> Any idea?

Some questions: Could you provide your .config? How is your X configured
(which driver, any acceleration etc.), or did you see the problem also
without X? Did you try to catch potential oopses via serial console?

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: [Adeos-main] Problems with last adeos patches

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 722 bytes --]

Roberto Bucher wrote:
> Attached you can find my config. Basically, this is the same config the works 
> with version 2.6.24.7 (the only differencies are given by the new CONFIG_XXX of 
> the new linux version).
> 
> I have a nvidia card and I use the commercial NVIDIA driver 177.80 under X in 
> Ubuntu 8.10.

Thanks for the .config, will see if it triggers anything strange as well
(ideally in kvm...), though I don't have nvidia nor its binary drivers
around.

Still some questions remain: Have you seen the problem without the
nvidia driver, too? And is there a chance to capture the kernel console
via serial or netconsole? Seeing an oops would help a lot.

Jan

PS: Please keep the list in CC.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: [Adeos-main] Problems with last adeos patches

[Roberto Bucher]

Some results:

1) I changed the nvidia driver from 177.80 to 177.82: problem still remains
2) I changed the driver to "vesa" in the xorg.conf file: X starts but the 
problem remains!
3) I connect using ssh -X from another terminal to the RT linux machine. I 
can't reproduce the problem. When I start scilab on the RT machine the RT 
system freezes and the connection fails.

I installed the kernel 2.6.25.20 with patch  derived from adeos-ipipe-2.6.25-
x86-2.0-09.patch and the system seems to work fine.

Thanks in advance

Roberto 

On Thursday 11 December 2008 09:29:54 Jan Kiszka wrote:
> Roberto Bucher wrote:
> > Attached you can find my config. Basically, this is the same config the
> > works with version 2.6.24.7 (the only differencies are given by the new
> > CONFIG_XXX of the new linux version).
> >
> > I have a nvidia card and I use the commercial NVIDIA driver 177.80 under
> > X in Ubuntu 8.10.
>
> Thanks for the .config, will see if it triggers anything strange as well
> (ideally in kvm...), though I don't have nvidia nor its binary drivers
> around.
>
> Still some questions remain: Have you seen the problem without the
> nvidia driver, too? And is there a chance to capture the kernel console
> via serial or netconsole? Seeing an oops would help a lot.
>
> Jan
>
> PS: Please keep the list in CC.

-- 
University of Applied Sciences of Southern Switzerland
Dept. Innovative Technologies
CH-6928 Lugano-Manno
http://web.dti.supsi.ch/~bucher

Re: [Adeos-main] Problems with last adeos patches

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 734 bytes --]

Roberto Bucher wrote:
> Some results:
> 
> 1) I changed the nvidia driver from 177.80 to 177.82: problem still remains
> 2) I changed the driver to "vesa" in the xorg.conf file: X starts but the 
> problem remains!
> 3) I connect using ssh -X from another terminal to the RT linux machine. I 
> can't reproduce the problem. When I start scilab on the RT machine the RT 
> system freezes and the connection fails.
> 
> I installed the kernel 2.6.25.20 with patch  derived from adeos-ipipe-2.6.25-
> x86-2.0-09.patch and the system seems to work fine.

Another question (before I actually start looking at code): Are all test
boxes using > 1 CPUs? Or is the issue also present when there is only a
single CPU?.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]

Re: [Adeos-main] Problems with last adeos patches

[Roberto Bucher]

I have the problem on a Dual Core Machine (2xT9600 Processor). 

On a Centrino machine with 1 processor, the same kernel+patch seems to be ok.

Paolo Mantegazza has a similar problem on 3 Dual core machines.

Thanks

Roberto

On Saturday 13 December 2008 12:55:27 Jan Kiszka wrote:
> Roberto Bucher wrote:
> > Some results:
> >
> > 1) I changed the nvidia driver from 177.80 to 177.82: problem still
> > remains 2) I changed the driver to "vesa" in the xorg.conf file: X starts
> > but the problem remains!
> > 3) I connect using ssh -X from another terminal to the RT linux machine.
> > I can't reproduce the problem. When I start scilab on the RT machine the
> > RT system freezes and the connection fails.
> >
> > I installed the kernel 2.6.25.20 with patch  derived from
> > adeos-ipipe-2.6.25- x86-2.0-09.patch and the system seems to work fine.
>
> Another question (before I actually start looking at code): Are all test
> boxes using > 1 CPUs? Or is the issue also present when there is only a
> single CPU?.
>
> Jan

-- 
University of Applied Sciences of Southern Switzerland
Dept. Innovative Technologies
CH-6928 Lugano-Manno
http://web.dti.supsi.ch/~bucher

Re: [Adeos-main] Problems with last adeos patches

[Andre Marais]

I seem to have the same/a similar problem, only using Dual and Quad core
machines. Sometimes the machine(s) hangs hard, networking is lost as well.

I can only verify it happens when running Xorg on the machine(s), seems
quite stable in console only. Machine's got a Silicon Motion Lynx3DM
card that I last had issues on.

Running Xenomai 2.4.6.1, Adeos 2.2-01, Kernel 2.6.27.7. Will have to
wait for a .config for a week or so though :-(

Regards,
Andre


Roberto Bucher wrote:
> I have the problem on a Dual Core Machine (2xT9600 Processor). 
> 
> On a Centrino machine with 1 processor, the same kernel+patch seems to be ok.
> 
> Paolo Mantegazza has a similar problem on 3 Dual core machines.
> 
> Thanks
> 
> Roberto
> 
> On Saturday 13 December 2008 12:55:27 Jan Kiszka wrote:
>> Roberto Bucher wrote:
>>> Some results:
>>>
>>> 1) I changed the nvidia driver from 177.80 to 177.82: problem still
>>> remains 2) I changed the driver to "vesa" in the xorg.conf file: X starts
>>> but the problem remains!
>>> 3) I connect using ssh -X from another terminal to the RT linux machine.
>>> I can't reproduce the problem. When I start scilab on the RT machine the
>>> RT system freezes and the connection fails.
>>>
>>> I installed the kernel 2.6.25.20 with patch  derived from
>>> adeos-ipipe-2.6.25- x86-2.0-09.patch and the system seems to work fine.
>> Another question (before I actually start looking at code): Are all test
>> boxes using > 1 CPUs? Or is the issue also present when there is only a
>> single CPU?.
>>
>> Jan
> 


-- 
Reading, after a certain age, diverts the mind too much from its
creative pursuits. Any man who read too much and uses his own brain
too little falls into lazy habits of thinking.
 - Albert Einstein.

 -----------------------------------------------------------------------
 LEGAL DISCLAIMER:
 The views or representations contained in this message, whether express
 or implied, are those of the sender only, unless that sender expressly
 states them to be the view or representations of an entity or person,
 who shall be named by the sender and the sender shall state to
 represent. No liability shall otherwise attach to any other entity or
 person.
 -----------------------------------------------------------------------

Re: [Adeos-main] Problems with last adeos patches

[Roberto Bucher]

I installed a new machine with Dual Core processor, but with the same chipset 
as the working centrino mono processor machine (which has been working fine for 
a week with kernel 2.6.27.8+patch), and this PC freezes exactly like the other 
dual core laptop.

It seems that the problem happens only on system with CPU>1.

Best regards

Roberto

-- 
University of Applied Sciences of Southern Switzerland
Dept. Innovative Technologies
CH-6928 Lugano-Manno
http://web.dti.supsi.ch/~bucher

Re: [Adeos-main] Problems with last adeos patches

[Jan Kiszka]

Roberto Bucher wrote:
> I installed a new machine with Dual Core processor, but with the same chipset 
> as the working centrino mono processor machine (which has been working fine for 
> a week with kernel 2.6.27.8+patch), and this PC freezes exactly like the other 
> dual core laptop.
> 
> It seems that the problem happens only on system with CPU>1.

I'm currently reviewing the x86 2.6.27 patch /wrt to potential SMP
issues. One thing I came across is our old friend, the MTRR subsystem.
There are a few local_irq_disable_hw spots, some are even
local_irq_disable_full, but others are still plain vanilla, and I'm not
100% sure that this is correct. I will check this carefully, probably
coming up with some correction patch.

In the meantime, could you check if disabling CONFIG_MTRR changes the
picture you (or anyone else with a similar problem)?

TiA,
Jan

-- 
Siemens AG, Corporate Technology, CT SE 26
Corporate Competence Center Embedded Linux

Re: [Adeos-main] Problems with last adeos patches

[Roberto Bucher]

I'll try it ASAP.

Roberto

On Monday 15 December 2008 18:07:29 Jan Kiszka wrote:
> Roberto Bucher wrote:
> > I installed a new machine with Dual Core processor, but with the same
> > chipset as the working centrino mono processor machine (which has been
> > working fine for a week with kernel 2.6.27.8+patch), and this PC freezes
> > exactly like the other dual core laptop.
> >
> > It seems that the problem happens only on system with CPU>1.
>
> I'm currently reviewing the x86 2.6.27 patch /wrt to potential SMP
> issues. One thing I came across is our old friend, the MTRR subsystem.
> There are a few local_irq_disable_hw spots, some are even
> local_irq_disable_full, but others are still plain vanilla, and I'm not
> 100% sure that this is correct. I will check this carefully, probably
> coming up with some correction patch.
>
> In the meantime, could you check if disabling CONFIG_MTRR changes the
> picture you (or anyone else with a similar problem)?
>
> TiA,
> Jan

-- 
University of Applied Sciences of Southern Switzerland
Dept. Innovative Technologies
CH-6928 Lugano-Manno
http://web.dti.supsi.ch/~bucher

Re: [Adeos-main] Problems with last adeos patches

[Roberto Bucher]

I've unchecked the CONFIG_MTRR flag in kernel, but the behaviour is exactly the 
same (scilab->freeze)

Roberto

On Monday 15 December 2008 18:07:29 Jan Kiszka wrote:
> Roberto Bucher wrote:
> > I installed a new machine with Dual Core processor, but with the same
> > chipset as the working centrino mono processor machine (which has been
> > working fine for a week with kernel 2.6.27.8+patch), and this PC freezes
> > exactly like the other dual core laptop.
> >
> > It seems that the problem happens only on system with CPU>1.
>
> I'm currently reviewing the x86 2.6.27 patch /wrt to potential SMP
> issues. One thing I came across is our old friend, the MTRR subsystem.
> There are a few local_irq_disable_hw spots, some are even
> local_irq_disable_full, but others are still plain vanilla, and I'm not
> 100% sure that this is correct. I will check this carefully, probably
> coming up with some correction patch.
>
> In the meantime, could you check if disabling CONFIG_MTRR changes the
> picture you (or anyone else with a similar problem)?
>
> TiA,
> Jan

-- 
University of Applied Sciences of Southern Switzerland
Dept. Innovative Technologies
CH-6928 Lugano-Manno
http://web.dti.supsi.ch/~bucher

Re: [Adeos-main] Problems with last adeos patches

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 543 bytes --]

Roberto Bucher wrote:
> I've unchecked the CONFIG_MTRR flag in kernel, but the behaviour is exactly the 
> same (scilab->freeze)
> 

Mmmh, would have been too easy.

Next try: Gilles posted the attached fix for a quad system boot issue
yesterday (also available as adeos-ipipe-2.6.27-x86-2.2-02.patch). Maybe
it has some effect for you as well.

Nevertheless, I will continue to dig into the patch as I see some SMP
affinity issue for Linux IRQs.

Jan

-- 
Siemens AG, Corporate Technology, CT SE 26
Corporate Competence Center Embedded Linux

[-- Attachment #2: Nachricht als Anhang --]
[-- Type: message/rfc822, Size: 6216 bytes --]

From: Gilles Chanteperdrix <gilles.chanteperdrix@xenomai.org>
To: Thomas Schaefer <thomas_schaefer_canada@domain.hid>
Cc: jan.kiszka@domain.hid, xenomai@xenomai.org
Subject: Re: [Xenomai-help] BUG: soft lockup
Date: Mon, 15 Dec 2008 18:59:31 +0000
Message-ID: <4946A913.1000201@domain.hid>

Thomas Schaefer wrote:
> Hi,
> 
>> Date: Mon, 15 Dec 2008 17:09:25 +0100
>> From: jan.kiszka@domain.hid
>> To: thomas_schaefer_canada@domain.hid
>> CC: xenomai@xenomai.org
>> Subject: Re: BUG: soft lockup
>>
>> Hi,
>>
>> please don't post HTML mails, they are not parsable unless one switches
>> HTML mode on for mail reading (which at least I dislike to do).
>>
>  
> Sorry about that.
> 
>> Thomas Schaefer wrote:
>>> Hi,
>>>
>>> I am trying to use kernel version 2.6.27.8 and the latest stable version
>>> of xenomai 2.4.6.1 on a quad core xeon 5400.
>>>
>>> The kernel seems to load OK.
>>> [ 1.306673] I-pipe: Domain Xenomai registered.
>>> [ 1.311242] Xenomai: hal/x86_64 started.
>>> [ 1.316744] Xenomai: real-time nucleus v2.4.6 (Bamboo) loaded.
>>> [ 1.322786] Xenomai: starting native API services.
>>> [ 1.327580] Xenomai: starting POSIX services.
>>> [ 1.331973] Xenomai: starting RTDM services.
>>>
>>> But as soon as init starts the PC hangs in an endlass loop and I get:
>>> [ 69.793501] BUG: soft lockup - CPU#3 stuck for 61s! [init:1]
>>> [ 69.793501] Modules linked in:
>>> [ 69.793501] CPU 3:
>>> [ 69.793501] Modules linked in:
>>> [ 69.793501] Pid: 1, comm: init Not tainted 2.6.27.8 #1
>>> [ 69.793501] RIP: 0010:[] []
>>> native_flush
>>> _tlb_others+0xa5/0xe0
>>> [ 69.793501] RSP: 0000:ffff88007f849c58 EFLAGS: 00000202
>>> [ 69.793501] RAX: 0000000000000000 RBX: 0000000000000000 RCX:
>>> 0000000000000000
>>> [ 69.793501] RDX: 0000000000000000 RSI: 0000000000000000 RDI:
>>> 0000000000000000
>>> [ 69.793501] RBP: ffff88007f840000 R08: 0000000000000000 R09:
>>> 0000000000000000
>>> [ 69.793501] R10: 0000000000000000 R11: 0000000000000000 R12:
>>> 0000000000000000
>>> [ 69.793501] R13: 0000000000000000 R14: 0000000000000000 R15:
>>> 0000000000000000
>>> [ 69.793501] FS: 00007fbcbf7486e0(0000) GS:ffff88007f806a80(0000)
>>> knlGS:00000
>>> 00000000000
>>> [ 69.793501] CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
>>> [ 69.793501] CR2: 0000000000672b08 CR3: 000000007eb84000 CR4:
>>> 00000000000006e0
>>> [ 69.793501] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
>>> 0000000000000000
>>> [ 69.793501] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
>>> 0000000000000400
>>> [ 69.793501]
>>> [ 69.793501] Call Trace:
>>> [ 69.793501]
>> Really no Stack trace available?
>>
> 
> Yes that's all there is.
> 
>>> The kernel runs fine without the Xenomai patch.
>>> I also tried the version in the current svn repository with the same result.
>>> Please let me know you what I can do to help to fix this or what
>>> additional info I should provide.
>> Does this problem also occurs when you
>> - disable CONFIG_XENOMAI?
>> - boot with maxcpus=2, and then maxcpus=1?
>  
> Disabling CONFIG_XENOMAI doesn't help but using maxcpus=2 or maxcpus=1 makes it boot again.

Could you try the following patch?

--- linux/include/asm-x86/irq_vectors.h~	2008-12-09 18:05:38.000000000 +0100
+++ linux/include/asm-x86/irq_vectors.h	2008-12-15 19:48:53.000000000 +0100
@@ -80,7 +80,7 @@
 #ifdef CONFIG_IPIPE
 #define INVALIDATE_TLB_VECTOR_END	0xf2
 #define INVALIDATE_TLB_VECTOR_START	0xf0	/* f0-f2 used for TLB flush */
-#define NUM_INVALIDATE_TLB_VECTORS	4	/* f3-f7 used by I-pipe */
+#define NUM_INVALIDATE_TLB_VECTORS	3	/* f3-f7 used by I-pipe */
 #else /* !CONFIG_IPIPE */
 #define INVALIDATE_TLB_VECTOR_END	0xf7
 #define INVALIDATE_TLB_VECTOR_START	0xf0	/* f0-f7 used for TLB flush */

-- 
                                                 Gilles.

Re: [Adeos-main] Problems with last adeos patches

[Roberto Bucher]

The patch from Gilles seems to be promising. I started scilab 10 times without 
freezing the PC!!!

I need more tests (and time) to be shure that the system is ok. I'll give you 
other feedback tomorrow. I'll try to work for some time with the present 
kernel.

Thanks 

Roberto

On Tuesday 16 December 2008 10:20:43 Jan Kiszka wrote:
> Roberto Bucher wrote:
> > I've unchecked the CONFIG_MTRR flag in kernel, but the behaviour is
> > exactly the same (scilab->freeze)
>
> Mmmh, would have been too easy.
>
> Next try: Gilles posted the attached fix for a quad system boot issue
> yesterday (also available as adeos-ipipe-2.6.27-x86-2.2-02.patch). Maybe
> it has some effect for you as well.
>
> Nevertheless, I will continue to dig into the patch as I see some SMP
> affinity issue for Linux IRQs.
>
> Jan

-- 
University of Applied Sciences of Southern Switzerland
Dept. Innovative Technologies
CH-6928 Lugano-Manno
http://web.dti.supsi.ch/~bucher

Re: [Adeos-main] Problems with last adeos patches

[Jan Kiszka]

[-- Attachment #1: Type: text/plain, Size: 844 bytes --]

Parashkev Penev wrote:
> Hi,
> 
> I also have this issue. I am using the 2.6.26 kernel + adeos. The
> .config file is the default from the CentOS 5.2/Redhat 5.2 with ACPI,
> CPU frequency scaling, and APM turned off. The video card is an Nvidia,
> using the nv module on one of the machines. the other one has an ATI
> using the ati-glfrx. I haven't tried with vesa yet.

Cross-checking with an open source driver would be good, though seeing
the issue with two likely quite different binary blobs suggest that they
are likely not to blame here.

> 
> Let me know if you ned something else. (Those are laptops, thus no
> serial console).

netconsole? Also here: Getting hold of an oops is very important,
specifically if the problem turns out to be not easily reproducible for us.

Jan

PS: Please keep the list in CC.


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 258 bytes --]

Re: [Adeos-main] Problems with last adeos patches

[Roberto Bucher]

Ok! I'll try the following steps:

a) login from a network terminal with my system in graphical mode
b) login from a network terminal with my system in non graphical mode
c) modify the X11.org file to have for example vesa driver

Some additional info:

- kernel 2.6.25.X seems to be ok (it doesn't have the same problem with 
scilab)
- kernel 2.6.27.8 with patch seems to be ok on a UP machine with Intel 
graphics card

Best regards

Roberto

On Thursday 11 December 2008 09:30:47 Jan Kiszka wrote:
> Parashkev Penev wrote:
> > Hi,
> >
> > I also have this issue. I am using the 2.6.26 kernel + adeos. The
> > .config file is the default from the CentOS 5.2/Redhat 5.2 with ACPI,
> > CPU frequency scaling, and APM turned off. The video card is an Nvidia,
> > using the nv module on one of the machines. the other one has an ATI
> > using the ati-glfrx. I haven't tried with vesa yet.
>
> Cross-checking with an open source driver would be good, though seeing
> the issue with two likely quite different binary blobs suggest that they
> are likely not to blame here.
>
> > Let me know if you ned something else. (Those are laptops, thus no
> > serial console).
>
> netconsole? Also here: Getting hold of an oops is very important,
> specifically if the problem turns out to be not easily reproducible for us.
>
> Jan
>
> PS: Please keep the list in CC.

-- 
University of Applied Sciences of Southern Switzerland
Dept. Innovative Technologies
CH-6928 Lugano-Manno
http://web.dti.supsi.ch/~bucher