[Kernel Panic] 3.10.10-rt7

[Kernel Panic] 3.10.10-rt7

[Frederich, Jens]

Hello,

I got a Kernel panic by some long time stress tests. Look here: http://dy.cx/pqP7M.

Furthermore I got sporadic some BUG messages:

[ 1128.358971] BUG: scheduling while atomic: swapper/0/0/0x00010002
[ 1128.358982] Preemption disabled at:[<ffffffff810dfb8a>] cpu_startup_entry+0x17a/0x300

Hardware
--------

My system is a i7 with 4 GB RAM congatec evalboard.

Setup
-----

On the system runs Linux RT (text mode) 3.10.10-rt7 with 1 GB RAM and a
Windows 7 KVM/QEMU (Qemu 1.6.1) guest with 3 GB RAM. Each system has it own
ethernet card. The Win 7 guest uses it per pci pass-through. The Win 7 is
pinned to core 2,3.

Scenario
--------

We sending UDP packages every 1 msec on the Linux side. The UDP sender process
has real time prio 80 and the ethernet driver 90. At the same time we are
stressing Win 7. Memory stress - allocate as much as possible and do memset(),
then CPU stress - high CPU load, then Kernel DPC stress - high load thread at
DPC level and ethernet stress - send UDP packages every milliseconds. I got the
panic after 45 minutes. It seems that is correlates with the ethernet stress.

Can anybody help me?

thanks
Jens

Re: [Kernel Panic] 3.10.10-rt7

[Jan Kiszka]

On 2013-11-20 13:26, Frederich, Jens wrote:
> Hello,
> 
> I got a Kernel panic by some long time stress tests. Look here: http://dy.cx/pqP7M.

Ah, that should be http://thread.gmane.org/gmane.linux.kernel/1564783

Paul, was there any follow-up patch on this topic?

Jan

> 
> Furthermore I got sporadic some BUG messages:
> 
> [ 1128.358971] BUG: scheduling while atomic: swapper/0/0/0x00010002
> [ 1128.358982] Preemption disabled at:[<ffffffff810dfb8a>] cpu_startup_entry+0x17a/0x300
> 
> Hardware
> --------
> 
> My system is a i7 with 4 GB RAM congatec evalboard.
> 
> Setup
> -----
> 
> On the system runs Linux RT (text mode) 3.10.10-rt7 with 1 GB RAM and a
> Windows 7 KVM/QEMU (Qemu 1.6.1) guest with 3 GB RAM. Each system has it own
> ethernet card. The Win 7 guest uses it per pci pass-through. The Win 7 is
> pinned to core 2,3.
> 
> Scenario
> --------
> 
> We sending UDP packages every 1 msec on the Linux side. The UDP sender process
> has real time prio 80 and the ethernet driver 90. At the same time we are
> stressing Win 7. Memory stress - allocate as much as possible and do memset(),
> then CPU stress - high CPU load, then Kernel DPC stress - high load thread at
> DPC level and ethernet stress - send UDP packages every milliseconds. I got the
> panic after 45 minutes. It seems that is correlates with the ethernet stress.
> 
> Can anybody help me?
> 
> thanks
> Jens

-- 
Siemens AG, Corporate Technology, CT RTC ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [Kernel Panic] 3.10.10-rt7

[Paul Gortmaker]

On 13-11-20 07:34 AM, Jan Kiszka wrote:
> On 2013-11-20 13:26, Frederich, Jens wrote:
>> Hello,
>>
>> I got a Kernel panic by some long time stress tests. Look here: http://dy.cx/pqP7M.
> 
> Ah, that should be http://thread.gmane.org/gmane.linux.kernel/1564783
> 
> Paul, was there any follow-up patch on this topic?

Yeah, from the rt-summit [ http://lwn.net/Articles/572740/ ] it is on me
to get those simple wait queue patches mainlined (and then the local_bh
stuff).  Once I've got that done, I'll look at spending some time on 3.12-rt
kvm patches -- in the meantime, you are right, the screenshot shows that his
3.10-rt needs the straightforward conversion of vpcu->wq to simple wait.

Paul.
--

> 
> Jan
> 
>>
>> Furthermore I got sporadic some BUG messages:
>>
>> [ 1128.358971] BUG: scheduling while atomic: swapper/0/0/0x00010002
>> [ 1128.358982] Preemption disabled at:[<ffffffff810dfb8a>] cpu_startup_entry+0x17a/0x300
>>
>> Hardware
>> --------
>>
>> My system is a i7 with 4 GB RAM congatec evalboard.
>>
>> Setup
>> -----
>>
>> On the system runs Linux RT (text mode) 3.10.10-rt7 with 1 GB RAM and a
>> Windows 7 KVM/QEMU (Qemu 1.6.1) guest with 3 GB RAM. Each system has it own
>> ethernet card. The Win 7 guest uses it per pci pass-through. The Win 7 is
>> pinned to core 2,3.
>>
>> Scenario
>> --------
>>
>> We sending UDP packages every 1 msec on the Linux side. The UDP sender process
>> has real time prio 80 and the ethernet driver 90. At the same time we are
>> stressing Win 7. Memory stress - allocate as much as possible and do memset(),
>> then CPU stress - high CPU load, then Kernel DPC stress - high load thread at
>> DPC level and ethernet stress - send UDP packages every milliseconds. I got the
>> panic after 45 minutes. It seems that is correlates with the ethernet stress.
>>
>> Can anybody help me?
>>
>> thanks
>> Jens
> 

RE: [Kernel Panic] 3.10.10-rt7

[Frederich, Jens]

>-----Original Message-----
>From: Paul Gortmaker [mailto:paul.gortmaker@windriver.com]
>Sent: Wednesday, November 20, 2013 4:46 PM
>To: Jan Kiszka; Frederich, Jens; linux-rt-users@vger.kernel.org
>Cc: linux-kernel@vger.kernel.org
>Subject: Re: [Kernel Panic] 3.10.10-rt7
>
>On 13-11-20 07:34 AM, Jan Kiszka wrote:
>> On 2013-11-20 13:26, Frederich, Jens wrote:
>>> Hello,
>>>
>>> I got a Kernel panic by some long time stress tests. Look here:
>http://dy.cx/pqP7M.
>>
>> Ah, that should be
>http://thread.gmane.org/gmane.linux.kernel/1564783
>>
>> Paul, was there any follow-up patch on this topic?
>
>Yeah, from the rt-summit [ http://lwn.net/Articles/572740/ ] it is on
>me
>to get those simple wait queue patches mainlined (and then the
>local_bh
>stuff).  Once I've got that done, I'll look at spending some time on
>3.12-rt
>kvm patches -- in the meantime, you are right, the screenshot shows
>that his
>3.10-rt needs the straightforward conversion of vpcu->wq to simple
>wait.
>

It's good to hear.

Paul, can you estimate when the kvm patches is ready, so that I can test it? 

thanks
Jens

>Paul.
>--
>
>>
>> Jan
>>
>>>
>>> Furthermore I got sporadic some BUG messages:
>>>
>>> [ 1128.358971] BUG: scheduling while atomic: swapper/0/0/0x00010002
>>> [ 1128.358982] Preemption disabled at:[<ffffffff810dfb8a>]
>cpu_startup_entry+0x17a/0x300
>>>
>>> Hardware
>>> --------
>>>
>>> My system is a i7 with 4 GB RAM congatec evalboard.
>>>
>>> Setup
>>> -----
>>>
>>> On the system runs Linux RT (text mode) 3.10.10-rt7 with 1 GB RAM
>and a
>>> Windows 7 KVM/QEMU (Qemu 1.6.1) guest with 3 GB RAM. Each system
>has it own
>>> ethernet card. The Win 7 guest uses it per pci pass-through. The
>Win 7 is
>>> pinned to core 2,3.
>>>
>>> Scenario
>>> --------
>>>
>>> We sending UDP packages every 1 msec on the Linux side. The UDP
>sender process
>>> has real time prio 80 and the ethernet driver 90. At the same time
>we are
>>> stressing Win 7. Memory stress - allocate as much as possible and
>do memset(),
>>> then CPU stress - high CPU load, then Kernel DPC stress - high load
>thread at
>>> DPC level and ethernet stress - send UDP packages every
>milliseconds. I got the
>>> panic after 45 minutes. It seems that is correlates with the
>ethernet stress.
>>>
>>> Can anybody help me?
>>>
>>> thanks
>>> Jens
>>