* [Xenomai-help] 2.5.2 + 2.6.32 patched kernel causes system reset
@ 2010-07-06 18:56 Peter Soetens
2010-07-06 19:11 ` Jan Kiszka
2010-07-06 19:16 ` Gilles Chanteperdrix
0 siblings, 2 replies; 4+ messages in thread
From: Peter Soetens @ 2010-07-06 18:56 UTC (permalink / raw)
To: xenomai-help
[-- Attachment #1: Type: text/plain, Size: 1133 bytes --]
Hi,
We have a single-core Pentium M industrial PC equiped with Xenomai
2.5.2 patched vanilla Linux 2.6.32.7.
We have two builds of the kernel: one that causes a reliable system
reset when starting a 'heavy' Orocos application (lot of thread
creation/deletions), the other is rock solid. We think it is related
to thread creation in the native skin, since we could at least twice
cause the reset by just creating a native thread (that was meant to
block on a semaphore) but this was hard to reproduce. Stack size was
set to 128k, but only trivial code was executed.We had the impression
that enlarging the stack size to 256k caused less resets, but it was
still very frequent. Setting the stack size too low caused segfaults
in the app, not a kernel crash.
The stable config and the diffs between both configs are in
attachment. Read it as: apply this patch to make it unstable. Any
clues what is the most likely suspect ? Could the xenomai watchdog
cause the reset ?
Thanks for any pointers, we're prepared to bisect this, if any one
knows a tool to do a bisect on a set of config options, we'll be glad
to use it.
Cheers,
Peter
[-- Attachment #2: stable-to-crash-config.diff --]
[-- Type: text/x-patch, Size: 8770 bytes --]
--- /boot/config-2.6.32.7-xenomai-2.5.2 2010-04-20 00:03:51.000000000 +0200
+++ /boot/config-2.6.32.7-xenomai-2.5.2-bas 2010-07-05 16:45:34.000000000 +0200
@@ -1,7 +1,7 @@
#
# Automatically generated make config: don't edit
-# Linux kernel version: 2.6.32.7-xenomai-2.5.2
-# Tue Apr 20 00:03:51 2010
+# Linux kernel version: 2.6.32.7-xenomai-2.5.2-bas
+# Mon Jul 5 16:45:34 2010
#
# CONFIG_64BIT is not set
CONFIG_X86_32=y
@@ -13,7 +13,6 @@
CONFIG_GENERIC_CMOS_UPDATE=y
CONFIG_CLOCKSOURCE_WATCHDOG=y
CONFIG_GENERIC_CLOCKEVENTS=y
-CONFIG_GENERIC_CLOCKEVENTS_BROADCAST=y
CONFIG_LOCKDEP_SUPPORT=y
CONFIG_STACKTRACE_SUPPORT=y
CONFIG_HAVE_LATENCYTOP_SUPPORT=y
@@ -46,11 +45,6 @@
CONFIG_GENERIC_HARDIRQS=y
CONFIG_GENERIC_HARDIRQS_NO__DO_IRQ=y
CONFIG_GENERIC_IRQ_PROBE=y
-CONFIG_GENERIC_PENDING_IRQ=y
-CONFIG_USE_GENERIC_SMP_HELPERS=y
-CONFIG_X86_32_SMP=y
-CONFIG_X86_HT=y
-CONFIG_X86_TRAMPOLINE=y
CONFIG_X86_32_LAZY_GS=y
CONFIG_KTIME_SCALAR=y
CONFIG_DEFCONFIG_LIST="/lib/modules/$UNAME_RELEASE/.config"
@@ -60,6 +54,7 @@
# General setup
#
CONFIG_EXPERIMENTAL=y
+CONFIG_BROKEN_ON_SMP=y
CONFIG_LOCK_KERNEL=y
CONFIG_INIT_ENV_ARG_LIMIT=32
CONFIG_LOCALVERSION=""
@@ -165,7 +160,6 @@
CONFIG_PROFILING=y
CONFIG_TRACEPOINTS=y
CONFIG_OPROFILE=m
-# CONFIG_OPROFILE_IBS is not set
# CONFIG_OPROFILE_EVENT_MULTIPLEX is not set
CONFIG_HAVE_OPROFILE=y
# CONFIG_KPROBES is not set
@@ -193,7 +187,6 @@
CONFIG_MODULE_FORCE_UNLOAD=y
CONFIG_MODVERSIONS=y
# CONFIG_MODULE_SRCVERSION_ALL is not set
-CONFIG_STOP_MACHINE=y
CONFIG_BLOCK=y
CONFIG_LBDAF=y
CONFIG_BLK_DEV_BSG=y
@@ -231,8 +224,16 @@
CONFIG_XENO_OPT_SEM_HEAPSZ=12
CONFIG_XENO_OPT_GLOBAL_SEM_HEAPSZ=12
CONFIG_XENO_OPT_STATS=y
-# CONFIG_XENO_OPT_DEBUG is not set
-# CONFIG_XENO_OPT_SHIRQ is not set
+CONFIG_XENO_OPT_DEBUG=y
+# CONFIG_XENO_OPT_DEBUG_NUCLEUS is not set
+# CONFIG_XENO_OPT_DEBUG_XNLOCK is not set
+# CONFIG_XENO_OPT_DEBUG_QUEUES is not set
+# CONFIG_XENO_OPT_DEBUG_REGISTRY is not set
+# CONFIG_XENO_OPT_DEBUG_TIMERS is not set
+# CONFIG_XENO_OPT_DEBUG_SYNCH_RELAX is not set
+CONFIG_XENO_OPT_WATCHDOG=y
+CONFIG_XENO_OPT_WATCHDOG_TIMEOUT=4
+CONFIG_XENO_OPT_SHIRQ=y
CONFIG_XENO_OPT_SELECT=y
#
@@ -258,7 +259,6 @@
#
# NMI watchdog
#
-# CONFIG_XENO_HW_NMI_DEBUG_LATENCY is not set
#
# SMI workaround
@@ -284,6 +284,7 @@
CONFIG_XENO_OPT_NATIVE_ALARM=y
CONFIG_XENO_OPT_NATIVE_MPS=y
CONFIG_XENO_OPT_NATIVE_INTR=y
+CONFIG_XENO_OPT_DEBUG_NATIVE=y
CONFIG_XENO_SKIN_POSIX=y
CONFIG_XENO_OPT_POSIX_PERIOD=0
CONFIG_XENO_OPT_POSIX_SHM=y
@@ -300,6 +301,8 @@
CONFIG_XENO_OPT_RTDM_PERIOD=0
CONFIG_XENO_OPT_RTDM_FILDES=128
# CONFIG_XENO_OPT_RTDM_SELECT is not set
+CONFIG_XENO_OPT_DEBUG_RTDM=y
+CONFIG_XENO_OPT_DEBUG_RTDM_APPL=y
#
# Drivers
@@ -361,10 +364,7 @@
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y
CONFIG_GENERIC_CLOCKEVENTS_BUILD=y
-CONFIG_SMP=y
-# CONFIG_SPARSE_IRQ is not set
-CONFIG_X86_MPPARSE=y
-# CONFIG_X86_BIGSMP is not set
+# CONFIG_SMP is not set
# CONFIG_X86_EXTENDED_PLATFORM is not set
CONFIG_X86_SUPPORTS_MEMORY_FAILURE=y
CONFIG_SCHED_OMIT_FRAME_POINTER=y
@@ -377,7 +377,7 @@
# CONFIG_M686 is not set
# CONFIG_MPENTIUMII is not set
# CONFIG_MPENTIUMIII is not set
-# CONFIG_MPENTIUMM is not set
+CONFIG_MPENTIUMM=y
# CONFIG_MPENTIUM4 is not set
# CONFIG_MK6 is not set
# CONFIG_MK7 is not set
@@ -392,7 +392,7 @@
# CONFIG_MVIAC3_2 is not set
# CONFIG_MVIAC7 is not set
# CONFIG_MPSC is not set
-CONFIG_MCORE2=y
+# CONFIG_MCORE2 is not set
# CONFIG_MATOM is not set
# CONFIG_GENERIC_CPU is not set
# CONFIG_X86_GENERIC is not set
@@ -424,9 +424,7 @@
CONFIG_DMI=y
# CONFIG_IOMMU_HELPER is not set
# CONFIG_IOMMU_API is not set
-CONFIG_NR_CPUS=8
-# CONFIG_SCHED_SMT is not set
-CONFIG_SCHED_MC=y
+CONFIG_NR_CPUS=1
# CONFIG_PREEMPT_NONE is not set
# CONFIG_PREEMPT_VOLUNTARY is not set
CONFIG_PREEMPT=y
@@ -435,19 +433,13 @@
CONFIG_IPIPE_COMPAT=y
CONFIG_IPIPE_DELAYED_ATOMICSW=y
# CONFIG_IPIPE_UNMASKED_CONTEXT_SWITCH is not set
-CONFIG_X86_LOCAL_APIC=y
-CONFIG_X86_IO_APIC=y
-# CONFIG_X86_REROUTE_FOR_BROKEN_BOOT_IRQS is not set
+# CONFIG_X86_UP_APIC is not set
CONFIG_X86_MCE=y
-CONFIG_X86_MCE_INTEL=y
-CONFIG_X86_MCE_AMD=y
# CONFIG_X86_ANCIENT_MCE is not set
-CONFIG_X86_MCE_THRESHOLD=y
# CONFIG_X86_MCE_INJECT is not set
-CONFIG_X86_THERMAL_VECTOR=y
CONFIG_VM86=y
-CONFIG_TOSHIBA=m
-CONFIG_I8K=m
+# CONFIG_TOSHIBA is not set
+# CONFIG_I8K is not set
# CONFIG_X86_REBOOTFIXUPS is not set
CONFIG_MICROCODE=m
CONFIG_MICROCODE_INTEL=y
@@ -506,7 +498,6 @@
CONFIG_PHYSICAL_START=0x1000000
# CONFIG_RELOCATABLE is not set
CONFIG_PHYSICAL_ALIGN=0x100000
-# CONFIG_HOTPLUG_CPU is not set
# CONFIG_COMPAT_VDSO is not set
# CONFIG_CMDLINE_BOOL is not set
CONFIG_ARCH_ENABLE_MEMORY_HOTPLUG=y
@@ -540,12 +531,10 @@
# CONFIG_PCIE_ECRC is not set
# CONFIG_PCIEAER_INJECT is not set
# CONFIG_PCIEASPM is not set
-CONFIG_ARCH_SUPPORTS_MSI=y
-# CONFIG_PCI_MSI is not set
+# CONFIG_ARCH_SUPPORTS_MSI is not set
CONFIG_PCI_LEGACY=y
# CONFIG_PCI_DEBUG is not set
# CONFIG_PCI_STUB is not set
-CONFIG_HT_IRQ=y
# CONFIG_PCI_IOV is not set
CONFIG_ISA_DMA_API=y
CONFIG_ISA=y
@@ -3547,7 +3108,16 @@
CONFIG_UNUSED_SYMBOLS=y
CONFIG_DEBUG_FS=y
# CONFIG_HEADERS_CHECK is not set
-# CONFIG_IPIPE_DEBUG is not set
+CONFIG_IPIPE_DEBUG=y
+CONFIG_IPIPE_DEBUG_CONTEXT=y
+CONFIG_IPIPE_DEBUG_INTERNAL=y
+CONFIG_IPIPE_TRACE=y
+CONFIG_IPIPE_TRACE_ENABLE=y
+CONFIG_IPIPE_TRACE_MCOUNT=y
+CONFIG_IPIPE_TRACE_IRQSOFF=y
+CONFIG_IPIPE_TRACE_SHIFT=14
+# CONFIG_IPIPE_TRACE_VMALLOC is not set
+CONFIG_IPIPE_TRACE_PANIC=y
CONFIG_DEBUG_KERNEL=y
# CONFIG_DEBUG_SHIRQ is not set
CONFIG_DETECT_SOFTLOCKUP=y
@@ -3586,7 +3156,7 @@
# CONFIG_DEBUG_NOTIFIERS is not set
# CONFIG_DEBUG_CREDENTIALS is not set
CONFIG_ARCH_WANT_FRAME_POINTERS=y
-# CONFIG_FRAME_POINTER is not set
+CONFIG_FRAME_POINTER=y
# CONFIG_BOOT_PRINTK_DELAY is not set
# CONFIG_RCU_TORTURE_TEST is not set
# CONFIG_RCU_CPU_STALL_DETECTOR is not set
@@ -3599,6 +3169,7 @@
# CONFIG_DEBUG_PAGEALLOC is not set
CONFIG_USER_STACKTRACE_SUPPORT=y
CONFIG_NOP_TRACER=y
+CONFIG_HAVE_FTRACE_NMI_ENTER=y
CONFIG_HAVE_FUNCTION_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_TRACER=y
CONFIG_HAVE_FUNCTION_GRAPH_FP_TEST=y
@@ -3607,6 +3178,7 @@
CONFIG_HAVE_FTRACE_MCOUNT_RECORD=y
CONFIG_HAVE_SYSCALL_TRACEPOINTS=y
CONFIG_RING_BUFFER=y
+CONFIG_FTRACE_NMI_ENTER=y
CONFIG_EVENT_TRACING=y
CONFIG_CONTEXT_SWITCH_TRACER=y
CONFIG_RING_BUFFER_ALLOW_SWAP=y
@@ -3614,7 +3186,7 @@
CONFIG_GENERIC_TRACER=y
CONFIG_TRACING_SUPPORT=y
CONFIG_FTRACE=y
-# CONFIG_FUNCTION_TRACER is not set
+CONFIG_FUNCTION_TRACER=y
# CONFIG_IRQSOFF_TRACER is not set
# CONFIG_PREEMPT_TRACER is not set
# CONFIG_SYSPROF_TRACER is not set
@@ -3629,6 +3201,9 @@
# CONFIG_KMEMTRACE is not set
# CONFIG_WORKQUEUE_TRACER is not set
CONFIG_BLK_DEV_IO_TRACE=y
+CONFIG_DYNAMIC_FTRACE=y
+# CONFIG_FUNCTION_PROFILER is not set
+CONFIG_FTRACE_MCOUNT_RECORD=y
# CONFIG_FTRACE_STARTUP_TEST is not set
# CONFIG_MMIOTRACE is not set
# CONFIG_RING_BUFFER_BENCHMARK is not set
@@ -3637,7 +3212,9 @@
# CONFIG_DMA_API_DEBUG is not set
# CONFIG_SAMPLES is not set
CONFIG_HAVE_ARCH_KGDB=y
-# CONFIG_KGDB is not set
+CONFIG_KGDB=y
+CONFIG_KGDB_SERIAL_CONSOLE=y
+# CONFIG_KGDB_TESTS is not set
CONFIG_HAVE_ARCH_KMEMCHECK=y
# CONFIG_STRICT_DEVMEM is not set
CONFIG_X86_VERBOSE_BOOTUP=y
@@ -3645,7 +3222,6 @@
# CONFIG_EARLY_PRINTK_DBGP is not set
# CONFIG_DEBUG_STACKOVERFLOW is not set
# CONFIG_DEBUG_STACK_USAGE is not set
-# CONFIG_DEBUG_PER_CPU_MAPS is not set
# CONFIG_X86_PTDUMP is not set
# CONFIG_DEBUG_RODATA is not set
# CONFIG_DEBUG_NX_TEST is not set
@@ -3687,12 +3263,6 @@
CONFIG_SECURITY_SELINUX_CHECKREQPROT_VALUE=1
# CONFIG_SECURITY_SELINUX_POLICYDB_VERSION_MAX is not set
# CONFIG_SECURITY_TOMOYO is not set
-CONFIG_XOR_BLOCKS=m
-CONFIG_ASYNC_CORE=m
-CONFIG_ASYNC_MEMCPY=m
-CONFIG_ASYNC_XOR=m
-CONFIG_ASYNC_PQ=m
-CONFIG_ASYNC_RAID6_RECOV=m
CONFIG_ASYNC_TX_DISABLE_PQ_VAL_DMA=y
CONFIG_ASYNC_TX_DISABLE_XOR_VAL_DMA=y
CONFIG_CRYPTO=y
@@ -3705,7 +3275,7 @@
CONFIG_CRYPTO_ALGAPI2=y
CONFIG_CRYPTO_AEAD=m
CONFIG_CRYPTO_AEAD2=y
-CONFIG_CRYPTO_BLKCIPHER=m
+CONFIG_CRYPTO_BLKCIPHER=y
CONFIG_CRYPTO_BLKCIPHER2=y
CONFIG_CRYPTO_HASH=y
CONFIG_CRYPTO_HASH2=y
@@ -3731,7 +3301,7 @@
#
# Block modes
#
-CONFIG_CRYPTO_CBC=m
+CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_CTR=m
CONFIG_CRYPTO_CTS=m
CONFIG_CRYPTO_ECB=m
@@ -3776,7 +3346,7 @@
CONFIG_CRYPTO_CAMELLIA=m
CONFIG_CRYPTO_CAST5=m
CONFIG_CRYPTO_CAST6=m
-CONFIG_CRYPTO_DES=m
+CONFIG_CRYPTO_DES=y
CONFIG_CRYPTO_FCRYPT=m
CONFIG_CRYPTO_KHAZAD=m
CONFIG_CRYPTO_SALSA20=m
@@ -3832,7 +3402,6 @@
CONFIG_DECOMPRESS_GZIP=y
CONFIG_DECOMPRESS_BZIP2=y
CONFIG_DECOMPRESS_LZMA=y
-CONFIG_GENERIC_ALLOCATOR=y
CONFIG_TEXTSEARCH=y
CONFIG_TEXTSEARCH_KMP=m
CONFIG_TEXTSEARCH_BM=m
[-- Attachment #3: stable-2.6.32.7-xenomai-2.5.2.gz --]
[-- Type: application/x-gzip, Size: 22658 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Xenomai-help] 2.5.2 + 2.6.32 patched kernel causes system reset
2010-07-06 18:56 [Xenomai-help] 2.5.2 + 2.6.32 patched kernel causes system reset Peter Soetens
@ 2010-07-06 19:11 ` Jan Kiszka
2010-07-06 19:16 ` Gilles Chanteperdrix
1 sibling, 0 replies; 4+ messages in thread
From: Jan Kiszka @ 2010-07-06 19:11 UTC (permalink / raw)
To: Peter Soetens; +Cc: xenomai-help
[-- Attachment #1: Type: text/plain, Size: 1410 bytes --]
Peter Soetens wrote:
> Hi,
>
> We have a single-core Pentium M industrial PC equiped with Xenomai
> 2.5.2 patched vanilla Linux 2.6.32.7.
> We have two builds of the kernel: one that causes a reliable system
> reset when starting a 'heavy' Orocos application (lot of thread
> creation/deletions), the other is rock solid. We think it is related
> to thread creation in the native skin, since we could at least twice
> cause the reset by just creating a native thread (that was meant to
> block on a semaphore) but this was hard to reproduce. Stack size was
> set to 128k, but only trivial code was executed.We had the impression
> that enlarging the stack size to 256k caused less resets, but it was
> still very frequent. Setting the stack size too low caused segfaults
> in the app, not a kernel crash.
>
> The stable config and the diffs between both configs are in
> attachment. Read it as: apply this patch to make it unstable. Any
> clues what is the most likely suspect ? Could the xenomai watchdog
> cause the reset ?
>
> Thanks for any pointers, we're prepared to bisect this, if any one
> knows a tool to do a bisect on a set of config options, we'll be glad
> to use it.
You could first of all try to enable CONFIG_X86_UP_APIC. Recommended for
performance reasons anyway.
Then try to disable debugging features that were enabled between the two
versions.
Jan
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 257 bytes --]
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Xenomai-help] 2.5.2 + 2.6.32 patched kernel causes system reset
2010-07-06 18:56 [Xenomai-help] 2.5.2 + 2.6.32 patched kernel causes system reset Peter Soetens
2010-07-06 19:11 ` Jan Kiszka
@ 2010-07-06 19:16 ` Gilles Chanteperdrix
2010-07-07 20:51 ` Peter Soetens
1 sibling, 1 reply; 4+ messages in thread
From: Gilles Chanteperdrix @ 2010-07-06 19:16 UTC (permalink / raw)
To: Peter Soetens; +Cc: xenomai-help
Peter Soetens wrote:
> Hi,
>
> We have a single-core Pentium M industrial PC equiped with Xenomai
> 2.5.2 patched vanilla Linux 2.6.32.7.
> We have two builds of the kernel: one that causes a reliable system
> reset when starting a 'heavy' Orocos application (lot of thread
> creation/deletions), the other is rock solid. We think it is related
> to thread creation in the native skin, since we could at least twice
> cause the reset by just creating a native thread (that was meant to
> block on a semaphore) but this was hard to reproduce. Stack size was
> set to 128k, but only trivial code was executed.We had the impression
> that enlarging the stack size to 256k caused less resets, but it was
> still very frequent. Setting the stack size too low caused segfaults
> in the app, not a kernel crash.
>
> The stable config and the diffs between both configs are in
> attachment. Read it as: apply this patch to make it unstable. Any
> clues what is the most likely suspect ? Could the xenomai watchdog
> cause the reset ?
>
> Thanks for any pointers, we're prepared to bisect this, if any one
> knows a tool to do a bisect on a set of config options, we'll be glad
> to use it.
Before investing any more time, please upgrade to Xenomai 2.5.3.
--
Gilles.
^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: [Xenomai-help] 2.5.2 + 2.6.32 patched kernel causes system reset
2010-07-06 19:16 ` Gilles Chanteperdrix
@ 2010-07-07 20:51 ` Peter Soetens
0 siblings, 0 replies; 4+ messages in thread
From: Peter Soetens @ 2010-07-07 20:51 UTC (permalink / raw)
To: Gilles Chanteperdrix; +Cc: xenomai-help
On Tue, Jul 6, 2010 at 9:16 PM, Gilles Chanteperdrix
<gilles.chanteperdrix@xenomai.org> wrote:
> Peter Soetens wrote:
>> Hi,
>>
>> We have a single-core Pentium M industrial PC equiped with Xenomai
>> 2.5.2 patched vanilla Linux 2.6.32.7.
>> We have two builds of the kernel: one that causes a reliable system
>> reset when starting a 'heavy' Orocos application (lot of thread
>> creation/deletions), the other is rock solid. We think it is related
>> to thread creation in the native skin, since we could at least twice
>> cause the reset by just creating a native thread (that was meant to
>> block on a semaphore) but this was hard to reproduce. Stack size was
>> set to 128k, but only trivial code was executed.We had the impression
>> that enlarging the stack size to 256k caused less resets, but it was
>> still very frequent. Setting the stack size too low caused segfaults
>> in the app, not a kernel crash.
>>
>> The stable config and the diffs between both configs are in
>> attachment. Read it as: apply this patch to make it unstable. Any
>> clues what is the most likely suspect ? Could the xenomai watchdog
>> cause the reset ?
>>
>> Thanks for any pointers, we're prepared to bisect this, if any one
>> knows a tool to do a bisect on a set of config options, we'll be glad
>> to use it.
>
> Before investing any more time, please upgrade to Xenomai 2.5.3.
X 2.5.3 has exactly the same behaviour. We enabled the UP_APIC, but
that didn't help. We then enabled COMFIG_SMP, which caused noticable
slowdown during booting (boot takes 2x as long compared to both
good/bad configs which have equal speed). We then disabled the Xenomai
watchdog, to no avail.
It's really a bisecting issue between the two configs. As I said
before, it's as good as 100% reproducible on our setup.
'They' are pulling me of this and someone else will do further testing
hopefully next week (you know how these things go though...).
We've struck something, we just have to corner it.
Peter
^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2010-07-07 20:51 UTC | newest]
Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2010-07-06 18:56 [Xenomai-help] 2.5.2 + 2.6.32 patched kernel causes system reset Peter Soetens
2010-07-06 19:11 ` Jan Kiszka
2010-07-06 19:16 ` Gilles Chanteperdrix
2010-07-07 20:51 ` Peter Soetens
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.