* [Xenomai] Fwd: Latency test fails. Problem during installation [not found] <CANBp=-hcC5DHb=HpMzqbewQswwwh-dF-+i+DoOd6KRHoXbv2qw@mail.gmail.com> @ 2016-03-07 10:33 ` Michele Belotti 2016-03-07 11:32 ` Henning Schild ` (2 more replies) 0 siblings, 3 replies; 7+ messages in thread From: Michele Belotti @ 2016-03-07 10:33 UTC (permalink / raw) To: xenomai Hello everybody, We recently migrated my project on a new and more powerful workstation but I can obtain the previous performance. We observed by running the standard Xenomai tools that there are very high latencies, hinting for an issue in the kernel configuration (or maybe unsupported HW features by Xenomai ?). We are trying to install Xenomai version 2.6.3 on a HP Z840 workstation Red Hat system 7.2. The Workstation has a double 12 cores CPU Intel Xeon E5-2690v3 on it. We follow the installation guide and the troubleshooting but the latency is still very high with some high jumps. We tried to deal with the different options described in the guide CONFIG_CPU_FREQ, CONFIG_CPU_IDLE, CONFIG_CONTEXT_TRACKING_FORCE XENO_OPT_STATS NO_HZ_FULL_ALL BSD_PROCESS_ACCT but none has impacted the system, only marginal improvements. We also tried to modify the SMI detection parameters adding the following string to kernel command line xeno_hal.smi = 1 but without any effect. Note: to notice that xenomai does not recognize the ethernet and we needed to install again the ethernet card driver. We don't know if this has some impact on the system. We also noticed some freeze when recovering from the sleep mode implying to restart the machine. Would you have any idea how we can proceed to isolate the issue? Thanks for the support Regards Michele ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Xenomai] Fwd: Latency test fails. Problem during installation 2016-03-07 10:33 ` [Xenomai] Fwd: Latency test fails. Problem during installation Michele Belotti @ 2016-03-07 11:32 ` Henning Schild 2016-03-08 8:50 ` Michele Belotti 2016-03-07 11:37 ` Gilles Chanteperdrix 2016-03-07 11:45 ` Philippe Gerum 2 siblings, 1 reply; 7+ messages in thread From: Henning Schild @ 2016-03-07 11:32 UTC (permalink / raw) To: Michele Belotti; +Cc: xenomai Hello Michele, you seem to be having multiple, possibly unrelated, issues. Some of them seem related to the kernel you are using. Please provide the kernel version and ipipe version you are talking about. For the high latency cases you should provide an event-trace. Henning On Mon, 7 Mar 2016 11:33:31 +0100 Michele Belotti <miki.belotti@gmail.com> wrote: > Hello everybody, > We recently migrated my project on a new and more powerful > workstation but I can obtain the previous performance. > > We observed by running the standard Xenomai tools that there are very > high latencies, hinting for an issue in the kernel configuration (or > maybe unsupported HW features by Xenomai ?). > > We are trying to install Xenomai version 2.6.3 on a HP Z840 > workstation Red Hat system 7.2. > The Workstation has a double 12 cores CPU Intel Xeon E5-2690v3 on it. > > We follow the installation guide and the troubleshooting but the > latency is still very high with some high jumps. > We tried to deal with the different options described in the guide > > CONFIG_CPU_FREQ, CONFIG_CPU_IDLE, CONFIG_CONTEXT_TRACKING_FORCE > XENO_OPT_STATS NO_HZ_FULL_ALL BSD_PROCESS_ACCT but none has > impacted the system, only marginal improvements. > > We also tried to modify the SMI detection parameters adding the > following string to kernel command line > xeno_hal.smi = 1 > but without any effect. > Note: to notice that xenomai does not recognize the ethernet and we > needed to install again the ethernet card driver. We don't know if > this has some impact on the system. We also noticed some freeze when > recovering from the sleep mode implying to restart the machine. > > Would you have any idea how we can proceed to isolate the issue? > > Thanks for the support > > Regards > > > Michele > _______________________________________________ > Xenomai mailing list > Xenomai@xenomai.org > https://xenomai.org/mailman/listinfo/xenomai ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Xenomai] Fwd: Latency test fails. Problem during installation 2016-03-07 11:32 ` Henning Schild @ 2016-03-08 8:50 ` Michele Belotti 2016-03-09 15:12 ` Michele Belotti 0 siblings, 1 reply; 7+ messages in thread From: Michele Belotti @ 2016-03-08 8:50 UTC (permalink / raw) To: Henning Schild, gilles.chanteperdrix, rpm; +Cc: xenomai Hello everybody, many thanks for the prompt support. I have tested yesterday the latest version of 2.6 branch but it does not help. The kernel I'm using is the v3.10.32 and I try ipipe 4 and 7 without any improvement. The latency is around 100 ns and always start with a max value of 30 and it degenerates during time. The worst case trace is hereafter obtaind with latency -f I-pipe worst-case tracing service on 3.10.32_xenomai-2.6_20160224_1/ipipe release #4 ------------------------------------------------------------- CPU: 0, Begin: 357409893008 cycles, Trace Points: 12 (-10/+1), Length: 31523 us Calibrated minimum trace-point overhead: 0.078 us +----- Hard IRQs ('|': locked) |+-- Xenomai ||+- Linux ('*': domain stalled, '+': current, '#': current+stalled) ||| +---------- Delay flag ('+': > 1 us, '!': > 10 us) ||| | +- NMI noise ('N') ||| | | Type User Val. Time Delay Function (Parent) | #begin 0x80000000 -143 0.123 ipipe_unstall_root+0x1c (_raw_spin_unlock_irq+0x11) | #func -143 0.185 ipipe_root_only+0x0 (ipipe_unstall_root+0x21) | +end 0x80000000 -143 0.114 ipipe_trace_end+0x19 (ipipe_unstall_root+0x59) +func -143 0.123 sub_preempt_count+0x0 (_raw_spin_unlock_irq+0x1b) +func -142 0.141 add_preempt_count+0x0 (cpu_stopper_thread+0x97) +func -142 0.134 stop_machine_cpu_stop+0x0 (cpu_stopper_thread+0x9d) | +begin 0x80000001 -142 0.160 debug_smp_processor_id+0x3f (stop_machine_cpu_stop+0x1a) | +end 0x80000001 -142 142.114 ipipe_trace_end+0x19 (debug_smp_processor_id+0x76) #func 0 0.235 mtrr_rendezvous_handler+0x0 (stop_machine_cpu_stop+0xe6) #func 0 0.174 generic_set_mtrr+0x0 (mtrr_rendezvous_handler+0x67) >| #begin 0x80000001 0 0.132 generic_set_mtrr+0x1af (mtrr_rendezvous_handler+0x67) :| #func 0 0.137 prepare_set+0x0 (generic_set_mtrr+0x80) :| #func 0 0.131 _raw_spin_lock+0x0 (prepare_set+0x15) :| #func 0! 30160.710 add_preempt_count+0x0 (_raw_spin_lock+0x17) :| #func 30161! 1072.522 mtrr_wrmsr+0x0 (prepare_set+0x7a) :| #func 31233! 77.451 mtrr_wrmsr+0x0 (generic_set_mtrr+0x149) :| #func 31311! 76.539 mtrr_wrmsr+0x0 (generic_set_mtrr+0x160) :| #func 31387! 78.262 post_set+0x0 (generic_set_mtrr+0x165) :| #func 31465! 55.673 mtrr_wrmsr+0x0 (post_set+0x25) :| #func 31521+ 1.141 _raw_spin_unlock+0x0 (post_set+0x52) :| #func 31522 0.496 sub_preempt_count+0x0 (_raw_spin_unlock+0x16) <| #end 0x80000001 31523 0.829 ipipe_trace_end+0x19 (generic_set_mtrr+0x17a) | #begin 0x000000ef 31524 0.000 apic_timer_interrupt+0x6d (generic_set_mtrr+0xc2) The dynamic parameters used are BOOT_IMAGE=/boot/vmlinuz-3.10.32_xenomai-2.6_20160224_1 root=UUID=2e4bd84d-b2ce-4ac9-8b10-377b2fa26437 ro crashkernel=auto rhgb quiet LANG=en_US.UTF-8 systemd.debug ro xeno_nucleus.xenomai_gid=1001 I also verify that the ACPI is disables and Hyperthreading is desactivated. About the Smi workaround what you intend that some message are triggered? Warning disappears and I can see that the message Xenomai: SMI workaround failed! is not present. I guess that it is not working. But what I need to do in the kernel ? Many thanks Michele > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Xenomai] Fwd: Latency test fails. Problem during installation 2016-03-08 8:50 ` Michele Belotti @ 2016-03-09 15:12 ` Michele Belotti 0 siblings, 0 replies; 7+ messages in thread From: Michele Belotti @ 2016-03-09 15:12 UTC (permalink / raw) Cc: xenomai Hello Everybody, I tried different option and I obtain some improvement switching off in the bios the Hyperthreading, but I'm still far from the result. Latency is till very high as you can see from the test. I verify defintely that SMI workaround is not working Partial deactivation of smi sources is also not working.(xeno_hal.smi_mask=<enable mask>). Actually it didn’t even recognize that it was tried but accepted the command(e.g. xeno_hal.smi_mask=0xFFFCFFFB ). To resume my situation I collect more information HP Z840 workstation Chipset : Intel C610/X99 CPU : 2 x Intel(R) Xeon(R) CPU E5-2690 v3 12Core Linux Kernel 3.10.32 Xenomai version 2.6.4 ipipe version 7 Linux distribution RHEL 7.1 *Message form BIOS* *[ 6.461268] I-pipe: head domain Xenomai registered.[ 6.462528] Xenomai: hal/x86_64 started.[ 6.462797] Xenomai: scheduling class idle registered.[ 6.462808] Xenomai: scheduling class rt registered.[ 6.584832] Xenomai: real-time nucleus v2.6.4 (Jumpin' Out) loaded.[ 6.584844] Xenomai: debug mode enabled.[ 6.585368] Xenomai: disabling automatic C1E state promotion on Intel processor[ 6.585603] Xenomai: SMI-enabled chipset found[ 6.585629] Xenomai: SMI workaround failed![ 6.587167] Xenomai: starting native API services.[ 6.587179] Xenomai: starting POSIX services.[ 6.587294] Xenomai: starting RTDM services.[ 6.461268] I-pipe: head domain Xenomai registered.[ 31.679848] I-pipe: spurious interrupt 32[ 45.024846] I-pipe: spurious interrupt 32[ 54.793524] I-pipe: spurious interrupt 32[ 54.793544] I-pipe: spurious interrupt 32[ 55.050552] I-pipe: spurious interrupt 32[ 55.050568] I-pipe: spurious interrupt 32[ 65.077412] I-pipe: spurious interrupt 32[ 75.107794] I-pipe: spurious interrupt 32[ 75.107813] I-pipe: spurious interrupt 32[ 85.128461] I-pipe: spurious interrupt 32[ 95.160067] I-pipe: spurious interrupt 32[ 95.160084] I-pipe: spurious interrupt 32[ 105.181079] I-pipe: spurious interrupt 32[ 125.234585] I-pipe: spurious interrupt 32* - *Configuration of Xenomai * *xeno-config --verbose --version="2.6.4" --cc="gcc" --arch="x86" --prefix="/usr/xenomai" --xeno-cflags="-I/usr/xenomai/include -D_GNU_SOURCE -D_REENTRANT -D__XENO__" --xeno-ldflags="-L/usr/xenomai/lib -lxenomai -lpthread -lrt " --posix-cflags="" --posix-ldflags="-Wl,@/usr/xenomai/lib/posix.wrappers -L/usr/xenomai/lib -lpthread_rt -lxenomai -lpthread -lrt " --library-dir="/usr/xenomai/lib"* # Generated by configure. # Compiler output produced by configure, useful for debugging # configure, is in config.log if it exists. configured by ./configure, generated by GNU Autoconf 2.69, ac_configure_extra_args= ac_configure_extra_args="$ac_configure_extra_args --silent" set X /bin/sh './configure' $ac_configure_extra_args --no-create --no-recursion # on some systems where configure will not decide to define it. # Let's still pretend it is `configure' which instantiates (i.e., don't configure_input='Generated from '` `' by configure.' configure_input="$ac_file. $configure_input" case $configure_input in #( ac_sed_conf_input=`$as_echo "$configure_input" | *) ac_sed_conf_input=$configure_input;; s|@configure_input@|$ac_sed_conf_input|;t t $as_echo "/* $configure_input */" \ $as_echo "/* $configure_input */" \ # Libtool was configured on host `(hostname || uname -n) 2>/dev/null | sed 1q`: - Dynamic information to the kernel BOOT_IMAGE=/boot/vmlinuz-3.10.32_xenomai-2.6_20160224_1 root=UUID=2e4bd84d-b2ce-4ac9-8b10-377b2fa26437 ro crashkernel=auto rhgb quiet LANG=en_US.UTF-8 systemd.debug ro xeno_nucleus.xenomai_gid=1001 Itrace Max I-pipe worst-case tracing service on 3.10.32x20160309/ipipe release #7 ------------------------------------------------------------- CPU: 0, Begin: 43934326523000 cycles, Trace Points: 12 (-10/+1), Length: 37832 us Calibrated minimum trace-point overhead: 0.078 us +----- Hard IRQs ('|': locked) |+-- Xenomai ||+- Linux ('*': domain stalled, '+': current, '#': current+stalled) ||| +---------- Delay flag ('+': > 1 us, '!': > 10 us) ||| | +- NMI noise ('N') ||| | | Type User Val. Time Delay Function (Parent) #func -391 0.143 rcu_irq_exit+0x0 (irq_exit+0x8f) | #begin 0x80000001 -391 0.137 debug_smp_processor_id+0x3f (rcu_irq_exit+0x55) | #end 0x80000001 -391 0.126 ipipe_trace_end+0x14 (debug_smp_processor_id+0x76) #func -391 0.115 ipipe_restore_root+0x0 (rcu_irq_exit+0x8e) #func -391 0.118 ipipe_root_only+0x0 (ipipe_restore_root+0x12) | #begin 0x80000001 -391 0.155 ipipe_root_only+0x9c (ipipe_restore_root+0x12) | #end 0x80000001 -391 0.376 ipipe_trace_end+0x14 (ipipe_root_only+0x86) | +end 0x000000ef -390 389.590 ipipe_trace_end+0x14 (apic_timer_interrupt+0x8f) #func -1 0.580 mtrr_rendezvous_handler+0x0 (stop_machine_cpu_stop+0xe4) #func 0 0.473 generic_set_mtrr+0x0 (mtrr_rendezvous_handler+0x58) >| #begin 0x80000001 0 0.168 generic_set_mtrr+0x1a0 (mtrr_rendezvous_handler+0x58) :| #func 0 0.256 prepare_set+0x0 (generic_set_mtrr+0x80) :| #func 0 0.121 _raw_spin_lock+0x0 (prepare_set+0x15) :| #func 0! 36321.797 add_preempt_count+0x0 (_raw_spin_lock+0x17) :| #func 36322! 1263.460 mtrr_wrmsr+0x0 (prepare_set+0x7a) :| #func 37585! 66.500 mtrr_wrmsr+0x0 (generic_set_mtrr+0x142) :| #func 37652! 65.266 mtrr_wrmsr+0x0 (generic_set_mtrr+0x159) :| #func 37717! 65.372 post_set+0x0 (generic_set_mtrr+0x15e) :| #func 37782! 47.720 mtrr_wrmsr+0x0 (post_set+0x25) :| #func 37830 0.951 _raw_spin_unlock+0x0 (post_set+0x52) :| #func 37831 0.464 sub_preempt_count+0x0 (_raw_spin_unlock+0x16) <| #end 0x80000001 37832 1.318 ipipe_trace_end+0x14 (generic_set_mtrr+0x173) | #begin 0x000000ef 37833 0.000 apic_timer_interrupt+0x6d (generic_set_mtrr+0xc2) I also verified the other suggested parameters but they don't have any effects. Could you please hint me on how I can activated the SMI workaround? I don't know if it helps but UEFI is deactivated for other reason and were running on legacy bios. Thanks a lot for the precious help and please me indicate if you need more information. Michele 2016-03-08 9:50 GMT+01:00 Michele Belotti <miki.belotti@gmail.com>: > Hello everybody, many thanks for the prompt support. > I have tested yesterday the latest version of 2.6 branch but it does not > help. > > The kernel I'm using is the v3.10.32 and I try ipipe 4 and 7 without any > improvement. > The latency is around 100 ns and always start with a max value of 30 and > it degenerates during time. > > > The worst case trace is hereafter obtaind with latency -f > > I-pipe worst-case tracing service on 3.10.32_xenomai-2.6_20160224_1/ipipe > release #4 > ------------------------------------------------------------- > CPU: 0, Begin: 357409893008 cycles, Trace Points: 12 (-10/+1), Length: > 31523 us > Calibrated minimum trace-point overhead: 0.078 us > > +----- Hard IRQs ('|': locked) > |+-- Xenomai > ||+- Linux ('*': domain stalled, '+': current, '#': current+stalled) > ||| +---------- Delay flag ('+': > 1 us, '!': > 10 > us) > ||| | +- NMI noise ('N') > ||| | | > Type User Val. Time Delay Function (Parent) > | #begin 0x80000000 -143 0.123 ipipe_unstall_root+0x1c > (_raw_spin_unlock_irq+0x11) > | #func -143 0.185 ipipe_root_only+0x0 > (ipipe_unstall_root+0x21) > | +end 0x80000000 -143 0.114 ipipe_trace_end+0x19 > (ipipe_unstall_root+0x59) > +func -143 0.123 sub_preempt_count+0x0 > (_raw_spin_unlock_irq+0x1b) > +func -142 0.141 add_preempt_count+0x0 > (cpu_stopper_thread+0x97) > +func -142 0.134 stop_machine_cpu_stop+0x0 > (cpu_stopper_thread+0x9d) > | +begin 0x80000001 -142 0.160 debug_smp_processor_id+0x3f > (stop_machine_cpu_stop+0x1a) > | +end 0x80000001 -142 142.114 ipipe_trace_end+0x19 > (debug_smp_processor_id+0x76) > #func 0 0.235 mtrr_rendezvous_handler+0x0 > (stop_machine_cpu_stop+0xe6) > #func 0 0.174 generic_set_mtrr+0x0 > (mtrr_rendezvous_handler+0x67) > >| #begin 0x80000001 0 0.132 generic_set_mtrr+0x1af > (mtrr_rendezvous_handler+0x67) > :| #func 0 0.137 prepare_set+0x0 > (generic_set_mtrr+0x80) > :| #func 0 0.131 _raw_spin_lock+0x0 > (prepare_set+0x15) > :| #func 0! 30160.710 add_preempt_count+0x0 > (_raw_spin_lock+0x17) > :| #func 30161! 1072.522 mtrr_wrmsr+0x0 (prepare_set+0x7a) > :| #func 31233! 77.451 mtrr_wrmsr+0x0 > (generic_set_mtrr+0x149) > :| #func 31311! 76.539 mtrr_wrmsr+0x0 > (generic_set_mtrr+0x160) > :| #func 31387! 78.262 post_set+0x0 > (generic_set_mtrr+0x165) > :| #func 31465! 55.673 mtrr_wrmsr+0x0 (post_set+0x25) > :| #func 31521+ 1.141 _raw_spin_unlock+0x0 > (post_set+0x52) > :| #func 31522 0.496 sub_preempt_count+0x0 > (_raw_spin_unlock+0x16) > <| #end 0x80000001 31523 0.829 ipipe_trace_end+0x19 > (generic_set_mtrr+0x17a) > | #begin 0x000000ef 31524 0.000 apic_timer_interrupt+0x6d > (generic_set_mtrr+0xc2) > > > The dynamic parameters used are > BOOT_IMAGE=/boot/vmlinuz-3.10.32_xenomai-2.6_20160224_1 > root=UUID=2e4bd84d-b2ce-4ac9-8b10-377b2fa26437 ro crashkernel=auto rhgb > quiet LANG=en_US.UTF-8 systemd.debug ro xeno_nucleus.xenomai_gid=1001 > > > I also verify that the ACPI is disables and Hyperthreading is > desactivated. > About the Smi workaround what you intend that some message are triggered? > Warning disappears and I can see that the message > > Xenomai: SMI workaround failed! > > is not present. I guess that it is not working. But what I need to do in the kernel ? > > > Many thanks > > Michele > > > > >> > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Xenomai] Fwd: Latency test fails. Problem during installation 2016-03-07 10:33 ` [Xenomai] Fwd: Latency test fails. Problem during installation Michele Belotti 2016-03-07 11:32 ` Henning Schild @ 2016-03-07 11:37 ` Gilles Chanteperdrix 2016-03-07 12:51 ` Gilles Chanteperdrix 2016-03-07 11:45 ` Philippe Gerum 2 siblings, 1 reply; 7+ messages in thread From: Gilles Chanteperdrix @ 2016-03-07 11:37 UTC (permalink / raw) To: Michele Belotti; +Cc: xenomai On Mon, Mar 07, 2016 at 11:33:31AM +0100, Michele Belotti wrote: > Hello everybody, > We recently migrated my project on a new and more powerful workstation but > I can obtain the previous performance. You mean you can NOT. > > We observed by running the standard Xenomai tools that there are very high > latencies, hinting for an issue in the kernel configuration (or maybe > unsupported HW features by Xenomai ?). How high? > > We are trying to install Xenomai version 2.6.3 on a HP Z840 workstation Red > Hat system 7.2. > The Workstation has a double 12 cores CPU Intel Xeon E5-2690v3 on > it. Why installing Xenomai 2.6.3 ? It is not even the latest release in the 2.6 branch. > > We follow the installation guide and the troubleshooting but the latency is > still very high with some high jumps. > We tried to deal with the different options described in the guide > > CONFIG_CPU_FREQ, CONFIG_CPU_IDLE, CONFIG_CONTEXT_TRACKING_FORCE > XENO_OPT_STATS NO_HZ_FULL_ALL BSD_PROCESS_ACCT but none has impacted the > system, only marginal improvements. What about CONFIG_ACPI_CPUFREQ? > > We also tried to modify the SMI detection parameters adding the following > string to kernel command line > xeno_hal.smi = 1 > but without any effect. I am not sure 2.6.3 already had the kernel parameters, you may have to configure the kernel to enable the SMI workaround. In any case, using the module parameter, if this works, should trigger some messages in the boot logs, as explained in the troubleshooting guide. If you do not see the messages, you are doing it wrong. Regards. -- Gilles. https://click-hack.org ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Xenomai] Fwd: Latency test fails. Problem during installation 2016-03-07 11:37 ` Gilles Chanteperdrix @ 2016-03-07 12:51 ` Gilles Chanteperdrix 0 siblings, 0 replies; 7+ messages in thread From: Gilles Chanteperdrix @ 2016-03-07 12:51 UTC (permalink / raw) To: Michele Belotti; +Cc: xenomai On Mon, Mar 07, 2016 at 12:37:08PM +0100, Gilles Chanteperdrix wrote: > > > > We follow the installation guide and the troubleshooting but the latency is > > still very high with some high jumps. > > We tried to deal with the different options described in the guide > > > > CONFIG_CPU_FREQ, CONFIG_CPU_IDLE, CONFIG_CONTEXT_TRACKING_FORCE > > XENO_OPT_STATS NO_HZ_FULL_ALL BSD_PROCESS_ACCT but none has impacted the > > system, only marginal improvements. > > What about CONFIG_ACPI_CPUFREQ? Sorry, CONFIG_ACPI_PROCESSOR. -- Gilles. https://click-hack.org ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [Xenomai] Fwd: Latency test fails. Problem during installation 2016-03-07 10:33 ` [Xenomai] Fwd: Latency test fails. Problem during installation Michele Belotti 2016-03-07 11:32 ` Henning Schild 2016-03-07 11:37 ` Gilles Chanteperdrix @ 2016-03-07 11:45 ` Philippe Gerum 2 siblings, 0 replies; 7+ messages in thread From: Philippe Gerum @ 2016-03-07 11:45 UTC (permalink / raw) To: Michele Belotti, xenomai On 03/07/2016 11:33 AM, Michele Belotti wrote: > Hello everybody, > We recently migrated my project on a new and more powerful workstation but > I can obtain the previous performance. > > We observed by running the standard Xenomai tools that there are very high > latencies, hinting for an issue in the kernel configuration (or maybe What does "high" mean? The output of a latency test showing the peak would help. > unsupported HW features by Xenomai ?). > > We are trying to install Xenomai version 2.6.3 on a HP Z840 workstation Red > Hat system 7.2. 2.6.3 is an outdated release of a Xenomai series which is EOL. Any reason to use a totally unmaintained version? Also, Xenomai releases are validated on mainline kernels. You are basically on your own when running it over vendor kernels. > The Workstation has a double 12 cores CPU Intel Xeon E5-2690v3 on it. > - SMI are always a problem, affecting many x86 workstations and laptops - Hyperthreading is always a problem - Power management should be carefully audited Please note that running Xenomai in dual kernel mode over so many cores won't fly. You will likely have to restrict the number of cores usable in real-time operations to get better performances at some point. > We follow the installation guide and the troubleshooting but the latency is > still very high with some high jumps. > We tried to deal with the different options described in the guide > > CONFIG_CPU_FREQ, CONFIG_CPU_IDLE, CONFIG_CONTEXT_TRACKING_FORCE > XENO_OPT_STATS NO_HZ_FULL_ALL BSD_PROCESS_ACCT but none has impacted the > system, only marginal improvements. Which guide mentions these ones: NO_HZ_FULL_ALL BSD_PROCESS_ACCT? > > We also tried to modify the SMI detection parameters adding the following > string to kernel command line > xeno_hal.smi = 1 > but without any effect. .smi = 1 just tells Xenomai to try enabling the work around. A message in the kernel log should tell you whether the work around was enabled successfully. I suspect it was not for various reasons. Other ways to detect SMIs available on the net: - the hwlat_detector module and script. - reading MSR 0x34 on each CPU, using https://01.org/msr-tools Check your BIOS: - find out whether digital thermal sensors are present. If they are, you may want to talk to your hw supplier about the potential implication of DTS in SMI events, and see what might be done for fixing their BIOS in case they are involved. - disable any PS/2 emulation for mouse/kbd. > Note: to notice that xenomai does not recognize the ethernet and we needed > to install again the ethernet card driver. What do you mean by "installing"? We don't know if this has some > impact on the system. We also noticed some freeze when recovering from the > sleep mode implying to restart the machine. > There is no support for this, neither the interrupt pipeline or Xenomai care for it in any way. > Would you have any idea how we can proceed to isolate the issue? > A lot of information is missing. Following those guidelines may help you in getting some answers: http://xenomai.org/asking-for-help/ -- Philippe. ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2016-03-09 15:12 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <CANBp=-hcC5DHb=HpMzqbewQswwwh-dF-+i+DoOd6KRHoXbv2qw@mail.gmail.com>
2016-03-07 10:33 ` [Xenomai] Fwd: Latency test fails. Problem during installation Michele Belotti
2016-03-07 11:32 ` Henning Schild
2016-03-08 8:50 ` Michele Belotti
2016-03-09 15:12 ` Michele Belotti
2016-03-07 11:37 ` Gilles Chanteperdrix
2016-03-07 12:51 ` Gilles Chanteperdrix
2016-03-07 11:45 ` Philippe Gerum
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.