From mboxrd@z Thu Jan 1 00:00:00 1970 Subject: Re: hith latency when cpu is fully loaded References: <25135.7772549213$1574125359@news.gmane.org> <22cf8336-ae20-1744-aa10-ff7a3a91c322@siemens.com> <5DD39DA9.2010605@kylinos.cn> <0abc0864-6324-2eb7-f5cf-52ca539f1d2d@siemens.com> From: chensong Message-ID: <5DD3A671.2060408@kylinos.cn>+B69FEEDD10462DE8 Date: Tue, 19 Nov 2019 16:23:13 +0800 MIME-Version: 1.0 In-Reply-To: <0abc0864-6324-2eb7-f5cf-52ca539f1d2d@siemens.com> Content-Type: text/plain; charset="utf-8"; format="flowed" Content-Transfer-Encoding: 8bit List-Id: Discussions about the Xenomai project List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Jan Kiszka , xenomai@xenomai.org On 2019年11月19日 16:03, Jan Kiszka wrote: > On 19.11.19 08:45, chensong wrote: >> >> >> On 2019年11月19日 14:49, Jan Kiszka wrote: >>> On 19.11.19 02:01, chensong via Xenomai wrote: >>>> Dear experts, >>>> >>>> i'm new in xenomai, i got an issue, here is the detail: >>>> >>>> Main processor architect: ARM64 phytium ft2000ahk >>>> Kernel release number: 4.14.4 >>>> cmdline:BOOT_IMAGE=/Image-tmp >>>> root=UUID=9fea0634-a9c9-4e9f-906c-9c36b7249822 console=ttyS1,115200 >>>> earlyprintk=uart8250-32bit,0x28001000 rw rootdelay=10 KEYBOARDTYPE=pc >>>> KEYTABLE=us security= >>>> xenomai release number:3.1-devel >>>> xenomai configuration: >>>> kylin@kylin-os:~/workspace/code/nudt-hgj-xenomai-tjrd$ grep >>>> configure config.status >>>> # Generated by configure. >>>> # Compiler output produced by configure, useful for debugging >>>> # configure, is in config.log if it exists. >>>> configured by ./configure, generated by GNU Autoconf 2.69, >>>> ac_configure_extra_args= >>>> ac_configure_extra_args="$ac_configure_extra_args --silent" >>>> set X /bin/bash './configure' '--with-core=cobalt' >>>> '--enable-smp' '--enable-pshared' $ac_configure_extra_args --no-create >>>> --no-recursion >>>> configure_time_dlsearch_path='/lib /usr/lib >>>> /lib/aarch64-linux-gnu /usr/lib/aarch64-linux-gnu >>>> /usr/lib/aarch64-linux-gnu/mesa-egl /usr/lib/aarch64-linux-gnu/mesa >>>> /usr/local/lib ' >>>> configure_time_lt_sys_library_path='' >>>> for var in reload_cmds old_postinstall_cmds >>>> old_postuninstall_cmds old_archive_cmds extract_expsyms_cmds >>>> old_archive_from_new_cmds old_archive_from_expsyms_cmds archive_cmds >>>> archive_expsym_cmds module_cmds module_expsym_cmds export_symbols_cmds >>>> prelink_cmds postlink_cmds postinstall_cmds postuninstall_cmds >>>> finish_cmds sys_lib_search_path_spec configure_time_dlsearch_path >>>> configure_time_lt_sys_library_path; do >>>> # on some systems where configure will not decide to define it. >>>> # Let's still pretend it is `configure' which instantiates >>>> (i.e., don't >>>> configure_input='Generated from '` >>>> `' by configure.' >>>> configure_input="$ac_file. $configure_input" >>>> case $configure_input in #( >>>> ac_sed_conf_input=`$as_echo "$configure_input" | >>>> *) ac_sed_conf_input=$configure_input;; >>>> s|@configure_input@|$ac_sed_conf_input|;t t >>>> $as_echo "/* $configure_input */" \ >>>> $as_echo "/* $configure_input */" \ >>>> # Libtool was configured on host `(hostname || uname -n) >>>> 2>/dev/null | sed 1q`: >>>> : \${LT_SYS_LIBRARY_PATH="$configure_time_lt_sys_library_path"} >>>> sys_lib_dlsearch_path_spec=$lt_configure_time_dlsearch_path >>>> # Explicit LT_SYS_LIBRARY_PATH set during ./configure time. >>>> configure_time_lt_sys_library_path=$lt_configure_time_lt_sys_library_path >>>> >>>> >>>> OR: >>>> kylin@kylin-os:~$ xeno-config --info >>>> Xenomai version: Xenomai/cobalt v3.1-devel -- # () >>>> Linux kylin-os 4.14.4.kylin.rt-1118-ipipe-trace+ #2 SMP >>>> PREEMPT Mon Nov 18 18:28:17 CST 2019 aarch64 aarch64 aarch64 GNU/Linux >>>> Kernel parameters: BOOT_IMAGE=/Image-tmp >>>> root=UUID=9fea0634-a9c9-4e9f-906c-9c36b7249822 console=ttyS1,115200 >>>> earlyprintk=uart8250-32bit,0x28001000 rw rootdelay=10 KEYBOARDTYPE=pc >>>> KEYTABLE=us security= >>>> I-pipe release #2 detected >>>> Cobalt core 3.1-devel detected >>>> Compiler: gcc version 5.4.0 20160609 (Ubuntu/Linaro >>>> 5.4.0-6kord1~16.04.10) >>>> Build args: --prefix=/usr --includedir=/usr/include/xenomai >>>> --mandir=/usr/share/man --with-testdir=/usr/lib/xenomai/testsuite >>>> --enable-smp --build aarch64-linux-gnu build_alias=aarch64-linux-gnu >>>> >>>> Desktop: kylin 4.0.2 (ubuntu likely desktop) >>>> >>>> Issue description: >>>> latency and cyclictest work fine in my system in most of cases, >>>> the worst latency is around 100us ~ 200us. however, when i ran a >>>> script to increase the cpu load in the system, the worst latency >>>> reached 2000us ~ 5000us or even worse. Basically, the script forks 6 >>>> processes by default and each process applies a four-pages buffer and >>>> keeps writing without any breath, no warning or error messages in >>>> dmesg. below is the script: >>>> >>>> #include >>>> #include >>>> #include >>>> #include >>>> #include >>>> #include >>>> >>>> #define PAGE_SIZE 4096 >>>> #define TEST_THREADS 6 >>>> >>>> unsigned int test_threads; >>>> >>>> void do_thread_test(void) >>>> { >>>> void *mm; >>>> char i = 0; >>>> >>>> printf("mem test thread start \n"); >>>> mm = malloc(PAGE_SIZE * 4); // 1M >>>> while(1) { >>>> for (i = 0; i<100; i++) >>>> memset(mm, i, PAGE_SIZE * 4); >>>> } >>> >>> You cannot run Xenomai threads at 100% on Linux. You need to leave some >>> time for the rest of the system to do housekeeping. That explains the >>> "deadlock" you see. If you turn on CONFIG_XENO_OPT_WATCHDOG, it will >>> detect such mistakes and kick the task out of RT. >>> >>> Jan >>> >> >> "do_thread_test" is not an RT task, it's running in linux domain and >> latency running in xenomai domain was affected. > > Ok, I missed that. > > Are you using an official I-pipe release kernel (doesn't look like)? > Please try 4.19.y for that. If it's a downstream kernel (from I-pipe > perspective), your local BSP enablings may not be correct yet, breaking > RT subtly. > > Power or also thermal management are further areas of disturbance. They > may either unneeded degrade performance and, thus, latencies or need to > throttle overloaded/overheated CPUs. > > Jan > good idea, i will try an official I-pipe release kernel to see if it can be reproduced and get you back soon, many thanks. /song