Re: hith latency when cpu is fully loaded

[chensong <chensong@kylinos.cn>]

On 2019年11月19日 16:03, Jan Kiszka wrote:
> On 19.11.19 08:45, chensong wrote:
>>
>>
>> On 2019年11月19日 14:49, Jan Kiszka wrote:
>>> On 19.11.19 02:01, chensong via Xenomai wrote:
>>>> Dear experts,
>>>>
>>>> i'm new in xenomai, i got an issue, here is the detail:
>>>>
>>>> Main processor architect: ARM64 phytium ft2000ahk
>>>> Kernel release number: 4.14.4
>>>> cmdline:BOOT_IMAGE=/Image-tmp
>>>> root=UUID=9fea0634-a9c9-4e9f-906c-9c36b7249822 console=ttyS1,115200
>>>> earlyprintk=uart8250-32bit,0x28001000 rw rootdelay=10 KEYBOARDTYPE=pc
>>>> KEYTABLE=us security=
>>>> xenomai release number:3.1-devel
>>>> xenomai configuration:
>>>>         kylin@kylin-os:~/workspace/code/nudt-hgj-xenomai-tjrd$ grep
>>>> configure config.status
>>>>         # Generated by configure.
>>>>        # Compiler output produced by configure, useful for debugging
>>>>         # configure, is in config.log if it exists.
>>>>         configured by ./configure, generated by GNU Autoconf 2.69,
>>>>        ac_configure_extra_args=
>>>>        ac_configure_extra_args="$ac_configure_extra_args --silent"
>>>>        set X /bin/bash './configure'  '--with-core=cobalt'
>>>> '--enable-smp' '--enable-pshared' $ac_configure_extra_args --no-create
>>>> --no-recursion
>>>>        configure_time_dlsearch_path='/lib /usr/lib
>>>> /lib/aarch64-linux-gnu /usr/lib/aarch64-linux-gnu
>>>> /usr/lib/aarch64-linux-gnu/mesa-egl /usr/lib/aarch64-linux-gnu/mesa
>>>> /usr/local/lib '
>>>>        configure_time_lt_sys_library_path=''
>>>>       for var in reload_cmds old_postinstall_cmds
>>>> old_postuninstall_cmds old_archive_cmds extract_expsyms_cmds
>>>> old_archive_from_new_cmds old_archive_from_expsyms_cmds archive_cmds
>>>> archive_expsym_cmds module_cmds module_expsym_cmds export_symbols_cmds
>>>> prelink_cmds postlink_cmds postinstall_cmds postuninstall_cmds
>>>> finish_cmds sys_lib_search_path_spec configure_time_dlsearch_path
>>>> configure_time_lt_sys_library_path; do
>>>>        # on some systems where configure will not decide to define it.
>>>>        # Let's still pretend it is `configure' which instantiates
>>>> (i.e., don't
>>>>        configure_input='Generated from '`
>>>>         `' by configure.'
>>>>        configure_input="$ac_file.  $configure_input"
>>>>         case $configure_input in #(
>>>>         ac_sed_conf_input=`$as_echo "$configure_input" |
>>>>         *) ac_sed_conf_input=$configure_input;;
>>>>          s|@configure_input@|$ac_sed_conf_input|;t t
>>>>        $as_echo "/* $configure_input  */" \
>>>>      $as_echo "/* $configure_input  */" \
>>>>       # Libtool was configured on host `(hostname || uname -n)
>>>> 2>/dev/null | sed 1q`:
>>>>        : \${LT_SYS_LIBRARY_PATH="$configure_time_lt_sys_library_path"}
>>>>         sys_lib_dlsearch_path_spec=$lt_configure_time_dlsearch_path
>>>>        # Explicit LT_SYS_LIBRARY_PATH set during ./configure time.
>>>> configure_time_lt_sys_library_path=$lt_configure_time_lt_sys_library_path
>>>>
>>>>
>>>> OR:
>>>>         kylin@kylin-os:~$ xeno-config --info
>>>>          Xenomai version: Xenomai/cobalt v3.1-devel -- # ()
>>>>         Linux kylin-os 4.14.4.kylin.rt-1118-ipipe-trace+ #2 SMP
>>>> PREEMPT Mon Nov 18 18:28:17 CST 2019 aarch64 aarch64 aarch64 GNU/Linux
>>>>         Kernel parameters: BOOT_IMAGE=/Image-tmp
>>>> root=UUID=9fea0634-a9c9-4e9f-906c-9c36b7249822 console=ttyS1,115200
>>>> earlyprintk=uart8250-32bit,0x28001000 rw rootdelay=10 KEYBOARDTYPE=pc
>>>> KEYTABLE=us security=
>>>>         I-pipe release #2 detected
>>>>         Cobalt core 3.1-devel detected
>>>>         Compiler: gcc version 5.4.0 20160609 (Ubuntu/Linaro
>>>> 5.4.0-6kord1~16.04.10)
>>>>         Build args: --prefix=/usr --includedir=/usr/include/xenomai
>>>> --mandir=/usr/share/man --with-testdir=/usr/lib/xenomai/testsuite
>>>> --enable-smp --build aarch64-linux-gnu build_alias=aarch64-linux-gnu
>>>>
>>>> Desktop: kylin 4.0.2 (ubuntu likely desktop)
>>>>
>>>> Issue description:
>>>>        latency and cyclictest work fine in my system in most of cases,
>>>> the worst latency is around 100us ~ 200us. however, when i ran a
>>>> script to increase the cpu load in the system, the worst latency
>>>> reached 2000us ~ 5000us or even worse. Basically, the script forks 6
>>>> processes by default and each process applies a four-pages buffer and
>>>> keeps writing without any breath, no warning or error messages in
>>>> dmesg. below is the script:
>>>>
>>>> #include <stdio.h>
>>>> #include <stdlib.h>
>>>> #include <sys/types.h>
>>>> #include <sys/wait.h>
>>>> #include <sys/stat.h>
>>>> #include <fcntl.h>
>>>>
>>>> #define PAGE_SIZE 4096
>>>> #define TEST_THREADS 6
>>>>
>>>> unsigned int test_threads;
>>>>
>>>> void do_thread_test(void)
>>>> {
>>>>      void *mm;
>>>>      char i = 0;
>>>>
>>>>      printf("mem test thread start \n");
>>>>      mm = malloc(PAGE_SIZE * 4); // 1M
>>>>      while(1) {
>>>>          for (i = 0; i<100; i++)
>>>>              memset(mm, i, PAGE_SIZE * 4);
>>>>      }
>>>
>>> You cannot run Xenomai threads at 100% on Linux. You need to leave some
>>> time for the rest of the system to do housekeeping. That explains the
>>> "deadlock" you see. If you turn on CONFIG_XENO_OPT_WATCHDOG, it will
>>> detect such mistakes and kick the task out of RT.
>>>
>>> Jan
>>>
>>
>> "do_thread_test" is not an RT task, it's running in linux domain and
>> latency running in xenomai domain was affected.
>
> Ok, I missed that.
>
> Are you using an official I-pipe release kernel (doesn't look like)?
> Please try 4.19.y for that. If it's a downstream kernel (from I-pipe
> perspective), your local BSP enablings may not be correct yet, breaking
> RT subtly.
>
> Power or also thermal management are further areas of disturbance. They
> may either unneeded degrade performance and, thus, latencies or need to
> throttle overloaded/overheated CPUs.
>
> Jan
>

good idea, i will try an official I-pipe release kernel to see if it can 
be reproduced and get you back soon, many thanks.

/song