[Xenomai-help] Busy Loop Hang

[Xenomai-help] Busy Loop Hang

[Matthew Fornero]

[-- Attachment #1: Type: text/plain, Size: 2376 bytes --]

Hello,

I've created the following test case to illustrate linux and any tasks
in the secondary domain hanging while executing a busy loop on one CPU
in an SMP system.

Xenomai 2.4.5, Linux 2.6.26.5, x86_64
Tests on quad-core AMD Opteron and dual-core Intel Core 2
kernel booted with isolcpus=1-31

Task 1 (busy loop): Priority 99, CPU 1
polls rt_timer_tsc until a timeout value is reached (1 ms)
increments a counter (master counter)
updates a global variable for the elapsed time
every four minutes, goes to sleep with rt_task_sleep

Task 2 (print loop): Priority 1, CPU 0
sets itself up as periodic using rt_task_set_periodic (1 sec)
goes to sleep with rt_task_wait_period
increments a counter (slave counter)
prints out: (1) the slave counter (2) the master counter (3) the
elapsed time since the start of the test

After some period of time (usually a couple minutes) the system will
appear to lock up. Linux (X or any terminals) will stop responding.
Once the busy loop goes to sleep, linux will begin responding again.
During the time that linux is unresponsive, the print loop does not
execute, which can be seen from looking at the slave counter values:

Slave Cnt: 195 Master Cnt: 194994 Elapsed Time: 194.999156932 s
Slave Cnt: 196 Master Cnt: 195994 Elapsed Time: 195.999183809 s
Slave Cnt: 197 Master Cnt: 196994 Elapsed Time: 196.999211373 s
<-------- linux hangs
Slave Cnt: 201 Master Cnt: 240898 Elapsed Time: 242.999124197 s
<-------- linux comes back
Slave Cnt: 202 Master Cnt: 241898 Elapsed Time: 243.999151533 s
Slave Cnt: 203 Master Cnt: 242898 Elapsed Time: 244.999178747 s

On the quad-core system, I see the same behavior if I run the print
loop on CPU 2 or 3 (CPUs isolated by the isolcpus kernel parameter).

While the busy loop example is fairly trivial, our systems will use a
similar setup for a real application:

We'd like to be able to dedicate CPU(s) in an SMP system to polling
some memory location while waiting for a flag to be DMA'd. Once the
flag is detected, the task would process the data, and then begin
polling for the flag again. Adding sleep states to the task increases
our minimum cycle time, so is undesirable. Ideally, CPU 0 having
"idle" time would be all that's required for the linux kernel and RT
tasks that switch to the secondary domain to continue running.

I've attached the test code and makefile.
--
Matthew Fornero

[-- Attachment #2: Makefile --]
[-- Type: application/octet-stream, Size: 2225 bytes --]

###### CONFIGURATION ######

### List of applications to be build
APPLICATIONS = busyloop

### Note: to override the search path for the xeno-config script, use "make XENO=..."


### List of modules to be build
MODULES =

### Note: to override the kernel source path, use "make KSRC=..."



###### USER SPACE BUILD (no change required normally) ######
ifeq ($(KERNELRELEASE),)
ifneq ($(APPLICATIONS),)

### Default Xenomai installation path
XENO ?= /usr/xenomai

XENOCONFIG=$(shell PATH=$(XENO):$(XENO)/bin:$(PATH) which xeno-config 2>/dev/null)

### Sanity check
ifeq ($(XENOCONFIG),)
all::
	@echo ">>> Invoke make like this: \"make XENO=/path/to/xeno-config\" <<<"
	@echo
endif


CC=$(shell $(XENOCONFIG) --cc)

CFLAGS=$(shell $(XENOCONFIG) --xeno-cflags) $(MY_CFLAGS)

LDFLAGS=$(shell $(XENOCONFIG) --xeno-ldflags) $(MY_LDFLAGS) -lnative

# This includes the library path of given Xenomai into the binary to make live
# easier for beginners if Xenomai's libs are not in any default search path.
LDFLAGS+=-Xlinker -rpath -Xlinker $(shell $(XENOCONFIG) --libdir)

all:: $(APPLICATIONS)

clean::
	$(RM) $(APPLICATIONS) *.o

endif
endif



###### SPECIAL TARGET RULES ######
rtprint: rtprint.c
	$(CC) $(CFLAGS) $? $(LDFLAGS) -lrtdk -o $@



###### KERNEL MODULE BUILD (no change required normally) ######
ifneq ($(MODULES),)

### Default to sources of currently running kernel
KSRC ?= /lib/modules/$(shell uname -r)/build

OBJS     := ${patsubst %, %.o, $(MODULES)}
CLEANMOD := ${patsubst %, .%*, $(MODULES)}
PWD      := $(shell if [ "$$PWD" != "" ]; then echo $$PWD; else pwd; fi)

### Kernel 2.6
ifeq ($(findstring 2.6,$(KSRC)),2.6)

obj-m        := $(OBJS)
EXTRA_CFLAGS := -I$(KSRC)/include/xenomai -I$(KSRC)/include/xenomai/posix $(ADD_CFLAGS)

all::
	$(MAKE) -C $(KSRC) SUBDIRS=$(PWD) modules

### Kernel 2.4
else

ARCH    ?= $(shell uname -i)
INCLUDE := -I$(KSRC)/include/xenomai -I$(KSRC)/include/xenomai/compat -I$(KSRC)/include/xenomai/posix
CFLAGS  += $(shell $(MAKE) -s -C $(KSRC) CC=$(CC) ARCH=$(ARCH) SUBDIRS=$(PWD) modules) $(INCLUDE)

all:: $(OBJS)

endif

## Target for capturing 2.4 module CFLAGS
modules:
	@echo "$(CFLAGS)"

clean::
	$(RM) $(CLEANMOD) *.o *.ko *.mod.c Module*.symvers
	$(RM) -R .tmp*

endif

[-- Attachment #3: busyloop.c --]
[-- Type: text/plain, Size: 2838 bytes --]

#include <stdio.h>
#include <signal.h>
#include <unistd.h>
#include <sys/mman.h>

#include <native/task.h>
#include <native/timer.h>

int end =0;
unsigned long master_cnt = 0;
RT_TASK busy_task, print_task;

RTIME current, start;

/* NOTE: error handling omitted. */

void busyloop(void *arg)
{
     RTIME now, wait_time, next, sleep_time, previous;

     sleep_time = rt_timer_ns2tsc(1000000000); // 1 second
     wait_time = rt_timer_ns2tsc(1000000); // 1 ms
     now = rt_timer_tsc(); // get the current tsc
     next = now + wait_time; // timeout value
     start = now; //start time of the task
     while (!end) {
	  while(now < next) {
	       now = rt_timer_tsc();
	       if (now < previous){ //rollover[unlikely...]
		    break;
	       }
	   
	  }
	  current = now; // update the global variables for printing
	  previous = now; // update for rollever checking
	  master_cnt++; // increment the counter
	  next = now + wait_time; // find the next timeout
	  if(master_cnt % 240000 == 0){ 
	       // break the busy loop every ~ 4 minutes, see if linux comes back
	       rt_task_sleep(sleep_time);
	       now = rt_timer_tsc();
	       next = now + wait_time;
	       previous = now;
	  }
     }
}

void printloop(void *arg)
{

     unsigned long old_master_cnt, slave_cnt;

     slave_cnt = 0;

     rt_task_set_periodic(NULL, TM_NOW, 1000000000); // 1 sec

     old_master_cnt = master_cnt;
     while (!end) {
	  rt_task_wait_period(NULL);
	  slave_cnt++;
	  /*
	   * NOTE: printf may have unexpected impact on the timing of
	   *       your program. It is used here in the critical loop
	   *       only for demonstration purposes.
	   */
		
	  if(master_cnt != old_master_cnt){
	       printf("Slave Cnt: %ld Master Cnt: %ld Elapsed Time: %ld.%06ld s\n",
		      slave_cnt,
		      master_cnt,
		      (long)rt_timer_tsc2ns(current - start) / 1000000000,
		      (long)rt_timer_tsc2ns(current - start) % 1000000000);
	       old_master_cnt = master_cnt;
	  }
     }
}

void catch_signal(int sig)
{
     end=1;
}

int main(int argc, char* argv[])
{
     signal(SIGTERM, catch_signal);
     signal(SIGINT, catch_signal);

     /* Avoids memory swapping for this program */
     mlockall(MCL_CURRENT|MCL_FUTURE);

     /*
      * Arguments: &task,
      *            name,
      *            stack size (0=default),
      *            priority,
      *            mode (FPU, start suspended, ...)
      */
     rt_task_create(&busy_task, "busy", 0, 99, T_CPU(1));
     rt_task_create(&print_task, "print", 0, 1, T_CPU(0));

     /*
      * Arguments: &task,
      *            task function,
      *            function argument
      */
     rt_task_start(&busy_task, &busyloop, NULL);
     rt_task_start(&print_task, &printloop, NULL);

     pause();

     rt_task_delete(&busy_task);
     //rt_task_delete(&print_task);
}

Re: [Xenomai-help] Busy Loop Hang

[Gilles Chanteperdrix]

Matthew Fornero wrote:
> Hello,
> 
> I've created the following test case to illustrate linux and any tasks
> in the secondary domain hanging while executing a busy loop on one CPU
> in an SMP system.
> 
> Xenomai 2.4.5, Linux 2.6.26.5, x86_64
> Tests on quad-core AMD Opteron and dual-core Intel Core 2
> kernel booted with isolcpus=1-31
> 
> Task 1 (busy loop): Priority 99, CPU 1
> polls rt_timer_tsc until a timeout value is reached (1 ms)
> increments a counter (master counter)
> updates a global variable for the elapsed time
> every four minutes, goes to sleep with rt_task_sleep

This will not work. You can not prevent linux from working during 4
minutes and hope that it continue to work normally. For a minimal
perturbation of Linux, I would advise to suspend real-time activities
every 1/HZ seconds.

-- 
                                                 Gilles.