timer signal loss on RT

timer signal loss on RT

[Mark Hounschell]

[-- Attachment #1: Type: text/plain, Size: 1231 bytes --]

I'm trying to run one of our applications while using the RT kernel. I am 
having a problem receiving timer signals in a RT thread pinned to a 
processor. The thread is a spinning cpu hog. I am not able to provide 
source for this application but have written a simple test case that 
_seems_ to fail the same way.  This works fine on a non-RT kernel. I have 
attached the test case. It basically sets up a 60Hz repeating timer, runs 
for 10 seconds, then displays how many signals the RT thread caught. Should 
be around 600. And it is on a vanilla kernel but not on an RT kernel. It 
does require an SMP box to run. It will NOT lockup your machine if you run 
it. Worse case would be for 10 seconds. The 2 RT priorities in use are less 
than 50 BTW.

I'm sure you can find many things wrong with this test case but I'm not 
sending it for discussion on the merits, or certainly the lack of, an 
application that contains a cpu hog, but only to show what I think is a 
problem with RT kernels.

compile with "cc rtc.c -o rtc -lrt -lpthread"

run it and in 10 seconds it should stop and indicate 600 timers signals 
were caught by the RT thread. On RT, I usually get 0 but often I get some, 
just not 600.

Thanks in advance
Mark


[-- Attachment #2: rtc.c --]
[-- Type: text/x-csrc, Size: 2995 bytes --]

#include <signal.h>
#include <stdio.h>
#include <stdint.h>
#include <stdlib.h>
#include <unistd.h>
#include <time.h>
#include <errno.h>
#include <sys/time.h>
#include <sched.h>

#define __NR_sched_set_affinity 241     
#define __NR_sched_get_affinity 242     

int32_t sig_occured = 0;
int32_t sig_count   = 0;
int32_t Sig;
int32_t exit_flg = 0;
int32_t thread_running = 0;

uint32_t pri_mask = 0x01;
uint32_t sec_mask = 0x02;

sigset_t pri_sset;
sigset_t sec_sset;
timer_t timerid;
pthread_t thread_id;

struct itimerspec RTC_value;
struct sigevent  sig_event;
struct sigaction sigaction_block;
struct sched_param sparam;

void RTC_isr() 
{
    sig_occured = 1;
}

void *thread_code(void *args)
{
    uint64_t nsecs;

    //
    // Pin self to a different processor 
    //
    syscall(__NR_sched_set_affinity, 0, sizeof(sec_mask), &sec_mask);

    //
    // Set self to SCHED_FIFO
    //
    sparam.sched_priority = 30;
    pthread_setschedparam(thread_id, SCHED_FIFO, &sparam);

    Sig = SIGRTMIN;
    sigaction_block.sa_handler = (__sighandler_t)RTC_isr;
    sigaction_block.sa_flags = SA_RESTART;

    if (sigaction(Sig, &sigaction_block, NULL) < 0) {
        perror("sigaction failed: ");
        goto out;
    }

    sig_event.sigev_signo  = Sig;
    sig_event.sigev_notify = SIGEV_SIGNAL;

    if (timer_create(CLOCK_MONOTONIC, &sig_event, &timerid) < 0) {
        perror("timer_create failed: ");
        goto out;
    }

    //
    // UNBlock signals
    //
    sigfillset(&sec_sset);
    pthread_sigmask(SIG_UNBLOCK, &sec_sset, NULL);

    //
    // start the 60 Hz repeating timer running
    //
    nsecs = (uint64_t) (1000000000LL / 60); 

    RTC_value.it_value.tv_sec     = nsecs / 1000000000LL;
    RTC_value.it_value.tv_nsec    = nsecs % 1000000000LL;
    RTC_value.it_interval.tv_sec  = RTC_value.it_value.tv_sec;
    RTC_value.it_interval.tv_nsec = RTC_value.it_value.tv_nsec;

    if (timer_settime(timerid, 0, &RTC_value, NULL) < 0) {
        perror("timer_settime failed: ");
        goto out;
    }

    thread_running = 1;

    while (!exit_flg) {
        if (sig_occured) {
            sig_occured = 0;
            sig_count++;
    }}

  out:

    if (timerid)
        timer_delete(timerid);

    return (0);
}

int32_t main(int argc, char **argv)
{
    int32_t i = 0;
    int32_t sleep_cnt = 0;

    //
    // Pin self to a processor 
    //
    syscall(__NR_sched_set_affinity, 0, sizeof(pri_mask), &pri_mask);

    //
    // Set self to SCHED_FIFO
    //
    sparam.sched_priority = 35;
    sched_setscheduler(0, SCHED_FIFO, &sparam);

    //
    // Block signals to self
    //
    sigfillset(&pri_sset);
    sigprocmask(SIG_BLOCK, &pri_sset, NULL);

    pthread_create(&thread_id, NULL, thread_code, NULL);

    while(!thread_running)
        usleep(100);

    while (sleep_cnt < 10) {
        sleep(1);
        sleep_cnt++;
    }

    exit_flg = 1;
    pthread_cancel(thread_id);

    printf("Thread process received %d signals\n", sig_count);

    exit (0);
}

Re: timer signal loss on RT

[Mike Galbraith]

On Sat, 2012-03-10 at 05:26 -0500, Mark Hounschell wrote:

> compile with "cc rtc.c -o rtc -lrt -lpthread"
> 
> run it and in 10 seconds it should stop and indicate 600 timers signals 
> were caught by the RT thread. On RT, I usually get 0 but often I get some, 
> just not 600.

Increase the priority of ksoftirqd.

-Mike

Re: timer signal loss on RT

[Thomas Gleixner]

On Sat, 10 Mar 2012, Mark Hounschell wrote:
> I'm sure you can find many things wrong with this test case but I'm not
> sending it for discussion on the merits, or certainly the lack of, an
> application that contains a cpu hog, but only to show what I think is a
> problem with RT kernels.

Nope, PEBKAC.

Timer signals are delivered via ksoftirqd, so your cpu hog prevents
ksoftirqd frum running.

Thanks,

	tglx

Re: timer signal loss on RT

[Mark Hounschell]

On 03/10/2012 06:34 AM, Thomas Gleixner wrote:
> On Sat, 10 Mar 2012, Mark Hounschell wrote:
>> I'm sure you can find many things wrong with this test case but I'm not
>> sending it for discussion on the merits, or certainly the lack of, an
>> application that contains a cpu hog, but only to show what I think is a
>> problem with RT kernels.
>
> Nope, PEBKAC.
>
> Timer signals are delivered via ksoftirqd, so your cpu hog prevents
> ksoftirqd frum running.
>

Does ksoftirqd deliver timer signals in vanilla?

Thanks
Mark

Re: timer signal loss on RT

[Steven Rostedt]

On Sat, 2012-03-10 at 06:53 -0500, Mark Hounschell wrote:
> >
> 
> Does ksoftirqd deliver timer signals in vanilla?

Sometimes :-)

Softirqs in vanilla run in interrupt context. Mostly on return from an
hardware interrupt handler, the softirqs are called and run anytime
interrupts are enabled (and bottom halves enabled). But if there's too
much work to do from the soft interrupt, then it pushes the rest of the
work to ksoftirqd. Which on vanilla runs as a normal SCHED_OTHER task.

In -rt, the hardware handlers are run as thread, and all softirqs are
handled by ksoftirqd. -rt does not process any softirqs in interrupt
context.

The answer to your question is, the timerd softirq runs mostly from
interrupt context, which will preempt your high priority CPU hog, but if
too much work is being done, it may defer it to ksoftirqd, where your
thread might block it. But chances are it wont.

-- Steve