cyclictest hangs with posix interval timers !?

cyclictest hangs with posix interval timers !?

[Steven Scholz]

Hi,

These are my very first steps into CONFIG_PREEMPT_RT.

I built a kernel 2.6.20-rc3-rt0, booted it on a x86 based single board
computer and tried to run cyclictest.

"cyclictest -t 1 -p 80 -i 10000 -l 10000 -n" runs fine.

But "cyclictest -t 1 -p 80 -i 10000 -l 10000" hangs.

Putting in some printfs I got the impression that the timerthread() blocks
at sigwait()

		case MODE_SYS_ITIMER:
			if (sigwait(&sigset, &sigs) < 0) {
				goto out;
			}
			break;

So I I booted the OSADL Live CD provided at

	http://www.osadl.org/projects-live-cd.0.html

on the same board and cyclictest runs fine in both modes.

I build a kernel 2.6.17 + 2.6.17-rt3 patch with the same kernel config used
on this CD. But cyclictest hangs when using interval timer.

So I recon it has something to do with the rootfs I am using (created with
PTXdist).

Will I need a special version of glibc or libpthread? NTPL?
I am using a cross tool chain gcc-3.4.4-glibc-2.3.5 for i686 built with
"crosstool".

Any ideas what I could be missing? Or how I could find out.

Thanks a million!

--
Steven

Re: cyclictest hangs with posix interval timers !?

[Robert Schwebel]

On Thu, Jan 04, 2007 at 12:43:11PM +0100, Steven Scholz wrote:
> Will I need a special version of glibc or libpthread? NTPL?
> I am using a cross tool chain gcc-3.4.4-glibc-2.3.5 for i686 built with
> "crosstool".

You usually need NPTL, and if I remember correctly you also needed the
hrtimer-support lib for older glibcs.

Robert 
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

Robert,

Thanks for your fast reply.

>> Will I need a special version of glibc or libpthread? NTPL?
>> I am using a cross tool chain gcc-3.4.4-glibc-2.3.5 for i686 built with
>> "crosstool".
> 
> You usually need NPTL, and if I remember correctly you also needed the
> hrtimer-support lib for older glibcs.

How could I find out if the glibc on the OASDL CD uses a patch glibc? I only
see libc-2.3.5.so ...

Steven

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

Robert Schwebel wrote:
> On Thu, Jan 04, 2007 at 12:43:11PM +0100, Steven Scholz wrote:
>> Will I need a special version of glibc or libpthread? NTPL?
>> I am using a cross tool chain gcc-3.4.4-glibc-2.3.5 for i686 built with
>> "crosstool".
> 
> You usually need NPTL, and if I remember correctly you also needed the
> hrtimer-support lib for older glibcs.
Any pointers where I could find this hrtimer-support?

Steven

Re: cyclictest hangs with posix interval timers !?

[Thomas Gleixner]

On Thu, 2007-01-04 at 14:15 +0100, Steven Scholz wrote:
> Robert Schwebel wrote:
> > On Thu, Jan 04, 2007 at 12:43:11PM +0100, Steven Scholz wrote:
> >> Will I need a special version of glibc or libpthread? NTPL?
> >> I am using a cross tool chain gcc-3.4.4-glibc-2.3.5 for i686 built with
> >> "crosstool".
> > 
> > You usually need NPTL, and if I remember correctly you also needed the
> > hrtimer-support lib for older glibcs.
> Any pointers where I could find this hrtimer-support?

cyclictest does not need any extra functionality. You only need a new
glibc if you want to use priority inheritance mutexes.

	tglx

Re: cyclictest hangs with posix interval timers !?

[Thomas Gleixner]

On Thu, 2007-01-04 at 12:43 +0100, Steven Scholz wrote:
> So I recon it has something to do with the rootfs I am using (created with
> PTXdist).

Can you try to use a standard root filesystem (fedora, debian or
whatever you have handy) ?

	tglx

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

Hi Thomas,

>>>> Will I need a special version of glibc or libpthread? NTPL?
>>>> I am using a cross tool chain gcc-3.4.4-glibc-2.3.5 for i686 built with
>>>> "crosstool".
>>> You usually need NPTL, and if I remember correctly you also needed the
>>> hrtimer-support lib for older glibcs.
>> Any pointers where I could find this hrtimer-support?
> 
> cyclictest does not need any extra functionality. You only need a new
> glibc if you want to use priority inheritance mutexes.

Ok. Then any idea how I could find out why sigwait() is blocking?

Thanks!

Steven

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

Thomas Gleixner wrote:
> On Thu, 2007-01-04 at 12:43 +0100, Steven Scholz wrote:
>> So I recon it has something to do with the rootfs I am using (created with
>> PTXdist).
> 
> Can you try to use a standard root filesystem (fedora, debian or
> whatever you have handy) ?

Nothing handy. :-(

But of course I could download some and install it.

Which would you recommend? Shell I try some recent Fedora?

Steven

Re: cyclictest hangs with posix interval timers !?

[Thomas Gleixner]

On Thu, 2007-01-04 at 14:31 +0100, Steven Scholz wrote:
> > cyclictest does not need any extra functionality. You only need a new
> > glibc if you want to use priority inheritance mutexes.
> 
> Ok. Then any idea how I could find out why sigwait() is blocking?

Can you strace cyclictest and send me the output ?

# strace -f cyclictest -t 1 -p 80 -i 10000 -l 10 -q

	tglx

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

Thomas Gleixner wrote:
> On Thu, 2007-01-04 at 14:15 +0100, Steven Scholz wrote:
>> Robert Schwebel wrote:
>>> On Thu, Jan 04, 2007 at 12:43:11PM +0100, Steven Scholz wrote:
>>>> Will I need a special version of glibc or libpthread? NTPL?
>>>> I am using a cross tool chain gcc-3.4.4-glibc-2.3.5 for i686 built with
>>>> "crosstool".
>>> You usually need NPTL, and if I remember correctly you also needed the
>>> hrtimer-support lib for older glibcs.
>> Any pointers where I could find this hrtimer-support?
> 
> cyclictest does not need any extra functionality. You only need a new
> glibc if you want to use priority inheritance mutexes.

So IIUC the cyclictest should run and give results even on kernels with no
real time, preempt, lowlatency support at all?

Just tried it on a plain 2.6.19. posix timer version still hangs ...
--
Steven

Re: cyclictest hangs with posix interval timers !?

[Thomas Gleixner]

On Thu, 2007-01-04 at 16:01 +0100, Steven Scholz wrote:
> > cyclictest does not need any extra functionality. You only need a new
> > glibc if you want to use priority inheritance mutexes.
> 
> So IIUC the cyclictest should run and give results even on kernels with no
> real time, preempt, lowlatency support at all?

Right.

> Just tried it on a plain 2.6.19. posix timer version still hangs ...

That's caused by user space breakage. Looking at the strace you sent:

[pid   932] getpid()                    = 932
[pid   932] rt_sigprocmask(SIG_SETMASK, [ALRM RTMIN], NULL, 8) = 0
[pid   932] gettid()                    = 932
[pid   932] rt_sigprocmask(SIG_BLOCK, [ALRM], NULL, 8) = 0
[pid   932] sched_setscheduler(0, SCHED_FIFO, { 80 }) = 0
[pid   932] clock_gettime(CLOCK_MONOTONIC, {325, 223507890}) = 0
[pid   932] rt_sigaction(SIGALRM, {0xb7ed72c0, ~[], 0}, NULL, 8) = 0
[pid   932] rt_sigprocmask(SIG_BLOCK, NULL, [ALRM RTMIN], 8) = 0
[pid   932] rt_sigsuspend(~[ALRM RT_1] <unfinished ...>
[pid   930] <... nanosleep resumed> NULL) = 0

There is neither a call of timer_create() nor of timer_settime(). So the
posix timer is not setup, which results in the endless wait for the
signal.

This looks like a serious problem with your cross compiler / glibc
setup. Can you compile cyclictest on one of your desktop machines and
link it statically ?

	tglx

Re: cyclictest hangs with posix interval timers !?

[Robert Schwebel]

On Fri, Jan 05, 2007 at 09:48:02AM +0100, Thomas Gleixner wrote:
> This looks like a serious problem with your cross compiler / glibc
> setup. Can you compile cyclictest on one of your desktop machines and
> link it statically?

Steven, if this is a standard ptxdist toolchain it is built with stock
crosstool, so if there is really a problem with the toolchain you might
want to check if crosstool does everything right. However, we havn't
seen such effects, although all our realtime testing is currently done
with the soon-to-be-released OSELAS.Toolchain() toolchains, which are
built with native PTXdist instead of crosstool.

Robert
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

Hi Thomas,

>> Just tried it on a plain 2.6.19. posix timer version still hangs ...
> 
> That's caused by user space breakage. Looking at the strace you sent:
> 
> [pid   932] getpid()                    = 932
> [pid   932] rt_sigprocmask(SIG_SETMASK, [ALRM RTMIN], NULL, 8) = 0
> [pid   932] gettid()                    = 932
> [pid   932] rt_sigprocmask(SIG_BLOCK, [ALRM], NULL, 8) = 0
> [pid   932] sched_setscheduler(0, SCHED_FIFO, { 80 }) = 0
> [pid   932] clock_gettime(CLOCK_MONOTONIC, {325, 223507890}) = 0
> [pid   932] rt_sigaction(SIGALRM, {0xb7ed72c0, ~[], 0}, NULL, 8) = 0
> [pid   932] rt_sigprocmask(SIG_BLOCK, NULL, [ALRM RTMIN], 8) = 0
> [pid   932] rt_sigsuspend(~[ALRM RT_1] <unfinished ...>
> [pid   930] <... nanosleep resumed> NULL) = 0

Ah. Thanks for that hint!

> This looks like a serious problem with your cross compiler / glibc
> setup. Can you compile cyclictest on one of your desktop machines and
> link it statically ?

gcc -Wall -O2  -lpthread -lrt cyclictest.c -o cyclictest -static

results in

/tmp/ccIsApGI.o: In function `main':
cyclictest.c:(.text+0x4a2): undefined reference to `pthread_create'
cyclictest.c:(.text+0x6d9): undefined reference to `pthread_kill'
cyclictest.c:(.text+0x6f0): undefined reference to `pthread_join'
/tmp/ccIsApGI.o: In function `timerthread':
cyclictest.c:(.text+0x839): undefined reference to `clock_gettime'
cyclictest.c:(.text+0x881): undefined reference to `timer_settime'
cyclictest.c:(.text+0x90c): undefined reference to `clock_gettime'
cyclictest.c:(.text+0xa7b): undefined reference to `clock_gettime'
cyclictest.c:(.text+0xa8c): undefined reference to `clock_nanosleep'
cyclictest.c:(.text+0xb08): undefined reference to `timer_delete'
cyclictest.c:(.text+0xb79): undefined reference to `clock_gettime'
cyclictest.c:(.text+0xbd8): undefined reference to `clock_nanosleep'
cyclictest.c:(.text+0xc37): undefined reference to `timer_create'
collect2: ld returned 1 exit status

???

Shouldn't pthread_* be in libpthread?

Steven

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

Thomas,

> That's caused by user space breakage. Looking at the strace you sent:
> 
> [pid   932] getpid()                    = 932
> [pid   932] rt_sigprocmask(SIG_SETMASK, [ALRM RTMIN], NULL, 8) = 0
> [pid   932] gettid()                    = 932
> [pid   932] rt_sigprocmask(SIG_BLOCK, [ALRM], NULL, 8) = 0
> [pid   932] sched_setscheduler(0, SCHED_FIFO, { 80 }) = 0
> [pid   932] clock_gettime(CLOCK_MONOTONIC, {325, 223507890}) = 0
> [pid   932] rt_sigaction(SIGALRM, {0xb7ed72c0, ~[], 0}, NULL, 8) = 0
> [pid   932] rt_sigprocmask(SIG_BLOCK, NULL, [ALRM RTMIN], 8) = 0
> [pid   932] rt_sigsuspend(~[ALRM RT_1] <unfinished ...>
> [pid   930] <... nanosleep resumed> NULL) = 0
> 
> There is neither a call of timer_create() nor of timer_settime(). So the
> posix timer is not setup, which results in the endless wait for the
> signal.

cyclictest runs fine on my SuSE 10.0 host system. But: there are *no* calls
to any timer_* function!!!
Has it to with using TLS?

:~/Projekte/Latency/cyclictest> ldd ./cyclictest
        linux-gate.so.1 =>  (0xffffe000)
        libpthread.so.0 => /lib/tls/libpthread.so.0 (0x4003a000)
        librt.so.1 => /lib/tls/librt.so.1 (0x4004c000)
        libc.so.6 => /lib/tls/libc.so.6 (0x40055000)
        /lib/ld-linux.so.2 (0x40000000)

Steven

Re: cyclictest hangs with posix interval timers !?

[Thomas Gleixner]

Steven,

On Sun, 2007-01-07 at 13:51 +0100, Steven Scholz wrote:
> > There is neither a call of timer_create() nor of timer_settime(). So the
> > posix timer is not setup, which results in the endless wait for the
> > signal.
> 
> cyclictest runs fine on my SuSE 10.0 host system. But: there are *no* calls
> to any timer_* function!!!

Did you check that with strace ?

> Has it to with using TLS?

No.

> :~/Projekte/Latency/cyclictest> ldd ./cyclictest
>         linux-gate.so.1 =>  (0xffffe000)
>         libpthread.so.0 => /lib/tls/libpthread.so.0 (0x4003a000)
>         librt.so.1 => /lib/tls/librt.so.1 (0x4004c000)
>         libc.so.6 => /lib/tls/libc.so.6 (0x40055000)
>         /lib/ld-linux.so.2 (0x40000000)

The timer_* functions are in librt.

	tglx

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

[-- Attachment #1: Type: text/plain, Size: 1277 bytes --]

Thomas,

>>> There is neither a call of timer_create() nor of timer_settime(). So the
>>> posix timer is not setup, which results in the endless wait for the
>>> signal.
>> cyclictest runs fine on my SuSE 10.0 host system. But: there are *no* calls
>> to any timer_* function!!!
> 
> Did you check that with strace ?

Yes. I attach that run.

 gcc -Wall -O2  -lpthread -lrt cyclictest.c -o cyclictest

 ldd ./cyclictest
        linux-gate.so.1 =>  (0xffffe000)
        libpthread.so.0 => /lib/tls/libpthread.so.0 (0x4003a000)
        librt.so.1 => /lib/tls/librt.so.1 (0x4004c000)
        libc.so.6 => /lib/tls/libc.so.6 (0x40055000)
        /lib/ld-linux.so.2 (0x40000000)

 strace ./cyclictest -t 1 -p 80 -i 10000 -l 10 -v 2> cyclictest.txt

 grep timer cyclictest.txt
 <nothing>

> 
>> Has it to with using TLS?
> No.
IIUC TLS is only usefull for SMP systems?

>> :~/Projekte/Latency/cyclictest> ldd ./cyclictest
>>         linux-gate.so.1 =>  (0xffffe000)
>>         libpthread.so.0 => /lib/tls/libpthread.so.0 (0x4003a000)
>>         librt.so.1 => /lib/tls/librt.so.1 (0x4004c000)
>>         libc.so.6 => /lib/tls/libc.so.6 (0x40055000)
>>         /lib/ld-linux.so.2 (0x40000000)
> 
> The timer_* functions are in librt.
Then why does the static linking not work?

Steven

[-- Attachment #2: cyclictest.txt --]
[-- Type: text/plain, Size: 7992 bytes --]

execve("./cyclictest", ["./cyclictest", "-t", "1", "-p", "80", "-i", "10000", "-l", "10", "-v"], [/* 69 vars */]) = 0
uname({sys="Linux", node="pinguin", ...}) = 0
brk(0)                                  = 0x804c000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=135508, ...}) = 0
old_mmap(NULL, 135508, PROT_READ, MAP_PRIVATE, 3, 0) = 0x40017000
close(3)                                = 0
open("/lib/tls/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0pH\0\000"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=93266, ...}) = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40039000
old_mmap(NULL, 70104, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x4003a000
madvise(0x4003a000, 70104, MADV_SEQUENTIAL|0x1) = 0
old_mmap(0x40048000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0xd000) = 0x40048000
old_mmap(0x4004a000, 4568, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x4004a000
close(3)                                = 0
open("/lib/tls/librt.so.1", O_RDONLY)   = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0000\36\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=40833, ...}) = 0
old_mmap(NULL, 33384, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x4004c000
madvise(0x4004c000, 33384, MADV_SEQUENTIAL|0x1) = 0
old_mmap(0x40053000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6000) = 0x40053000
close(3)                                = 0
open("/lib/tls/libc.so.6", O_RDONLY)    = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\240O\1"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1417095, ...}) = 0
old_mmap(NULL, 1174524, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x40055000
madvise(0x40055000, 1174524, MADV_SEQUENTIAL|0x1) = 0
old_mmap(0x4016e000, 16384, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x119000) = 0x4016e000
old_mmap(0x40172000, 7164, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x40172000
close(3)                                = 0
old_mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40174000
mprotect(0x4016e000, 4096, PROT_READ)   = 0
set_thread_area({entry_number:-1 -> 6, base_addr:0x401746c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
munmap(0x40017000, 135508)              = 0
set_tid_address(0x40174708)             = 5192
rt_sigaction(SIGRTMIN, {0x4003e3c0, [], SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x4003e440, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFINITY}) = 0
_sysctl({{CTL_KERN, KERN_VERSION, 0, 20529, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, 2, 0xbf853c00, 34, (nil), 0}) = 0
geteuid32()                             = 0
open("/proc/version", O_RDONLY)         = 3
read(3, "Linux version 2.6.13-15.8-smp (g"..., 255) = 138
close(3)                                = 0
rt_sigprocmask(SIG_BLOCK, [ALRM], NULL, 8) = 0
rt_sigaction(SIGINT, {0x8048c00, [INT], SA_RESTART}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGTERM, {0x8048c00, [TERM], SA_RESTART}, {SIG_DFL}, 8) = 0
brk(0)                                  = 0x804c000
brk(0x806d000)                          = 0x806d000
mmap2(NULL, 2101248, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40175000
mprotect(0x40175000, 4096, PROT_NONE)   = 0
clone(child_stack=0x403754c4, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x40375bf8, {entry_number:6, base_addr:0x40375bb0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}, child_tidptr=0x40375bf8) = 5193
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 10000000}, NULL)          = 0
fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40017000
write(1, "       0:       0:       0\n", 27) = 27
nanosleep({0, 10000000}, NULL)          = 0
write(1, "       0:       1:    3508\n", 27) = 27
nanosleep({0, 10000000}, NULL)          = 0
write(1, "       0:       2:    5517\n", 27) = 27
write(1, "       0:       3:    7510\n", 27) = 27
nanosleep({0, 10000000}, NULL)          = 0
write(1, "       0:       4:    9508\n", 27) = 27
nanosleep({0, 10000000}, NULL)          = 0
write(1, "       0:       5:   11503\n", 27) = 27
nanosleep({0, 10000000}, NULL)          = 0
write(1, "       0:       6:   13517\n", 27) = 27
write(1, "       0:       7:   15512\n", 27) = 27
nanosleep({0, 10000000}, NULL)          = 0
write(1, "       0:       8:   17518\n", 27) = 27
nanosleep({0, 10000000}, NULL)          = 0
write(1, "       0:       9:   19519\n", 27) = 27
nanosleep({0, 10000000}, NULL)          = 0
nanosleep({0, 50000000}, NULL)          = 0
munmap(0x40017000, 4096)                = 0
exit_group(0)                           = ?

Re: cyclictest hangs with posix interval timers !?

[Thomas Gleixner]

Steven,

On Sun, 2007-01-07 at 15:38 +0100, Steven Scholz wrote:
>  strace ./cyclictest -t 1 -p 80 -i 10000 -l 10 -v 2> cyclictest.txt

You need to run "strace -f ...." otherwise you don't get the thread
strace.

> >> Has it to with using TLS?
> > No.
> IIUC TLS is only usefull for SMP systems?

Thread Local Storage is used to store per thread local information. This
is needed as the threads of a process share the same address space. NPTL
needs TLS support.
 
> > The timer_* functions are in librt.
> Then why does the static linking not work?

Are the static libs available ?

	tglx

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

Thomas Gleixner wrote:
> Steven,
> 
> On Sun, 2007-01-07 at 15:38 +0100, Steven Scholz wrote:
>>  strace ./cyclictest -t 1 -p 80 -i 10000 -l 10 -v 2> cyclictest.txt
> 
> You need to run "strace -f ...." otherwise you don't get the thread
> strace.

Ok. My fault. Sorry. Of course now the trace contains

[pid 20910] timer_create(CLOCK_MONOTONIC, {(nil), 14, SIGEV_THREAD_ID,
{20910}}, {(nil)}) = 0
[pid 20910] timer_settime(0, TIMER_ABSTIME, {it_interval={0, 10000000},
it_value={3908778, 132879754}}, NULL) = 0
[pid 20910] timer_delete(0)             = 0

>>> The timer_* functions are in librt.
>> Then why does the static linking not work?
> 
> Are the static libs available ?
Ah. Maybe not. Which would that be? libpthread.a, librt.a, libc.a?

Steven

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

Thomas,

>>> The timer_* functions are in librt.
>> Then why does the static linking not work?
> 
> Are the static libs available ?

Well. On the host system with /lib/tls/librt.so.1 there is only a
/usr/lib/librt.a. So maybe that does not work.

But the crosstool chain (built with crosstool) has all the static libs ...

Steven

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

Thomas,

> This looks like a serious problem with your cross compiler / glibc
> setup. Can you compile cyclictest on one of your desktop machines and
> link it statically ?

As I wrote I have problems linking it statically!

 gcc -Wall -O2  -lpthread -lrt cyclictest.c -o cyclictest -static

results in

/tmp/cc30ryGd.o: In function `main':
cyclictest.c:(.text+0x4a2): undefined reference to `pthread_create'
cyclictest.c:(.text+0x6d9): undefined reference to `pthread_kill'
cyclictest.c:(.text+0x6f0): undefined reference to `pthread_join'
/tmp/cc30ryGd.o: In function `timerthread':
cyclictest.c:(.text+0x839): undefined reference to `clock_gettime'
cyclictest.c:(.text+0x881): undefined reference to `timer_settime'
cyclictest.c:(.text+0x90c): undefined reference to `clock_gettime'
cyclictest.c:(.text+0xa7b): undefined reference to `clock_gettime'
cyclictest.c:(.text+0xa8c): undefined reference to `clock_nanosleep'
cyclictest.c:(.text+0xb08): undefined reference to `timer_delete'
cyclictest.c:(.text+0xb79): undefined reference to `clock_gettime'
cyclictest.c:(.text+0xbd8): undefined reference to `clock_nanosleep'
cyclictest.c:(.text+0xc37): undefined reference to `timer_create'
collect2: ld returned 1 exit status

while

 gcc -Wall -O2 cyclictest.c -o cyclictest -lrt -lpthread -static

compiles and links fine!

Any idea why the order matters?

--
Steven

Re: cyclictest hangs with posix interval timers !?

[Robert Schwebel]

On Mon, Jan 08, 2007 at 11:10:16AM +0100, Steven Scholz wrote:
> As I wrote I have problems linking it statically!
> 
>  gcc -Wall -O2  -lpthread -lrt cyclictest.c -o cyclictest -static
> 
> results in

...
 
> while
> 
>  gcc -Wall -O2 cyclictest.c -o cyclictest -lrt -lpthread -static
> 
> compiles and links fine!
> 
> Any idea why the order matters?

The order does always matter. From 'info gcc': 

     It makes a difference where in the command you write this option;
     the linker searches and processes libraries and object files in
     the order they are specified.  Thus, `foo.o -lz bar.o' searches
     library `z' after file `foo.o' but before `bar.o'.  If `bar.o'
     refers to functions in `z', those functions may not be loaded.

So in your first case, run with -v to see the linker options:

rsc@isonoe:~/tmp/cyclictest$ gcc -v -Wall -O2  -lpthread -lrt cyclictest.c -o cyclictest -static 2>&1 | grep collect2
 /usr/lib/gcc/i486-linux-gnu/4.1.2/collect2 -m elf_i386 -static -o
cyclictest /usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../lib/crt1.o
/usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../lib/crti.o
/usr/lib/gcc/i486-linux-gnu/4.1.2/crtbeginT.o
-L/usr/lib/gcc/i486-linux-gnu/4.1.2 -L/usr/lib/gcc/i486-linux-gnu/4.1.2
-L/usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../lib -L/lib/../lib
-L/usr/lib/../lib -lpthread -lrt /tmp/cc2ooNdo.o --start-group -lgcc
-lgcc_eh -lc --end-group /usr/lib/gcc/i486-linux-gnu/4.1.2/crtend.o
/usr/lib/gcc/i486-linux-gnu/4.1.2/../../../../lib/crtn.o
collect2: ld returned 1 exit status

... crt1.o crti.o crtbeginT.o -lpthread -lrt cc2ooNdo.o crtend.o

Robert
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

[-- Attachment #1: Type: text/plain, Size: 631 bytes --]

Hi,

> There is neither a call of timer_create() nor of timer_settime(). So the
> posix timer is not setup, which results in the endless wait for the
> signal.
> 
> This looks like a serious problem with your cross compiler / glibc
> setup. Can you compile cyclictest on one of your desktop machines and
> link it statically ?

Well the statically linked version on SuSE 10.0 hangs as well.

Using the crosstool chains

	gcc-4.1.1-glibc-2.3.6-linux-2.6.15.1 (2.6.12 headers)
	gcc-3.4.4-glibc-2.3.5-linux-2.6.12

the cyclictest segafaults on the SuSE host and target system. strace attached.

Any more ideas? Maybe? Please?

Steven

[-- Attachment #2: cyclictest.txt --]
[-- Type: text/plain, Size: 4637 bytes --]

execve("./cyclictest", ["./cyclictest", "-t", "1", "-p", "80", "-i", "10000", "-l", "10", "-v"], [/* 69 vars */]) = 0
uname({sys="Linux", node="pinguin", ...}) = 0
set_thread_area({entry_number:-1 -> 6, base_addr:0x80b9580, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
getpid()                                = 17701
rt_sigaction(SIGRTMIN, {0x804db08, [], 0}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x804dc31, [RTMIN], 0}, NULL, 8) = 0
rt_sigaction(SIGRT_2, {0x804dddc, [], 0}, NULL, 8) = 0
rt_sigprocmask(SIG_BLOCK, [RTMIN], NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RT_1], NULL, 8) = 0
_sysctl({{CTL_KERN, KERN_VERSION, 0, 20431, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0}, 2, 0xbfd40714, 34, (nil), 0}) = 0
brk(0)                                  = 0x8109000
brk(0x812a000)                          = 0x812a000
geteuid32()                             = 0
open("/proc/version", O_RDONLY)         = 3
read(3, "Linux version 2.6.13-15.8-smp (g"..., 255) = 138
close(3)                                = 0
rt_sigprocmask(SIG_BLOCK, [ALRM], NULL, 8) = 0
rt_sigaction(SIGINT, {0x804f724, [INT], SA_RESTART}, {SIG_DFL}, 8) = 0
rt_sigaction(SIGTERM, {0x804f724, [TERM], SA_RESTART}, {SIG_DFL}, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=RLIM_INFINITY, rlim_max=RLIM_INFINITY}) = 0
pipe([3, 4])                            = 0
clone(Process 17702 attached
child_stack=0x811b304, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND) = 17702
[pid 17701] write(4, "\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\20{J\5\10\0\0\0\0\17\0\0"..., 148 <unfinished ...>
[pid 17702] set_thread_area({entry_number: <unfinished ...>
[pid 17701] <... write resumed> )       = 148
[pid 17702] <... set_thread_area resumed> 6, base_addr:0x80b99a0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
[pid 17701] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid 17702] rt_sigprocmask(SIG_SETMASK, ~[TRAP RT_1],  <unfinished ...>
[pid 17701] <... rt_sigprocmask resumed> [ALRM RTMIN], 8) = 0
[pid 17702] <... rt_sigprocmask resumed> NULL, 8) = 0
[pid 17701] write(4, "\200\225\v\10\0\0\0\0\0\0\0\0\0\202\4\10\350\222\20\10"..., 148 <unfinished ...>
[pid 17702] read(3,  <unfinished ...>
[pid 17701] <... write resumed> )       = 148
[pid 17702] <... read resumed> "\0\0\0\0\5\0\0\0\0\0\0\0\0\0\0\20{J\5\10\0\0\0\0\17\0\0"..., 148) = 148
[pid 17701] rt_sigprocmask(SIG_SETMASK, NULL,  <unfinished ...>
[pid 17702] poll( <unfinished ...>
[pid 17701] <... rt_sigprocmask resumed> [ALRM RTMIN], 8) = 0
[pid 17702] <... poll resumed> [{fd=3, events=POLLIN, revents=POLLIN}], 1, 2000) = 1
[pid 17701] rt_sigsuspend([ALRM] <unfinished ...>
[pid 17702] getppid()                   = 17701
[pid 17702] read(3, "\200\225\v\10\0\0\0\0\0\0\0\0\0\202\4\10\350\222\20\10"..., 148) = 148
[pid 17702] mmap2(NULL, 8388608, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x40000000
[pid 17702] mprotect(0x40000000, 4096, PROT_NONE) = 0
[pid 17702] clone(Process 17703 attached
child_stack=0x407ffbc4, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|SIGRT_1) = 17703
[pid 17702] kill(17701, SIGRTMIN <unfinished ...>
[pid 17703] set_thread_area({entry_number: <unfinished ...>
[pid 17701] --- SIGRTMIN (Unknown signal 32) @ 0 (0) ---
[pid 17702] <... kill resumed> )        = 0
[pid 17703] <... set_thread_area resumed> 6, base_addr:0x407ffbe0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
[pid 17701] <... rt_sigsuspend resumed> ) = -1 EINTR (Interrupted system call)
[pid 17702] poll( <unfinished ...>
[pid 17703] getpid( <unfinished ...>
[pid 17701] sigreturn( <unfinished ...>
[pid 17703] <... getpid resumed> )      = 17703
[pid 17701] <... sigreturn resumed> )   = ? (mask now [ALRM RTMIN])
[pid 17703] rt_sigprocmask(SIG_SETMASK, [ALRM RTMIN],  <unfinished ...>
[pid 17701] nanosleep({0, 10000000},  <unfinished ...>
[pid 17703] <... rt_sigprocmask resumed> NULL, 8) = 0
[pid 17703] gettid()                    = 17703
[pid 17703] rt_sigprocmask(SIG_BLOCK, [ALRM], NULL, 8) = 0
[pid 17703] sched_setscheduler(0, SCHED_FIFO, { 80 }) = 0
[pid 17703] clock_gettime(CLOCK_MONOTONIC, {3983601, 531609754}) = 0
[pid 17703] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
Process 17703 detached
[pid 17701] <... nanosleep resumed> 0)  = ? ERESTART_RESTARTBLOCK (To be restarted)
[pid 17702] <... poll resumed> [{fd=3, events=POLLIN}], 1, 2000) = -1 EINTR (Interrupted system call)
[pid 17701] +++ killed by SIGKILL +++
+++ killed by SIGKILL +++

Re: cyclictest hangs with posix interval timers !?

[Robert Schwebel]

On Mon, Jan 08, 2007 at 12:37:34PM +0100, Steven Scholz wrote:
> Well the statically linked version on SuSE 10.0 hangs as well.
> 
> Using the crosstool chains
> 
> 	gcc-4.1.1-glibc-2.3.6-linux-2.6.15.1 (2.6.12 headers)
> 	gcc-3.4.4-glibc-2.3.5-linux-2.6.12
> 
> the cyclictest segafaults on the SuSE host and target system. strace attached.
> 
> Any more ideas? Maybe? Please?

How much RAM does your box have?

Robert
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: cyclictest hangs with posix interval timers !?

[Steven Scholz]

Robert Schwebel wrote:
> On Mon, Jan 08, 2007 at 12:37:34PM +0100, Steven Scholz wrote:
>> Well the statically linked version on SuSE 10.0 hangs as well.
>>
>> Using the crosstool chains
>>
>> 	gcc-4.1.1-glibc-2.3.6-linux-2.6.15.1 (2.6.12 headers)
>> 	gcc-3.4.4-glibc-2.3.5-linux-2.6.12
>>
>> the cyclictest segafaults on the SuSE host and target system. strace attached.
>>
>> Any more ideas? Maybe? Please?
> 
> How much RAM does your box have?

The desktop has 1GB. The kontron cp306 target has 512MB.

--
Steven