hang in creating the trinity-testfileX

hang in creating the trinity-testfileX

[Toralf Förster]

Trinity sometimes hangs infinitely in a KVM while trying to create the testfiles, eg .:

...
[main] fd[668] = timerfd
[main] fd[669] = fopen("trinity-testfile1", O_RDWR)
[main] fd[670] = fopen("trinity-testfile2", O_RDWR)


Ctrl-C works fortunately

-- 
Toralf
pgp key: 0076 E94E

Re: hang in creating the trinity-testfileX

[Dave Jones]

On Mon, Sep 01, 2014 at 07:53:47PM +0200, Toralf Förster wrote:
 > Trinity sometimes hangs infinitely in a KVM while trying to create the testfiles, eg .:
 > 
 > ...
 > [main] fd[668] = timerfd
 > [main] fd[669] = fopen("trinity-testfile1", O_RDWR)
 > [main] fd[670] = fopen("trinity-testfile2", O_RDWR)

so creating the 3rd file is where it hangs ? weird.
 
 > Ctrl-C works fortunately

Can you gdb it and find out where it's looping ?

	Dave

Re: hang in creating the trinity-testfileX

[Toralf Förster]

On 09/02/2014 07:37 PM, Dave Jones wrote:
> On Mon, Sep 01, 2014 at 07:53:47PM +0200, Toralf Förster wrote:
>  > Trinity sometimes hangs infinitely in a KVM while trying to create the testfiles, eg .:
>  > 
>  > ...
>  > [main] fd[668] = timerfd
>  > [main] fd[669] = fopen("trinity-testfile1", O_RDWR)
>  > [main] fd[670] = fopen("trinity-testfile2", O_RDWR)
> 
> so creating the 3rd file is where it hangs ? weird.
>  
>  > Ctrl-C works fortunately
> 
> Can you gdb it and find out where it's looping ?
> 
> 	Dave
> 
> 

this is from one ssh terminal of the KVM :

...
[main] fd[654] = fopen /proc/376/task/376/net/snmp6 (read-only) flags:0
[main] fd[655] = fopen /sys/devices/virtual/net/lo/statistics/collisions (read-only) flags:0
[main] fd[656] = open /proc/376/net/fib_triestat (read-only) flags:30b380
[main] fd[657] = open /proc/365/fdinfo/1 (read-only) flags:281800
[main] fd[658] = fopen /proc/21162/task/21162/wchan (read-only) flags:0
[main] fd[659] = fopen /proc/1971/attr/keycreate (read-write) flags:2
[main] fd[660] = fopen /proc/786/net/snmp (read-only) flags:0
[main] fd[661] = timerfd
[main] fd[662] = timerfd
[main] fd[663] = timerfd
[main] fd[664] = timerfd
[main] fd[665] = timerfd
[main] fd[666] = timerfd
[main] fd[667] = timerfd
[main] fd[668] = timerfd
[main] fd[669] = open("trinity-testfile1", flags:1000)          <-------------- here it hangs



and this info I derived within another ssh session to the KVM :

tfoerste@n22kvm ~ $ ps fx -eo pid,start_time,command | grep -e trinity -e sleep | grep -v grep
  375 19:40  |           \_ trinity -C 3 -q
  376 19:40  |               \_ trinity -C 3 -q
tfoerste@n22kvm ~ $ gdb trinity 375 -n -batch -ex 'thread apply all bt'

warning: Could not load shared library symbols for linux-gate.so.1.
Do you need "set solib-search-path" or "set sysroot"?

warning: no loadable sections found in added symbol-file system-supplied DSO at 0xb772a000
0xb772acec in ?? ()

Thread 1 (process 375):
#0  0xb772acec in ?? ()
#1  0xb7591ee5 in __libc_start_main (main=0x8053930 <main>, argc=4, argv=0xbff5b8d4, init=0x80672c0 <__libc_csu_init>, fini=0x8067330 <__libc_csu_fini>, rtld_fini=0xb773b4e0 <_dl_fini>, stack_end=0xbff5b8cc) at libc-start.c:285
#2  0x08054125 in _start ()
tfoerste@n22kvm ~ $ gdb trinity 376 -n -batch -ex 'thread apply all bt'

warning: Could not load shared library symbols for linux-gate.so.1.
Do you need "set solib-search-path" or "set sysroot"?

warning: no loadable sections found in added symbol-file system-supplied DSO at 0xb772a000
0xb772acec in ?? ()

Thread 1 (process 376):
#0  0xb772acec in ?? ()
#1  0xb7591ee5 in __libc_start_main (main=0x8053930 <main>, argc=4, argv=0xbff5b8d4, init=0x80672c0 <__libc_csu_init>, fini=0x8067330 <__libc_csu_fini>, rtld_fini=0xb773b4e0 <_dl_fini>, stack_end=0xbff5b8cc) at libc-start.c:285
#2  0x08054125 in _start ()
tfoerste@n22kvm ~ $ sudo cat /proc/375/stack
[<c5a4a574>] do_wait+0x1b4/0x200
[<c5a4b3a2>] SyS_waitpid+0x72/0xe0
[<c60cc761>] syscall_after_call+0x0/0x4
[<ffffffff>] 0xffffffff
tfoerste@n22kvm ~ $ sudo cat /proc/376/stack
[<c5a97ac8>] hrtimer_nanosleep+0xa8/0x1b0
[<c5a97c26>] SyS_nanosleep+0x56/0x70
[<c60cc66b>] sysenter_after_call+0x0/0x21
[<ffffffff>] 0xffffffff


-- 
Toralf
pgp key: 0076 E94E

Re: hang in creating the trinity-testfileX

[Dave Jones]

On Mon, Sep 01, 2014 at 07:53:47PM +0200, Toralf Förster wrote:
 > Trinity sometimes hangs infinitely in a KVM while trying to create the testfiles, eg .:
 > 
 > ...
 > [main] fd[668] = timerfd
 > [main] fd[669] = fopen("trinity-testfile1", O_RDWR)
 > [main] fd[670] = fopen("trinity-testfile2", O_RDWR)
 > 
 > 
 > Ctrl-C works fortunately

I just hit this myself. It happens if the test files already exist.
For now just clear out tmp/* before re-running.

I'll push out some changes later to deal with this (and other problems).

	Dave