Re: trinity-main is in an endless loop

public inbox for trinity@vger.kernel.org
 help / color / mirror / Atom feed

* Re: trinity-main is in an endless loop
       [not found] ` <20130413145849.GA32144@redhat.com>
@ 2013-04-13 16:44   ` Toralf Förster
  2013-04-13 17:14   ` Toralf Förster
  1 sibling, 0 replies; 2+ messages in thread
From: Toralf Förster @ 2013-04-13 16:44 UTC (permalink / raw)
  To: trinity; +Cc: Dave Jones

On 04/13/2013 04:58 PM, Dave Jones wrote:
> On Sat, Apr 13, 2013 at 03:26:10PM +0200, Toralf Förster wrote:
>  > tl;dr;
>  > 
>  > 
>  > Running a recent trinity within a user mode linux guest (3.9-rc6-...) without parameter -N
>  > yields into an endless loop of the trinity-main process - it occupies 100% CPU time
>  > 
>  > The last lines of the command line
>  > 
>  > trinity ~ #  ps axf | grep trinity
>  >   910 ?        SNs    0:01  |       \_ bash -c pkill -9 trinity; cd ~; T=/mnt/n22/v/victims; mkdir -p $T; for i in $(seq -w 1 100); do touch $T/f$i; mkdir $T/d$i; done; cd $T; trinity --children 4 --victims $T -x mremap -r -q
>  >  1115 ?        SN     0:00  |           \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -r -q
>  >  1116 ?        SN     0:00  |               \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -r -q
>  >  1117 ?        RN   257:20  |               \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -r -q
>  >  1359 ?        SNL    0:01  |                   \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -r -q
>  >  1377 ?        SNL    0:00  |                   \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -r -q
>  >  1392 ?        SNL    0:00  |                   \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -r -q
>  >  1395 ?        SN     0:00  |                   \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -r -q
>  >  1744 pts/0    S+     0:00          \_ grep --colour=auto trinity
> 
> ok, all 4 children are stuck sleeping. it would be interesting to get sysrq-T backtraces of those to see where
> they're sleeping. 
> 
> just to be sure, make sure they aren't sleeping/waking up very quickly by doing a tail -f *.log
> If that's not making any progress, we're definitly hung.

The culprit of the hung are these directory permissions I fear :

# ls -l /mnt/n22/v/
total 0
d--------- 102 tfoerste users 4040 Apr 13 17:16 victims

and because I start trinity from within the directory "victims"
all processes just stuck b/c trinity locked out itself.

IMO trinity shouldn't change the directory permissions above its own log
files - and/or should not change the perms of the victims directory



-- 
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3

^ permalink raw reply	[flat|nested] 2+ messages in thread

* Re: trinity-main is in an endless loop
       [not found] ` <20130413145849.GA32144@redhat.com>
  2013-04-13 16:44   ` trinity-main is in an endless loop Toralf Förster
@ 2013-04-13 17:14   ` Toralf Förster
  1 sibling, 0 replies; 2+ messages in thread
From: Toralf Förster @ 2013-04-13 17:14 UTC (permalink / raw)
  To: trinity; +Cc: Dave Jones

On 04/13/2013 04:58 PM, Dave Jones wrote:
> ok, all 4 children are stuck sleeping. it would be interesting to get sysrq-T backtraces of those to see where
> they're sleeping. 

Well, changing the directory perms doesn't help always, now I've this situation :

#  ps axf | grep trinity | grep -v grep
  904 ?        Ss     0:00  |       \_ bash -c pkill -9 trinity; cd ~; T=/mnt/n22/v/victims; mkdir -p $T; for i in $(seq -w 1 100); do touch $T/f$i; mkdir $T/d$i; done; cd $T/../; trinity --children 4 --victims $T -x mremap -q
 1109 ?        S      0:00  |           \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -q
 1110 ?        S      0:01  |               \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -q
 1111 ?        R     19:08  |               \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -q
 1287 ?        SNL    0:00  |                   \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -q
 1291 ?        SN     0:00  |                   \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -q
 1300 ?        SN     0:00  |                   \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -q
 1301 ?        S      0:05  |                   \_ trinity --children 4 --victims /mnt/n22/v/victims -x mremap -q



and an tail -f at the logfiles shows no activity :

...
id=0x40371000m, egid=0m, sgid=0x80c4000ge_zeros]= -1 (Bad address)[1291] [242] pread64=uf=0x80ca000ge_rand]ount=0m, pos=682ae8am) = 0[1291] [243] getitimerich=75ac268m, value=126m) = -1 (Invalid argument)[1291] [244] readahead=[0m, offset=660ca5bm, count=219m) = -1 (Invalid argument)[1291] [245] fchown16=ser=0000m, group=[0m= -1 (Operation not permitted)[1291] [246] fanotify_marknotify_fd=lags=62m, mask=58m, dfd=athname=""= -1 (Invalid argument)[1291] [247] getdents=irent=4m, count=4095m) = -1 (Not a directory)[1291] [248] getcwdf=4097m, size=0m) = -1 (Numerical result out of range)tail: unrecognized file system type 0x00c0ffee for ‘/mnt/n22/v/trinity-child3.log’. please report this to bug-coreutils@gnu.org. reverting to polling

==> /mnt/n22/v/trinity.log <==
[1111] Random reseed: 1226440414
[1111] Random reseed: 1856582971
[1111] Random reseed: 1841156799
[watchdog] pid 1257 hasn't made progress in 30 seconds! (last:1365871959 now:1365871989 diff:30). Stuck in syscall 267:clock_nanosleep. Sending SIGKILL.
[1111] Random reseed: 2124266190
[1111] Random reseed: 775945931
[1111] Random reseed: 725547373
[1111] Random reseed: 2055143371
[1111] Regenerating random pages, fd's etc.
[watchdog] Triggering periodic reseed.
tail: unrecognized file system type 0x00c0ffee for ‘/mnt/n22/v/trinity.log’. please report this to bug-coreutils@gnu.org. reverting to polling



The sysrq-t gives :


2013-04-13T19:10:22.416+02:00 trinity kernel: 
2013-04-13T19:10:22.416+02:00 trinity kernel: trinity         S 40001282     0  1109    904 0x00000000
2013-04-13T19:10:22.416+02:00 trinity kernel: 3ff876ac 00000001 00000000 3fd3bdd8 0806194f 3ff87c6c 3ff876ac 3fd38000 
2013-04-13T19:10:22.416+02:00 trinity kernel: 080a50a0 3fd3be04 083710dc 3ff877c0 3ff87200 00000001 400f1f10 3ff87200 
2013-04-13T19:10:22.416+02:00 trinity kernel: 3fd38000 3ff877b8 3fd3be54 3ff878bc 3fd3be18 08371175 3ff87200 3fd3be54 3fd3bdc4:  [<0806194f>] __switch_to+0x5f/0x100
2013-04-13T19:10:22.416+02:00 trinity kernel: 3fd3bddc:  [<083710dc>] __schedule+0x2cc/0x310
2013-04-13T19:10:22.418+02:00 trinity kernel: 3fd3be08:  [<08371175>] schedule+0x55/0x60
2013-04-13T19:10:22.418+02:00 trinity kernel: 3fd3be1c:  [<080838d6>] do_wait+0x186/0x220
2013-04-13T19:10:22.418+02:00 trinity kernel: 3fd3be48:  [<080848ce>] sys_wait4+0xae/0xd0
2013-04-13T19:10:22.418+02:00 trinity kernel: 3fd3be94:  [<08084917>] sys_waitpid+0x27/0x30
2013-04-13T19:10:22.418+02:00 trinity kernel: 3fd3beac:  [<08064a92>] handle_syscall+0x82/0xb0
2013-04-13T19:10:22.418+02:00 trinity kernel: 3fd3bef4:  [<08076e6d>] userspace+0x46d/0x590
2013-04-13T19:10:22.418+02:00 trinity kernel: 3fd3bfec:  [<080617bc>] fork_handler+0x6c/0x70
2013-04-13T19:10:22.418+02:00 trinity kernel: 3fd3bffc:  [<00000000>] 0x0
2013-04-13T19:10:22.418+02:00 trinity kernel: 
2013-04-13T19:10:22.418+02:00 trinity kernel: trinity-watchdo S 40001282     0  1110   1109 0x00000000
2013-04-13T19:10:22.421+02:00 trinity kernel: 4107b06c 00000001 00000000 3fc7bdac 0806194f 3ff5052c 4107b06c 3fc78000 
2013-04-13T19:10:22.421+02:00 trinity kernel: 080a50a0 3fc7bdd8 083710dc 3ff50080 4107abc0 00000001 00000000 3fc7be48 
2013-04-13T19:10:22.421+02:00 trinity kernel: 3fc78000 3ff50080 3fc7be48 00000001 3fc7bdec 08371175 0000c350 00000001 3fc7bd98:  [<0806194f>] __switch_to+0x5f/0x100
2013-04-13T19:10:22.421+02:00 trinity kernel: 3fc7bdb0:  [<083710dc>] __schedule+0x2cc/0x310
2013-04-13T19:10:22.421+02:00 trinity kernel: 3fc7bddc:  [<08371175>] schedule+0x55/0x60
2013-04-13T19:10:22.421+02:00 trinity kernel: 3fc7bdf0:  [<083706c4>] do_nanosleep+0xb4/0x110
2013-04-13T19:10:22.421+02:00 trinity kernel: 3fc7be24:  [<0809e0d8>] hrtimer_nanosleep+0xc8/0x140
2013-04-13T19:10:22.421+02:00 trinity kernel: 3fc7be88:  [<0809e1b7>] sys_nanosleep+0x67/0x70
2013-04-13T19:10:22.421+02:00 trinity kernel: 3fc7beac:  [<08064a92>] handle_syscall+0x82/0xb0
2013-04-13T19:10:22.421+02:00 trinity kernel: 3fc7bef4:  [<08076e6d>] userspace+0x46d/0x590
2013-04-13T19:10:22.424+02:00 trinity kernel: 3fc7bfec:  [<080617bc>] fork_handler+0x6c/0x70
2013-04-13T19:10:22.424+02:00 trinity kernel: 3fc7bffc:  [<00000000>] 0x0
2013-04-13T19:10:22.424+02:00 trinity kernel: 
2013-04-13T19:10:22.424+02:00 trinity kernel: trinity-main    R running      0  1111   1109 0x00000000
2013-04-13T19:10:22.424+02:00 trinity kernel: 0843f6b4 00000457 00000455 00000000 00000000 3ff87200 0843f6e8 080a39bf 
2013-04-13T19:10:22.424+02:00 trinity kernel: 3ff87200 3ff872ac 00000074 0845f1e4 00000000 0843f6f4 0827d0b2 00000000 
2013-04-13T19:10:22.424+02:00 trinity kernel: 0843f718 0827d3f6 00000074 084038f2 00000001 01000000 08501c80 0806ad50 0843f688:  [<08062ccf>] show_stack+0xcf/0x100
2013-04-13T19:10:22.424+02:00 trinity kernel: 0843f6ac:  [<080a3920>] sched_show_task+0xc0/0x120
2013-04-13T19:10:22.424+02:00 trinity kernel: 0843f6d0:  [<080a39bf>] show_state_filter+0x3f/0x80
2013-04-13T19:10:22.424+02:00 trinity kernel: 0843f6ec:  [<0827d0b2>] sysrq_handle_showstate+0x12/0x20
2013-04-13T19:10:22.427+02:00 trinity kernel: 0843f6f8:  [<0827d3f6>] __handle_sysrq+0xa6/0x150
2013-04-13T19:10:22.427+02:00 trinity kernel: 0843f71c:  [<0827d57c>] handle_sysrq+0x2c/0x30
2013-04-13T19:10:22.427+02:00 trinity kernel: 0843f72c:  [<0806ad64>] sysrq_proc+0x14/0x20
2013-04-13T19:10:22.427+02:00 trinity kernel: 0843f738:  [<0806a971>] with_console+0x51/0xc0
2013-04-13T19:10:22.427+02:00 trinity kernel: 0843f770:  [<0806ad14>] mconsole_sysrq+0x64/0x70
2013-04-13T19:10:22.427+02:00 trinity kernel: 0843f790:  [<0806a70e>] mconsole_interrupt+0x2e/0x100
2013-04-13T19:10:22.427+02:00 trinity kernel: 0843f7b4:  [<080c4c8f>] handle_irq_event_percpu+0x2f/0x150
2013-04-13T19:10:22.427+02:00 trinity kernel: 0843f7e4:  [<080c4dd7>] handle_irq_event+0x27/0x40
2013-04-13T19:10:22.427+02:00 trinity kernel: 0843f7f8:  [<080c70b0>] handle_edge_irq+0xd0/0x100
2013-04-13T19:10:22.427+02:00 trinity kernel: 0843f808:  [<080c4648>] generic_handle_irq+0x28/0x30
2013-04-13T19:10:22.508+02:00 trinity kernel: 0843f818:  [<08060e55>] do_IRQ+0x25/0x40
2013-04-13T19:10:22.508+02:00 trinity kernel: 0843f828:  [<08060ed5>] sigio_handler+0x65/0x80
2013-04-13T19:10:22.508+02:00 trinity kernel: 0843f840:  [<08073a28>] sig_handler_common+0xb8/0xe0
2013-04-13T19:10:22.508+02:00 trinity kernel: 0843fac4:  [<08073b6d>] sig_handler+0x3d/0x50
2013-04-13T19:10:22.508+02:00 trinity kernel: 0843fad4:  [<080736bb>] hard_handler+0x6b/0xa0
2013-04-13T19:10:22.508+02:00 trinity kernel: 0843fafc:  [<b772040c>] 0xb772040c
2013-04-13T19:10:22.508+02:00 trinity kernel: 
2013-04-13T19:10:22.508+02:00 trinity kernel: trinity-child0  S 40001282     0  1287   1111 0x00000000
2013-04-13T19:10:22.508+02:00 trinity kernel: 4107b06c 00000001 00000000 4133fd34 0806194f 3ff870ec 4107b06c 4133c000 
2013-04-13T19:10:22.508+02:00 trinity kernel: 080a50a0 4133fd60 083710dc 3ff86c40 4107abc0 00000001 11e1a378 00000000 
2013-04-13T19:10:22.520+02:00 trinity kernel: 4133c000 00000000 4133fd98 4133fe18 4133fd74 08371175 00000000 4133fd98 4133fd20:  [<0806194f>] __switch_to+0x5f/0x100
2013-04-13T19:10:22.520+02:00 trinity kernel: 4133fd38:  [<083710dc>] __schedule+0x2cc/0x310
2013-04-13T19:10:22.520+02:00 trinity kernel: 4133fd64:  [<08371175>] schedule+0x55/0x60
2013-04-13T19:10:22.520+02:00 trinity kernel: 4133fd78:  [<0809bf45>] do_cpu_nanosleep+0x105/0x1b0
2013-04-13T19:10:22.520+02:00 trinity kernel: 4133fe2c:  [<0809c112>] posix_cpu_nsleep+0x62/0x110
2013-04-13T19:10:22.520+02:00 trinity kernel: 4133fe68:  [<0809c1e7>] process_cpu_nsleep+0x27/0x30
2013-04-13T19:10:22.520+02:00 trinity kernel: 4133fe80:  [<08099be8>] sys_clock_nanosleep+0xc8/0xd0
2013-04-13T19:10:22.520+02:00 trinity kernel: 4133feac:  [<08064a92>] handle_syscall+0x82/0xb0
2013-04-13T19:10:22.520+02:00 trinity kernel: 4133fef4:  [<08076e6d>] userspace+0x46d/0x590
2013-04-13T19:10:22.520+02:00 trinity kernel: 4133ffec:  [<080617bc>] fork_handler+0x6c/0x70
2013-04-13T19:10:22.531+02:00 trinity kernel: 4133fffc:  [<00000000>] 0x0
2013-04-13T19:10:22.531+02:00 trinity kernel: 
2013-04-13T19:10:22.531+02:00 trinity kernel: trinity-child3  S 40001282     0  1291   1111 0x00000000
2013-04-13T19:10:22.531+02:00 trinity kernel: 3ff870ec 00000001 00000000 41437cc4 0806194f 41264b2c 3ff870ec 41434000 
2013-04-13T19:10:22.531+02:00 trinity kernel: 080a50a0 41437cf0 083710dc 41264680 3ff86c40 00000001 41437cf0 3ff86c40 
2013-04-13T19:10:22.531+02:00 trinity kernel: 41434000 41437d7c 00000000 41437d88 41437d04 08371175 41437d7c 00000000 41437cb0:  [<0806194f>] __switch_to+0x5f/0x100
2013-04-13T19:10:22.531+02:00 trinity kernel: 41437cc8:  [<083710dc>] __schedule+0x2cc/0x310
2013-04-13T19:10:22.531+02:00 trinity kernel: 41437cf4:  [<08371175>] schedule+0x55/0x60
2013-04-13T19:10:22.531+02:00 trinity kernel: 41437d08:  [<080af449>] futex_wait_queue_me+0xe9/0x110
2013-04-13T19:10:22.531+02:00 trinity kernel: 41437d3c:  [<080afca0>] futex_wait+0x160/0x2b0
2013-04-13T19:10:22.542+02:00 trinity kernel: 41437df8:  [<080b1619>] do_futex+0xa9/0x6e0
2013-04-13T19:10:22.542+02:00 trinity kernel: 41437e64:  [<080b1d71>] sys_futex+0x121/0x140
2013-04-13T19:10:22.542+02:00 trinity kernel: 41437eac:  [<08064a92>] handle_syscall+0x82/0xb0
2013-04-13T19:10:22.542+02:00 trinity kernel: 41437ef4:  [<08076e6d>] userspace+0x46d/0x590
2013-04-13T19:10:22.542+02:00 trinity kernel: 41437fec:  [<080617bc>] fork_handler+0x6c/0x70
2013-04-13T19:10:22.542+02:00 trinity kernel: 41437ffc:  [<00000000>] 0x0
2013-04-13T19:10:22.542+02:00 trinity kernel: 
2013-04-13T19:10:22.542+02:00 trinity kernel: trinity-child2  S 40001282     0  1300   1111 0x00000000
2013-04-13T19:10:22.542+02:00 trinity kernel: 3ff876ac 00000001 00000000 41113dac 0806194f 3ff50aec 3ff876ac 41110000 
2013-04-13T19:10:22.542+02:00 trinity kernel: 080a50a0 41113dd8 083710dc 3ff50640 3ff87200 00000001 00000000 3ff87200 
2013-04-13T19:10:22.550+02:00 trinity kernel: 41110000 3ff50640 41113e48 00000001 41113dec 08371175 0000c350 00000001 41113d98:  [<0806194f>] __switch_to+0x5f/0x100
2013-04-13T19:10:22.550+02:00 trinity kernel: 41113db0:  [<083710dc>] __schedule+0x2cc/0x310
2013-04-13T19:10:22.550+02:00 trinity kernel: 41113ddc:  [<08371175>] schedule+0x55/0x60
2013-04-13T19:10:22.550+02:00 trinity kernel: 41113df0:  [<083706c4>] do_nanosleep+0xb4/0x110
2013-04-13T19:10:22.550+02:00 trinity kernel: 41113e24:  [<0809e0d8>] hrtimer_nanosleep+0xc8/0x140
2013-04-13T19:10:22.550+02:00 trinity kernel: 41113e88:  [<0809e1b7>] sys_nanosleep+0x67/0x70
2013-04-13T19:10:22.550+02:00 trinity kernel: 41113eac:  [<08064a92>] handle_syscall+0x82/0xb0
2013-04-13T19:10:22.550+02:00 trinity kernel: 41113ef4:  [<08076e6d>] userspace+0x46d/0x590
2013-04-13T19:10:22.550+02:00 trinity kernel: 41113fec:  [<080617bc>] fork_handler+0x6c/0x70
2013-04-13T19:10:22.550+02:00 trinity kernel: 41113ffc:  [<00000000>] 0x0
2013-04-13T19:10:22.554+02:00 trinity kernel: 
2013-04-13T19:10:22.554+02:00 trinity kernel: trinity-child1  S 40001282     0  1301   1111 0x00000000
2013-04-13T19:10:22.554+02:00 trinity kernel: 3ff876ac 00000001 00000000 41433dac 0806194f 41265c6c 3ff876ac 41430000 
2013-04-13T19:10:22.554+02:00 trinity kernel: 080a50a0 41433dd8 083710dc 412657c0 3ff87200 00000001 00000000 3ff87200 
2013-04-13T19:10:22.554+02:00 trinity kernel: 41430000 412657c0 41433e48 00000001 41433dec 08371175 0000c350 00000001 41433d98:  [<0806194f>] __switch_to+0x5f/0x100
2013-04-13T19:10:22.554+02:00 trinity kernel: 41433db0:  [<083710dc>] __schedule+0x2cc/0x310
2013-04-13T19:10:22.554+02:00 trinity kernel: 41433ddc:  [<08371175>] schedule+0x55/0x60
2013-04-13T19:10:22.554+02:00 trinity kernel: 41433df0:  [<083706c4>] do_nanosleep+0xb4/0x110
2013-04-13T19:10:22.554+02:00 trinity kernel: 41433e24:  [<0809e0d8>] hrtimer_nanosleep+0xc8/0x140
2013-04-13T19:10:22.554+02:00 trinity kernel: 41433e88:  [<0809e1b7>] sys_nanosleep+0x67/0x70
2013-04-13T19:10:22.559+02:00 trinity kernel: 41433eac:  [<08064a92>] handle_syscall+0x82/0xb0
2013-04-13T19:10:22.559+02:00 trinity kernel: 41433ef4:  [<08076e6d>] userspace+0x46d/0x590
2013-04-13T19:10:22.559+02:00 trinity kernel: 41433fec:  [<080617bc>] fork_handler+0x6c/0x70
2013-04-13T19:10:22.559+02:00 trinity kernel: 41433ffc:  [<00000000>] 0x0

-- 
MfG/Sincerely
Toralf Förster
pgp finger print: 7B1A 07F4 EC82 0F90 D4C2 8936 872A E508 7DB6 9DA3

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2013-04-13 17:14 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <51695CF2.2010108@gmx.de>
     [not found] ` <20130413145849.GA32144@redhat.com>
2013-04-13 16:44   ` trinity-main is in an endless loop Toralf Förster
2013-04-13 17:14   ` Toralf Förster

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox