trinity cores in watchdog while syslog'ing

trinity cores in watchdog while syslog'ing

[Toralf Förster]

Just FWIW :

Fuzzying a 32 bit Gentoo Linux let the trinity process in side the UML guest sometimes core dump with this back trace:

Core was generated by `trinity -C 4 -N 100000 -x mremap -q'.
Program terminated with signal 11, Segmentation fault.
#0  __GI__IO_fflush (fp=0x8154200) at iofflush.c:40
40      iofflush.c: No such file or directory.
(gdb) bt
#0  __GI__IO_fflush (fp=0x8154200) at iofflush.c:40
#1  0x080582c3 in synclogs () at log.c:163
#2  0x0805e93d in watchdog () at watchdog.c:363
#3  init_watchdog () at watchdog.c:430
#4  0x080538d9 in main (argc=8, argv=0xbfc8fd94) at trinity.c:132
(gdb) quit

-- 
Toralf

Re: trinity cores in watchdog while syslog'ing

[Dave Jones]

On Wed, Aug 06, 2014 at 08:16:27PM +0200, Toralf Förster wrote:
 > Just FWIW :
 > 
 > Fuzzying a 32 bit Gentoo Linux let the trinity process in side the UML guest sometimes core dump with this back trace:
 > 
 > Core was generated by `trinity -C 4 -N 100000 -x mremap -q'.
 > Program terminated with signal 11, Segmentation fault.
 > #0  __GI__IO_fflush (fp=0x8154200) at iofflush.c:40
 > 40      iofflush.c: No such file or directory.
 > (gdb) bt
 > #0  __GI__IO_fflush (fp=0x8154200) at iofflush.c:40
 > #1  0x080582c3 in synclogs () at log.c:163
 > #2  0x0805e93d in watchdog () at watchdog.c:363
 > #3  init_watchdog () at watchdog.c:430
 > #4  0x080538d9 in main (argc=8, argv=0xbfc8fd94) at trinity.c:132
 > (gdb) quit

ugh, when I switched the log file creation to happen from within
the child, I forgot to take care of this function.

I'm surprised it's taken this long for anyone to notice, as it can't
possibly work.

I think I'm just going to tear that whole thing out, as syncing
from watchdog context just isn't going to be possible.

	Dave

Re: trinity cores in watchdog while syslog'ing

[Toralf Förster]

On 08/06/2014 08:28 PM, Dave Jones wrote:
> I'm surprised it's taken this long for anyone to notice, as it can't
> possibly work.
Well, I noticed that few days ago, but was distrcted by another bug ...
Ok, with latest git I see now this core :


warning: core file may not match specified executable file.
[New LWP 1292]
Core was generated by `trinity -C 4 -N 100000 -x mremap -q'.
Program terminated with signal 6, Aborted.
#0  0x400011d2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb) bt
#0  0x400011d2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1  0x4005ab5f in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#2  0x4005c3af in __GI_abort () at abort.c:90
#3  0x4009cad5 in __libc_message (do_abort=do_abort@entry=2, 
    fmt=fmt@entry=0x4019403f "*** %s ***: %s terminated\n")
    at ../sysdeps/unix/sysv/linux/libc_fatal.c:196
#4  0x40134d65 in __GI___fortify_fail (
    msg=msg@entry=0x40198d30 "invalid open call: O_CREAT without mode") at fortify_fail.c:31
#5  0x40113c07 in __open_2 (file=0x80eb3b8 "trinity-testfile1", oflag=20544)
    at ../sysdeps/unix/sysv/linux/open_2.c:28
#6  0x08056f61 in open (__path=0x80eb3b8 "trinity-testfile1", __oflag=20544)
    at /usr/include/bits/fcntl2.h:57
#7  open_testfile (i=1) at fd-testfiles.c:40
#8  open_testfile_fds () at fd-testfiles.c:54
#9  0x080572d5 in setup_fds () at fds.c:70
#10 0x08053962 in main (argc=8, argv=0xbfedbd94) at trinity.c:146
(gdb) quit



which correspond with this output at the console :

$ nice trinity.sh -t "-C 4 -N 100000 -x mremap -q" /tmp
>>> Creating Manifest for /usr/local/portage/dev-util/trinity
 * --tree is broken with --nodeps. Disabling...
>>> Verifying ebuild manifests
>>> Emerging (1 of 1) dev-util/trinity-9999 from toralf
>>> Installing (1 of 1) dev-util/trinity-9999
>>> Jobs: 1 of 1 complete                           Load avg: 2.29, 2.24, 2.44
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * GNU info directory index is up-to-date.
 starting UML ...  UML started

 prepare trinity job at Wed Aug  6 20:55:20 CEST 2014 1#-1, M=
Trinity 1.5pre  Dave Jones <davej@redhat.com>
shm: redzone:0x4023b000. shmdata:0x40259000. redzone:0x4025e000 end:0x4027c000.
[init] Done parsing arguments.
Marking all syscalls as enabled.
[init] Disabling syscalls marked as disabled by command line options
[init] Marked syscall mremap (163) as deactivated.
[init] Enabled 353 syscalls. Disabled 1 syscalls.
[init] Using pid_max = 32768
[init] Started watchdog process, PID is 1291
[main] Main thread is alive.
[main] Registered 8 fd providers.
[main] Couldn't find socket cachefile. Regenerating.
[main] created 375 sockets
[main] Generating file descriptors
[main] Added 14 filenames from /dev
[main] Added 6127 filenames from /proc
[main] Added 1320 filenames from /sys
*** invalid open call: O_CREAT without mode ***: trinity terminated
======= Backtrace: =========
/lib/libc.so.6(__fortify_fail+0x45)[0x40134d65]
/lib/libc.so.6(+0xeac07)[0x40113c07]
trinity[0x8056f61]
trinity(setup_fds+0xd5)[0x80572d5]
trinity(main+0x222)[0x8053962]
/lib/libc.so.6(__libc_start_main+0xf5)[0x40045cc5]
trinity[0x8053f35]
======= Memory map: ========
00100000-00102000 r-xp 00000000 00:00 0 
08048000-08079000 r-xp 00000000 62:00 45745      /usr/bin/trinity
08079000-0807a000 r--p 00030000 62:00 45745      /usr/bin/trinity
0807a000-080e2000 rw-p 00031000 62:00 45745      /usr/bin/trinity
080e2000-08103000 rw-p 00000000 00:00 0          [heap]
08103000-081a7000 rw-p 00000000 00:00 0          [heap]
40000000-40020000 r-xp 00000000 62:00 1676       /lib/ld-2.17.so
40020000-40021000 r--p 0001f000 62:00 1676       /lib/ld-2.17.so
40021000-40022000 rw-p 00020000 62:00 1676       /lib/ld-2.17.so
40022000-40024000 rw-p 00000000 00:00 0 
40024000-40026000 rw-s 00000000 00:04 1477       /dev/zero (deleted)
40026000-40028000 r--s 00000000 00:04 1478       /dev/zero (deleted)
40028000-40029000 rw-p 00000000 00:00 0 
40029000-401d1000 r-xp 00000000 62:00 1769       /lib/libc-2.17.so
401d1000-401d3000 r--p 001a8000 62:00 1769       /lib/libc-2.17.so
401d3000-401d4000 rw-p 001aa000 62:00 1769       /lib/libc-2.17.so
401d4000-401d8000 rw-p 00000000 00:00 0 
401d8000-4023b000 rw-s 00000000 00:04 1470       /dev/zero (deleted)
4023b000-40259000 ---s 00000000 00:04 1471       /dev/zero (deleted)
40259000-4025e000 rw-s 0001e000 00:04 1471       /dev/zero (deleted)
4025e000-4027c000 ---s 00023000 00:04 1471       /dev/zero (deleted)
4027c000-40289000 rw-s 00000000 00:04 1472       /dev/zero (deleted)
40289000-40296000 rw-s 00000000 00:04 1473       /dev/zero (deleted)
40296000-402a3000 rw-s 00000000 00:04 1474       /dev/zero (deleted)
402a3000-402b0000 rw-s 00000000 00:04 1475       /dev/zero (deleted)
402b0000-402b2000 -w-s 00000000 00:04 1479       /dev/zero (deleted)
402b2000-403b2000 rw-s 00000000 00:04 1480       /dev/zero (deleted)
403b2000-404b2000 r--s 00000000 00:04 1481       /dev/zero (deleted)
404b2000-405b2000 -w-s 00000000 00:04 1482       /dev/zero (deleted)
405b2000-407b2000 rw-s 00000000 00:04 1483       /dev/zero (deleted)
407b2000-409b2000 r--s 00000000 00:04 1484       /dev/zero (deleted)
409b2000-40bb2000 -w-s 00000000 00:04 1485       /dev/zero (deleted)
40bb2000-40fb2000 rw-s 00000000 00:04 1486       /dev/zero (deleted)
40fb2000-413b2000 r--s 00000000 00:04 1487       /dev/zero (deleted)
413b2000-417b2000 -w-s 00000000 00:04 1488       /dev/zero (deleted)
417b2000-421b2000 rw-s 00000000 00:04 1489       /dev/zero (deleted)
421b2000-42bb2000 r--s 00000000 00:04 1490       /dev/zero (deleted)
42bb2000-435b2000 -w-s 00000000 00:04 1491       /dev/zero (deleted)
435b2000-435b3000 rw-p 00000000 00:00 0
435b9000-435d3000 r-xp 00000000 62:00 26842      /usr/lib/gcc/i686-pc-linux-gnu/4.7.3/libgcc_s.so.1
435d3000-435d4000 r--p 00019000 62:00 26842      /usr/lib/gcc/i686-pc-linux-gnu/4.7.3/libgcc_s.so.1
435d4000-435d5000 rw-p 0001a000 62:00 26842      /usr/lib/gcc/i686-pc-linux-gnu/4.7.3/libgcc_s.so.1
bfebb000-bfedc000 rw-p 00000000 00:00 0


-- 
Toralf

Re: trinity cores in watchdog while syslog'ing

[Dave Jones]

On Wed, Aug 06, 2014 at 09:14:20PM +0200, Toralf Förster wrote:
 > On 08/06/2014 08:28 PM, Dave Jones wrote:
 > > I'm surprised it's taken this long for anyone to notice, as it can't
 > > possibly work.
 > Well, I noticed that few days ago, but was distrcted by another bug ...
 > Ok, with latest git I see now this core :

duh, dumb cut and paste error. Should be fixed now.

thanks,

	Dave