ftrace help for NYSE

ftrace help for NYSE

[Clark Williams]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Arnaldo, Luis and Steven (and everyone else :)),

Had a good conversation with NYSE this morning, including Shak, Larry Woodman and Ade
Lee, concerning tuning efforts for their CCG application (Java message routing
application). They're running the first iteration of this on the stock RHEL5.2 kernel
and have managed to tune most of the latency spikes out, but they're seeing some
serious spikes following Java GC. The GC takes 5ms (yeah, five milliseconds) and then
they see another 5ms before the app actually cranks back up. The question is whether
the GC triggers some sort of VM transactions such as page reclamation, or something
else that would basically hold off the JVM from running for 5ms.

Shak and Larry are getting them to send us SAR reports and vmstat info, but one thing
they were interested in was looking at using ftrace with a MRG trace kernel to see if
they can get a clue as to what's going on during that 10ms period.

What I'd like to do is send them a trace kernel rpm and an example script that:

	1. Mounts the debugfs (if required)
	2. turns on appropriate switches in the tracing dir
		- turn on ftrace
		- set the current tracer
		- other stuff?
	3. starts tracing
	4. runs something (cyclictest? ping?)
	5. turns off tracing
	6. dumps the trace info

- From that script, the NYSE programmers can instrument their Java app to do the same
stuff.  I'd really like it if they could just boot the trace kernel, instrument their
app, get a trace and send us the info for analysis.

Is there anything else we can do to make using ftrace easier for these guys?

Clark
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org

iEYEARECAAYFAkhj7nUACgkQHyuj/+TTEp04fgCgvp7mjP1XQcR3XtXq43UDnPps
6scAn2ZbU7jdwZoip3TtAh1PajFAZlax
=HJYk
-----END PGP SIGNATURE-----