Re: [PATCH][RFC] sched: Isochronous class for unprivileged soft rt scheduling

["Jack O'Quin" <joq@io.com>]

[-- Attachment #1: Type: text/plain, Size: 1146 bytes --]


[including Rui Nuno Capela in cc: list]

>> Con Kolivas <kernel@kolivas.org> writes:
>>
>>>This patch for 2.6.11-rc1 provides a method of providing real time
>>>scheduling to unprivileged users which increasingly is desired for
>>>multimedia workloads.

> Jack O'Quin wrote:
>> I ran some jack_test3.2 runs with this, using all the default
>> settings.  The results of three runs differ quite significantly for no
>> obvious reason.  I can't figure out why the DSP load should vary so
>> much.

Con Kolivas <kernel@kolivas.org> writes:
> I installed a fresh jack installation and got the test suite. I tried
> running the test suite and found it only ran to completion if I
> changed the run time right down to 30 seconds from 300. Otherwise it
> bombed out almost instantly at the default of 300. I don't know if
> that helps you debug the problem or not but it might be worth
> mentioning.

Here are a couple of bug fixes for the test package.  I still had not
gotten them to Rui (the test author).  He has created several newer
versions, but I'm still using this modified jack_test3.2, so the
number comparisons will continue to make sense.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: joq mods to jack_test3.2 --]
[-- Type: text/x-patch, Size: 5454 bytes --]

--- jack_test3.2/jack_test3_client.cpp	Thu Dec  9 10:26:12 2004
+++ jack_test/jack_test3_client.cpp	Sat Jan 15 20:48:20 2005
@@ -13,20 +13,25 @@
 unsigned int seconds_to_run = 60;
 unsigned int num_of_ports   = 4;
 
+/* mix all the input ports to each output port */
 int process(jack_nframes_t frames, void *arg) 
 {
 	// std::cout << "process callback" << std::endl;
 	jack_default_audio_sample_t *ibuf;
 	jack_default_audio_sample_t *obuf;
 	for (int i = 0; i < num_of_ports; i++) {
-		obuf = (jack_default_audio_sample_t*) jack_port_get_buffer(oports[i], frames);
+		obuf = (jack_default_audio_sample_t*)
+			jack_port_get_buffer(oports[i], frames);
 		for (int j = 0; j < num_of_ports; j++) {
-			ibuf = (jack_default_audio_sample_t*) jack_port_get_buffer(iports[j], frames);
-			for (jack_nframes_t frame = 0; frame < frames; frame++) {
+			ibuf = (jack_default_audio_sample_t*)
+				jack_port_get_buffer(iports[j], frames);
+			for (jack_nframes_t frame = 0;
+			     frame < frames; frame++) {
 				if (j == 0)
 					obuf[frame] = 0.0; 
 				if (ibuf[frame] < -1E-6 || ibuf[frame] > +1E-6)
-					obuf[frame] += ibuf[frame] / (float) num_of_ports;
+					obuf[frame] += ibuf[frame]
+						/ (float) num_of_ports;
 			}
 		}
 	}
@@ -68,10 +73,23 @@
 	for (int i = 0; i < num_of_ports; i++) {
 		std::stringstream iport_name;
 		iport_name << "in_" << i;
-		iports[i] = jack_port_register(client,  iport_name.str().c_str(), JACK_DEFAULT_AUDIO_TYPE, JackPortIsInput|JackPortIsTerminal, 0);
+		iports[i] =
+			jack_port_register(client,
+					   iport_name.str().c_str(),
+					   JACK_DEFAULT_AUDIO_TYPE,
+					   JackPortIsInput|JackPortIsTerminal,
+					   0);
 		std::stringstream oport_name;
 		oport_name << "out_" << i;
-		oports[i] = jack_port_register(client, oport_name.str().c_str(), JACK_DEFAULT_AUDIO_TYPE, JackPortIsTerminal|JackPortIsOutput, 0);
+		oports[i] = jack_port_register(client,
+					       oport_name.str().c_str(),
+					       JACK_DEFAULT_AUDIO_TYPE,
+					       JackPortIsTerminal|JackPortIsOutput,
+					       0);
+		if ((iports[i] == NULL) || (oports[i] == NULL)) {
+			std::cout << "port registration failed" << std::endl;
+			goto exit;
+		}
 	}
 	std::cout << "set_process_callback" << std::endl;
 	jack_set_process_callback(client, process, 0);
@@ -85,6 +103,8 @@
 	sleep(seconds_to_run);
 
 	jack_deactivate(client);
+
+exit:
 	jack_client_close(client);
 	
 	delete [] iports;
--- jack_test3.2/jack_test3_nmeter.c	Sat Nov 27 09:08:01 2004
+++ jack_test/jack_test3_nmeter.c	Thu Dec 30 07:50:38 2004
@@ -21,7 +21,9 @@
 //==============
 #define NL "\n"
 typedef unsigned long long ullong;
+#ifndef __USE_MISC
 typedef unsigned long ulong;
+#endif
 
 typedef ulong sample_t;
 
--- jack_test3.2/jack_test3_run.sh	Fri Dec 10 06:21:21 2004
+++ jack_test/jack_test3_run.sh	Sat Jan 15 22:23:06 2005
@@ -6,12 +6,14 @@
 CLIENTS=$2
 PORTS=$3
 PERIOD=$4
+PLYBK_PORTS=$5
 
 # Parameter defaults.
 [ -z "${SECS}"    ] && SECS=300
 [ -z "${CLIENTS}" ] && CLIENTS=20
 [ -z "${PORTS}"   ] && PORTS=4
 [ -z "${PERIOD}"  ] && PERIOD=64
+[ -z "${PLYBK_PORTS}"  ] && PLYBK_PORTS=32
 
 # Local tools must be on same directory...
 BASEDIR=`dirname $0`
@@ -24,13 +26,13 @@
 NMETER_ARGS="t c i x b m"
 
 # jackd configuration.
-JACKD="/usr/bin/jackd"
+JACKD="jackd"
 JACKD_DRIVER="alsa"
 JACKD_DEVICE="hw:0"
 JACKD_PRIO=60
 JACKD_RATE=44100
-JACKD_PORTS=$(((${CLIENTS} + 1) * ${PORTS} * 2))
-JACKD_ARGS="-v -R -P${JACKD_PRIO} -p${JACKD_PORTS} -d${JACKD_DRIVER} -d${JACKD_DEVICE} -r${JACKD_RATE} -p${PERIOD} -n2 -S -P"
+JACKD_PORTS=$((${CLIENTS} * ${PORTS} * 2 + ${PLYBK_PORTS}))
+JACKD_ARGS="-Rv -p${JACKD_PORTS} -d${JACKD_DRIVER} -d${JACKD_DEVICE} -r${JACKD_RATE} -p${PERIOD} -n2 -P"
 
 # client test configuration.
 CLIENT="${BASEDIR}/jack_test3_client"
@@ -85,6 +87,7 @@
 echo "Number of clients  (CLIENTS) = ${CLIENTS}"	>> ${LOG} 2>&1
 echo "Ports per client     (PORTS) = ${PORTS}"		>> ${LOG} 2>&1
 echo "Frames per buffer   (PERIOD) = ${PERIOD}"		>> ${LOG} 2>&1
+echo "Playback ports (PLYBK_PORTS) = ${PLYBK_PORTS}"	>> ${LOG} 2>&1
 echo >> ${LOG} 2>&1
 
 exec_log "uname -a"
@@ -102,7 +105,7 @@
 ilist_log 
 
 #
-# Lauch nmeter in the background...
+# Launch nmeter in the background...
 #
 echo -n "Launching `basename ${NMETER}`..."
 (${NMETER} ${NMETER_ARGS} >> ${LOG} 2>&1) &
@@ -110,7 +113,7 @@
 echo "done."
 sleep 1
 
-# Lauch the test client suite...
+# Launch the test client suite...
 SLEEP=6
 echo -n "Launching ${CLIENTS} ${CLIENT}(s) instance(s)..."
 for NUM in `seq 1 ${CLIENTS}`; do
@@ -129,7 +132,7 @@
 (sleep ${SLEEP}; killall `basename ${JACKD}`) &
 
 #
-# Lauch jackd and wait for it...
+# Launch jackd and wait for it...
 #
 echo -n "Running `basename ${JACKD}`..."
 exec_log ${JACKD} ${JACKD_ARGS}
@@ -149,7 +152,7 @@
 sleep 1
 sync
 
-# Summary log analynis...
+# Summary log analysis...
 cat ${LOG} | awk -f ${BASEDIR}/jack_test3_summary.awk | tee -a ${LOG}
 
 # Finally, plot log output...
--- jack_test3.2/Makefile	Thu Nov 25 05:28:47 2004
+++ jack_test/Makefile	Sat Jan 15 20:50:04 2005
@@ -1,10 +1,10 @@
 all:	jack_test3_nmeter jack_test3_client
 
 jack_test3_nmeter:	jack_test3_nmeter.c
-	gcc -o jack_test3_nmeter jack_test3_nmeter.c
+	$(CC) -g -o jack_test3_nmeter jack_test3_nmeter.c
 
 jack_test3_client:	jack_test3_client.cpp
-	g++ -o jack_test3_client jack_test3_client.cpp -ljack
+	$(CXX) -g -o jack_test3_client jack_test3_client.cpp -ljack
 
 clean:
 	rm -vf jack_test3_nmeter jack_test3_client

[-- Attachment #3: Type: text/plain, Size: 100 bytes --]


For completeness, here's Rui's unmodified test package against which
these diff's were generated.


[-- Attachment #4: JACK test version 3.2 (from rncbc) --]
[-- Type: archive/tar, Size: 11383 bytes --]

[-- Attachment #5: Type: text/plain, Size: 359 bytes --]


There's also a modified script for running with nice --20, called
jack_test3_nice.sh.  I hacked it for Ingo's tests.  It really should
be integrated into the main jack_test3_run.sh as an option, but I have
not bothered.  I think we're beyond that at this point, anyway.
Mainly, it proves that per-thread granularity is essential for making
this stuff work.


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #6: hacked up jack_test3_run.sh for running with nice --20 --]
[-- Type: text/x-sh, Size: 3794 bytes --]

#!/bin/sh
#

# Command line arguments.
SECS=$1
CLIENTS=$2
PORTS=$3
PERIOD=$4
PLYBK_PORTS=$5

# Parameter defaults.
[ -z "${SECS}"    ] && SECS=300
[ -z "${CLIENTS}" ] && CLIENTS=20
[ -z "${PORTS}"   ] && PORTS=4
[ -z "${PERIOD}"  ] && PERIOD=64
[ -z "${PLYBK_PORTS}"  ] && PLYBK_PORTS=32

# Local tools must be on same directory...
BASEDIR=`dirname $0`

# Make sure our local tools are in place.
(cd ${BASEDIR}; make all > /dev/null) || exit 1

# nmeter configuration.
NMETER="${BASEDIR}/jack_test3_nmeter"
NMETER_ARGS="t c i x b m"

# jackd configuration.
JACKD="jackd"
JACKD_DRIVER="alsa"
JACKD_DEVICE="hw:0"
JACKD_PRIO=60
JACKD_RATE=44100
JACKD_PORTS=$((${CLIENTS} * ${PORTS} * 2 + ${PLYBK_PORTS}))
JACKD_ARGS="-v -p${JACKD_PORTS} -d${JACKD_DRIVER} -d${JACKD_DEVICE} -r${JACKD_RATE} -p${PERIOD} -n2 -P"

# client test configuration.
CLIENT="${BASEDIR}/jack_test3_client"
CLIENT_ARGS="${SECS} ${PORTS}"

# process/thread list configuration.
PIDOF="/sbin/pidof"
PLIST="ps -o pid,tid,class,rtprio,ni,pri,pcpu,stat,comm"

# Log filename.
LOG="jack_test3-`uname -r`-`date +'%Y%m%d%H%M'`.log"

# Command executive and logger.
exec_log ()
{
	CMD="$*"
	echo "-----------------------"		>> ${LOG} 2>&1
	echo "# ${CMD}"				>> ${LOG} 2>&1
	${CMD}					>> ${LOG} 2>&1
	echo					>> ${LOG} 2>&1
}

# Process/thread status loggers.
plist_log ()
{
	PIDS="$*"
	if [ -n "${PIDS}" ]; then
		echo "- - - - - - - - - - - -"	>> ${LOG} 2>&1
		${PLIST} -m ${PIDS}		>> ${LOG} 2>&1
		echo				>> ${LOG} 2>&1
	fi
}


# IRQ thread status loggers.
ilist_log ()
{
	echo "- - - - - - - - - - - -"	>> ${LOG} 2>&1
	${PLIST} --sort -rtprio -e \
	| egrep '(^[ ]+PID|IRQ|jack)'	>> ${LOG} 2>&1
	echo				>> ${LOG} 2>&1
}


#
# Log headings -- show some relevant info...
#
echo "*** Started `date` ***" >> ${LOG} 2>&1

echo >> ${LOG} 2>&1
echo "Seconds to run        (SECS) = ${SECS}"		>> ${LOG} 2>&1
echo "Number of clients  (CLIENTS) = ${CLIENTS}"	>> ${LOG} 2>&1
echo "Ports per client     (PORTS) = ${PORTS}"		>> ${LOG} 2>&1
echo "Frames per buffer   (PERIOD) = ${PERIOD}"		>> ${LOG} 2>&1
echo "Playback ports (PLYBK_PORTS) = ${PLYBK_PORTS}"	>> ${LOG} 2>&1
echo >> ${LOG} 2>&1

exec_log "uname -a"

exec_log "cat /proc/asound/version"
exec_log "cat /proc/asound/cards"

#exec_log "grep . /proc/sys/kernel/*_preemption"
#exec_log "grep . /proc/irq/*/*/threaded"
#exec_log "grep . /sys/block/hd*/queue/max_sectors_kb"

exec_log "cat /proc/interrupts"

# This is just about to be sure...
ilist_log 

#
# Launch nmeter in the background...
#
echo -n "Launching `basename ${NMETER}`..."
(nice -15 ${NMETER} ${NMETER_ARGS} >> ${LOG} 2>&1) &
sleep 2
echo "done."
sleep 1

# Launch the test client suite...
SLEEP=6
echo -n "Launching ${CLIENTS} ${CLIENT}(s) instance(s)..."
for NUM in `seq 1 ${CLIENTS}`; do
	CLIENT_CMDLINE="${CLIENT} ${CLIENT_ARGS}"
	SLEEP=$((${SLEEP} + 3))
	(sleep ${SLEEP}; echo -n "$NUM..."; exec_log ${CLIENT_CMDLINE}) &
done
echo "done."

# Let there be some evidence of process status...
SLEEP=$((${SLEEP} + 6))
(sleep ${SLEEP}; plist_log -C `basename ${JACKD}`; plist_log -C `basename ${CLIENT}`) &

# Let jackd live for extended but limited time...
SLEEP=$((${SLEEP} + ${SECS}))
(sleep ${SLEEP}; killall `basename ${JACKD}`) &

#
# Launch jackd and wait for it...
#
echo -n "Running `basename ${JACKD}`..."
exec_log ${JACKD} ${JACKD_ARGS}
echo "done."
sleep 1

#echo -n "Killing `basename ${CLIENT}`..."
#killall `basename ${CLIENT}` && echo "OK." || echo "FAILED."
#sleep 1

echo -n "Killing `basename ${NMETER}`..."
killall `basename ${NMETER}` && echo "OK." || echo "FAILED."

echo "*** Terminated `date` ***" >> ${LOG} 2>&1

sync
sleep 1
sync

# Summary log analynis...
cat ${LOG} | awk -f ${BASEDIR}/jack_test3_summary.awk | tee -a ${LOG}

# Finally, plot log output...
${BASEDIR}/jack_test3_plot.sh ${LOG}

[-- Attachment #7: Type: text/plain, Size: 1433 bytes --]


What version of JACK do you have?  (jackd --version)

You need a recent CVS version to get all the data collection Rui and
Lee have added.  Looks like you're missing that.  Delay Max is an
important statistic.

You need a *very* recent version (about 10 minutes ago) to fix that
jack_deactivate() segfault.  (see URL below)

> It occasionally would segfault on client exit as well (as you've
> already mentioned). I think we're still in the dark here to be honest.

Try again with JACK 0.99.48.  It's in CVS now, but you probably need
this tarball to get around the dreaded SourceForge anon CVS lag...

   http://www.joq.us/jack/tarballs/jack-audio-connection-kit-0.99.48.tar.gz

The results I get with these versions are a lot more stable.  But,
there are still some puzzles about the scheduling.  Do you distinguish
different SCHED_ISO priorities?  

JACK runs with three different SCHED_FIFO priorities:

  (1) The main jackd audio thread uses the -P parameter.  The JACK
  default is 10, but Rui's script sets it with -P60.  I don't think
  the absolute value matters, but the value relative to the other JACK
  realtime threads probably does.

  (2) The clients' process threads run at a priority one less (59).

  (3) The watchdog timer thread runs at a priority 10 greater (70).

  (4) LinuxThreads creates a manager thread in each process running
  one level higher than the highest user realtime thread priority.
-- 
  joq