ktest.pl always returns 0?

ktest.pl always returns 0?

[Greg KH]

Hi Steven,

I'm trying to use ktest to do build tests of the stable patch series to
verify I didn't mess anything up, but I'm finding that ktest always
returns 0 when finished, no matter if the build test was successful or
failed.

Is this right?  Is there some other way to determine if ktest fails
other than greping the output log?

Oh, and any hints on kicking off a ktest process on a remote machine in
a "simple" way?  I'm just using ssh to copy over a script that runs
there, wrapping ktest.pl up with other stuff, I didn't miss the fact
that ktest itself can run remotely already, did I?

thanks,

greg k-h

Re: ktest.pl always returns 0?

[Steven Rostedt]

On Tue, 2012-09-25 at 11:00 -0700, Greg KH wrote:
> Hi Steven,

Note, emailing my RH account is hit or miss. If I'm traveling I don't
read it, and I wont return messages until I'm back. It's best to email
my rostedt@goodmis.org account, as I have better access to that account.
I author my patches by the email of the people that pay me to write
them. This isn't for your "who wrote the kernel" scripts. This is for
anyone that happens to do a git log.

> 
> I'm trying to use ktest to do build tests of the stable patch series to
> verify I didn't mess anything up, but I'm finding that ktest always
> returns 0 when finished, no matter if the build test was successful or
> failed.

Hmm, I should fix that. Yeah, I agree, if it fails a test it should
return something other than zero. But I think that only happens if you
have DIE_ON_FAILURE = 0. As IIRC, the perl "die" command should exit the
application with an error code.

But yeah, I agree, if one of the tests fail, the error code should not
be zero. I'll write up a patch to fix that. Or at least add an option to
make that happen.


> 
> Is this right?  Is there some other way to determine if ktest fails
> other than greping the output log?

If you have DIE_ON_FAILURE = 1 (default) it should exit with non zero.

> 
> Oh, and any hints on kicking off a ktest process on a remote machine in
> a "simple" way?  I'm just using ssh to copy over a script that runs
> there, wrapping ktest.pl up with other stuff, I didn't miss the fact
> that ktest itself can run remotely already, did I?

I'm a little confused by this question. Do you want a server ktest? That
is, have a ktest daemon that listens for clients that sends it config
files and then runs them? That would actually be a fun project ;-)

You're not running ktest on the target machine are you? The way I use it
is the following:

I have a server that I ssh to and run ktest from. It does all the builds
there on the server and this server has a means to monitor some target.
I use ttywatch that connects to the serial of the target, in which ktest
uses to read from.

Sometimes this "server" is the machine I'm logged in to.  And I just run
ktest directly.

Can you explain more of what you are looking for?

Thanks!

-- Steve

Re: ktest.pl always returns 0?

[Greg KH]

[-- Attachment #1: Type: text/plain, Size: 3756 bytes --]

On Tue, Sep 25, 2012 at 02:15:17PM -0400, Steven Rostedt wrote:
> On Tue, 2012-09-25 at 11:00 -0700, Greg KH wrote:
> > Hi Steven,
> 
> Note, emailing my RH account is hit or miss. If I'm traveling I don't
> read it, and I wont return messages until I'm back. It's best to email
> my rostedt@goodmis.org account, as I have better access to that account.
> I author my patches by the email of the people that pay me to write
> them. This isn't for your "who wrote the kernel" scripts. This is for
> anyone that happens to do a git log.

Hey, it's not my fault your employer has a crummy email system that
can't handle remote access well, I just went off of the Author: line in
your ktest.pl kernel commits :)

> > I'm trying to use ktest to do build tests of the stable patch series to
> > verify I didn't mess anything up, but I'm finding that ktest always
> > returns 0 when finished, no matter if the build test was successful or
> > failed.
> 
> Hmm, I should fix that. Yeah, I agree, if it fails a test it should
> return something other than zero. But I think that only happens if you
> have DIE_ON_FAILURE = 0. As IIRC, the perl "die" command should exit the
> application with an error code.
> 
> But yeah, I agree, if one of the tests fail, the error code should not
> be zero. I'll write up a patch to fix that. Or at least add an option to
> make that happen.

That would be great.

> > Is this right?  Is there some other way to determine if ktest fails
> > other than greping the output log?
> 
> If you have DIE_ON_FAILURE = 1 (default) it should exit with non zero.

It doesn't do that, test it and see (this is with what is in Linus's
3.6-rc7 tree, I didn't test linux-next if that is newer, my apologies.)

> > Oh, and any hints on kicking off a ktest process on a remote machine in
> > a "simple" way?  I'm just using ssh to copy over a script that runs
> > there, wrapping ktest.pl up with other stuff, I didn't miss the fact
> > that ktest itself can run remotely already, did I?
> 
> I'm a little confused by this question. Do you want a server ktest? That
> is, have a ktest daemon that listens for clients that sends it config
> files and then runs them? That would actually be a fun project ;-)
> 
> You're not running ktest on the target machine are you? The way I use it
> is the following:
> 
> I have a server that I ssh to and run ktest from. It does all the builds
> there on the server and this server has a means to monitor some target.
> I use ttywatch that connects to the serial of the target, in which ktest
> uses to read from.
> 
> Sometimes this "server" is the machine I'm logged in to.  And I just run
> ktest directly.
> 
> Can you explain more of what you are looking for?

I want to be able to say:
	- take this set of stable patches and go run a 'make
	  allmodconfig' build on a remote machine and email me back the
	  answer because I might not be able to keep an internet
	  connection open for the next 5-15 minutes it might take to
	  complete that task.

I don't do boot tests with these kernel build tests, although sometime
in the future it would be nice to do that.  Right now I do that testing
manually, as it's pretty infrequent (once per release usually.)

So yes, a 'ktest' server would be nice.  I've attached the (horrible)
script below that I'm using for this so far.  It seems to work well, and
I can do builds on a "cloud" server as well as my local build server
just fine, only thing needed to do is change the user and machine name
in the script.

I know ktest doesn't handle quilt patches yet, which is why I apply them
"by hand" now to a given git tree branch, if you ever do add that
option, I'll gladly test it out and change my script to use whatever
format it needs.

thanks,

greg k-h

[-- Attachment #2: ts --]
[-- Type: text/plain, Size: 4647 bytes --]

#!/bin/bash
#
# Testing script to take a stable kernel patch set, build it on a remote
# machine using ktest, and email back the results.
#
# Copyright 2012 Greg Kroah-Hartman <gregkh@linuxfoundation.org>
#
# Released under the GPLv2 only.
#
#
# Some variables you might want to mess with are:
#
# EMAIL: who to send the email to
# REMOTE_STABLE_GIT: on the remote machine, where the linux-stable git tree is located
# REMOTE_WORK: on the remote machine, what temporary location we can use to create a subdirectory and do our work in
# REMOTE_SERVER: the remote machine name
# REMOTE_USER: the username to run the script on the remote machine
# LOCAL_WORK: temporay location on the local machine to create some files before we copy them to the remote machine
# LOCAL_KTEST: local location of a version of ktest.pl that you want to run remotely (usually better than the version in the stable tree under testing due to age issues.)
# QUEUE_DIR: local location of the stable-queue git tree we are wanting to test


EMAIL="greg@kroah.com"

REMOTE_STABLE_GIT="/home/gregkh/linux/stable/linux-stable/"
REMOTE_WORK="/home/gregkh/tmp/"
REMOTE_SERVER="build"
REMOTE_USER="gregkh"

LOCAL_WORK="/tmp/"
LOCAL_KTEST="/home/gregkh/linux/gregkh/tools/testing/ktest/ktest.pl"
QUEUE_DIR="/home/gregkh/linux/stable/stable-queue/"

#######################################################
# I doubt you need to touch anything below this line,
# unless you want to fix my bugs, or rewrite the scripts
# to be saner (hint, feel free to do so...)
#######################################################


# grab the kernel version from the command line
KERNEL_VERSION="$1"

if [ "${KERNEL_VERSION}" == "" ] ; then
	echo "$0 KERNEL_VERSION"
	exit;
fi

# create the local temporary directory to do the work in.
SCRIPT_HOME=`mktemp -d ${LOCAL_WORK}/stable_test_XXXXX`
echo "Using ${SCRIPT_HOME} for our local files"

# create the remote directory
REMOTE_DIR=`ssh ${REMOTE_USER}@${REMOTE_SERVER} mktemp -d ${REMOTE_WORK}/stable_test_XXXXX`
echo "Using ${REMOTE_DIR} for the remote directory"


# tar up stable patch queue for just this kernel version
cd ${QUEUE_DIR}/queue-${KERNEL_VERSION}/
tar -c . | gzip > ${SCRIPT_HOME}/stable_queue.tar.gz

# create the script to run remotely

cat << __EOF__ > ${SCRIPT_HOME}/run_test.sh
#!/bin/bash
# Test script for the ${KERNEL_VERSION}-stable kernel
# autogenerated, do not edit by hand
#
STABLE_GIT="${REMOTE_STABLE_GIT}"
KERNEL_VERSION="${KERNEL_VERSION}"
PATCHES="stable_queue.tar.gz"

cd ${REMOTE_DIR}

# create the linux clone
git clone -s ${REMOTE_STABLE_GIT} linux
cd linux

# checkout the branch we need
git checkout -t -b linux-${KERNEL_VERSION}.y origin/linux-${KERNEL_VERSION}.y

# remove the .git directory as we don't need that space hanging around anymore
#rm -rf .git/

# create a patches/ directory for the stable patches to apply
mkdir patches
cd patches
tar -zxvf ${REMOTE_DIR}/\${PATCHES}
cd ..

# Apply the patch queue
QUILT_PATCHES=patches QUILT_SERIES=patches/series quilt push -aq --quiltrc

# build stuff
perl ../ktest.pl ../ktest.conf

KTEST_RUN=$?
echo "KTEST_RUN = \${KTEST_RUN}"

cd ..

if [ "\${KTEST_RUN}" = "0" ]; then
	# test succeeded
	SUBJECT="Build of ${KERNEL_VERSION} was good"
else
	SUBJECT="Build of ${KERNEL_VERSION} FAILED!"
fi

mutt -s "\${SUBJECT}" -a linux/log -- ${EMAIL} < output_log

# now that we are done, clean up after ourselves
rm -rf ${REMOTE_DIR}


__EOF__
chmod 755 ${SCRIPT_HOME}/run_test.sh

# create the ktest.conf file
cat <<__EOF__ > ${SCRIPT_HOME}/ktest.conf
TEST_START
TEST_TYPE = build

DEFAULTS
BUILD_TYPE = allmodconfig
OUTPUT_DIR = \${PWD}/output
LOG_FILE = \${PWD}/log
LOCALVERSION = -test
BUILD_OPTIONS = -j16
MACHINE = aws
BUILD_DIR = \${PWD}
__EOF__


# copy a version of ktest that we want to run
cp ${LOCAL_KTEST} ${SCRIPT_HOME}

# copy the files we created to the remote machine
scp -q ${SCRIPT_HOME}/* ${REMOTE_USER}@${REMOTE_SERVER}:${REMOTE_DIR}

# remove the local files as we don't need them anymore
rm -rf ${SCRIPT_HOME}

# execute the script on the remote machine

#ssh ${REMOTE_USER}@${REMOTE_SERVER} "cd ${REMOTE_DIR} && nohup \"cd ${REMOTE_DIR} && ./run_test.sh & > ${REMOTE_DIR} output_log \" "
#ssh ${REMOTE_USER}@${REMOTE_SERVER} "cd ${REMOTE_DIR} && nohup \"${REMOTE_DIR}/run_test.sh & > ${REMOTE_DIR}/output_log\""
#ssh ${REMOTE_USER}@${REMOTE_SERVER} "nohup ${REMOTE_DIR}/run_test.sh > ${REMOTE_DIR}/output_log &"
#ssh ${REMOTE_USER}@${REMOTE_SERVER} "${REMOTE_DIR}/run_test.sh > ${REMOTE_DIR}/output_log"
ssh -n -f ${REMOTE_USER}@${REMOTE_SERVER} "sh -c \"cd ${REMOTE_DIR}; nohup ./run_test.sh > ${REMOTE_DIR}/output_log 2>&1 &\" "


exit

Re: ktest.pl always returns 0?

[Steven Rostedt]

On Tue, 2012-09-25 at 12:40 -0700, Greg KH wrote:

> Hey, it's not my fault your employer has a crummy email system that
> can't handle remote access well, I just went off of the Author: line in
> your ktest.pl kernel commits :)

Yeah, I'm not upset by it. I just want to warn people that there's times
I may spend long periods of not answering that email.

> 
> > > I'm trying to use ktest to do build tests of the stable patch series to
> > > verify I didn't mess anything up, but I'm finding that ktest always
> > > returns 0 when finished, no matter if the build test was successful or
> > > failed.
> > 
> > Hmm, I should fix that. Yeah, I agree, if it fails a test it should
> > return something other than zero. But I think that only happens if you
> > have DIE_ON_FAILURE = 0. As IIRC, the perl "die" command should exit the
> > application with an error code.
> > 
> > But yeah, I agree, if one of the tests fail, the error code should not
> > be zero. I'll write up a patch to fix that. Or at least add an option to
> > make that happen.
> 
> That would be great.
> 
> > > Is this right?  Is there some other way to determine if ktest fails
> > > other than greping the output log?
> > 
> > If you have DIE_ON_FAILURE = 1 (default) it should exit with non zero.
> 
> It doesn't do that, test it and see (this is with what is in Linus's
> 3.6-rc7 tree, I didn't test linux-next if that is newer, my apologies.)

This should have been something from day one. I'll go ahead and try it
out. According to the perl-doc man pages the "die" command has:

           If an uncaught exception results in interpreter exit, the exit
           code is determined from the values of $! and $? with this
           pseudocode:

               exit $! if $!;              # errno
               exit $? >> 8 if $? >> 8;    # child exit status
               exit 255;                   # last resort

I'll investigate this further.

> 
> > > Oh, and any hints on kicking off a ktest process on a remote machine in
> > > a "simple" way?  I'm just using ssh to copy over a script that runs
> > > there, wrapping ktest.pl up with other stuff, I didn't miss the fact
> > > that ktest itself can run remotely already, did I?
> > 
> > I'm a little confused by this question. Do you want a server ktest? That
> > is, have a ktest daemon that listens for clients that sends it config
> > files and then runs them? That would actually be a fun project ;-)
> > 
> > You're not running ktest on the target machine are you? The way I use it
> > is the following:
> > 
> > I have a server that I ssh to and run ktest from. It does all the builds
> > there on the server and this server has a means to monitor some target.
> > I use ttywatch that connects to the serial of the target, in which ktest
> > uses to read from.
> > 
> > Sometimes this "server" is the machine I'm logged in to.  And I just run
> > ktest directly.
> > 
> > Can you explain more of what you are looking for?
> 
> I want to be able to say:
> 	- take this set of stable patches and go run a 'make
> 	  allmodconfig' build on a remote machine and email me back the
> 	  answer because I might not be able to keep an internet
> 	  connection open for the next 5-15 minutes it might take to
> 	  complete that task.

I cheat and run all my ktests in screen sessions ;-)

> 
> I don't do boot tests with these kernel build tests, although sometime
> in the future it would be nice to do that.  Right now I do that testing
> manually, as it's pretty infrequent (once per release usually.)
> 
> So yes, a 'ktest' server would be nice.  I've attached the (horrible)
> script below that I'm using for this so far.  It seems to work well, and
> I can do builds on a "cloud" server as well as my local build server
> just fine, only thing needed to do is change the user and machine name
> in the script.

This looks like my next "when I have time" project ;-).


> 
> I know ktest doesn't handle quilt patches yet, which is why I apply them
> "by hand" now to a given git tree branch, if you ever do add that
> option, I'll gladly test it out and change my script to use whatever
> format it needs.
> 

Yeah, I need to make ktest work with quilt, as I'm still a fan.

But currently the ones that pay me actually are giving me things to do.
Something about satisfying customers or some other crap. Thus, my "down
time" is limited at the moment :-(  But when things on the customer side
slows down again, I'll definitely work on these changes.

Thanks for the ideas! I'm actually looking forward to working on this.
But in the mean time, I will test the next time ktest fails on me to see
what the result of $? is.

-- Steve

Re: ktest.pl always returns 0?

[Steven Rostedt]

On Tue, 2012-09-25 at 16:33 -0400, Steven Rostedt wrote:
> On Tue, 2012-09-25 at 12:40 -0700, Greg KH wrote:
> 
> > Hey, it's not my fault your employer has a crummy email system that
> > can't handle remote access well, I just went off of the Author: line in
> > your ktest.pl kernel commits :)
> 
> Yeah, I'm not upset by it. I just want to warn people that there's times
> I may spend long periods of not answering that email.
> 
> > 
> > > > I'm trying to use ktest to do build tests of the stable patch series to
> > > > verify I didn't mess anything up, but I'm finding that ktest always
> > > > returns 0 when finished, no matter if the build test was successful or
> > > > failed.
> > > 
> > > Hmm, I should fix that. Yeah, I agree, if it fails a test it should
> > > return something other than zero. But I think that only happens if you
> > > have DIE_ON_FAILURE = 0. As IIRC, the perl "die" command should exit the
> > > application with an error code.
> > > 
> > > But yeah, I agree, if one of the tests fail, the error code should not
> > > be zero. I'll write up a patch to fix that. Or at least add an option to
> > > make that happen.
> > 
> > That would be great.
> > 
> > > > Is this right?  Is there some other way to determine if ktest fails
> > > > other than greping the output log?
> > > 
> > > If you have DIE_ON_FAILURE = 1 (default) it should exit with non zero.
> > 
> > It doesn't do that, test it and see (this is with what is in Linus's
> > 3.6-rc7 tree, I didn't test linux-next if that is newer, my apologies.)
> 
> This should have been something from day one. I'll go ahead and try it
> out. According to the perl-doc man pages the "die" command has:
> 
>            If an uncaught exception results in interpreter exit, the exit
>            code is determined from the values of $! and $? with this
>            pseudocode:
> 
>                exit $! if $!;              # errno
>                exit $? >> 8 if $? >> 8;    # child exit status
>                exit 255;                   # last resort
> 
> I'll investigate this further.
> 
> > 
> > > > Oh, and any hints on kicking off a ktest process on a remote machine in
> > > > a "simple" way?  I'm just using ssh to copy over a script that runs
> > > > there, wrapping ktest.pl up with other stuff, I didn't miss the fact
> > > > that ktest itself can run remotely already, did I?
> > > 
> > > I'm a little confused by this question. Do you want a server ktest? That
> > > is, have a ktest daemon that listens for clients that sends it config
> > > files and then runs them? That would actually be a fun project ;-)
> > > 
> > > You're not running ktest on the target machine are you? The way I use it
> > > is the following:
> > > 
> > > I have a server that I ssh to and run ktest from. It does all the builds
> > > there on the server and this server has a means to monitor some target.
> > > I use ttywatch that connects to the serial of the target, in which ktest
> > > uses to read from.
> > > 
> > > Sometimes this "server" is the machine I'm logged in to.  And I just run
> > > ktest directly.
> > > 
> > > Can you explain more of what you are looking for?
> > 
> > I want to be able to say:
> > 	- take this set of stable patches and go run a 'make
> > 	  allmodconfig' build on a remote machine and email me back the
> > 	  answer because I might not be able to keep an internet
> > 	  connection open for the next 5-15 minutes it might take to
> > 	  complete that task.
> 
> I cheat and run all my ktests in screen sessions ;-)
> 
> > 
> > I don't do boot tests with these kernel build tests, although sometime
> > in the future it would be nice to do that.  Right now I do that testing
> > manually, as it's pretty infrequent (once per release usually.)
> > 
> > So yes, a 'ktest' server would be nice.  I've attached the (horrible)
> > script below that I'm using for this so far.  It seems to work well, and
> > I can do builds on a "cloud" server as well as my local build server
> > just fine, only thing needed to do is change the user and machine name
> > in the script.
> 
> This looks like my next "when I have time" project ;-).
> 
> 
> > 
> > I know ktest doesn't handle quilt patches yet, which is why I apply them
> > "by hand" now to a given git tree branch, if you ever do add that
> > option, I'll gladly test it out and change my script to use whatever
> > format it needs.
> > 
> 
> Yeah, I need to make ktest work with quilt, as I'm still a fan.
> 
> But currently the ones that pay me actually are giving me things to do.
> Something about satisfying customers or some other crap. Thus, my "down
> time" is limited at the moment :-(  But when things on the customer side
> slows down again, I'll definitely work on these changes.
> 
> Thanks for the ideas! I'm actually looking forward to working on this.
> But in the mean time, I will test the next time ktest fails on me to see
> what the result of $? is.
> 

I just forced a build failure to see what ktest would show. This is my
result:

cp /home/rostedt/work/git/configs/ixf/config-use /home/rostedt/work/git/nobackup/ixf/trace/.config ... SUCCESS
touch /home/rostedt/work/git/nobackup/ixf/trace/.config ... SUCCESS
Applying minimum configurations into /home/rostedt/work/git/nobackup/ixf/trace/.config.new
mv /home/rostedt/work/git/nobackup/ixf/trace/.config.new /home/rostedt/work/git/nobackup/ixf/trace/.config ... SUCCESS
GCC_VERSION=4.6.0 distmake-64 O=/home/rostedt/work/git/nobackup/ixf/trace oldnoconfig ... SUCCESS
GCC_VERSION=4.6.0 distmake-64 O=/home/rostedt/work/git/nobackup/ixf/trace -j40 ... FAILED!
CRITICAL FAILURE... failed build
 See /home/rostedt/work/git/nobackup/ixf/ixf.log for more info.
failed build
[rostedt@fedora git]$ echo $?
25


So it seems that if DIE_ON_FAILURE is set, it returns non-zero error.
But if it doesn't then it wont. But for that, I have this patch:


--- ktest.pl.orig	2012-09-25 19:43:08.608524543 -0400
+++ ktest.pl	2012-09-25 19:46:21.795560499 -0400
@@ -3709,4 +3709,5 @@
 
 doprint "\n    $successes of $opt{NUM_TESTS} tests were successful\n\n";
 
-exit 0;
+# Return number of failures (Yeah if we had 256 tests fail this breaks)
+exit $opt{NUM_TESTS} - $successes

Re: ktest.pl always returns 0?

[Greg KH]

On Tue, Sep 25, 2012 at 07:48:30PM -0400, Steven Rostedt wrote:
> I just forced a build failure to see what ktest would show. This is my
> result:
> 
> cp /home/rostedt/work/git/configs/ixf/config-use /home/rostedt/work/git/nobackup/ixf/trace/.config ... SUCCESS
> touch /home/rostedt/work/git/nobackup/ixf/trace/.config ... SUCCESS
> Applying minimum configurations into /home/rostedt/work/git/nobackup/ixf/trace/.config.new
> mv /home/rostedt/work/git/nobackup/ixf/trace/.config.new /home/rostedt/work/git/nobackup/ixf/trace/.config ... SUCCESS
> GCC_VERSION=4.6.0 distmake-64 O=/home/rostedt/work/git/nobackup/ixf/trace oldnoconfig ... SUCCESS
> GCC_VERSION=4.6.0 distmake-64 O=/home/rostedt/work/git/nobackup/ixf/trace -j40 ... FAILED!
> CRITICAL FAILURE... failed build
>  See /home/rostedt/work/git/nobackup/ixf/ixf.log for more info.
> failed build
> [rostedt@fedora git]$ echo $?
> 25
> 
> 
> So it seems that if DIE_ON_FAILURE is set, it returns non-zero error.

Ugh, you are right, my mistake, it was a bug in my script that I was not
checking the correct return value (for those wondering, my bash script
was writing out a bash script that then got run, I forgot to escape the
"$?" properly, so it was being intrepreted when written out and not when
run.

So sorry for the noise, all is now working just fine.

greg "I should stick to C" k-h