[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[Jerry Van Baren]

This uses PyUnit and python-serial <http://pyserial.sourceforge.net/>
to do unit testing on u-boot commands.

Signed-off-by: Gerald Van Baren <vanbaren@cideas.com>
---

...and now, for something completely different.

The concept here is to use PyUnit testing in conjunction with python
serial I/O (theoretically, ethernet-based command I/O could be handled
too).

This is a concept probably sufficiently implemented for the "help"
command and a good start for the "fdt" command.  

What does The List think?
* Useful to have regression tests for the u-boot commands?
* Implementation-wise, am I heading in the right direction?

Best regards,
gvb

 u-unit/.gitignore |    2 +
 u-unit/fdt.dts    |   30 +++++++++
 u-unit/fdt.py     |  171 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 u-unit/help.py    |   70 ++++++++++++++++++++++
 u-unit/ubootio.py |   55 +++++++++++++++++
 5 files changed, 328 insertions(+), 0 deletions(-)
 create mode 100644 u-unit/.gitignore
 create mode 100644 u-unit/fdt.dts
 create mode 100755 u-unit/fdt.py
 create mode 100755 u-unit/help.py
 create mode 100755 u-unit/ubootio.py

diff --git a/u-unit/.gitignore b/u-unit/.gitignore
new file mode 100644
index 0000000..88e6bec
--- /dev/null
+++ b/u-unit/.gitignore
@@ -0,0 +1,2 @@
+*.pyc
+*.dtb
diff --git a/u-unit/fdt.dts b/u-unit/fdt.dts
new file mode 100644
index 0000000..66efcdd
--- /dev/null
+++ b/u-unit/fdt.dts
@@ -0,0 +1,30 @@
+/dts-v1/;
+
+/ {
+	compatible = "fdt";
+	prop-int = <0xdeadbeef>;
+	prop-str = "hello world";
+
+	subnode at 1 {
+		compatible = "subnode1";
+		prop-int = <0x01234567>;
+
+		subsubnode {
+			compatible = "subsubnode1", "subsubnode";
+			prop-int = <0x12345678>;
+			prop-empty;
+		};
+	};
+
+	subnode at 2 {
+		linux,phandle = <0x2000>;
+		prop-int = <0x23456789>;
+
+		subsubnode at 0 {
+			linux,phandle = <0x2001>;
+			compatible = "subsubnode2", "subsubnode";
+			prop-empty;
+			prop-int = <0x3456789a>;
+		};
+	};
+};
diff --git a/u-unit/fdt.py b/u-unit/fdt.py
new file mode 100755
index 0000000..53c1117
--- /dev/null
+++ b/u-unit/fdt.py
@@ -0,0 +1,171 @@
+#!/usr/bin/python
+#
+# Test suite for the fdt u-boot command
+#
+
+import unittest
+import re
+import ubootio
+
+class FdtTestCase(unittest.TestCase):
+	"""
+	Run regression tests the "fdt" command and subcommands
+
+	This requires a test fdt blob to be loaded.
+
+	Subcommands to be tested...
+help fdt                                                      
+fdt addr   <addr> [<length>]        - Set the fdt location to <addr>
+fdt boardsetup                      - Do board-specific set up
+fdt move   <fdt> <newaddr> <length> - Copy the fdt to <addr>
+fdt print  <path> [<prop>]          - Recursive print starting at <path>
+fdt list   <path> [<prop>]          - Print one level starting at <path>
+fdt set    <path> <prop> [<val>]    - Set <property> [to <val>]
+fdt mknode <path> <node>            - Create a new node after <path>
+fdt rm     <path> [<prop>]          - Delete the node or <property>
+fdt chosen - Add/update the /chosen branch in the tree
+fdt env    - Add/replace the /u-boot-env branch in the tree
+fdt bd_t   - Add/replace the /bd_t branch in the tree
+	"""
+
+	#
+	# Configuration
+	#
+	scratchram	= "0x400000"
+	serverip	= "192.168.47.8"
+	ipaddr		= "192.168.47.214"
+
+
+	def setUp(self):
+		"""fdt test case setup."""
+
+		# Configuration
+		self.fdt_setup = "\
+set serverip " + self.serverip + " ; \
+set ipaddr " + self.ipaddr + " ; \
+tftp " + self.scratchram + " fdt.dtb ; \
+fdt address " + self.scratchram + "\n"
+
+		self.fdt_dts = """\
+/ {
+	compatible = "fdt";
+	prop-int = <0xdeadbeef>;
+	prop-str = "hello world";
+	subnode at 1 {
+		compatible = "subnode1";
+		prop-int = <0x01234567>;
+		subsubnode {
+			compatible = "subsubnode1", "subsubnode";
+			prop-int = <0x12345678>;
+			prop-empty;
+		};
+	};
+	subnode at 2 {
+		linux,phandle = <0x00002000>;
+		prop-int = <0x23456789>;
+		subsubnode at 0 {
+			linux,phandle = <0x00002001>;
+			compatible = "subsubnode2", "subsubnode";
+			prop-empty;
+			prop-int = <0x3456789a>;
+		};
+	};
+};"""
+		self.fdt_subnode2_dts = """\
+subnode at 2 {
+	linux,phandle = <0x00002000>;
+	prop-int = <0x23456789>;
+	subsubnode at 0 {
+		linux,phandle = <0x00002001>;
+		compatible = "subsubnode2", "subsubnode";
+		prop-empty;
+		prop-int = <0x3456789a>;
+	};
+};"""
+		self.fdt_list_dts = """\
+/ {
+	compatible = "fdt";
+	prop-int = <0xdeadbeef>;
+	prop-str = "hello world";
+	subnode at 1 {
+	};
+	subnode at 2 {
+	};
+};"""
+
+		self.re_dts = re.compile(self.fdt_dts)
+		self.re_subnode2_dts = re.compile(self.fdt_subnode2_dts)
+		self.re_list_dts = re.compile(self.fdt_list_dts)
+
+		self.IO = ubootio.uBootIO()
+
+		self.IO.sendcmd(self.fdt_setup);
+		n = self.IO.waitresponse()
+		if (n <= 0):
+			print "FAIL: Timeout"
+			raise AssertionError
+		s = self.IO.getresponse(n)
+
+		# Verify that the dtb was loaded.
+		if (not re.search("Bytes transferred =", s)):
+			print "FAIL: could not load the test dtb.\n", s
+			raise AssertionError
+
+	def tearDown(self):
+		pass
+
+	def testPrintRoot(self):
+		"""
+		Print starting@the root node.
+		"""
+		self.IO.sendcmd("fdt print /\n");
+		n = self.IO.waitresponse()
+		assert n > 0, "Timeout"
+
+		s = self.IO.getresponse(n)
+		assert self.re_dts.search(s), \
+			"fdt print did not match.\nIS:\n" + s + \
+			"SHOULD BE:\n" + fdt_dts
+
+	def testPrintUnspecified(self):
+		"""
+		Print with no node specified, defaults to the root node.
+		"""
+		self.IO.sendcmd("fdt print\n");
+		n = self.IO.waitresponse()
+		assert n > 0, "Timeout"
+
+		s = self.IO.getresponse(n)
+		assert self.re_dts.search(s), \
+			"fdt print did not match.\nIS:\n" + s + \
+			"SHOULD BE:\n" + fdt_dts
+
+	def testPrintSubnode(self):
+		"""
+		Print a subnode.
+		"""
+		self.IO.sendcmd("fdt p /subnode at 2\n");
+		n = self.IO.waitresponse()
+		assert n > 0, "Timeout"
+
+		s = self.IO.getresponse(n)
+		assert self.re_subnode2_dts.search(s), \
+			"fdt print did not match.\nIS:\n" + s + \
+			"SHOULD BE:\n" + fdt_subnode2_dts
+
+	def testListRoot(self):
+		"""
+		List starting at the root node.
+		"""
+		self.IO.sendcmd("fdt list\n");
+		n = self.IO.waitresponse()
+		assert n > 0, "Timeout"
+
+		s = self.IO.getresponse(n)
+		assert self.re_list_dts.search(s), \
+			"fdt list did not match.\nIS:\n" + s + \
+			"SHOULD BE:\n" + fdt_list_dts
+
+
+if __name__ == "__main__":
+	unittest.main()
diff --git a/u-unit/help.py b/u-unit/help.py
new file mode 100755
index 0000000..dbd0e0e
--- /dev/null
+++ b/u-unit/help.py
@@ -0,0 +1,70 @@
+#!/usr/bin/python
+#
+# Test suite for the fdt u-boot command
+#
+
+import unittest
+import re
+import ubootio
+
+class HelpTestCase(unittest.TestCase):
+        """ Run regression tests the "help" command """
+
+
+	def setUp(self):
+		"""help test case setup."""
+
+		self.IO = ubootio.uBootIO()
+
+
+	def testHelp(self):
+		"""
+		Run regression tests the "help" command
+
+		This runs the "help" command and verifies that it returns
+		information on a subset of the commands.
+		"""
+
+		# These commands should be on all configurations
+		tests = [ \
+			"askenv", \
+			"base", \
+			"boot", \
+			"cp", \
+			"crc32", \
+			"echo", \
+			"erase", \
+			"exit", \
+			"go", \
+			"help", \
+			"icrc32", \
+			"md", \
+			"mm", \
+			"mw", \
+			"nm", \
+			"printenv", \
+			"protect", \
+			"reset", \
+			"run", \
+			"saveenv", \
+			"setenv", \
+			"sleep", \
+			"test", \
+			"version", \
+			"=>" ]
+
+		self.IO.sendcmd("help\n")
+		n = self.IO.waitresponse()
+		if (n <= 0):
+			print "FAIL: Timeout"
+			testpassed = False
+		else:
+			s = self.IO.getresponse(n)
+			for test in tests:
+				if (not re.search(test, s)):
+					print "FAIL: ", test, " not found in\n", s
+					testpassed = False
+
+if __name__ == "__main__":
+        unittest.main()
+
diff --git a/u-unit/ubootio.py b/u-unit/ubootio.py
new file mode 100755
index 0000000..3dad300
--- /dev/null
+++ b/u-unit/ubootio.py
@@ -0,0 +1,55 @@
+#!/usr/bin/python
+#
+# I/O for the u-boot command regression tests
+#
+
+import serial
+import time
+import re
+
+
+class uBootIO:
+	#
+	# Configuration
+	#
+	chantimeout	= 0.5
+
+	def __init__(self):
+		self.chan = serial.Serial(0, 115200, timeout=5.0, rtscts=0, \
+			parity=serial.PARITY_NONE)
+
+
+	def waitresponse(self):
+		"""
+		Waits for a response from the target.
+
+		Returns the number of characters waiting to be received.
+		"""
+
+		n = -1
+		time.sleep(self.chantimeout)
+		while (self.chan.inWaiting() > n):
+			time.sleep(self.chantimeout)
+			n = self.chan.inWaiting()
+		if (n <= 0):
+			print "FAIL: Timeout"
+		return n
+
+	def getresponse(self, n):
+		"""
+		Returns the response from the target.
+
+		Post-processes the response for line ends (changes to newlines).
+		"""
+
+		canonical = re.compile("[\r\n]+")
+
+		s = self.chan.read(n)
+		return canonical.sub("\n", s)
+
+	def sendcmd(self, s):
+		"""
+		Send a command string to the target.
+		"""
+
+		self.chan.write(s)
-- 
1.5.3.4

[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[Wolfgang Denk]

Hello,

in message <20071124010338.GA27628@cideas.com> you wrote:
> 
> ...and now, for something completely different.
> 
> The concept here is to use PyUnit testing in conjunction with python
> serial I/O (theoretically, ethernet-based command I/O could be handled
> too).
> 
> This is a concept probably sufficiently implemented for the "help"
> command and a good start for the "fdt" command.  
> 
> What does The List think?
> * Useful to have regression tests for the u-boot commands?

It is not only useful, but IMHO absolutely necessary to do regular
regression testing on U-Boot.

> * Implementation-wise, am I heading in the right direction?

I wish you had asked that question before spending efforts  on  this.
The point is, that there is already a pretty extensive and extandable
regression  test suite for U-Boot (and Linux, and some more). We just
failed to announce it loudly enough (which clearly shows that DENX is
not marketing driven - and funny enough, I'm not a bit ashamed  about
that).  However,  I'm sad about the avoidable duplication of efforts,
so I have to say I'm sorry.

Please have a look at the DUTS, our regression test suite. See the
git repo at http://www.denx.de/cgi-bin/gitweb.cgi?p=duts.git;a=summary
and some documentation at http://www.denx.de/wiki/DUTS/DUTSDocs

I'd really appreciate if this was picked up by others  as  well.  [It
took  us  a  long and winded road to get there - trying some existing
solutions like dejagnu etc. and finally ending with a  do-it-yourself
solution.]

Note: there is a non-abvious benefit from the DUTS: once you  got  it
running  on  a  board, it will generate a set of log files which just
need to be uploaded to our web server to generate a  new  version  of
the  DULG for this board. The idea is that if you have a new software
version you just need to run the DUTS to get (1) a confirmation  that
everything  is working as expected fromt he regression test suite and
(2) an updated version of the documentation  with  eexamples  showing
exacly this software version on exactly that hardware.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
Es gibt immer genug fuer die Beduerfnisse aller, aber  niemals  genug
fuer die Gier einzelner.                                    -- Ghandi

[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[Jerry Van Baren]

Wolfgang Denk wrote:
> Hello,
> 
> in message <20071124010338.GA27628@cideas.com> you wrote:
>> ...and now, for something completely different.
>>
>> The concept here is to use PyUnit testing in conjunction with python
>> serial I/O (theoretically, ethernet-based command I/O could be handled
>> too).
>>
>> This is a concept probably sufficiently implemented for the "help"
>> command and a good start for the "fdt" command.  
>>
>> What does The List think?
>> * Useful to have regression tests for the u-boot commands?
> 
> It is not only useful, but IMHO absolutely necessary to do regular
> regression testing on U-Boot.
> 
>> * Implementation-wise, am I heading in the right direction?
> 
> I wish you had asked that question before spending efforts  on  this.
> The point is, that there is already a pretty extensive and extandable
> regression  test suite for U-Boot (and Linux, and some more). We just
> failed to announce it loudly enough (which clearly shows that DENX is
> not marketing driven - and funny enough, I'm not a bit ashamed  about
> that).  However,  I'm sad about the avoidable duplication of efforts,
> so I have to say I'm sorry.

Oh no apology necessary.  It was a learning exercise, useful in its own 
right, WRT Python, PyUnit, PySerial, and seeing what I could do with them.

I've used tk/tcl/expect and am not wild about them.  The older scripting 
languages are so.... 70s. ;-)  I haven't used python much, but I've come 
to like it a lot, kind of a rational perl. :-/

My experience with expect is that it worked well for simple cases, but 
when I started matching more patterns and more complex patterns, it 
would get harder and harder to match reliably.  Also, its asynchronous 
timing-based capture (which causes "capture breaks" in different places 
when you least expect it) drove me crazy.  Then to debug the pattern 
matching (or not matching, or mismatching, as the case usually was) was 
difficult.

Having said that, that was a few years ago and maybe I'm smarter now...

> Please have a look at the DUTS, our regression test suite. See the
> git repo at http://www.denx.de/cgi-bin/gitweb.cgi?p=duts.git;a=summary
> and some documentation at http://www.denx.de/wiki/DUTS/DUTSDocs
> 
> I'd really appreciate if this was picked up by others  as  well.  [It
> took  us  a  long and winded road to get there - trying some existing
> solutions like dejagnu etc. and finally ending with a  do-it-yourself
> solution.]
> 
> Note: there is a non-abvious benefit from the DUTS: once you  got  it
> running  on  a  board, it will generate a set of log files which just
> need to be uploaded to our web server to generate a  new  version  of
> the  DULG for this board. The idea is that if you have a new software
> version you just need to run the DUTS to get (1) a confirmation  that
> everything  is working as expected fromt he regression test suite and
> (2) an updated version of the documentation  with  eexamples  showing
> exacly this software version on exactly that hardware.
> 
> Best regards,
> Wolfgang Denk

Thanks for the pointer.  I've poked around a bit and will continue to do so.

Best regards,
gvb

[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[Wolfgang Denk]

Hi,

in message <474ACEEC.1080801@ge.com> you wrote:
> 
> I've used tk/tcl/expect and am not wild about them.  The older scripting 
> languages are so.... 70s. ;-)  I haven't used python much, but I've come 
> to like it a lot, kind of a rational perl. :-/

I think I know what you mean.

> My experience with expect is that it worked well for simple cases, but 
> when I started matching more patterns and more complex patterns, it 
> would get harder and harder to match reliably.  Also, its asynchronous 
> timing-based capture (which causes "capture breaks" in different places 
> when you least expect it) drove me crazy.  Then to debug the pattern 
> matching (or not matching, or mismatching, as the case usually was) was 
> difficult.
> 
> Having said that, that was a few years ago and maybe I'm smarter now...

I'm open to all suggestions for improvement, but there is  a  certain
complexity  in  the  setup  itself,  and I cannot say how much can be
avoided by using other tools.

Best regards,

Wolfgang Denk

-- 
DENX Software Engineering GmbH,     MD: Wolfgang Denk & Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: wd at denx.de
If you believe that feeling bad or worrying long enough will change a
past or future event, then you are residing on another planet with  a
different reality system.

[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[Robert Schwebel]

On Mon, Nov 26, 2007 at 03:09:35PM +0100, Wolfgang Denk wrote:
> in message <474ACEEC.1080801@ge.com> you wrote:
> > 
> > I've used tk/tcl/expect and am not wild about them.  The older scripting 
> > languages are so.... 70s. ;-)  I haven't used python much, but I've come 
> > to like it a lot, kind of a rational perl. :-/
> 
> I think I know what you mean.

Same here, we are through all these iterations as well:

- we've written a test suite in python with xml specs (oo design, but
  very complex in the end, without the advantages we expected from
  an oo language)

- expect/tcl based test suite; causes brain cancer because of tcl

Now we do the u-boot tests/scripting with the tcl/expect stuff and all
other Linux test suites in normal shell scripts, remote-controlled via
ssh.

While the latter one has finally turned out to be a good way to write
test cases, we think about adding something like a dbus module to
u-boot-v2, which would make it possible to do real and well defined RPC
between a host and u-boot.

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[gvb.uboot]

Robert Schwebel wrote:
> On Mon, Nov 26, 2007 at 03:09:35PM +0100, Wolfgang Denk wrote:
>> in message <474ACEEC.1080801@ge.com> you wrote:
>>> I've used tk/tcl/expect and am not wild about them.  The older scripting 
>>> languages are so.... 70s. ;-)  I haven't used python much, but I've come 
>>> to like it a lot, kind of a rational perl. :-/
>> I think I know what you mean.
> 
> Same here, we are through all these iterations as well:
> 
> - we've written a test suite in python with xml specs (oo design, but
>   very complex in the end, without the advantages we expected from
>   an oo language)
> 
> - expect/tcl based test suite; causes brain cancer because of tcl
> 
> Now we do the u-boot tests/scripting with the tcl/expect stuff and all
> other Linux test suites in normal shell scripts, remote-controlled via
> ssh.
> 
> While the latter one has finally turned out to be a good way to write
> test cases, we think about adding something like a dbus module to
> u-boot-v2, which would make it possible to do real and well defined RPC
> between a host and u-boot.
> 
> rsc

Hmmm, I seem to have hit a resonance.

More background trivia... my weekend experiment was using python-serial. 
  I chose that and used it to capture whole serial transactions (waited 
until it appeared no more serial characters were being sent) so I could 
then use python regexp matching to do simpler, more elaborate, and more 
sane (yes, all three) pattern matching than what I was able to do with 
expect due to expect breaking the serial stream in semi-random time 
based chunks.

Another option that I have not experimented with is a python-based expect:
   <http://sourceforge.net/projects/pexpect/>

Best regards,
gvb

[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[Robert Schwebel]

On Tue, Nov 27, 2007 at 09:32:08PM -0500, gvb.uboot wrote:
> Hmmm, I seem to have hit a resonance.

:-)

> More background trivia... my weekend experiment was using python-serial. 
>  I chose that and used it to capture whole serial transactions (waited 
> until it appeared no more serial characters were being sent) so I could 
> then use python regexp matching to do simpler, more elaborate, and more 
> sane (yes, all three) pattern matching than what I was able to do with 
> expect due to expect breaking the serial stream in semi-random time 
> based chunks.
> 
> Another option that I have not experimented with is a python-based expect:
>   <http://sourceforge.net/projects/pexpect/>

That was what we used; two python sub-versions later they had changed
some internal interfaces and everything stopped working. Nothing that
looks like a stable platform to depend on ...

Robert
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[Kumar Gala]

I assume you'll be dropping this patch from your current libfdt tree  
in the testing branch.

- k

[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[gvb.uboot]

Kumar Gala wrote:
> I assume you'll be dropping this patch from your current libfdt tree  
> in the testing branch.
> 
> - k

Yes.  (In retrospect, I should have made a separate branch as well.)  I 
just dropped the spurious patch and re-pushed the testing branch (I kept 
the trivial "spaces around the =" print format change that I added just 
before the regression testing patch).

Thanks for reminding me to clean up,
gvb

[U-Boot-Users] [RFC PATCH] Add u-boot command regression tests.

[Mike Frysinger]

On Monday 26 November 2007, Jerry Van Baren wrote:
> My experience with expect is that it worked well for simple cases, but
> when I started matching more patterns and more complex patterns, it
> would get harder and harder to match reliably.  Also, its asynchronous
> timing-based capture (which causes "capture breaks" in different places
> when you least expect it) drove me crazy.  Then to debug the pattern
> matching (or not matching, or mismatching, as the case usually was) was
> difficult.

i think that's really nature of the beast and not a deficiency in expect.  how 
do you know when things stop coming in ?  you dont ... so write properly 
anchored input matches and things work nicely
-mike
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 827 bytes
Desc: This is a digitally signed message part.
Url : http://lists.denx.de/pipermail/u-boot/attachments/20071128/8dda484f/attachment.pgp