* [U-Boot] [PATCH] test/run: Exit on error from tests
@ 2017-11-27 21:21 Tom Rini
2017-11-27 21:45 ` Simon Glass
0 siblings, 1 reply; 5+ messages in thread
From: Tom Rini @ 2017-11-27 21:21 UTC (permalink / raw)
To: u-boot
Currently, result isn't saved outside of the scope of run_test, so we
always exit on error here. An easier way to deal with having test/run
show failures is to exit on failure.
Cc: Simon Glass <sjg@chromium.org>
Fixes: 2f52018c3cbb ("test/run: Report and return failure")
Signed-off-by: Tom Rini <trini@konsulko.com>
---
Automation is only as good as the humans behind it. If you forget, like
I did, that you finally fixed Jenkins to email you every time, that lack
of emails about my non-travis jobs failing wasn't "no problems", it was
"You forgot to migrate all the Docker stuff to the new build server".
So, I'm going to apply this ASAP as I would not have let in the
u-boot-dm PR, had my setup been functioning as expected.
---
test/run | 18 ++++--------------
1 file changed, 4 insertions(+), 14 deletions(-)
diff --git a/test/run b/test/run
index caee4f83f2c1..47dfed7bdca7 100755
--- a/test/run
+++ b/test/run
@@ -1,24 +1,14 @@
#!/bin/bash
-run_test() {
- $@
- [ $? -ne 0 ] && result=$((result+1))
- echo "result $result"
-}
+# Exit on error
+set +e
# Run all tests that the standard sandbox build can support
-run_test ./test/py/test.py --bd sandbox --build
+./test/py/test.py --bd sandbox --build
# Run tests which require sandbox_spl
-run_test ./test/py/test.py --bd sandbox_spl --build -k \
+./test/py/test.py --bd sandbox_spl --build -k \
test/py/tests/test_ofplatdata.py
# Run tests for the flat DT version of sandbox
./test/py/test.py --bd sandbox_flattree --build
-
-if [ $result == 0 ]; then
- echo "Tests passed!"
-else
- echo "Tests FAILED"
- exit 1
-fi
--
2.7.4
^ permalink raw reply related [flat|nested] 5+ messages in thread* [U-Boot] [PATCH] test/run: Exit on error from tests
2017-11-27 21:21 [U-Boot] [PATCH] test/run: Exit on error from tests Tom Rini
@ 2017-11-27 21:45 ` Simon Glass
2017-11-28 1:11 ` Tom Rini
0 siblings, 1 reply; 5+ messages in thread
From: Simon Glass @ 2017-11-27 21:45 UTC (permalink / raw)
To: u-boot
Hi Tom,
On 27 November 2017 at 14:21, Tom Rini <trini@konsulko.com> wrote:
> Currently, result isn't saved outside of the scope of run_test, so we
> always exit on error here. An easier way to deal with having test/run
> show failures is to exit on failure.
>
> Cc: Simon Glass <sjg@chromium.org>
> Fixes: 2f52018c3cbb ("test/run: Report and return failure")
> Signed-off-by: Tom Rini <trini@konsulko.com>
> ---
> Automation is only as good as the humans behind it. If you forget, like
> I did, that you finally fixed Jenkins to email you every time, that lack
> of emails about my non-travis jobs failing wasn't "no problems", it was
> "You forgot to migrate all the Docker stuff to the new build server".
> So, I'm going to apply this ASAP as I would not have let in the
> u-boot-dm PR, had my setup been functioning as expected.
> ---
> test/run | 18 ++++--------------
> 1 file changed, 4 insertions(+), 14 deletions(-)
I'm OK with this if it's what you want. This means that we will not
run all the tests if one of the earlier one fails.
Did you see this one?
http://patchwork.ozlabs.org/patch/841453/
Regards,
Simon
^ permalink raw reply [flat|nested] 5+ messages in thread* [U-Boot] [PATCH] test/run: Exit on error from tests
2017-11-27 21:45 ` Simon Glass
@ 2017-11-28 1:11 ` Tom Rini
2017-11-28 4:32 ` Simon Glass
0 siblings, 1 reply; 5+ messages in thread
From: Tom Rini @ 2017-11-28 1:11 UTC (permalink / raw)
To: u-boot
On Mon, Nov 27, 2017 at 02:45:14PM -0700, Simon Glass wrote:
> Hi Tom,
>
> On 27 November 2017 at 14:21, Tom Rini <trini@konsulko.com> wrote:
> > Currently, result isn't saved outside of the scope of run_test, so we
> > always exit on error here. An easier way to deal with having test/run
> > show failures is to exit on failure.
> >
> > Cc: Simon Glass <sjg@chromium.org>
> > Fixes: 2f52018c3cbb ("test/run: Report and return failure")
> > Signed-off-by: Tom Rini <trini@konsulko.com>
> > ---
> > Automation is only as good as the humans behind it. If you forget, like
> > I did, that you finally fixed Jenkins to email you every time, that lack
> > of emails about my non-travis jobs failing wasn't "no problems", it was
> > "You forgot to migrate all the Docker stuff to the new build server".
> > So, I'm going to apply this ASAP as I would not have let in the
> > u-boot-dm PR, had my setup been functioning as expected.
> > ---
> > test/run | 18 ++++--------------
> > 1 file changed, 4 insertions(+), 14 deletions(-)
>
> I'm OK with this if it's what you want. This means that we will not
> run all the tests if one of the earlier one fails.
>
> Did you see this one?
>
> http://patchwork.ozlabs.org/patch/841453/
I'm playing a little catch-up and that's fine too. I've unearthed
another bit of breakage to fix now (because the time PRs break my setup
is the time I break runtime of my setup and assume it's working).
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20171127/dd53b42d/attachment.sig>
^ permalink raw reply [flat|nested] 5+ messages in thread* [U-Boot] [PATCH] test/run: Exit on error from tests
2017-11-28 1:11 ` Tom Rini
@ 2017-11-28 4:32 ` Simon Glass
2017-11-28 14:33 ` Tom Rini
0 siblings, 1 reply; 5+ messages in thread
From: Simon Glass @ 2017-11-28 4:32 UTC (permalink / raw)
To: u-boot
Hi Tom,
On 27 November 2017 at 18:11, Tom Rini <trini@konsulko.com> wrote:
> On Mon, Nov 27, 2017 at 02:45:14PM -0700, Simon Glass wrote:
>> Hi Tom,
>>
>> On 27 November 2017 at 14:21, Tom Rini <trini@konsulko.com> wrote:
>> > Currently, result isn't saved outside of the scope of run_test, so we
>> > always exit on error here. An easier way to deal with having test/run
>> > show failures is to exit on failure.
>> >
>> > Cc: Simon Glass <sjg@chromium.org>
>> > Fixes: 2f52018c3cbb ("test/run: Report and return failure")
>> > Signed-off-by: Tom Rini <trini@konsulko.com>
>> > ---
>> > Automation is only as good as the humans behind it. If you forget, like
>> > I did, that you finally fixed Jenkins to email you every time, that lack
>> > of emails about my non-travis jobs failing wasn't "no problems", it was
>> > "You forgot to migrate all the Docker stuff to the new build server".
>> > So, I'm going to apply this ASAP as I would not have let in the
>> > u-boot-dm PR, had my setup been functioning as expected.
>> > ---
>> > test/run | 18 ++++--------------
>> > 1 file changed, 4 insertions(+), 14 deletions(-)
>>
>> I'm OK with this if it's what you want. This means that we will not
>> run all the tests if one of the earlier one fails.
>>
>> Did you see this one?
>>
>> http://patchwork.ozlabs.org/patch/841453/
>
> I'm playing a little catch-up and that's fine too. I've unearthed
> another bit of breakage to fix now (because the time PRs break my setup
> is the time I break runtime of my setup and assume it's working).
Yes, OK, sorry about the churn. As you can tell I'm trying to get it
so all the tests we have run automatically both with 'make tests' and
with travis-ci. Still a way to go but it is closer.
Regards,
Simon
^ permalink raw reply [flat|nested] 5+ messages in thread* [U-Boot] [PATCH] test/run: Exit on error from tests
2017-11-28 4:32 ` Simon Glass
@ 2017-11-28 14:33 ` Tom Rini
0 siblings, 0 replies; 5+ messages in thread
From: Tom Rini @ 2017-11-28 14:33 UTC (permalink / raw)
To: u-boot
On Mon, Nov 27, 2017 at 09:32:06PM -0700, Simon Glass wrote:
> Hi Tom,
>
> On 27 November 2017 at 18:11, Tom Rini <trini@konsulko.com> wrote:
> > On Mon, Nov 27, 2017 at 02:45:14PM -0700, Simon Glass wrote:
> >> Hi Tom,
> >>
> >> On 27 November 2017 at 14:21, Tom Rini <trini@konsulko.com> wrote:
> >> > Currently, result isn't saved outside of the scope of run_test, so we
> >> > always exit on error here. An easier way to deal with having test/run
> >> > show failures is to exit on failure.
> >> >
> >> > Cc: Simon Glass <sjg@chromium.org>
> >> > Fixes: 2f52018c3cbb ("test/run: Report and return failure")
> >> > Signed-off-by: Tom Rini <trini@konsulko.com>
> >> > ---
> >> > Automation is only as good as the humans behind it. If you forget, like
> >> > I did, that you finally fixed Jenkins to email you every time, that lack
> >> > of emails about my non-travis jobs failing wasn't "no problems", it was
> >> > "You forgot to migrate all the Docker stuff to the new build server".
> >> > So, I'm going to apply this ASAP as I would not have let in the
> >> > u-boot-dm PR, had my setup been functioning as expected.
> >> > ---
> >> > test/run | 18 ++++--------------
> >> > 1 file changed, 4 insertions(+), 14 deletions(-)
> >>
> >> I'm OK with this if it's what you want. This means that we will not
> >> run all the tests if one of the earlier one fails.
> >>
> >> Did you see this one?
> >>
> >> http://patchwork.ozlabs.org/patch/841453/
> >
> > I'm playing a little catch-up and that's fine too. I've unearthed
> > another bit of breakage to fix now (because the time PRs break my setup
> > is the time I break runtime of my setup and assume it's working).
>
> Yes, OK, sorry about the churn. As you can tell I'm trying to get it
> so all the tests we have run automatically both with 'make tests' and
> with travis-ci. Still a way to go but it is closer.
I'm looking forward to it all being in there :) Thanks!
--
Tom
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://lists.denx.de/pipermail/u-boot/attachments/20171128/cc3e3ed9/attachment.sig>
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2017-11-28 14:33 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2017-11-27 21:21 [U-Boot] [PATCH] test/run: Exit on error from tests Tom Rini
2017-11-27 21:45 ` Simon Glass
2017-11-28 1:11 ` Tom Rini
2017-11-28 4:32 ` Simon Glass
2017-11-28 14:33 ` Tom Rini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox