[Buildroot] [PATCH 1/1] support/testing: improve mosquitto test reliability

[Buildroot] [PATCH 1/1] support/testing: improve mosquitto test reliability

[Julien Olivain]

The mosquitto runtime test can randomly fail on slow
runners, see [1].

This commit improves this test in the following ways:

- the mosquitto_sub subscriber process is now started in a subshell
to suppress the job control messages (to prevent any spurious
messages when the job stops),

- the standard error is redirected to /dev/null, to prevent the
printing of any messages,

- the mosquitto_pub publisher process is started later, by increasing
the sleep time,

- finally, a new sleep time is introduced between the mosquitto_pub
publisher process and the check of the mosquitto_sub subscriber, to
make sure it will have time to write its output and exit.

Fixes: [1]

[1] https://gitlab.com/buildroot.org/buildroot/-/jobs/8453386454

Signed-off-by: Julien Olivain <ju.o@free.fr>
---
 support/testing/tests/package/test_mosquitto.py | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/support/testing/tests/package/test_mosquitto.py b/support/testing/tests/package/test_mosquitto.py
index ba10b30b00..6b256e5e27 100644
--- a/support/testing/tests/package/test_mosquitto.py
+++ b/support/testing/tests/package/test_mosquitto.py
@@ -25,13 +25,19 @@ class TestMosquitto(infra.basetest.BRTest):
         msg = "Hello Buildroot!"
 
         # We subscribe to a topic and write one message to a log file.
-        self.assertRunOk(f"mosquitto_sub -t {topic} -C 1 > {log} &")
+        cmd = f"( mosquitto_sub -t {topic} -C 1 > {log} 2> /dev/null & )"
+        self.assertRunOk(cmd)
 
-        time.sleep(1)
+        # We give some time to the subscriber process to settle.
+        time.sleep(5)
 
         # We publish a message.
         self.assertRunOk(f"mosquitto_pub -t {topic} -m '{msg}'")
 
+        # We give some more time to the subscriber process to write
+        # the log and quit.
+        time.sleep(5)
+
         # We check the log file contains our message.
         out, ret = self.emulator.run(f"cat {log}")
         self.assertEqual(ret, 0)
-- 
2.47.1

_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] support/testing: improve mosquitto test reliability

[Thomas Petazzoni via buildroot]

Hello Julien,

On Tue, 26 Nov 2024 19:28:55 +0100
Julien Olivain <ju.o@free.fr> wrote:

> The mosquitto runtime test can randomly fail on slow
> runners, see [1].
> 
> This commit improves this test in the following ways:
> 
> - the mosquitto_sub subscriber process is now started in a subshell
> to suppress the job control messages (to prevent any spurious
> messages when the job stops),
> 
> - the standard error is redirected to /dev/null, to prevent the
> printing of any messages,
> 
> - the mosquitto_pub publisher process is started later, by increasing
> the sleep time,
> 
> - finally, a new sleep time is introduced between the mosquitto_pub
> publisher process and the check of the mosquitto_sub subscriber, to
> make sure it will have time to write its output and exit.
> 
> Fixes: [1]
> 
> [1] https://gitlab.com/buildroot.org/buildroot/-/jobs/8453386454
> 
> Signed-off-by: Julien Olivain <ju.o@free.fr>
> ---
>  support/testing/tests/package/test_mosquitto.py | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)

Thanks a lot, I've applied to master. However, I'm not a huge fan of
those hardcoded "sleep 5": it makes the test longer even on fast
machines, and it doesn't necessarily resolve the problem in all
situations. But I think you had yourself mentioned in some other thread
that it would be nicer to have some kind of retry loop instead?

Best regards,

Thomas
-- 
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] support/testing: improve mosquitto test reliability

[Julien Olivain]

Hi Thomas,

On 28/11/2024 21:18, Thomas Petazzoni wrote:
> Hello Julien,
> 
> On Tue, 26 Nov 2024 19:28:55 +0100
> Julien Olivain <ju.o@free.fr> wrote:
> 
>> The mosquitto runtime test can randomly fail on slow
>> runners, see [1].
>> 
>> This commit improves this test in the following ways:
>> 
>> - the mosquitto_sub subscriber process is now started in a subshell
>> to suppress the job control messages (to prevent any spurious
>> messages when the job stops),
>> 
>> - the standard error is redirected to /dev/null, to prevent the
>> printing of any messages,
>> 
>> - the mosquitto_pub publisher process is started later, by increasing
>> the sleep time,
>> 
>> - finally, a new sleep time is introduced between the mosquitto_pub
>> publisher process and the check of the mosquitto_sub subscriber, to
>> make sure it will have time to write its output and exit.
>> 
>> Fixes: [1]
>> 
>> [1] https://gitlab.com/buildroot.org/buildroot/-/jobs/8453386454
>> 
>> Signed-off-by: Julien Olivain <ju.o@free.fr>
>> ---
>>  support/testing/tests/package/test_mosquitto.py | 10 ++++++++--
>>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> Thanks a lot, I've applied to master. However, I'm not a huge fan of
> those hardcoded "sleep 5": it makes the test longer even on fast
> machines, and it doesn't necessarily resolve the problem in all
> situations. But I think you had yourself mentioned in some other thread
> that it would be nicer to have some kind of retry loop instead?

I am not a fan either...  This will help for the time being.  I have
few patches about a retry loop I need polish before proposing.
I also don't want to include retry logic everywhere like this one:
https://gitlab.com/buildroot.org/buildroot/-/blob/2024.11-rc2/support/testing/tests/package/test_mdadm.py#L125

I currently add new functions in support/testing/infra/basetest.py for
those situations.

Best regards,

Julien.
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] support/testing: improve mosquitto test reliability

[Thomas Petazzoni via buildroot]

Hello,

On Thu, 28 Nov 2024 21:35:49 +0100
Julien Olivain <ju.o@free.fr> wrote:

> > Thanks a lot, I've applied to master. However, I'm not a huge fan of
> > those hardcoded "sleep 5": it makes the test longer even on fast
> > machines, and it doesn't necessarily resolve the problem in all
> > situations. But I think you had yourself mentioned in some other thread
> > that it would be nicer to have some kind of retry loop instead?  
> 
> I am not a fan either...  This will help for the time being.

Absolutely, that's why I applied even though it isn't the "perfect"
solution.

> I have few patches about a retry loop I need polish before proposing.
> I also don't want to include retry logic everywhere like this one:
> https://gitlab.com/buildroot.org/buildroot/-/blob/2024.11-rc2/support/testing/tests/package/test_mdadm.py#L125
> 
> I currently add new functions in support/testing/infra/basetest.py for
> those situations.

Good idea!

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot

Re: [Buildroot] [PATCH 1/1] support/testing: improve mosquitto test reliability

[Peter Korsgaard]

>>>>> "Julien" == Julien Olivain <ju.o@free.fr> writes:

 > The mosquitto runtime test can randomly fail on slow
 > runners, see [1].

 > This commit improves this test in the following ways:

 > - the mosquitto_sub subscriber process is now started in a subshell
 > to suppress the job control messages (to prevent any spurious
 > messages when the job stops),

 > - the standard error is redirected to /dev/null, to prevent the
 > printing of any messages,

 > - the mosquitto_pub publisher process is started later, by increasing
 > the sleep time,

 > - finally, a new sleep time is introduced between the mosquitto_pub
 > publisher process and the check of the mosquitto_sub subscriber, to
 > make sure it will have time to write its output and exit.

 > Fixes: [1]

 > [1] https://gitlab.com/buildroot.org/buildroot/-/jobs/8453386454

 > Signed-off-by: Julien Olivain <ju.o@free.fr>

Committed to 2024.02.x and 2024.08.x, thanks.

-- 
Bye, Peter Korsgaard
_______________________________________________
buildroot mailing list
buildroot@buildroot.org
https://lists.buildroot.org/mailman/listinfo/buildroot