Re: [OE-core] [PATCH 1/2] qemurunner: Impove stdout logging handling

["Alex Bennée" <alex.bennee@linaro.org>]

"Richard Purdie" <richard.purdie@linuxfoundation.org> writes:

> On Mon, 2023-12-18 at 10:07 -0800, Khem Raj wrote:
>> On Mon, Dec 18, 2023 at 9:58 AM Richard Purdie
>> <richard.purdie@linuxfoundation.org> wrote:
>> > 
>> > On Mon, 2023-12-18 at 09:45 -0800, Khem Raj wrote:
>> > > I tried the two patches in this series. It did improve the situation
>> > > but I am still getting SSH timeouts. But this time its 13 tests
>> > > earlier it used to be 40+
>> > > btw. my images are using systemd. So it might be good to see if we see
>> > > this with poky-altcfg as well or not.
>> > 
>> > Do you have the log.do_testimage and the ${WORKDIR}/testimage/ files?
>> 
>> yes, further I ran the failing tests in loop one after another still
>> one test gzip fails with ssh timeouts
>> 
>> https://busybox.net/~kraj/log.do_testimage.503
>> https://busybox.net/~kraj/testimage/
>> 
>> there are two runs in the testimages folder. In one you see the RCU
>> stall and in second you do not
>> but it fails with same ssh timeout issue.
>> 
>> > 
>> > Did you still see rcu stalls in the logs?
>
> What is interesting is there is ~3MB of nulls in the .2 serial log. The
> rcu stall is also:
>
> [   88.261687]  serial8250_tx_chars+0xea/0x2b0
> [   88.261689]  serial8250_handle_irq+0x1e9/0x330
> [   88.261691]  serial8250_default_handle_irq+0x4a/0x90
> [   88.261693]  serial8250_interrupt+0x66/0xc0
> [   88.261696]  __handle_irq_event_percpu+0x54/0x1c0
> [   88.261701]  handle_irq_event+0x3d/0x80
>
> i.e. it is stalled in the serial TX path.
>
> The big question is why is there so many nulls on the serial port. I
> see a few on my local x86 test runs but only ~4kb, not megabytes of
> them. I hadn't worked out where/what they are from yet.
>
> I suspect something in the serial/kernel/qemu space isn't interacting
> correctly.

We generally don't model FIFOs in QEMU and also we will block the main
thread if we can't write data out.

>
> Find the cause of the nulls and we might make progress on this.
>
> Cheers,
>
> Richard
>
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#192681): https://lists.openembedded.org/g/openembedded-core/message/192681
> Mute This Topic: https://lists.openembedded.org/mt/103244414/5085169
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.bennee@linaro.org]
> -=-=-=-=-=-=-=-=-=-=-=-

-- 
Alex Bennée
Virtualisation Tech Lead @ Linaro