serial hang in qemu-system-ppc64 -M pseries

serial hang in qemu-system-ppc64 -M pseries

[Rob Landley]

[-- Attachment #1: Type: text/plain, Size: 752 bytes --]

When I cut and paste 80-ish characters of text into the Linux serial console, it
reads 16 characters and stops. When I hit space, it reads another 16 characters,
and if I keep at it will eventually catch up without losing data. If I type,
every character shows up immediately.

(On other qemu targets and kernels I can cut and paste an entire uuencoded
binary and it goes through just fine in one go, but this target hangs with big
pastes until I hit keys.)

Is this a qemu-side bug, or a kernel-side bug?

Kernel config attached (linux 5.18-rc3 or thereabouts), qemu invocation is:

qemu-system-ppc64 -M pseries -vga none -nographic -no-reboot -m 256 -kernel
vmlinux -initrd powerpc64leroot.cpio.gz -append "panic=1 HOST=powerpc64le
console=hvc0"

Rob

[-- Attachment #2: linux-fullconfig.gz --]
[-- Type: application/gzip, Size: 13060 bytes --]

Re: serial hang in qemu-system-ppc64 -M pseries

[Thomas Huth]

On 26/04/2022 12.26, Rob Landley wrote:
> When I cut and paste 80-ish characters of text into the Linux serial console, it
> reads 16 characters and stops. When I hit space, it reads another 16 characters,
> and if I keep at it will eventually catch up without losing data. If I type,
> every character shows up immediately.

That "16" certainly comes from VTERM_BUFSIZE in hw/char/spapr_vty.c in the 
QEMU sources, I think.

> (On other qemu targets and kernels I can cut and paste an entire uuencoded
> binary and it goes through just fine in one go, but this target hangs with big
> pastes until I hit keys.)
> 
> Is this a qemu-side bug, or a kernel-side bug?
> 
> Kernel config attached (linux 5.18-rc3 or thereabouts), qemu invocation is:
> 
> qemu-system-ppc64 -M pseries -vga none -nographic -no-reboot -m 256 -kernel
> vmlinux -initrd powerpc64leroot.cpio.gz -append "panic=1 HOST=powerpc64le
> console=hvc0"

Which version of QEMU are you using? Which frontend (GTK or terminal?) ... 
this rings a distant bell, but I thought we had fixed these issues long ago 
in the past... e.g.:

https://yhbt.net/lore/all/1380113886-16845-10-git-send-email-mdroth@linux.vnet.ibm.com/

https://git.qemu.org/?p=qemu.git;a=commitdiff;h=8a273cbe53221d28

... but maybe my memory also just fails and this has never been properly fixed.

  Thomas

Re: serial hang in qemu-system-ppc64 -M pseries

[Thomas Huth]

On 27/04/2022 17.27, Thomas Huth wrote:
> On 26/04/2022 12.26, Rob Landley wrote:
>> When I cut and paste 80-ish characters of text into the Linux serial 
>> console, it
>> reads 16 characters and stops. When I hit space, it reads another 16 
>> characters,
>> and if I keep at it will eventually catch up without losing data. If I type,
>> every character shows up immediately.
> 
> That "16" certainly comes from VTERM_BUFSIZE in hw/char/spapr_vty.c in the 
> QEMU sources, I think.
> 
>> (On other qemu targets and kernels I can cut and paste an entire uuencoded
>> binary and it goes through just fine in one go, but this target hangs with 
>> big
>> pastes until I hit keys.)
>>
>> Is this a qemu-side bug, or a kernel-side bug?
>>
>> Kernel config attached (linux 5.18-rc3 or thereabouts), qemu invocation is:
>>
>> qemu-system-ppc64 -M pseries -vga none -nographic -no-reboot -m 256 -kernel
>> vmlinux -initrd powerpc64leroot.cpio.gz -append "panic=1 HOST=powerpc64le
>> console=hvc0"
> 
> Which version of QEMU are you using? Which frontend (GTK or terminal?) ... 
> this rings a distant bell, but I thought we had fixed these issues long ago 
> in the past... e.g.:
> 
> https://yhbt.net/lore/all/1380113886-16845-10-git-send-email-mdroth@linux.vnet.ibm.com/ 
> 
> https://git.qemu.org/?p=qemu.git;a=commitdiff;h=8a273cbe53221d28
> 
> ... but maybe my memory also just fails and this has never been properly fixed.

In case you're using GTK, I think I just found the problem that I was 
remembering again:

https://lists.gnu.org/archive/html/qemu-ppc/2016-11/msg00142.html
https://lists.gnu.org/archive/html/qemu-ppc/2016-11/msg00143.html

I assume this has never been properly fixed.

  Thomas

Re: serial hang in qemu-system-ppc64 -M pseries

[Rob Landley]

On 4/27/22 10:27, Thomas Huth wrote:
> On 26/04/2022 12.26, Rob Landley wrote:
>> When I cut and paste 80-ish characters of text into the Linux serial console, it
>> reads 16 characters and stops. When I hit space, it reads another 16 characters,
>> and if I keep at it will eventually catch up without losing data. If I type,
>> every character shows up immediately.
> 
> That "16" certainly comes from VTERM_BUFSIZE in hw/char/spapr_vty.c in the 
> QEMU sources, I think.
> 
>> (On other qemu targets and kernels I can cut and paste an entire uuencoded
>> binary and it goes through just fine in one go, but this target hangs with big
>> pastes until I hit keys.)
>> 
>> Is this a qemu-side bug, or a kernel-side bug?
>> 
>> Kernel config attached (linux 5.18-rc3 or thereabouts), qemu invocation is:
>> 
>> qemu-system-ppc64 -M pseries -vga none -nographic -no-reboot -m 256 -kernel
>> vmlinux -initrd powerpc64leroot.cpio.gz -append "panic=1 HOST=powerpc64le
>> console=hvc0"
> 
> Which version of QEMU are you using?

$ qemu-system-ppc64 --version
QEMU emulator version 6.2.92 (v6.2.0-rc2)
Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

From november. I can pull and rebuild but it'll take a bit. (Hopefully
rebuilding would fix the need to echo -e '\e[?7h' afterwards to undo the "bash
command line history marches up the screen because qemu's x86 bios disabled line
wrap and then left it that way" issue...)

> Which frontend (GTK or terminal?) ... 

The above command line has -nographic, forcing terminal. Running ldd on the
binary doesn't pull up anything gtk. (It pulls up libncursesw though.)

If you want to reproduce my test locally:

wget https://landley.net/toybox/downloads/binaries/mkroot/0.8.5/powerpc64le.tgz
tar xvzf powerpc64le.tgz
cd powerpc64le
./qemu-*.sh

Then paste something longer than 16 characters at the eventual command prompt
once the kernel finishes booting.

If you want to reproduce it all from source:

git clone https://github.com/landley/toybox
cd toybox && mkdir ccc && cd ccc
wget
https://landley.net/toybox/downloads/binaries/toolchains/latest/powerpc64le-linux-musl-cross.tar.xz
-O - | tar xv
cd ..
CROSS=powerpc64le LINUX=~/linux scripts/mkroot.sh
cd root/powerpc64le
./qemu-*.sh

This assumes your linux kernel source directory is in ~/linux of course, and
that qemu-system-ppc64 is in the $PATH...

> this rings a distant bell, but I thought we had fixed these issues long ago 
> in the past... e.g.:
>
> https://yhbt.net/lore/all/1380113886-16845-10-git-send-email-mdroth@linux.vnet.ibm.com/
> 
> https://git.qemu.org/?p=qemu.git;a=commitdiff;h=8a273cbe53221d28

The qemu I'm running is newer than 2016. :)

Most targets are fine with this: I cut and paste entire uuencoded binaries into
the serial console as an easy way to insert a file into an initramfs. It can
usually take multiple megabytes without dropping a character, so you just
"uudecode" enter, and then paste.

Even my 32 bit powerpc target works fine with this. (Although -M g3beige is a
very different machine from -M pseries...)

Alas this target (and sh4 -m r2d) stop at 16 chars. (On sh4 the extra is
discarded, not delivered progressively as more interrupts are generated.)

> ... but maybe my memory also just fails and this has never been properly fixed.
> 
>   Thomas

Rob

Re: serial hang in qemu-system-ppc64 -M pseries

[Rob Landley]

On 4/28/22 00:41, Rob Landley wrote:
> On 4/27/22 10:27, Thomas Huth wrote:
>> On 26/04/2022 12.26, Rob Landley wrote:
>>> When I cut and paste 80-ish characters of text into the Linux serial console, it
>>> reads 16 characters and stops. When I hit space, it reads another 16 characters,
>>> and if I keep at it will eventually catch up without losing data. If I type,
>>> every character shows up immediately.
>> 
>> That "16" certainly comes from VTERM_BUFSIZE in hw/char/spapr_vty.c in the 
>> QEMU sources, I think.
>> 
>>> (On other qemu targets and kernels I can cut and paste an entire uuencoded
>>> binary and it goes through just fine in one go, but this target hangs with big
>>> pastes until I hit keys.)
>>> 
>>> Is this a qemu-side bug, or a kernel-side bug?
>>> 
>>> Kernel config attached (linux 5.18-rc3 or thereabouts), qemu invocation is:
>>> 
>>> qemu-system-ppc64 -M pseries -vga none -nographic -no-reboot -m 256 -kernel
>>> vmlinux -initrd powerpc64leroot.cpio.gz -append "panic=1 HOST=powerpc64le
>>> console=hvc0"
>> 
>> Which version of QEMU are you using?
> 
> $ qemu-system-ppc64 --version
> QEMU emulator version 6.2.92 (v6.2.0-rc2)
> Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers

Just confirmed it behaves the same with current git (commit cf6f26d6f9b2).

Rob

Re: serial hang in qemu-system-ppc64 -M pseries

[Fabiano Rosas]

Rob Landley <rob@landley.net> writes:

> On 4/27/22 10:27, Thomas Huth wrote:
>> On 26/04/2022 12.26, Rob Landley wrote:
>>> When I cut and paste 80-ish characters of text into the Linux serial console, it
>>> reads 16 characters and stops. When I hit space, it reads another 16 characters,
>>> and if I keep at it will eventually catch up without losing data. If I type,
>>> every character shows up immediately.
>> 
>> That "16" certainly comes from VTERM_BUFSIZE in hw/char/spapr_vty.c in the 
>> QEMU sources, I think.
>> 
>>> (On other qemu targets and kernels I can cut and paste an entire uuencoded
>>> binary and it goes through just fine in one go, but this target hangs with big
>>> pastes until I hit keys.)
>>> 
>>> Is this a qemu-side bug, or a kernel-side bug?
>>> 
>>> Kernel config attached (linux 5.18-rc3 or thereabouts), qemu invocation is:
>>> 
>>> qemu-system-ppc64 -M pseries -vga none -nographic -no-reboot -m 256 -kernel
>>> vmlinux -initrd powerpc64leroot.cpio.gz -append "panic=1 HOST=powerpc64le
>>> console=hvc0"
>> 
>> Which version of QEMU are you using?
>
> $ qemu-system-ppc64 --version
> QEMU emulator version 6.2.92 (v6.2.0-rc2)
> Copyright (c) 2003-2021 Fabrice Bellard and the QEMU Project developers
>
> From november. I can pull and rebuild but it'll take a bit. (Hopefully
> rebuilding would fix the need to echo -e '\e[?7h' afterwards to undo the "bash
> command line history marches up the screen because qemu's x86 bios disabled line
> wrap and then left it that way" issue...)
>
>> Which frontend (GTK or terminal?) ... 
>
> The above command line has -nographic, forcing terminal. Running ldd on the
> binary doesn't pull up anything gtk. (It pulls up libncursesw though.)
>
> If you want to reproduce my test locally:
>
> wget https://landley.net/toybox/downloads/binaries/mkroot/0.8.5/powerpc64le.tgz
> tar xvzf powerpc64le.tgz
> cd powerpc64le
> ./qemu-*.sh
>
> Then paste something longer than 16 characters at the eventual command prompt
> once the kernel finishes booting.

I suspect this is due to how the tty driver (n_tty.c) interacts with
the console (hvc_console.c) driver's buffer size.

This is the stack:

#0 hvc_push          <-- calls into KVM/QEMU to write up to 16 bytes
#1 hvc_write
#2 tty_put_char
#3 do_output_char
#4 __process_echoes          <-- writes up to tty_write_room() chars
#5 flush_echoes              <-- returns early if ~ECHO && ~ECHONL
#6 n_tty_receive_buf_common  <-- buffers more than 16 bytes
#7 tty_ldisc_receive_buf
#8 tty_port_default_receive_buf
#9 receive_buf
#10 process_one_work

In my busybox instance which does not have this issue I can see that
termios.c_lflag = 0x447, so in the stack above #4 is not called (ECHO
is 0x8, ECHONL is 0x10).

In the bug scenario: termios.c_lflag = 0x5cf, so we go into #4 which
is supposed to write (say) 17 bytes, but ends up writing only 16
because that is what tty_write_room() returns.

What I think is causing this issue is that the hvc_vio.c driver is
configured with hp->outbuf_size = 16. That number comes from the
H_PUT_TERM_CHAR hypercall spec which reads two registers at a
time. However, the hvc_write function supports writes larger than 16
bytes so it seems we're needlessly propagating the 16 byte limitation
to the upper layer.

The driver is also not buffering the write, so if it gets called to
write one char at a time (like __process_echoes does) there should be no
limit to how much it can write.

I think if we increase hp->outbuf_size to a value that is larger than
N_TTY_BUF_SIZE=4096 the echo buffer would drain before reaching this new
hvc driver limit.

I tested that and it seems to work, but I'm not sure if it's the right
fix, there are some things I couldn't figure out:

   i) if a driver actually runs out of buffer space, what
      __process_echoes should do about it;

  ii) why the bug sometimes happens only at the 32 characters boundary
      (or other multiple of 16);

 iii) why the ECHO flag differs from the working case.

> If you want to reproduce it all from source:
>
> git clone https://github.com/landley/toybox
> cd toybox && mkdir ccc && cd ccc
> wget
> https://landley.net/toybox/downloads/binaries/toolchains/latest/powerpc64le-linux-musl-cross.tar.xz
> -O - | tar xv
> cd ..
> CROSS=powerpc64le LINUX=~/linux scripts/mkroot.sh
> cd root/powerpc64le
> ./qemu-*.sh
>
> This assumes your linux kernel source directory is in ~/linux of course, and
> that qemu-system-ppc64 is in the $PATH...
>
>> this rings a distant bell, but I thought we had fixed these issues long ago 
>> in the past... e.g.:
>>
>> https://yhbt.net/lore/all/1380113886-16845-10-git-send-email-mdroth@linux.vnet.ibm.com/
>> 
>> https://git.qemu.org/?p=qemu.git;a=commitdiff;h=8a273cbe53221d28
>
> The qemu I'm running is newer than 2016. :)
>
> Most targets are fine with this: I cut and paste entire uuencoded binaries into
> the serial console as an easy way to insert a file into an initramfs. It can
> usually take multiple megabytes without dropping a character, so you just
> "uudecode" enter, and then paste.
>
> Even my 32 bit powerpc target works fine with this. (Although -M g3beige is a

> very different machine from -M pseries...)
>
> Alas this target (and sh4 -m r2d) stop at 16 chars. (On sh4 the extra is
> discarded, not delivered progressively as more interrupts are generated.)
>
>> ... but maybe my memory also just fails and this has never been properly fixed.
>> 
>>   Thomas
>
> Rob

Re: serial hang in qemu-system-ppc64 -M pseries

[Rob Landley]

On 4/29/22 16:43, Fabiano Rosas wrote:
> Rob Landley <rob@landley.net> writes:
>> Then paste something longer than 16 characters at the eventual command prompt
>> once the kernel finishes booting.
> 
> I suspect this is due to how the tty driver (n_tty.c) interacts with
> the console (hvc_console.c) driver's buffer size.
> 
> This is the stack:
> 
> #0 hvc_push          <-- calls into KVM/QEMU to write up to 16 bytes
> #1 hvc_write
> #2 tty_put_char
> #3 do_output_char
> #4 __process_echoes          <-- writes up to tty_write_room() chars
> #5 flush_echoes              <-- returns early if ~ECHO && ~ECHONL
> #6 n_tty_receive_buf_common  <-- buffers more than 16 bytes
> #7 tty_ldisc_receive_buf
> #8 tty_port_default_receive_buf
> #9 receive_buf
> #10 process_one_work
> 
> In my busybox instance which does not have this issue I can see that
> termios.c_lflag = 0x447, so in the stack above #4 is not called (ECHO
> is 0x8, ECHONL is 0x10).
> 
> In the bug scenario: termios.c_lflag = 0x5cf, so we go into #4 which
> is supposed to write (say) 17 bytes, but ends up writing only 16
> because that is what tty_write_room() returns.

I think my init script left the terminal wherever the hardware defaults
initialized it to?

(I note that sh4 also has a variant of this problem, but instead of the
stutter-and-flush behavior it just hard discards everything after the first 16
characters of a paste. Large pastes seemsto work without issue on all the other
targets I've tried so far, I.E. x86, arm, mips, powerpc -M g3beige, s390x, and
m68k. And by "large" I mean I've fed half a megabyte of uuencode output into
uudecode as a single paste.)

> What I think is causing this issue is that the hvc_vio.c driver is
> configured with hp->outbuf_size = 16. That number comes from the
> H_PUT_TERM_CHAR hypercall spec which reads two registers at a
> time. However, the hvc_write function supports writes larger than 16
> bytes so it seems we're needlessly propagating the 16 byte limitation
> to the upper layer.

Looks like the call is:

  hp = hvc_alloc(termno, vdev->irq, ops, MAX_VIO_PUT_CHARS);

MAX_VIO_PUT_CHARS implies it's the maximum allowed write size. Understandable if
writes bigger than that didn't get a lot of testing. (There's an identical call
in hvc_opal.c, by the way.)

> The driver is also not buffering the write, so if it gets called to
> write one char at a time (like __process_echoes does) there should be no
> limit to how much it can write.
> 
> I think if we increase hp->outbuf_size to a value that is larger than
> N_TTY_BUF_SIZE=4096 the echo buffer would drain before reaching this new
> hvc driver limit.

How is this handled on any of the architectures where it works? (Or do their tty
flags just start at different defaults so I don't see it there?)

> I tested that and it seems to work, but I'm not sure if it's the right
> fix, there are some things I couldn't figure out:

Let me know if I can help, although this sounds like it's moved a bit beyond
areas I'm familiar with.

Thanks,

Rob