Slow performance of VUART on AST2500 with 5.15.5

Slow performance of VUART on AST2500 with 5.15.5

[Oskar Senft]

Hi everyone

I'm doing some more validation work with meta-tyan
(https://gerrit.openbmc-project.xyz/c/openbmc/openbmc/+/49181) on the
current 5.15.5 kernel. I noticed that when using ttyS0 from the host
(being the VUART in the AST2500), that the transfer host->BMC is
really slow (like 300 baud slow). This is true even after stopping
obmc-console-server and just doing `cat /dev/ttyVUART0`, so I figured
it must be a kernel problem. When I then tried kernel 5.2.11 (with the
DTS from 5.15.5 minus the uart_routing node), then VUART behaved
normally. After having done this comparison, I think that 5.15.5 is
generally just much slower.

Is anyone aware of AST2500 VUART (or something else that would affect
performance on an AST2500) having gotten broken somewhere between
5.2.11 and 5.15.5?

Thanks
Oskar.

Re: Slow performance of VUART on AST2500 with 5.15.5

[Zev Weiss]

On Wed, Dec 15, 2021 at 07:38:36PM PST, Oskar Senft wrote:
>Hi everyone
>
>I'm doing some more validation work with meta-tyan
>(https://gerrit.openbmc-project.xyz/c/openbmc/openbmc/*/49181 ) on the
>current 5.15.5 kernel. I noticed that when using ttyS0 from the host
>(being the VUART in the AST2500), that the transfer host->BMC is
>really slow (like 300 baud slow). This is true even after stopping
>obmc-console-server and just doing `cat /dev/ttyVUART0`, so I figured
>it must be a kernel problem. When I then tried kernel 5.2.11 (with the
>DTS from 5.15.5 minus the uart_routing node), then VUART behaved
>normally. After having done this comparison, I think that 5.15.5 is
>generally just much slower.
>
>Is anyone aware of AST2500 VUART (or something else that would affect
>performance on an AST2500) having gotten broken somewhere between
>5.2.11 and 5.15.5?
>
>Thanks
>Oskar.

Yes, this is a known issue.

Prior to commit 54da3e381c2 ("serial: 8250_aspeed_vuart: use UPF_IOREMAP
to set up register mapping"), a long-standing bug in the aspeed-vuart
driver had a side-effect of unintentionally leaving the VUART's FIFOs
disabled.  With that patch applied to fix the bug, the FIFOs get enabled
as they were originally intended to be, but that in turn seems to expose
another bug with the host-side THRE bit not getting set when it should,
so the host-side driver's write routine ends up waiting for a timeout to
expire on every character (when it would otherwise continue on to the
next character upon seeing THRE asserted).  I think this may be a VUART
hardware problem, though that hasn't been officially confirmed thus far.

I'm hoping we can land on a better solution, but as a temporary
band-aid, hacking the driver to treat the device as an 8250 instead of a
16550A (effectively just re-disabling the FIFOs) should speed things
back up:

diff --git a/drivers/tty/serial/8250/8250_aspeed_vuart.c b/drivers/tty/serial/8250/8250_aspeed_vuart.c
index 2350fb3bb5e4..c335f2b482bd 100644
--- a/drivers/tty/serial/8250/8250_aspeed_vuart.c
+++ b/drivers/tty/serial/8250/8250_aspeed_vuart.c
@@ -487,7 +487,7 @@ static int aspeed_vuart_probe(struct platform_device *pdev)
 	port.port.irq = irq_of_parse_and_map(np, 0);
 	port.port.handle_irq = aspeed_vuart_handle_irq;
 	port.port.iotype = UPIO_MEM;
-	port.port.type = PORT_16550A;
+	port.port.type = PORT_8250;
 	port.port.uartclk = clk;
 	port.port.flags = UPF_SHARE_IRQ | UPF_BOOT_AUTOCONF | UPF_IOREMAP
 		| UPF_FIXED_PORT | UPF_FIXED_TYPE | UPF_NO_THRE_TEST;

--

Zev

Re: Slow performance of VUART on AST2500 with 5.15.5

[Oskar Senft]

Hi Zev

> >Is anyone aware of AST2500 VUART (or something else that would affect
> >performance on an AST2500) having gotten broken somewhere between
> >5.2.11 and 5.15.5?
>
> Yes, this is a known issue.

And I felt that I was going crazy! Thanks for confirming that this is
a known issue.

> Prior to commit 54da3e381c2 ("serial: 8250_aspeed_vuart: use UPF_IOREMAP
> to set up register mapping"), a long-standing bug in the aspeed-vuart
> driver had a side-effect of unintentionally leaving the VUART's FIFOs
> disabled.  With that patch applied to fix the bug, the FIFOs get enabled
> as they were originally intended to be, but that in turn seems to expose
> another bug with the host-side THRE bit not getting set when it should,
> so the host-side driver's write routine ends up waiting for a timeout to
> expire on every character (when it would otherwise continue on to the
> next character upon seeing THRE asserted).  I think this may be a VUART
> hardware problem, though that hasn't been officially confirmed thus far.

Did you reach out to Aspeed yet? I've had some good experiences when
talking with them directly.


> I'm hoping we can land on a better solution, but as a temporary
> band-aid, hacking the driver to treat the device as an 8250 instead of a
> 16550A (effectively just re-disabling the FIFOs) should speed things
> back up:

I can confirm that this fixes things, thank you! Is it worth dropping
that into upstream for the time being, or do you think a "better
solution" is imminent?

I've been trying REAL hard to keep meta-tyan (and our other downstream
stuff) patch free, but I can't leave it broken since it affects boot
time (both BIOS and Linux synchronize on writes to the serial
console).

Thanks
Oskar.

Re: Slow performance of VUART on AST2500 with 5.15.5

[Zev Weiss]

On Wed, Dec 15, 2021 at 10:54:19PM CST, Oskar Senft wrote:
>Hi Zev
>
>> >Is anyone aware of AST2500 VUART (or something else that would affect
>> >performance on an AST2500) having gotten broken somewhere between
>> >5.2.11 and 5.15.5?
>>
>> Yes, this is a known issue.
>
>And I felt that I was going crazy! Thanks for confirming that this is
>a known issue.
>
>> Prior to commit 54da3e381c2 ("serial: 8250_aspeed_vuart: use UPF_IOREMAP
>> to set up register mapping"), a long-standing bug in the aspeed-vuart
>> driver had a side-effect of unintentionally leaving the VUART's FIFOs
>> disabled.  With that patch applied to fix the bug, the FIFOs get enabled
>> as they were originally intended to be, but that in turn seems to expose
>> another bug with the host-side THRE bit not getting set when it should,
>> so the host-side driver's write routine ends up waiting for a timeout to
>> expire on every character (when it would otherwise continue on to the
>> next character upon seeing THRE asserted).  I think this may be a VUART
>> hardware problem, though that hasn't been officially confirmed thus far.
>
>Did you reach out to Aspeed yet? I've had some good experiences when
>talking with them directly.
>

I've discussed it a little with Troy on Discord, though I haven't yet
heard any conclusive statements about the recommended course of action.

>
>> I'm hoping we can land on a better solution, but as a temporary
>> band-aid, hacking the driver to treat the device as an 8250 instead of a
>> 16550A (effectively just re-disabling the FIFOs) should speed things
>> back up:
>
>I can confirm that this fixes things, thank you! Is it worth dropping
>that into upstream for the time being, or do you think a "better
>solution" is imminent?
>

I don't really know what the timeline on a better workaround might be
(or even if one is likely to exist); hopefully someone from Aspeed might
be more likely to have some insight on that...

I wouldn't be opposed to dropping that hack into the mainline driver at
least for the time being though; seems like it shouldn't leave things
any worse off than they were before the accidental bugfix side-effect of
the UPF_IOREMAP patch, anyway.  Joel or Andrew, any particular opinions
here? 


Zev