Issues attempting to use Raspberry Pi 4 serial console on mainline

Issues attempting to use Raspberry Pi 4 serial console on mainline

[Nathan Chancellor]

Hi all,

Thank you for all of the hard work that has been done on supporting the
Raspberry Pi 4 upstream. It has been a great platform so far for testing
various clang technologies on actual hardware.

I am investigating a panic that occurs when running a guest under KVM
when clang's Shadow Call Stack is enabled and it would be handy to grab
the kernel panic via serial console as the device panics and I lose my
ssh connection. I picked up a USB to TTL cable and I am able to get
connected with the following config.txt options and cmdline.txt options:

$ head -n4 /boot/config.txt
enable_uart=1
kernel=Image
os_prefix=custom-mainline-arm64/
upstream_kernel=1

$ cat /boot/custom-mainline-arm64/cmdline.txt
console=ttyS1,115200 console=tty1 root=PARTUUID=45a8dd8a-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait plymouth.ignore-serial-consoles

However, when I connect to the serial console via PuTTY, I get nothing
but garbage output: https://imgur.com/a/ekFlLYq

As I understand it, that is due to the mini UART not being as good as a
regular PL011. On the downstream kernel, one would apply
'dtoverlay=disable-bt' which would free up the first PL011 to be used as
the primary UART but the device tree overlays do not work on a mainline
kernel. Is there an easy way to disable Bluetooth via the device tree?
If not, is there any recommended or documented way to use the mini UART
successfully? I have seen information around using 'core_freq=...' but
the documentation says that does not work for the Raspberry Pi 4.

Any guidance or documentation that you could point me to would be much
appreciated.

Cheers,
Nathan

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Issues attempting to use Raspberry Pi 4 serial console on mainline

[Nicolas Saenz Julienne]

Hi Nathan,

On Tue, 2020-07-21 at 15:57 -0700, Nathan Chancellor wrote:
> Hi all,
> 
> Thank you for all of the hard work that has been done on supporting
> the
> Raspberry Pi 4 upstream. It has been a great platform so far for
> testing
> various clang technologies on actual hardware.

I'm glad it's useful :)

> I am investigating a panic that occurs when running a guest under KVM
> when clang's Shadow Call Stack is enabled and it would be handy to
> grab
> the kernel panic via serial console as the device panics and I lose
> my
> ssh connection. I picked up a USB to TTL cable and I am able to get
> connected with the following config.txt options and cmdline.txt
> options:
> 
> $ head -n4 /boot/config.txt
> enable_uart=1
> kernel=Image
> os_prefix=custom-mainline-arm64/
>
> upstream_kernel=1

I'd say for rpi4 this property isn't necessary as you're creating a custom
os_prefix anyway. But it should be harmless.

That aside, the only thing I'm missing is:

arm_64bit=1

> $ cat /boot/custom-mainline-arm64/cmdline.txt
> console=ttyS1,115200 console=tty1 root=PARTUUID=45a8dd8a-02

Which device tree are you using? This is the right tty device if using the
upstream one, if using the one provided by the RPi foundation it should be
ttyS0.

> rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
> plymouth.ignore-serial-consoles

For reference I just booted linux-next with this setup:

boot partition:
	...Latest firmware files taken from the RPi firmware repo [1]...
	Image			#Copied from linux build
	bcm2711-rpi-4-b.dtb	#Copied from linux build
	config.txt
	cmdline.txt

config.txt:
	kernel=Image
	enable_uart=1
	arm_64bit=1

cmdline.txt:
	 console=tty console=ttyS1,115200 text root=/dev/nfs
	 nfsroot=10.42.0.1:/home/nico/netboot/root,vers=3 rw ip=dhcp
rootwait
	 elevator=deadline

> However, when I connect to the serial console via PuTTY, I get
> nothing
> but garbage output: https://imgur.com/a/ekFlLYq
>
> As I understand it, that is due to the mini UART not being as good as
> a
> regular PL011. On the downstream kernel, one would apply
> 'dtoverlay=disable-bt' which would free up the first PL011 to be used
> as
> the primary UART but the device tree overlays do not work on a
> mainline
> kernel. Is there an easy way to disable Bluetooth via the device
> tree?
> If not, is there any recommended or documented way to use the mini
> UART
> successfully? I have seen information around using 'core_freq=...'
> but
> the documentation says that does not work for the Raspberry Pi 4.

The issue with the mini UART is its clock, which is derived from VPU's, which
is itself controlled by RPi's firmware. Changes might happen behind the
kernel's back, and the mini UART divisors will not be updated accordingly.
This is an area the we could do better, but no one found a good solution yet.
That said, for now, when using the upstream kernel, VPU's clock should be
stable as we forbid the firmware from performing frequency scaling on that
clock.

In the future, once Maxime Rippards vc4/HDMI code is merged we will most
probably hit this issue as the core clock has to be upscaled when feeding big
screen resolutions. As you mention, a solution to this is fixing the core
frequency in config.txt, for example I tested this successfully:

core_freq=500
core_freq_min=500

Regards,
Nicolas

[1] https://github.com/raspberrypi/firmware


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Issues attempting to use Raspberry Pi 4 serial console on mainline

[Peter Robinson]

On Wed, Jul 22, 2020 at 2:27 PM Nicolas Saenz Julienne
<nsaenzjulienne@suse.de> wrote:
>
> Hi Nathan,
>
> On Tue, 2020-07-21 at 15:57 -0700, Nathan Chancellor wrote:
> > Hi all,
> >
> > Thank you for all of the hard work that has been done on supporting
> > the
> > Raspberry Pi 4 upstream. It has been a great platform so far for
> > testing
> > various clang technologies on actual hardware.
>
> I'm glad it's useful :)
>
> > I am investigating a panic that occurs when running a guest under KVM
> > when clang's Shadow Call Stack is enabled and it would be handy to
> > grab
> > the kernel panic via serial console as the device panics and I lose
> > my
> > ssh connection. I picked up a USB to TTL cable and I am able to get
> > connected with the following config.txt options and cmdline.txt
> > options:
> >
> > $ head -n4 /boot/config.txt
> > enable_uart=1
> > kernel=Image
> > os_prefix=custom-mainline-arm64/
> >
> > upstream_kernel=1
>
> I'd say for rpi4 this property isn't necessary as you're creating a custom
> os_prefix anyway. But it should be harmless.
>
> That aside, the only thing I'm missing is:
>
> arm_64bit=1
>
> > $ cat /boot/custom-mainline-arm64/cmdline.txt
> > console=ttyS1,115200 console=tty1 root=PARTUUID=45a8dd8a-02
>
> Which device tree are you using? This is the right tty device if using the
> upstream one, if using the one provided by the RPi foundation it should be
> ttyS0.
>
> > rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait
> > plymouth.ignore-serial-consoles
>
> For reference I just booted linux-next with this setup:
>
> boot partition:
>         ...Latest firmware files taken from the RPi firmware repo [1]...
>         Image                   #Copied from linux build
>         bcm2711-rpi-4-b.dtb     #Copied from linux build
>         config.txt
>         cmdline.txt
>
> config.txt:
>         kernel=Image
>         enable_uart=1
>         arm_64bit=1
>
> cmdline.txt:
>          console=tty console=ttyS1,115200 text root=/dev/nfs
>          nfsroot=10.42.0.1:/home/nico/netboot/root,vers=3 rw ip=dhcp
> rootwait
>          elevator=deadline
>
> > However, when I connect to the serial console via PuTTY, I get
> > nothing
> > but garbage output: https://imgur.com/a/ekFlLYq
> >
> > As I understand it, that is due to the mini UART not being as good as
> > a
> > regular PL011. On the downstream kernel, one would apply
> > 'dtoverlay=disable-bt' which would free up the first PL011 to be used
> > as
> > the primary UART but the device tree overlays do not work on a
> > mainline
> > kernel. Is there an easy way to disable Bluetooth via the device
> > tree?
> > If not, is there any recommended or documented way to use the mini
> > UART
> > successfully? I have seen information around using 'core_freq=...'
> > but
> > the documentation says that does not work for the Raspberry Pi 4.
>
> The issue with the mini UART is its clock, which is derived from VPU's, which
> is itself controlled by RPi's firmware. Changes might happen behind the
> kernel's back, and the mini UART divisors will not be updated accordingly.
> This is an area the we could do better, but no one found a good solution yet.
> That said, for now, when using the upstream kernel, VPU's clock should be
> stable as we forbid the firmware from performing frequency scaling on that
> clock.

There has actually been a regression in the firmware here, prior to
mid April if the enable_uart=1 I always had clean output on the serial
console, since that date on the rpi3/3b+/4 I get a whole bunch of
junk, revert to an older firmware (I've been using April 1st ) and it
all goes back to being fine. I've not had time to actually report it
yet, and it's still a problem with the latest firmware but a quick
look it likes similar to this issue reported in late April:

https://github.com/raspberrypi/firmware/issues/1376

Peter

> In the future, once Maxime Rippards vc4/HDMI code is merged we will most
> probably hit this issue as the core clock has to be upscaled when feeding big
> screen resolutions. As you mention, a solution to this is fixing the core
> frequency in config.txt, for example I tested this successfully:
>
> core_freq=500
> core_freq_min=500
>
> Regards,
> Nicolas
>
> [1] https://github.com/raspberrypi/firmware
>
>
> _______________________________________________
> linux-rpi-kernel mailing list
> linux-rpi-kernel@lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-rpi-kernel

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Issues attempting to use Raspberry Pi 4 serial console on mainline

[Nicolas Saenz Julienne]

Hi Nathan, Peter,

On Wed, 2020-07-22 at 15:41 +0100, Peter Robinson wrote:

[...]

> On Wed, Jul 22, 2020 at 2:27 PM Nicolas Saenz Julienne
> > For reference I just booted linux-next with this setup:
> > 
> > boot partition:
> >         ...Latest firmware files taken from the RPi firmware repo [1]...

Natan, it seems I lied here and been using an outdated firmware version. I've
just validated what Peter comments below, and the last working firmware release
with this setup is Feb12[1]. Given your use case I'd just use that and forget
about it.

> >         Image                   #Copied from linux build
> >         bcm2711-rpi-4-b.dtb     #Copied from linux build
> >         config.txt
> >         cmdline.txt
> > 
> > config.txt:
> >         kernel=Image
> >         enable_uart=1
> >         arm_64bit=1
> > 
> > cmdline.txt:
> >          console=tty console=ttyS1,115200 text root=/dev/nfs
> >          nfsroot=10.42.0.1:/home/nico/netboot/root,vers=3 rw ip=dhcp
> > rootwait
> >          elevator=deadline
> > 

[...]

> > 
> > The issue with the mini UART is its clock, which is derived from VPU's, which
> > is itself controlled by RPi's firmware. Changes might happen behind the
> > kernel's back, and the mini UART divisors will not be updated accordingly.
> > This is an area the we could do better, but no one found a good solution yet.
> > That said, for now, when using the upstream kernel, VPU's clock should be
> > stable as we forbid the firmware from performing frequency scaling on that
> > clock.
> 
> There has actually been a regression in the firmware here, prior to
> mid April if the enable_uart=1 I always had clean output on the serial
> console, since that date on the rpi3/3b+/4 I get a whole bunch of
> junk, revert to an older firmware (I've been using April 1st ) and it
> all goes back to being fine. I've not had time to actually report it
> yet, and it's still a problem with the latest firmware but a quick
> look it likes similar to this issue reported in late April:
> 
> https://github.com/raspberrypi/firmware/issues/1376

Thanks for pointing it out! I missed that bug report. I'll try to get the gist
of it.

Regards,
Nicolas

[1] https://github.com/raspberrypi/firmware/releases


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Issues attempting to use Raspberry Pi 4 serial console on mainline

[Peter Robinson]

On Wed, Jul 22, 2020 at 4:46 PM Nicolas Saenz Julienne
<nsaenzjulienne@suse.de> wrote:
>
> Hi Nathan, Peter,
>
> On Wed, 2020-07-22 at 15:41 +0100, Peter Robinson wrote:
>
> [...]
>
> > On Wed, Jul 22, 2020 at 2:27 PM Nicolas Saenz Julienne
> > > For reference I just booted linux-next with this setup:
> > >
> > > boot partition:
> > >         ...Latest firmware files taken from the RPi firmware repo [1]...
>
> Natan, it seems I lied here and been using an outdated firmware version. I've
> just validated what Peter comments below, and the last working firmware release
> with this setup is Feb12[1]. Given your use case I'd just use that and forget
> about it.
>
> > >         Image                   #Copied from linux build
> > >         bcm2711-rpi-4-b.dtb     #Copied from linux build
> > >         config.txt
> > >         cmdline.txt
> > >
> > > config.txt:
> > >         kernel=Image
> > >         enable_uart=1
> > >         arm_64bit=1
> > >
> > > cmdline.txt:
> > >          console=tty console=ttyS1,115200 text root=/dev/nfs
> > >          nfsroot=10.42.0.1:/home/nico/netboot/root,vers=3 rw ip=dhcp
> > > rootwait
> > >          elevator=deadline
> > >
>
> [...]
>
> > >
> > > The issue with the mini UART is its clock, which is derived from VPU's, which
> > > is itself controlled by RPi's firmware. Changes might happen behind the
> > > kernel's back, and the mini UART divisors will not be updated accordingly.
> > > This is an area the we could do better, but no one found a good solution yet.
> > > That said, for now, when using the upstream kernel, VPU's clock should be
> > > stable as we forbid the firmware from performing frequency scaling on that
> > > clock.
> >
> > There has actually been a regression in the firmware here, prior to
> > mid April if the enable_uart=1 I always had clean output on the serial
> > console, since that date on the rpi3/3b+/4 I get a whole bunch of
> > junk, revert to an older firmware (I've been using April 1st ) and it
> > all goes back to being fine. I've not had time to actually report it
> > yet, and it's still a problem with the latest firmware but a quick
> > look it likes similar to this issue reported in late April:
> >
> > https://github.com/raspberrypi/firmware/issues/1376
>
> Thanks for pointing it out! I missed that bug report. I'll try to get the gist
> of it.

Thanks, let me know if you need any more info from me, I would like to
get this issue solved, I've just not had nearly enough time for RPi of
late.

Peter

> Regards,
> Nicolas
>
> [1] https://github.com/raspberrypi/firmware/releases
>

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: Issues attempting to use Raspberry Pi 4 serial console on mainline

[Nathan Chancellor]

On Wed, Jul 22, 2020 at 05:46:22PM +0200, Nicolas Saenz Julienne wrote:
> Hi Nathan, Peter,
> 
> On Wed, 2020-07-22 at 15:41 +0100, Peter Robinson wrote:
> 
> [...]
> 
> > On Wed, Jul 22, 2020 at 2:27 PM Nicolas Saenz Julienne
> > > For reference I just booted linux-next with this setup:
> > > 
> > > boot partition:
> > >         ...Latest firmware files taken from the RPi firmware repo [1]...
> 
> Natan, it seems I lied here and been using an outdated firmware version. I've
> just validated what Peter comments below, and the last working firmware release
> with this setup is Feb12[1]. Given your use case I'd just use that and forget
> about it.

Excellent, that worked perfectly with the settings that I had initially.

> > >         Image                   #Copied from linux build
> > >         bcm2711-rpi-4-b.dtb     #Copied from linux build
> > >         config.txt
> > >         cmdline.txt
> > > 
> > > config.txt:
> > >         kernel=Image
> > >         enable_uart=1
> > >         arm_64bit=1
> > > 
> > > cmdline.txt:
> > >          console=tty console=ttyS1,115200 text root=/dev/nfs
> > >          nfsroot=10.42.0.1:/home/nico/netboot/root,vers=3 rw ip=dhcp
> > > rootwait
> > >          elevator=deadline
> > > 
> 
> [...]
> 
> > > 
> > > The issue with the mini UART is its clock, which is derived from VPU's, which
> > > is itself controlled by RPi's firmware. Changes might happen behind the
> > > kernel's back, and the mini UART divisors will not be updated accordingly.
> > > This is an area the we could do better, but no one found a good solution yet.
> > > That said, for now, when using the upstream kernel, VPU's clock should be
> > > stable as we forbid the firmware from performing frequency scaling on that
> > > clock.
> > 
> > There has actually been a regression in the firmware here, prior to
> > mid April if the enable_uart=1 I always had clean output on the serial
> > console, since that date on the rpi3/3b+/4 I get a whole bunch of
> > junk, revert to an older firmware (I've been using April 1st ) and it
> > all goes back to being fine. I've not had time to actually report it
> > yet, and it's still a problem with the latest firmware but a quick
> > look it likes similar to this issue reported in late April:
> > 
> > https://github.com/raspberrypi/firmware/issues/1376
> 
> Thanks for pointing it out! I missed that bug report. I'll try to get the gist
> of it.
> 
> Regards,
> Nicolas
> 
> [1] https://github.com/raspberrypi/firmware/releases
> 

Same goes for me as Peter, let me know if you need any information or
debugging info.

Cheers,
Nathan

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel