MPC83xx console : no output after handover

MPC83xx console : no output after handover

[Matthew L. Creech]

(Sorry if this is a dupe, but I think my first attempt was filtered out)

Hi,

I'm trying to get Linux booting on a custom board based on the MPC8313
ERDB, but there's no output after the console is transferred.  I get
the following:

Using MPC831x RDB machine description
Linux version 2.6.29 (mlcreech@lap) (gcc version 4.3.2 (Sourcery G++ Lite 4.3-50
) ) #2 PREEMPT Tue Mar 31 14:54:27 EDT 2009
console [udbg0] enabled

... (normal boot output) ...

Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
serial8250.0: ttyS0 at MMIO 0xe0004500 (irq = 16) is a 16550A
console handover: boot [udbg0] -> real [ttyS0]
serial8250.0: ttyS1 at MMIO 0xe0004600 (irq = 17) is a 16550A

If I boot with "console=udbg0,115200", then it gets much further along
before printing "turn off boot console udbg0".  So presumably the
kernel is booting beyond this point, but my console isn't printing any
output.  I traced from that last line back up the call stack to the
end of drivers/serial/8250.c:serial8250_init(), and everything seems
okay (it returns 0, and pritnk() still works fine at that point).  I'm
not sure where to go from there to figure out where the problem lies.
Could someone point me in the right direction, or suggest what might
cause a console problem like this?

Thanks!

-- 
Matthew L. Creech

Re: MPC83xx console : no output after handover

[Gary Thomas]

Matthew L. Creech wrote:
> (Sorry if this is a dupe, but I think my first attempt was filtered out)

Nope, it made it.

> Hi,
> 
> I'm trying to get Linux booting on a custom board based on the MPC8313
> ERDB, but there's no output after the console is transferred.  I get
> the following:
> 
> Using MPC831x RDB machine description
> Linux version 2.6.29 (mlcreech@lap) (gcc version 4.3.2 (Sourcery G++ Lite 4.3-50
> ) ) #2 PREEMPT Tue Mar 31 14:54:27 EDT 2009
> console [udbg0] enabled
> 
> ... (normal boot output) ...
> 
> Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> serial8250.0: ttyS0 at MMIO 0xe0004500 (irq = 16) is a 16550A
> console handover: boot [udbg0] -> real [ttyS0]
> serial8250.0: ttyS1 at MMIO 0xe0004600 (irq = 17) is a 16550A
> 
> If I boot with "console=udbg0,115200", then it gets much further along
> before printing "turn off boot console udbg0".  So presumably the
> kernel is booting beyond this point, but my console isn't printing any
> output.  I traced from that last line back up the call stack to the
> end of drivers/serial/8250.c:serial8250_init(), and everything seems
> okay (it returns 0, and pritnk() still works fine at that point).  I'm
> not sure where to go from there to figure out where the problem lies.
> Could someone point me in the right direction, or suggest what might
> cause a console problem like this?

What does your command line (boot args) when it fails?  It should
probably have something like "console=ttyS0,115200" in it.

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: MPC83xx console : no output after handover

[Matthew L. Creech]

On Tue, Mar 31, 2009 at 4:45 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>
> What does your command line (boot args) when it fails? =A0It should
> probably have something like "console=3DttyS0,115200" in it.
>

Yes, that's what I'm using.  It also seems to be the default if none
is supplied.

For the sake of completeness, here's a full dump:

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Using MPC831x RDB machine description
Linux version 2.6.29 (mlcreech@lap) (gcc version 4.3.2 (Sourcery G++ Lite 4=
.3-50
) ) #3 PREEMPT Tue Mar 31 15:10:02 EDT 2009
console [udbg0] enabled
setup_arch: bootmem
mpc831x_rdb_setup_arch()
arch: exit
Zone PFN ranges:
  DMA      0x00000000 -> 0x00008000
  Normal   0x00008000 -> 0x00008000
Movable zone start PFN for each node
early_node_map[1] active PFN ranges
    0: 0x00000000 -> 0x00008000
Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32512
Kernel command line: root=3D/dev/mtdblock1 init=3D/bin/sh console=3DttyS0,1=
15200
IPIC (128 IRQ sources) at fdffd700
PID hash table entries: 512 (order: 9, 2048 bytes)
clocksource: timebase mult[7800001] shift[22] registered
Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
Memory: 126236k/131072k available (3260k kernel code, 4692k reserved, 136k =
data,
 98k bss, 160k init)
SLUB: Genslabs=3D12, HWalign=3D32, Order=3D0-3, MinObjects=3D0, CPUs=3D1, N=
odes=3D1
Calibrating delay loop... 66.30 BogoMIPS (lpj=3D33152)
Mount-cache hash table entries: 512
net_namespace: 708 bytes
NET: Registered protocol family 16

bio: create slab <bio-0> at 0
Freescale Elo / Elo Plus DMA driver
NET: Registered protocol family 2
IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
TCP established hash table entries: 4096 (order: 3, 32768 bytes)
TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
TCP: Hash tables configured (established 4096 bind 4096)
TCP reno registered
NET: Registered protocol family 1
WDT driver for MPC8xxx initialized. mode:reset timeout=3D65535 (32 seconds)
fsl-elo-dma e00082a8.dma: Probe the Freescale DMA driver for fsl,elo-dma co=
ntrol
ler at e00082a8...
fsl-elo-dma e00082a8.dma: #0 (fsl,elo-dma-channel), irq 71
fsl-elo-dma e00082a8.dma: #1 (fsl,elo-dma-channel), irq 71
fsl-elo-dma e00082a8.dma: #2 (fsl,elo-dma-channel), irq 71
fsl-elo-dma e00082a8.dma: #3 (fsl,elo-dma-channel), irq 71
squashfs: version 4.0 (2009/01/31) Phillip Lougher
Registering unionfs 2.5.1 (for 2.6.29-rc2)
yaffs Mar 31 2009 02:31:59 Installing.
msgmni has been set to 246
alg: No test for stdrng (krng)
io scheduler noop registered
io scheduler anticipatory registered
io scheduler deadline registered (default)
io scheduler cfq registered
Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
serial8250.0: ttyS0 at MMIO 0xe0004500 (irq =3D 16) is a 16550A
console handover: boot [udbg0] -> real [ttyS0]
serial8250.0: ttyS1 at MMIO 0xe0004600 (irq =3D 17) is a 16550A
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

(nothing after this)

Thanks

--=20
Matthew L. Creech

Re: MPC83xx console : no output after handover

[Gary Thomas]

Matthew L. Creech wrote:
> On Tue, Mar 31, 2009 at 4:45 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>> What does your command line (boot args) when it fails?  It should
>> probably have something like "console=ttyS0,115200" in it.
>>
> 
> Yes, that's what I'm using.  It also seems to be the default if none
> is supplied.
> 
> For the sake of completeness, here's a full dump:
> 
> ============
> Using MPC831x RDB machine description
> Linux version 2.6.29 (mlcreech@lap) (gcc version 4.3.2 (Sourcery G++ Lite 4.3-50
> ) ) #3 PREEMPT Tue Mar 31 15:10:02 EDT 2009
> console [udbg0] enabled
> setup_arch: bootmem
> mpc831x_rdb_setup_arch()
> arch: exit
> Zone PFN ranges:
>   DMA      0x00000000 -> 0x00008000
>   Normal   0x00008000 -> 0x00008000
> Movable zone start PFN for each node
> early_node_map[1] active PFN ranges
>     0: 0x00000000 -> 0x00008000
> Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 32512
> Kernel command line: root=/dev/mtdblock1 init=/bin/sh console=ttyS0,115200
> IPIC (128 IRQ sources) at fdffd700
> PID hash table entries: 512 (order: 9, 2048 bytes)
> clocksource: timebase mult[7800001] shift[22] registered
> Dentry cache hash table entries: 16384 (order: 4, 65536 bytes)
> Inode-cache hash table entries: 8192 (order: 3, 32768 bytes)
> Memory: 126236k/131072k available (3260k kernel code, 4692k reserved, 136k data,
>  98k bss, 160k init)
> SLUB: Genslabs=12, HWalign=32, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
> Calibrating delay loop... 66.30 BogoMIPS (lpj=33152)
> Mount-cache hash table entries: 512
> net_namespace: 708 bytes
> NET: Registered protocol family 16
> 
> bio: create slab <bio-0> at 0
> Freescale Elo / Elo Plus DMA driver
> NET: Registered protocol family 2
> IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
> TCP established hash table entries: 4096 (order: 3, 32768 bytes)
> TCP bind hash table entries: 4096 (order: 2, 16384 bytes)
> TCP: Hash tables configured (established 4096 bind 4096)
> TCP reno registered
> NET: Registered protocol family 1
> WDT driver for MPC8xxx initialized. mode:reset timeout=65535 (32 seconds)
> fsl-elo-dma e00082a8.dma: Probe the Freescale DMA driver for fsl,elo-dma control
> ler at e00082a8...
> fsl-elo-dma e00082a8.dma: #0 (fsl,elo-dma-channel), irq 71
> fsl-elo-dma e00082a8.dma: #1 (fsl,elo-dma-channel), irq 71
> fsl-elo-dma e00082a8.dma: #2 (fsl,elo-dma-channel), irq 71
> fsl-elo-dma e00082a8.dma: #3 (fsl,elo-dma-channel), irq 71
> squashfs: version 4.0 (2009/01/31) Phillip Lougher
> Registering unionfs 2.5.1 (for 2.6.29-rc2)
> yaffs Mar 31 2009 02:31:59 Installing.
> msgmni has been set to 246
> alg: No test for stdrng (krng)
> io scheduler noop registered
> io scheduler anticipatory registered
> io scheduler deadline registered (default)
> io scheduler cfq registered
> Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> serial8250.0: ttyS0 at MMIO 0xe0004500 (irq = 16) is a 16550A
> console handover: boot [udbg0] -> real [ttyS0]
> serial8250.0: ttyS1 at MMIO 0xe0004600 (irq = 17) is a 16550A
> ============

The fact that you get the ttyS1 line printed is interesting.  At
this point, the kernel is switching from raw console I/O (only
suitable for bring-up messages) to the general serial driver
(interrupt driven, etc).  I'm curious about what the ttyS1 driver
is causing to break...

A couple of things you could try:
  * Disable ttyS1 (take it out of your device tree)
  * Look at the console log when this happens.  Look in your system
    map for the symbol '__log_buf', e.g.
       c031ca54 b __log_buf
    This will get stored at physical location '0x31ca54' and will
    often contain data that didn't get a chance to print, for example
    if you have stuck interrupts that prevent the console from working.
    I'd just run it to this point and then examine the memory - either
    using a BDI if one is attached, or press RESET (I hope you have one!)
    and then look using your boot loader (uBoot, RedBoot, ...)



-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: MPC83xx console : no output after handover

[Matthew L. Creech]

On Tue, Mar 31, 2009 at 5:26 PM, Gary Thomas <gary@mlbassoc.com> wrote:
>
> The fact that you get the ttyS1 line printed is interesting. =A0At
> this point, the kernel is switching from raw console I/O (only
> suitable for bring-up messages) to the general serial driver
> (interrupt driven, etc). =A0I'm curious about what the ttyS1 driver
> is causing to break...
>
> A couple of things you could try:
> =A0* Disable ttyS1 (take it out of your device tree)

I tried taking ttyS1 out - the results are no different (other than
the fact that "console handover" is now the last line that prints).
My debug printk()s at the end of 8250_init() show up, which would seem
to imply that ttyS0 was initialized correctly, so I'm not sure what
else could be stomping on it.

> =A0* Look at the console log when this happens. =A0Look in your system
> =A0 =A0map for the symbol '__log_buf', e.g.
> =A0 =A0 =A0 c031ca54 b __log_buf
> =A0 =A0This will get stored at physical location '0x31ca54' and will
> =A0 =A0often contain data that didn't get a chance to print, for example
> =A0 =A0if you have stuck interrupts that prevent the console from working=
.
> =A0 =A0I'd just run it to this point and then examine the memory - either
> =A0 =A0using a BDI if one is attached, or press RESET (I hope you have on=
e!)
> =A0 =A0and then look using your boot loader (uBoot, RedBoot, ...)
>

I think I can get my hands on a BDI tomorrow, so I'll give this a try
then.  Thanks for the reply

--=20
Matthew L. Creech

Re: MPC83xx console : no output after handover

[Matthew L. Creech]

It turned out that the normally-initialized 8250 driver was getting
stomped on by my device tree's serial driver (which is also an 8250) -
apparently CONFIG_SERIAL_8250[_CONSOLE] and CONFIG_SERIAL_OF_PLATFORM
don't play well together.  Disabling the latter fixed my problem.

Thanks for the help, and sorry for the noise.

-- 
Matthew L. Creech

Re: MPC83xx console : no output after handover

[Gary Thomas]

Matthew L. Creech wrote:
> It turned out that the normally-initialized 8250 driver was getting
> stomped on by my device tree's serial driver (which is also an 8250) -
> apparently CONFIG_SERIAL_8250[_CONSOLE] and CONFIG_SERIAL_OF_PLATFORM
> don't play well together.  Disabling the latter fixed my problem.
> 
> Thanks for the help, and sorry for the noise.
> 

Can you share what you had and what you changed it to?  I'm
running on very similar systems and may face this exact problem
soon.

Thanks

-- 
------------------------------------------------------------
Gary Thomas                 |  Consulting for the
MLB Associates              |    Embedded world
------------------------------------------------------------

Re: MPC83xx console : no output after handover

[Matthew L. Creech]

On Thu, Apr 2, 2009 at 6:52 AM, Gary Thomas <gary@mlbassoc.com> wrote:
>
> Can you share what you had and what you changed it to? =A0I'm
> running on very similar systems and may face this exact problem
> soon.
>
> Thanks
>

I don't actually have a platform-specific bus, but I misread the help
text for CONFIG_SERIAL_OF_PLATFORM and thought that it applied to any
system whose serial port was configured via the DTS file.  So I had:

CONFIG_SERIAL_8250=3Dy
CONFIG_SERIAL_8250_CONSOLE=3Dy
...
CONFIG_SERIAL_OF_PLATFORM=3Dy

The result was that my console's serial port was initialized correctly
by the 8250 driver, but then of_platform_serial_probe() immediately
stomped on it.  (Not sure what it was doing exactly, I just traced
through far enough to see that printk() worked before
serial8250_register_port() was called in of_serial.c, and not
afterward).

The "fix" for me was simply to disable CONFIG_SERIAL_OF_PLATFORM, as
it's unnecessary in my case.

--=20
Matthew L. Creech

Re: MPC83xx console : no output after handover

[Scott Wood]

On Tue, Mar 31, 2009 at 04:30:26PM -0400, Matthew L. Creech wrote:
> (Sorry if this is a dupe, but I think my first attempt was filtered out)

No, it's a dupe. :-)

> I'm trying to get Linux booting on a custom board based on the MPC8313
> ERDB, but there's no output after the console is transferred.  I get
> the following:
[snip]
> Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
> serial8250.0: ttyS0 at MMIO 0xe0004500 (irq = 16) is a 16550A
> console handover: boot [udbg0] -> real [ttyS0]
> serial8250.0: ttyS1 at MMIO 0xe0004600 (irq = 17) is a 16550A

Make sure that u-boot for your board (or whatever you're using) is
properly filling in the clock-frequency property of the serial port
nodes.

-Scott