Kernel debugging??

Kernel debugging??

[Andreas Tobler]

Is there a possibility to debug the kernel, especially in its very early
load stage?
Is xmon the right tool to do so?
My situation is a not booting 2.3.28 kernel. All I know is, that I pass
at least the prom_init(). After that the screen stays black with the
'Welcome to kernel 2.3.28' and a few messages about addresses I think.
And at the end it says 'booting... '
(On a PM7200 and a Wallstreet I)

Any experiences around?

Thanks

Andreas
-- 
| Andreas Tobler				
| CH-8004 Zuerich 							
| E-Mail: andreas.tobler@alcatel.ch	| a.tobler@schweiz.ch
-----------------------------------------------------------

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Ani Joshi]

andreas,
	since you're on a powermac and using BootX, you can use
prom_print() to print debuging info before start_kernel, there after you
can just use printk().  instead of putting a million prints all over the
place, you should look at the patches between 2.3.25-2.3.28 and see what
was changed in ppc/kernel or ppc/mm , i've booted
2.3.25 on my 7200 but i have had no time lately to try 2.3.28, perhaps
later today i will work on it.



ani


On Tue, 16 Nov 1999, Andreas Tobler wrote:

> 
> Is there a possibility to debug the kernel, especially in its very early
> load stage?
> Is xmon the right tool to do so?
> My situation is a not booting 2.3.28 kernel. All I know is, that I pass
> at least the prom_init(). After that the screen stays black with the
> 'Welcome to kernel 2.3.28' and a few messages about addresses I think.
> And at the end it says 'booting... '
> (On a PM7200 and a Wallstreet I)
> 
> Any experiences around?
> 
> Thanks
> 
> Andreas
> -- 
> | Andreas Tobler				
> | CH-8004 Zuerich 							
> | E-Mail: andreas.tobler@alcatel.ch	| a.tobler@schweiz.ch
> -----------------------------------------------------------
> 


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Andreas Tobler]

Hi Ani,

Ani Joshi wrote:
> 
> andreas,
>         since you're on a powermac and using BootX, you can use
> prom_print() to print debuging info before start_kernel, there after you

This what I'm doing now.
But I feel a bit lost when it goes into asm.

> can just use printk().  instead of putting a million prints all over the
> place, you should look at the patches between 2.3.25-2.3.28 and see what
> was changed in ppc/kernel or ppc/mm ,

This I also did, on the mm side do I have to include both mm and
arch/ppc/mm?
Here many changes happened.

> i've booted 2.3.25 on my 7200 
And I could boot a stock 2.3.26 with changes in asm-ppc/fcntl.h
(O_LARGEFILE ....)

>but i have had no time lately to try 2.3.28, perhaps
> later today i will work on it.

On my side it's mostly a question of learning, a fixed problem is nice,
a self found and fixed problem is even nicer :-)

Thanks for your hints

Andreas

-- 
| Andreas Tobler				
| CH-8004 Zuerich 							
| E-Mail: andreas.tobler@alcatel.ch	| a.tobler@schweiz.ch
-----------------------------------------------------------

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Geert Uytterhoeven]

On Tue, 16 Nov 1999, Andreas Tobler wrote:
> Ani Joshi wrote:
> >         since you're on a powermac and using BootX, you can use
> > prom_print() to print debuging info before start_kernel, there after you
> 
> This what I'm doing now.
> But I feel a bit lost when it goes into asm.
> 
> > can just use printk().  instead of putting a million prints all over the
> > place, you should look at the patches between 2.3.25-2.3.28 and see what
> > was changed in ppc/kernel or ppc/mm ,

2.3.28 gets sufficiently far to get output on the serial console
(console=ttyS0).

> This I also did, on the mm side do I have to include both mm and
> arch/ppc/mm?
> Here many changes happened.
> 
> > i've booted 2.3.25 on my 7200 
> And I could boot a stock 2.3.26 with changes in asm-ppc/fcntl.h
> (O_LARGEFILE ....)

That's very interesting to know! It means we just have to analyse the 2.3.27
and 2.3.28 patches to find out what went wrong.

Gr{oetje,eeting}s,
--
Geert Uytterhoeven -- Linux/{m68k~Amiga,PPC~CHRP} -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Martin Costabel]

Geert Uytterhoeven wrote:

> > And I could boot a stock 2.3.26 with changes in asm-ppc/fcntl.h
> > (O_LARGEFILE ....)
> 
> That's very interesting to know! It means we just have to analyse the 2.3.27
> and 2.3.28 patches to find out what went wrong.

Just for information: Paul M's 2.3.27 rsync kernel compiles and runs
just fine on this Pmac 6400. The machine swaps like crazy and is
therefore rather slow, but I am not sure if this is the kernel's fault.

--
Martin

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Geert Uytterhoeven]

On Tue, 16 Nov 1999, Martin Costabel wrote:
> Geert Uytterhoeven wrote:
> > > And I could boot a stock 2.3.26 with changes in asm-ppc/fcntl.h
> > > (O_LARGEFILE ....)
> > 
> > That's very interesting to know! It means we just have to analyse the 2.3.27
> > and 2.3.28 patches to find out what went wrong.
> 
> Just for information: Paul M's 2.3.27 rsync kernel compiles and runs
> just fine on this Pmac 6400. The machine swaps like crazy and is
> therefore rather slow, but I am not sure if this is the kernel's fault.

Are you sure about the version? I just checked out this tree, and it's 2.3.22,
according to the Makefile. Perhaps I made a mistake? This is the first time I
used rsync (`rsync -auvz samba.anu.edu.au::linux-pmac-devel .').

Gr{oetje,eeting}s,
--
Geert Uytterhoeven -- Linux/{m68k~Amiga,PPC~CHRP} -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Martin Costabel]

Geert Uytterhoeven wrote:
> 

> > Just for information: Paul M's 2.3.27 rsync kernel compiles and runs
> > just fine on this Pmac 6400. The machine swaps like crazy and is
> > therefore rather slow, but I am not sure if this is the kernel's fault.

The swapping is now being blamed on BootX.
 
> Are you sure about the version? I just checked out this tree, and it's 2.3.22,
> according to the Makefile. Perhaps I made a mistake? This is the first time I
> used rsync (`rsync -auvz samba.anu.edu.au::linux-pmac-devel .').

>From BenH's page or some of his messages I learned the new address
linuxcare.com.au. There it's 2.3.27(*). Seems Paul left ANU for good.

(*) Correction. I ran rsync while typing this letter. It's 2.3.28 now. I
am curious to see how this will run. 

--
Martin

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Martin Costabel]

Martin Costabel wrote:
> 

> (*) Correction. I ran rsync while typing this letter. It's 2.3.28 now. I
> am curious to see how this will run.

Followup: Paul's 2.3.28 runs very well on my 6400.

One curious thing, though: I played with hdparm for the internal IDE
disk. Turns out that it is faster with DMA turned OFF (I did a PIO
autotune hdparm -p  before):

root[90]#hdparm -d0 /dev/hda
/dev/hda:
 setting using_dma to 0 (off)
 using_dma    =  0 (off)
root[91]#hdparm -tT /dev/hda
/dev/hda:
 Timing buffer-cache reads:   64 MB in  2.91 seconds =21.99 MB/sec
 Timing buffered disk reads:  32 MB in  5.62 seconds = 5.69 MB/sec
root[92]#hdparm -d1 /dev/hda
/dev/hda:
 setting using_dma to 1 (on)
 using_dma    =  1 (on)
root[93]#hdparm -tT /dev/hda
/dev/hda:
 Timing buffer-cache reads:   64 MB in  2.91 seconds =21.99 MB/sec
 Timing buffered disk reads:  32 MB in 10.21 seconds = 3.13 MB/sec

This is repeatable. And there is still some strange swapping going on
that shouldn't be necessary (no high charge, just some window switching
in X). But now that I went back to BootX-1.1.3, the overall speed is OK.

--
Martin

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Extreme swapping / new location of Paul's kernels [was Re: Kernel debugging??]

[phandel]

On Wed, 17 Nov 1999, Geert Uytterhoeven wrote:

> On Tue, 16 Nov 1999, Martin Costabel wrote:
> > Just for information: Paul M's 2.3.27 rsync kernel compiles and runs
> > just fine on this Pmac 6400. The machine swaps like crazy and is
> > therefore rather slow, but I am not sure if this is the kernel's fault.

I've noticed the same extreme swapping in 2.3.26, using SCSI disks
(PowerCenter Pro 210) and BootX.  Returning to my previous kernel, there
was no swapping. This was using BootX 1.2b1.

> Are you sure about the version? I just checked out this tree, and it's 2.3.22,
> according to the Makefile. Perhaps I made a mistake? This is the first time I
> used rsync (`rsync -auvz samba.anu.edu.au::linux-pmac-devel .').

rsync -auvz linuxcare.com.au::linux-pmac-devel /usr/src/linux-2.3.devel
(from Ben's post a few days ago)


Thanks,
Peter

-- 
Peter F. Handel          "[The anti-Christ] also forced everyone, small and
phandel@cise.ufl.edu      great ... to receive a mark [smart card?] on his
www.cise.ufl.edu/~phandel right hand or on his forehead, so that no one could
FAX: (561) 619-8051       buy or sell unless he had the mark"-Revelation13:16


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Extreme swapping / new location of Paul's kernels [was Re: Kernel debugging??]

[Benjamin Herrenschmidt]

On Wed, Nov 17, 1999,  <phandel@cise.ufl.edu> wrote:

>I've noticed the same extreme swapping in 2.3.26, using SCSI disks
>(PowerCenter Pro 210) and BootX.  Returning to my previous kernel, there
>was no swapping. This was using BootX 1.2b1.

And I have absolutely no idea about what in BootX can cause this. Could
you check if the amount of available RAM seen by the kenrel is correct ?


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Extreme swapping / new location of Paul's kernels [was Re: Kernel debugging??]

[phandel]

On Thu, 18 Nov 1999, Benjamin Herrenschmidt wrote:

> On Wed, Nov 17, 1999,  <phandel@cise.ufl.edu> wrote:
> 
> >I've noticed the same extreme swapping in 2.3.26, using SCSI disks
> >(PowerCenter Pro 210) and BootX.  Returning to my previous kernel, there
> >was no swapping. This was using BootX 1.2b1.
> 
> And I have absolutely no idea about what in BootX can cause this. Could
> you check if the amount of available RAM seen by the kenrel is correct ?

I'm having a hard time blaming BootX - top did reveal an incorrect amount
of memory (total/free/etc.) but I've heard that top is broken in the 2.3
series.  I'll grab Paul's new 2.3.28 rsync later tomorrow and look at the
memory.

The same version of BootX works perfectly with the 2.2 series.


Thanks,
Pete

-- 
Peter F. Handel          "[The anti-Christ] also forced everyone, small and
phandel@cise.ufl.edu      great ... to receive a mark [smart card?] on his
www.cise.ufl.edu/~phandel right hand or on his forehead, so that no one could
FAX: (561) 619-8051       buy or sell unless he had the mark"-Revelation13:16


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Extreme swapping / new location of Paul's kernels [was Re: Kernel debugging??]

[Benjamin Herrenschmidt]

On Thu, Nov 18, 1999,  <phandel@cise.ufl.edu> wrote:

>I'm having a hard time blaming BootX - top did reveal an incorrect amount
>of memory (total/free/etc.) but I've heard that top is broken in the 2.3
>series.  I'll grab Paul's new 2.3.28 rsync later tomorrow and look at the
>memory.

There is a possibility that BootX passed an incorrect device tree size or
something like that that would mess up the kenrel memory management. I
didn't see this happen, but I'll have a second look at the code.


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Extreme swapping / new location of Paul's kernels [was Re: Kernel debugging??]

[Martin Costabel]

Benjamin Herrenschmidt wrote:
> 
> On Wed, Nov 17, 1999,  <phandel@cise.ufl.edu> wrote:
> 
> >I've noticed the same extreme swapping in 2.3.26, using SCSI disks
> >(PowerCenter Pro 210) and BootX.  Returning to my previous kernel, there
> >was no swapping. This was using BootX 1.2b1.
> 
> And I have absolutely no idea about what in BootX can cause this. Could
> you check if the amount of available RAM seen by the kenrel is correct ?

Comparing boot messages from booting with "good" and "bad" versions of
BootX, I saw that the amount of RAM is OK in both cases, but then I
found one quite incredible thing, and this is essentially the only
difference in the boot messages. Booting with BootX version <= 1.1.3 and
any kernel, I get as usual

   kernel: Calibrating delay loop... 133.12 BogoMIPS 

Booting with BootX-1.2b2, I get with the same kernels

   kernel: Calibrating delay loop... 11.39 BogoMIPS

This is repeatable, but for me quite incomprehensible.

--
Martin

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Extreme swapping / new location of Paul's kernels [was Re: Kernel debugging??]

[Benjamin Herrenschmidt]

On Fri, Nov 19, 1999, Martin Costabel <costabel@wanadoo.fr> wrote:

>   kernel: Calibrating delay loop... 133.12 BogoMIPS 
>
>Booting with BootX-1.2b2, I get with the same kernels
>
>   kernel: Calibrating delay loop... 11.39 BogoMIPS
>
>This is repeatable, but for me quite incomprehensible.

Ouch. Could this be a problem with VIA-based calibration in the kernel ?

Paul ? Any clue ? I don't see what in BootX could cause this to happen,
probably MacOS playing tricks with the VIA behind my back... I just
looked at the calibration code, and it looks ok, except if interrupts are
enabled during the calibration (Actually, I don't know if MSR-EE is
enabled at this point, I should really add code to BootX that masks all
interrupts in the controller before launching the kernel anyway).

If I remember correctly, Martin, your machine has a 603. There is a
possibility that, for some reason, MacOS changes the PLL settings in HID1
(but I don't really see why this would happen with BootX 1.2x and not
with previous versions).


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Paul Mackerras]

On Thu, 18 Nov 1999, Geert Uytterhoeven wrote:

> Are you sure about the version? I just checked out this tree, and it's 2.3.22,
> according to the Makefile. Perhaps I made a mistake? This is the first time I
> used rsync (`rsync -auvz samba.anu.edu.au::linux-pmac-devel .').

There is a 2.3.28 tree in linuxcare.com.au::linux-pmac-devel, which boots
and runs for me but has some sort of major memory leak.  The tree on samba
is old, maybe I should just remove it.  I will be putting stuff on
linuxcare.com.au (== ftp.linuxcare.com.au) generally from now on.

Paul.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Michael Fenske]

Paul Mackerras wrote:

> is old, maybe I should just remove it.  I will be putting stuff on
> linuxcare.com.au (== ftp.linuxcare.com.au) generally from now on.

Is there something wrong with linuxcare.com.au ? If I do

 rsync -arvz linuxcare.com.au::linux-pmac-stable .

I get this:

  Welcome to the Linuxcare Australia rsync server
  For information about Linuxcare see http://linuxcare.com.au/


  receiving file list ... done
  ./
  COPYING
  deflate on token returned -2 (700 bytes left)
  unexpected EOF in read_timeout

rsync from samba.anu.edu.au works just fine.

Michael

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Benjamin Herrenschmidt]

On Thu, Nov 18, 1999, Michael Fenske <michael.fenske@arcormail.de> wrote:

>Is there something wrong with linuxcare.com.au ? If I do
>
> rsync -arvz linuxcare.com.au::linux-pmac-stable .
>
>I get this:
>
>  Welcome to the Linuxcare Australia rsync server
>  For information about Linuxcare see http://linuxcare.com.au/
>
>
>  receiving file list ... done
>  ./
>  COPYING
>  deflate on token returned -2 (700 bytes left)
>  unexpected EOF in read_timeout
>
>rsync from samba.anu.edu.au works just fine.

I had the same problem today. Just remove the "z" option, it will eat
more bandidth but it works. I beleive it's time to update my rsync or zlib...

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Michael Fenske]

Benjamin Herrenschmidt wrote:

> I had the same problem today. Just remove the "z" option, it will eat
> more bandidth but it works. I beleive it's time to update my rsync or zlib...

Thanks, rsync works without the "z" option. And FYI, I updated rsync to
version 2.3.2 (from linuxcare.com.au) after the first time the problem
occurs, but that doesn't help. Maybe an update of zlib could help ?!

Michael

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Joel Klecker]

At 10:10 +0100 1999-11-19, Michael Fenske wrote:
>Thanks, rsync works without the "z" option. And FYI, I updated rsync to
>version 2.3.2 (from linuxcare.com.au) after the first time the problem
>occurs, but that doesn't help. Maybe an update of zlib could help ?!

I think it's a problem on the server side.
-- 
Joel Klecker (aka Espy)                    Debian GNU/Linux Developer
<URL:mailto:jk@espy.org>                 <URL:mailto:espy@debian.org>
<URL:http://web.espy.org/>               <URL:http://www.debian.org/>

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Kevin Hendricks]

Hi,

> Is there something wrong with linuxcare.com.au ? If I do

>   receiving file list ... done
>   ./
>   COPYING
>   deflate on token returned -2 (700 bytes left)
>   unexpected EOF in read_timeout

I got the same thing.  Just drop the compression (the z) and see if that helps.
 It did for me.

By the way, I updated from the samba site after the linuxcare site went up (I
didn't know about the linuxcare site).  To get a good source tree I had to do
the following:

-arv --delete

The -u update kept thinking my samba version was newer than the linuxcare
version and wouldn't get the new version and there were lots of files that
needed to be deleted.


So the following worked for me

 rsync -arv --delete linuxcare.com.au::linux-pmac-stable .

I hope this helps.

Kevin

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Paul Mackerras]

On Fri, 19 Nov 1999, Kevin Hendricks wrote:

> By the way, I updated from the samba site after the linuxcare site went up (I
> didn't know about the linuxcare site).  To get a good source tree I had to do
> the following:
> 
> -arv --delete
> 
> The -u update kept thinking my samba version was newer than the linuxcare
> version and wouldn't get the new version and there were lots of files that
> needed to be deleted.

If you have done a compilation in the tree, it may have touched some of
the header files.  The way the kernel Makefiles take care of header
dependencies is that if a.h includes b.h, and b.h is newer than a.h, it
will touch a.h.

I usually keep separate trees and only compile in one of them.  I use
dirdiff to keep track of the differences between the trees.  Dirdiff is a
tool I wrote for seeing the differences between directory trees.  It can
handle up to 5 directory trees.  It's a tcl/tk script and it's available
at ftp://ftp.linuxcare.com.au/pub/ppclinux/dirdiff.

Paul.

** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Benjamin Herrenschmidt]

On Wed, Nov 17, 1999, Geert Uytterhoeven <geert@linux-m68k.org> wrote:

>Are you sure about the version? I just checked out this tree, and it's
2.3.22,
>according to the Makefile. Perhaps I made a mistake? This is the first time I
>used rsync (`rsync -auvz samba.anu.edu.au::linux-pmac-devel .').

To my knowledge, Paul's tree is now at

rsync -auvz linuxcare.com.au::linux-pmac-devel .


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/

Re: Kernel debugging??

[Geert Uytterhoeven]

On Thu, 18 Nov 1999, Benjamin Herrenschmidt wrote:
> On Wed, Nov 17, 1999, Geert Uytterhoeven <geert@linux-m68k.org> wrote:
> 
> >Are you sure about the version? I just checked out this tree, and it's
> 2.3.22,
> >according to the Makefile. Perhaps I made a mistake? This is the first time I
> >used rsync (`rsync -auvz samba.anu.edu.au::linux-pmac-devel .').
> 
> To my knowledge, Paul's tree is now at
> 
> rsync -auvz linuxcare.com.au::linux-pmac-devel .

Using this, 2.3.28 boots a bit further. It locks up right before the SCSI
device probing. After `scsi : 2 hosts.', I can still toggle the Caps Lock LED
for one second, but after that interrupts no longer seem to come through.

    ncr53c8xx: IO region 0x1400 to 0x147f is in use
    mesh: assuming 50MHz clock frequency
    scsi0 : sym53c8xx - version 1.3c
    scsi1 : MESH
    scsi : 2 hosts.
HERE ------------------>
      Vendor: QUANTUM   Model: VIKING II 4.5WLS  Rev: 3506
      Type:   Direct-Access                      ANSI SCSI revision: 02

Fortunately the kernel gets sufficiently far to resume working on frame buffer
device, cfr. the patch I just posted.

Gr{oetje,eeting}s,
--
Geert Uytterhoeven -- Linux/{m68k~Amiga,PPC~CHRP} -- geert@linux-m68k.org

In personal conversations with technical people, I call myself a hacker. But
when I'm talking to journalists I just say "programmer" or something like that.
							    -- Linus Torvalds


** Sent via the linuxppc-dev mail list. See http://lists.linuxppc.org/