[parisc-linux] help debuging HMPC

[parisc-linux] help debuging HMPC

[Ryan Bradetich]

Hello parisc-linux hackers,

Now that I am starting to have time to devote on this port again, 
I started to look into the HPMC crash when doing a cpio from the 
scsi cdrom drive to a hard drive again.  I origionally tried to
debug the problem using printk statements in the ccio code, and
John Marvin helped me identify a problem I was experiencing while
attempting to debug with printk's .

http://lists.parisc-linux.org/pipermail/parisc-linux/2001-May/012579.html

Now I am trying to debug the problem using a different approach 
that does not use printk statements.  The HPMC is extremely
reproducable, and the IAOQ always points to the _gsc_readl
function and GR 2 always points to the dino_in32 function as 
the _gsc_readl function caller.

This is the instruction that causes the HPMC (IAOQ address - 8):
	ldwa  r0(r26),r26

Looking up the value of GR26 from the register dump gives me random
addresses.  The last three addresses I recorded using this kernel are:

	GR26: 0x00ff9000
	GR26: 0x002e8800
	GR26: 0x000a1000

My (limited) understanding of appropriate values for an address
being passed to the _gsc_readl function would be an I/O address,
which is >= 0xf0000000.  If my understanding is correct, then the
C200+ is properly HPMC'ing because of the invalid I/O address.

To verify my assumptions about the bad value in the GR26, I added
a global variable rbrad_debug to the dino.c code which will save 
the value off in memory (since I can not use printk :() the address 
value being passed to _gsc_readl from the dino_in32 function and 
use the ser mr <addr> [len] of the pdc to read the contents of
the rbrad_debug variable after the machine had HPMC'd.

Doing this gave me unexpected results also. (The results did not
match GR26 and they were not always >= 0xf0000000).  So I made
rbrad_debug an array of u32 and set known values in array indexes
0, 2, and 4 and used array indexes 1 and 3 to store the addresses
being written by gsc_writel (I stored this value for curiosity sakes) 
and gsc_readl.  If the known addresses were present in the the proper
index of the array, then it would increase my trust in the values being
returned in the unknown indexes.

To my surprise, I am not getting the expected values back even in the
known indexes when I do the ser mr <addr> 5 in the boot menu after the
HPMC.

This leaves me to consider two things:
	1. Am I using the ser mr <addr> len incorrectly?

	I grabbed the address from the system map and in this case:
	102db05c D rbrad_debug

		Main Menu: Enter command > ser mr 0x102db05c 5

   		0x102db05c  3fa276cd  5faa1b4c  7f44ed9a  bf543698  
   		0x102db06c  fe89db34  


	Next I though of getting rid of the 0x10000000 since that is
	where I believe we relocate the kernel to in vm.
	
		Main Menu: Enter command > ser mr 0x002db05c 5

		0x002db05c  0fc1bc54  f97e75b0  1f8378a8  f2fceb60  
		0x002db06c  3f06f150  

	Index 0, 2, and 4 should all be 0 if I were going to trust
	these values.

	2. Is memory getting stomped again like it did before when
	I was trying to debug using printk statements.

	Two things kinda indicate this is happening again to me (but
	I could also be mis-reading the results).
		a. I am not getting the known results back in the
		   indexes I expected.

		b. The address present in the IAOQ is not present in
		   the stack trace that is dumped.  The IAOQ value is:
		
			IAOQ: 102200d0 102207e4	

		and this is the stack dump range:

			Dumping Stack from 10320000 to 10320a40:

		running the astk command from the build-tools also does
		not show the dino_in32 or the _gsc_readl addresses in
		the stack trace either.

I have a bunch of files saved off in ~rbrad on p.e.h.c for people to look
at if they are interested.  I'm not going to attach them to this email 
since I (appropriately) got spanked last time for doing that :).   If people
do not have access to p.e.h.c and would like to get a copy of the files,
send me private email (rbradetich@uswest.net), and we can arrange a 
method to get the files for you.

The files on p.e.h.c/~rbrad are:
	vmlinux - The kernel used for the c200.log
	System.map - The System.map for the vmlinux kernel
	dino.patch - My patch to dino.c which addes rbrad_debug
	c200.log - The entire console log of the HPMC including:
		- Commands used to create the HPMC
		- The Stack Trace and register dump
		- The results from the ser mr commands
		- and the PIM dump for the HPMC

Hopefully this provides enough information for people interested in helping
debug the problem, if it doesn't let me know what additional information
you need and I will working on getting the information to you.

Thanks!

- Ryan

-- 

Re: [parisc-linux] help debuging HMPC

[Grant Grundler]

Ryan Bradetich wrote:
> Hello parisc-linux hackers,
> 
> Now that I am starting to have time to devote on this port again, 
> I started to look into the HPMC crash when doing a cpio from the 
> scsi cdrom drive to a hard drive again.
...

I don't know what the problem is but I can help clarify some
of the issues below.

...
> This is the instruction that causes the HPMC (IAOQ address - 8):
> 	ldwa  r0(r26),r26

Given the IOAQ data below, I'm pretty sure GR26 is garbage.

Anyway, the IO port address is most likely still in GR24
since it's the third arg to dino_inX. The call to gsc_readX
won't (shouldn't) clobber that.


> My (limited) understanding of appropriate values for an address
> being passed to the _gsc_readl function would be an I/O address,
> which is >= 0xf0000000.  If my understanding is correct, then the
> C200+ is properly HPMC'ing because of the invalid I/O address.

That's right.
The arg to gsc_readX should be the address of Dino's PCI_DATA register.
That's what generates the IO Port cycle.

> To my surprise, I am not getting the expected values back even in the
> known indexes when I do the ser mr <addr> 5 in the boot menu after the
> HPMC.

That was a good idea!

> 	I grabbed the address from the system map and in this case:
> 	102db05c D rbrad_debug
> 
> 		Main Menu: Enter command > ser mr 0x102db05c 5

a500 console boot output shows:
| Segment 0 load 00100000 size 2163856 mediaptr 0x1000
| Segment 1 load 00312000 size 472144 mediaptr 0x212000
| Segment 2 load 00388000 size 302608 mediaptr 0x286000
| Segment 3 load 003d4000 size 16384 mediaptr 0x2d0000
| branching to kernel entry point 0x00100000

Not sure how to form the proper address from System.map.
I'd guess subtract 0x00100000?

> 		b. The address present in the IAOQ is not present in
> 		   the stack trace that is dumped.  The IAOQ value is:
> 		
> 			IAOQ: 102200d0 102207e4	

Do these values mean the CPU was branching?
Normally I expect them to be +4 apart.
But it suggests we executed the ldwa insn and trashed GR26
before the HPMC was recorded.

> 		and this is the stack dump range:
> 
> 			Dumping Stack from 10320000 to 10320a40:

The stack is just a temporary storage space for code to save registers.
IOAQ reflects the state of the CPU and has nothing to do with stack.

> 		running the astk command from the build-tools also does
> 		not show the dino_in32 or the _gsc_readl addresses in
> 		the stack trace either.

right - only time you'll see a function address (which was part
of the call chain) is when GR02 gets saved.
gsc_readX is a "leaf" routine and thus doesn't save GR02.
The last thing on the stack should be the caller to dino_inX().

> I have a bunch of files saved off in ~rbrad on p.e.h.c for people to look
> at if they are interested.

I'll take a look and see if anything obvious pops up...

grant

Grant Grundler
parisc-linux {PCI|IOMMU|SMP} hacker
+1.408.447.7253

Re: [parisc-linux] help debuging HMPC

[Ryan Bradetich]

On Sat, Jul 14, 2001 at 12:20:00AM -0600, Grant Grundler wrote:

> > This is the instruction that causes the HPMC (IAOQ address - 8):
> > 	ldwa  r0(r26),r26
> 
> Given the IOAQ data below, I'm pretty sure GR26 is garbage.
> 
> Anyway, the IO port address is most likely still in GR24
> since it's the third arg to dino_inX. The call to gsc_readX
> won't (shouldn't) clobber that.

Just a quick clarification:

I macro expanded dino_in32 into this:
	static u32 dino_in32 (struct pci_hba_data *d, u16 addr);

dino_in32 only takes two arguments, so I should check GR25 and manually
perform the calculations in the dino_in32 to get the proper address
passed to the _gsc_readl function, correct?  


> The stack is just a temporary storage space for code to save registers.
> IOAQ reflects the state of the CPU and has nothing to do with stack.

Doh .. ok.  This makes sense to me now. *sigh*

> > 		running the astk command from the build-tools also does
> > 		not show the dino_in32 or the _gsc_readl addresses in
> > 		the stack trace either.
> 
> right - only time you'll see a function address (which was part
> of the call chain) is when GR02 gets saved.
> gsc_readX is a "leaf" routine and thus doesn't save GR02.
> The last thing on the stack should be the caller to dino_inX().

This also make sense to me.  Thanks for your time and explaination Grant!

- Ryan

> grant
> 
> Grant Grundler
> parisc-linux {PCI|IOMMU|SMP} hacker
> +1.408.447.7253
> 
> 
> _______________________________________________
> parisc-linux mailing list
> parisc-linux@lists.parisc-linux.org
> http://lists.parisc-linux.org/cgi-bin/mailman/listinfo/parisc-linux
> 

-- 

Re: [parisc-linux] help debuging HMPC

[Grant Grundler]

Ryan Bradetich wrote:
> Just a quick clarification:
> 
> I macro expanded dino_in32 into this:
> 	static u32 dino_in32 (struct pci_hba_data *d, u16 addr);
> 
> dino_in32 only takes two arguments, so I should check GR25 and manually
> perform the calculations in the dino_in32 to get the proper address
> passed to the _gsc_readl function, correct?  

Erm, right...dino_outX takes three...but the address is the second arg
and should be found in GR25 too. ATM, I can't explain why I found the
IO port address in GR24 when debugging a PCI problem on the A500.

grant

Grant Grundler
parisc-linux {PCI|IOMMU|SMP} hacker
+1.408.447.7253

Re: [parisc-linux] help debuging HMPC

[Ryan Bradetich]

On Mon, Jul 16, 2001 at 07:38:00AM -0600, Grant Grundler wrote:
> Ryan Bradetich wrote:
> > Just a quick clarification:
> > 
> > I macro expanded dino_in32 into this:
> > 	static u32 dino_in32 (struct pci_hba_data *d, u16 addr);
> > 
> > dino_in32 only takes two arguments, so I should check GR25 and manually
> > perform the calculations in the dino_in32 to get the proper address
> > passed to the _gsc_readl function, correct?  
> 
> Erm, right...dino_outX takes three...but the address is the second arg
> and should be found in GR25 too. ATM, I can't explain why I found the
> IO port address in GR24 when debugging a PCI problem on the A500.

I am confused about this also.

This is from the Documentation/parisc/registers file:

    r23-r26: these are arg3-arg0, i.e. you can use them if you
        don't care about the values that were passed in anymore.


this is from my latest register dump:

	r24-27   0000ff40 f2000064 040067f0 102ac010

The first thing that threw me was GR25 was obviously larger then the
u16 address being passed into dino_in32 function.  The other thing is
that value looks like a valid IO address.

I then checked to see how the values being passed to the gsc_writel 
function (I macro-expanded the function and pasted it here for easy 
reference):  

static u32 dino_in32 (struct pci_hba_data *d, u16 addr) 
{
	u32 v;
	unsigned long flags;

	spin_lock_irqsave(&(DINO_DEV(d)->dinosaur_pen), flags);

	/* tell HW which IO Port address */ \
	gsc_writel((u32) addr & ~3, d->base_addr + DINO_PCI_ADDR);

	/* generate I/O PORT read cycle */
	v = gsc_readl(d->base_addr+DINO_IO_DATA+(addr&0));

	spin_unlock_irqrestore(&(DINO_DEV(d)->dinosaur_pen), flags);
	return le32_to_cpu(v); \
}

The register value for GR24 looks to be a 16 bit value, and the value
for GR25 is the calculated value for d->base_addr + DINO_PCI_ADDR.

Dino version 3.1 (bridge mode) found at 0xf2000000
#define DINO_PCI_ADDR		0x064
#define DINO_IO_DATA		0x06c


So a couple of questions :)

1.  Is the HPMC really in the gsc_writel function and just not showing up
until the I/O port read cycle (gsc_readl function)?

2. Is the documentation correct for the resgister usage?

Thanks,

- Ryan

> grant
> 
> Grant Grundler
> parisc-linux {PCI|IOMMU|SMP} hacker
> +1.408.447.7253
> 

--