mmap + segfaults on MPC8349E

mmap + segfaults on MPC8349E

[R. Ebersole (VTI - new)]

[-- Attachment #1: Type: text/plain, Size: 1462 bytes --]


Hi.

We've run into an .... interesting (and frustrating) occurance recently 
while testing
out a new board (and BSP).   We were wondering if anybody has seen anything
like this.

A description follows:

We wrote some simple drivers/modules to mmap() FPGA registers to user space.
At the moment, for testing, we reserve the upper x-MB of RAM, and mmap() 
there, instead.

We open 4 such devices, and mmap() 64KB of space (memory, in this case) 
for each.
Accessing the first device opened & mmapped was fine.  But accesssing 
the 2nd device
yielded some odd behavior.  We could read and write the first and 3rd 4K 
regions fine, but
if we read from the 2nd one (at offset 0x1000) before we wrote to it, we 
get a segmentation
fault.  If, however, we wrote first (to the 2nd region), then things 
were OK.

Addendum: With the real device, this same behavior happens at the 10th 
4-KB region of the device.
 Without any of the others mmap()-ed.  If we mmap() just that 4-KB 
region, things are fine.  We
can access the region with ioremap() in the driver w/o issues.

We are running on an MPC8349-E.

Any thoughts or ideas?


vti

--

This message contains confidential information and is intended only for the
individual named.  If you are not the intended recipient you should not
disseminate, distribute or copy this e-mail.  Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and delete
this e-mail from your system.



[-- Attachment #2: Type: text/html, Size: 1987 bytes --]

Re: mmap + segfaults on MPC8349E

[David Hawkins]

Hi,

You haven't really provided enough information.

> We wrote some simple drivers/modules to mmap() FPGA registers to user space.
> At the moment, for testing, we reserve the upper x-MB of RAM, and mmap() 
> there, instead.

1. The FPGA is located where? The local bus, or the PCI bus?
    What frequency are you trying to operate at?

2. If its on the local bus, do you access it using GPCM or UPM?
    Have you setup either correctly?
    Have you confirmed the bus timing with a logic analyzer?

3. Have you created a bus functional model of the processor bus
    that you have then run with your FPGA bus in ModelSim to
    confirm that your FPGA performs correctly.

4. Have you tried burst and non-burst access by either using
    DMA, or treating the memory area as cacheable or non-cacheable?
    Have you checked those cases with simulation and then
    with a scope or logic analyzer?

5. Did you try running stand-alone tests in U-Boot, to go for a
    more bare-metal debug approach?

No point in debugging software if you have no idea whether the
hardware behaves. So confirm that you have tested your hardware
first.

My board design uses the MPC8349EA, I have an Altera Stratix II
FPGA on the local bus. I use GPCM to access flash on the local
bus via the FPGA, and UPM to access FPGA registers. I don't
have boards yet, but I've got a pretty good handle on how the
interface should work.

Cheers,
Dave

Re: mmap + segfaults on MPC8349E

[R. Ebersole (VTI - new)]

David Hawkins wrote:

> Hi,


Hi.

>
> You haven't really provided enough information.


Sorry about that.  I grabbed one of the h/w guys to help out.

>
>> We wrote some simple drivers/modules to mmap() FPGA registers to user 
>> space.
>> At the moment, for testing, we reserve the upper x-MB of RAM, and 
>> mmap() there, instead.
>
>
> 1. The FPGA is located where? The local bus, or the PCI bus?
>    What frequency are you trying to operate at? 


On the local bus frequency is 33.0 MHz. (66.0 PCI_CLK_IN).  See the 
attached .cfg script for memory map.

>
>
> 2. If its on the local bus, do you access it using GPCM or UPM?
>    Have you setup either correctly?
>    Have you confirmed the bus timing with a logic analyzer? 


GPCM.
We have confirmed the timing both in functionality (in the debugger) and 
with a logic analyzer (plus oscope for setup and hold times).
We have stand-alone C-code that runs in CodeWarrior that bangs away at 
the registers of this device and the hardware runs perfectly using the 
.cfg script settings..

>
>
> 3. Have you created a bus functional model of the processor bus
>    that you have then run with your FPGA bus in ModelSim to
>    confirm that your FPGA performs correctly. 

Yes, we have.  Also, checked in h/w (see 2 above)

>
>
> 4. Have you tried burst and non-burst access by either using
>    DMA, or treating the memory area as cacheable or non-cacheable?
>    Have you checked those cases with simulation and then
>    with a scope or logic analyzer?


Always non-burst at this point.

>
> 5. Did you try running stand-alone tests in U-Boot, to go for a
>    more bare-metal debug approach?

I assume the bare-metal debug approach is using a JTAG debugger 
connection and CodeWarrior.  We started there and now are trying to 
access the same device(s) via linux drivers.

>
> No point in debugging software if you have no idea whether the
> hardware behaves. So confirm that you have tested your hardware
> first.
>
> My board design uses the MPC8349EA, I have an Altera Stratix II
> FPGA on the local bus. I use GPCM to access flash on the local
> bus via the FPGA, and UPM to access FPGA registers. I don't
> have boards yet, but I've got a pretty good handle on how the
> interface should work.
>
> Cheers,
> Dave
>
>
>

-- 

Sometimes I feel like a red shirt in the Star Trek episode of life.

--

This message contains confidential information and is intended only for the
individual named.  If you are not the intended recipient you should not
disseminate, distribute or copy this e-mail.  Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and delete
this e-mail from your system.

Re: mmap + segfaults on MPC8349E

[David Hawkins]

Hi,

>> You haven't really provided enough information.
> 
> Sorry about that.  I grabbed one of the h/w guys to help out.

No problem. Just wanted to check the basics had been covered.
Sounds like they have.

> On the local bus frequency is 33.0 MHz. (66.0 PCI_CLK_IN).  See the 
> attached .cfg script for memory map.

It got dropped by the mailer. Don't sweat it.

>> 2. If its on the local bus, do you access it using GPCM or UPM?
>>    Have you setup either correctly?
>>    Have you confirmed the bus timing with a logic analyzer? 
> 
> GPCM.
> We have confirmed the timing both in functionality (in the debugger) and 
> with a logic analyzer (plus oscope for setup and hold times).
> We have stand-alone C-code that runs in CodeWarrior that bangs away at 
> the registers of this device and the hardware runs perfectly using the 
> .cfg script settings..

Great.

>> 3. Have you created a bus functional model of the processor bus
>>    that you have then run with your FPGA bus in ModelSim to
>>    confirm that your FPGA performs correctly. 
> 
> Yes, we have.  Also, checked in h/w (see 2 above)

Nice :)

>> 4. Have you tried burst and non-burst access by either using
>>    DMA, or treating the memory area as cacheable or non-cacheable?
>>    Have you checked those cases with simulation and then
>>    with a scope or logic analyzer?
> 
> Always non-burst at this point.

Are you sure?

When I was playing with the Yosemite board, I was having trouble
with a PCI card due to caching ... here's a code snippet from
the mmap routine of a driver:


	/* Flags to stop the processor treating PCI memory as
	 * cacheable (see asm-ppc/pgtable.h)
	 *
	 * (thanks to Travis Sawyer from the ppc-embedded list)
	 *
	 * I could have used '#ifdef CONFIG_44x', but 40x uses
	 * these flags too, as do other processors. So just check
	 * whether the flag exists.
	 *
	 * TODO:
	 * p425 Rubini; use pgprot_noncached()
	 *
	 * asm-ppc/pgtable.h defines it as setting these two flags
	 *
	 * So, that appears to be the 'portable' way to do it.
	 *
	 * drivers/char/mem.c uses pgprot_noncached()
	 */
#ifdef pgprot_noncached
	vma->vm_page_prot = pgprot_noncached(vma->vm_page_prot);
#endif

In that particular case, I think I wasn't seeing data transferred
to the bus until a cache flush occurred. Its possible that in your
case a burst is triggered to your hardware.

How does your BFM+simulation react to bus bursts?

Can you probe the bus when you trigger the seg-fault to see
if caching is the problem?

I'm not sure that a bus error can cause a segfault. When I
get my boards, I'll generate bus parity errors to see what
kernel routines it triggers. So, chances are this is not
the source of your error.

>> 5. Did you try running stand-alone tests in U-Boot, to go for a
>>    more bare-metal debug approach?
> 
> I assume the bare-metal debug approach is using a JTAG debugger 
> connection and CodeWarrior.  We started there and now are trying to 
> access the same device(s) via linux drivers.

Yep, sounds good.

Ok, so assuming you come back with the fact that the seg-fault
occurs without bursting. Lets take a look at your driver code.

Cheers,
Dave

Re: mmap + segfaults on MPC8349E

[R. Ebersole (VTI - new)]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]


Dave,

Attached is the driver code that was in use when we had the problem.
(FPGA_orig.7z)

We think that we *may* have resolved the issue.
We have not, yet, completed testing the resolution, however.

Attached is the updated version of the driver (FPGArw.7z).
So far, this version appears OK when using mmap() without
MAP_FIXED.  We open() the device O_RDWR | O_SYNC.

We were going to switch to using pread(), pwrite(), and ioctl() to
access the FPGA's registers, but we are going to hold off on that
for now.



David Hawkins wrote:

> Hi,
>
> Ok, so assuming you come back with the fact that the seg-fault
> occurs without bursting. Lets take a look at your driver code.
>
> Cheers,
> Dave
>
>
>

-- 

Sometimes I feel like a red shirt in the Star Trek episode of life.

--

This message contains confidential information and is intended only for the
individual named.  If you are not the intended recipient you should not
disseminate, distribute or copy this e-mail.  Please notify the sender
immediately by e-mail if you have received this e-mail by mistake and delete
this e-mail from your system.


[-- Attachment #2: FPGArw.7z --]
[-- Type: application/octet-stream, Size: 3775 bytes --]

[-- Attachment #3: FPGA_orig.7z --]
[-- Type: application/octet-stream, Size: 2544 bytes --]

Re: mmap + segfaults on MPC8349E

[David Hawkins]

Hi,

> Attached is the driver code that was in use when we had the problem.
> (FPGA_orig.7z)
> 
> We think that we *may* have resolved the issue.
> We have not, yet, completed testing the resolution, however.
> 
> Attached is the updated version of the driver (FPGArw.7z).
> So far, this version appears OK when using mmap() without
> MAP_FIXED.  We open() the device O_RDWR | O_SYNC.
> 
> We were going to switch to using pread(), pwrite(), and ioctl() to
> access the FPGA's registers, but we are going to hold off on that
> for now.

I had a look at a few of the user-space examples I have written
for accessing via mmap() for my own drivers, and the standard
/dev/mem driver. I've include a generic /dev/mem interface program
that I've used on a TS-7300 board (ARM-based board from Technologic
Systems). Its inline below. It should work fine on a PowerPC too.

My example code has always used open with O_RDWR and in some
cases (O_RDWR | O_SYNC) (as in the code below). The call to
mmap() uses prot set to (PROT_READ|PROT_WRITE) to match the open
mode, and for the flags MAP_SHARED, never MAP_FIXED.

 From the man page for mmap()

http://linux.die.net/man/2/mmap

   MAP_FIXED
     Do not select a different address than the one specified.
     If the memory region specified by start and len overlaps pages
     of any existing mapping(s), then the overlapped part of the
     existing mapping(s) will be discarded. If the specified address
     cannot be used, mmap() will fail. If MAP_FIXED is specified,
     start must be a multiple of the page size. Use of this option
     is discouraged.

So I'm not sure why you would choose this option.

I can't say what problems selecting this option would create, as I've
never tried. It would take some walking through the kernel code to
see where this flag is used.

Did you really see a difference between the driver code versions, or
was it changing MAP_FIXED to MAP_SHARED that did the trick?

If you test with the /dev/mem driver below and have no problems,
then I would recommend using the /dev/mem mmap() routine in your
custom driver.

To track down which code change causes the SEGFAULT would require
finding out the source of the SEGFAULT. So if you can code up
something that consistently causes the error, then you should be
able to debug the kernel to see what is causing it. I haven't had
to debug this sort of thing, so can't offer much advice. If its
hard to trigger the error, then do something deliberate; generate
a parity error (eg. by reading a specific FPGA register), generate
a bus timeout, etc. If that has the same effect on your code, then
perhaps the driver can be instrumented to give you a little more
debug information.

Cheers,
Dave



/*
  * mem_debug.c
  *
  * 5/10/07 D. W. Hawkins (dwh@ovro.caltech.edu)
  *
  * A debug console for read/write access to /dev/mem mapped
  * areas.
  */
#include <stdio.h>
#include <errno.h>
#include <string.h>
#include <fcntl.h>
#include <sys/ioctl.h>
#include <signal.h>
#include <sys/mman.h>
#include <stdlib.h>
#include <unistd.h>

void display_help();
void change_mem(char *cmd);
void display_mem(char *cmd);
void process_command(char *cmd);

/* Mapped address and size */
static char        *mem_addr  = NULL;
static unsigned int mem_phys  = 0;
static unsigned int mem_pages = 0;
static unsigned int mem_size = 0;

static void show_usage()
{
	printf("\nUsage: mem_debug -a <address> -n <pages>\n"\
		 "  -h            Help (this message)\n"\
		 "  -a <address>  Hex address to start the map\n"\
		 "  -n <pages>    Number of pages to map\n\n");
}

int main(int argc, char *argv[])
{
	int opt;       /* getopt processing         */
	int fd;        /* /dev/mem file descriptor  */
	char buf[200]; /* command processing buffer */
	int len = 200;
	int page_size = getpagesize();
	int status;

	while ((opt = getopt(argc, argv, "a:hn:")) != -1) {
		switch (opt) {
			case 'a':
				status = sscanf(optarg, "%X", &mem_phys);
				if (status != 1) {
					printf("Parse error for -a option\n");
					show_usage();
					return -1;
				}
				break;
			case 'h':
				show_usage();
				return -1;
			case 'n':
				mem_pages = atoi(optarg);
				break;
			default:
				show_usage();
				return -1;
		}
	}
	if (mem_phys != (mem_phys/page_size)*page_size) {
		printf("Error: the address must be page-aligned (0x%X)\n",
				page_size);
		return -1;
	}
	if (mem_pages == 0) {
		mem_pages = 1;
	}
	mem_size = mem_pages*page_size;

	/* Startup */
	printf("\n------------------------\n");
	printf("/dev/mem debug interface\n");
	printf("========================\n\n");

	/* Open /dev/mem and map it */
	printf(" * open /dev/mem\n");
	fd = open("/dev/mem", O_RDWR | O_SYNC);
	if (fd < 0) {
		printf("Open /dev/mem failed - %s\n",
			strerror(errno));
		return -1;
	}
	printf(" * map %d page(s) (%d-bytes) at address 0x%.8X\n",
			mem_pages, mem_size, mem_phys);
	mem_addr = (char *)mmap(
		0,
		mem_size,
		PROT_READ|PROT_WRITE,
		MAP_SHARED,
		fd,
		mem_phys);
	if (mem_addr == (char *)MAP_FAILED) {
		printf("Error: mmap failed\n");
		close(fd);
		return -1;
	}

	/* Display help */
	display_help();

	/* Process commands */
	while (1) {
		printf("CMD> ");
		fflush(stdout);
		fgets(buf,len,stdin);
		process_command(buf);
	}

	/* Cleanup */
	munmap((void *)mem_addr, mem_pages*page_size);
	close(fd);
	return 0;
}

/*--------------------------------------------------------------------
  * User interface
  *--------------------------------------------------------------------
  */
void display_help()
{
	printf("\n  ?              Help\n");
	printf("  dw addr len    Display len words starting from  addr\n");
	printf("  db addr len    Display len bytes starting from addr\n");
	printf("  cw addr val    Change word at addr to val\n");
	printf("  cb addr val    Change byte at addr to val\n");
	printf("  q              Quit\n");
	printf("\n  Notes:\n");
	printf("    * addr, len, and val are interpreted as hex values\n");
	printf("    * addresses are always byte based\n");
	printf("    * addresses are offsets relative to the base address\n\n");
}

void process_command(char *cmd)
{
	if (cmd[0] == '\0') {
		return;
	}
	switch (cmd[0]) {
		case '?':
			display_help();
			break;
		case 'd':
		case 'D':
			display_mem(cmd);
			break;
		case 'c':
		case 'C':
			change_mem(cmd);
			break;
		case 'q':
		case 'Q':
			exit(0);
		default:
			break;
	}

	return;
}

void display_mem(char *cmd)
{
	char width = 0;
	int addr = 0;
	int len = 0;
	int status;
	int i, data;

	/* d, db, dw */
	status = sscanf(cmd, "%*c%c %x %x", &width, &addr, &len);
	if (status != 3) {
		printf("syntax error (use ? for help)\n");
		return;
	}
	if (len > mem_size) {
		len = mem_size;
	}
	/* Convert to offset if required */
	if (addr >= mem_phys) {
		addr -= mem_phys;
	}
	if (addr >= mem_size) {
		printf("Illegal address\n");
		return;
	}
	switch (width) {
		case 'b':
			for (i = 0; i < len; i++) {
				if ((i%16) == 0) {
					printf("\n%.8X: ", mem_phys + addr + i);
				}
				data = (int)(mem_addr[addr+i]) & 0xFF;
				printf("%.02X ", data);
			}
			printf("\n");
			break;
		default:
			for (i = 0; i < len; i+=4) {
				if ((i%16) == 0) {
					printf("\n%.8X: ", mem_phys + addr + i);
				}
				data = *(int *)(mem_addr + addr + i);
				printf("%.08X ", data);
			}
			printf("\n");
			break;
	}
	printf("\n");
	return;
}

void change_mem(char *cmd)
{
	char width = 0;
	int addr = 0;
	int data = 0;
	int status;

	/* c, cb, cw */
	status = sscanf(cmd, "%*c%c %x %x", &width, &addr, &data);
	if (status != 3) {
		printf("syntax error (use ? for help)\n");
		return;
	}
	/* Convert to offset if required */
	if (addr >= mem_phys) {
		addr -= mem_phys;
	}
	if (addr >= mem_size) {
		printf("Illegal address\n");
		return;
	}
	switch (width) {
		case 'b':
			mem_addr[addr] = data & 0xFF;
			break;
		default:
			*(int *)(mem_addr + addr) = data;
			break;
	}
	return;
}