Re: PCI resource unavailable on mips

Re: PCI resource unavailable on mips

[Ralf Baechle]

On Thu, Dec 13, 2007 at 09:56:46AM -0500, Jon Dufresne wrote:

> I've done a bit of linux driver development on x86 in the past.
> Currently I am working on my first ever linux driver for a mips box. I
> started by testing the device in an x86 box and got it reasonable stable
> and am now testing it in the mips box. There appears to be a major
> problem, one unlike I have ever seen before.
> 
> My PCI device has three BARS. This can be confirmed by the Technical
> documentation and the x86 code. When the pci device is first probed, I
> run a loop to printk out the bar information, this is just as a sanity
> check. Here is the output on the x86:
> 
> Bar0:PHYS=e0000000 LEN=04000000
> Bar1:PHYS=efa00000 LEN=00200000
> Bar2:PHYS=e8000000 LEN=04000000
> 
> but here is the output on the mips:
> Bar0:PHYS=20000000 LEN=04000000
> Bar1:PHYS=24000000 LEN=00200000
> Bar2:PHYS=00000000 LEN=00000000
> 
> notice, BAR2 has no valid information on the mips. I tried to run
> "pci_enable_device" before printing this information, as suggested by
> LDD but it did not help.

Resources are assigned on bootup by MIPS, not yet by pci_enable_device,
so that was expected.

> Has anyone seen a problem like this before and any idea how I can get
> BAR2 a proper address?
> 
> If I examine the config space directly there is an address in BAR2's
> register, however it isn't in the 0x20000000 range like the other two,
> instead it is 0x1c000000. Also if I do a ``cat /proc/iomem'' I correctly
> see BAR0 and BAR1 in the output, but not BAR2.

Odd.  I knew the resource allocation stuff has it's issues for some
non-trivial configuration but that one is a new one.  Which makes me
wonder if your platform runs the PCI code in probe-only mode where it
will not actually assign resources but only inherit the whole PCI setup
except interrupt routing from the firmware.

What MIPS platform do you use?  I'd like to take a look at its PCI setup
code.

  Ralf

Re: PCI resource unavailable on mips

[Jon Dufresne]

> 
> Odd.  I knew the resource allocation stuff has it's issues for some
> non-trivial configuration but that one is a new one.  Which makes me
> wonder if your platform runs the PCI code in probe-only mode where it
> will not actually assign resources but only inherit the whole PCI setup
> except interrupt routing from the firmware.
> 
> What MIPS platform do you use?  I'd like to take a look at its PCI setup
> code.
> 

I am using the MDS 810 STB as provided by MDS
(http://www.mds.com/products/product.asp?prod=MDS-810). The kernel and
entire environment (except my driver) was set up by MDS. uname output is
as follows.

 # uname -a
Linux 10.41.13.87 2.6.19PNX8550 #1 Wed Nov 21 14:55:52 EST 2007 mips
unknown

If I can provide additional information for you I'd be happy to help.

Thanks,
Jon

Re: PCI resource unavailable on mips

[Jon Dufresne]

> Odd.  I knew the resource allocation stuff has it's issues for some
> non-trivial configuration but that one is a new one.  Which makes me
> wonder if your platform runs the PCI code in probe-only mode where it
> will not actually assign resources but only inherit the whole PCI setup
> except interrupt routing from the firmware.


Hmm, I found more strange behavior with the bars that may or may not be
related. I wrote a function that does another sanity check. It does an
ioremap on one of the working bars, then reads one address for
correctness. This is just to confirm that iomem is working. This is what
the function looks like:

> void dump_mmio(struct pci_dev *dev)
> {
> 	unsigned long phys, size;
> 	struct resource *mem_region;
> 	void __iomem *mmio;
> 	u32 dword;
> 
> 	phys = pci_resource_start(dev, 1);
> 	size = pci_resource_len(dev, 1);
> 
> 	mem_region = request_mem_region(phys, size, MODULENAME);
> 	BUG_ON(!mem_region);
> 	mmio = ioremap_nocache(phys, size);
> 	BUG_ON(!mmio);
> 
> 	printk("******************MMIO*************************************\n");
> 	printk("mmio=0x%p phys=0x%08lx size=0x%08lx\n", mmio, phys, size);
> 	dword = ioread32(mmio + PCI_MMIO_BASE + 0x40);
> 	printk("dword=%x\n", dword);
> 	printk("***********************************************************\n");
> 
> 	iounmap(mmio);
> 	release_mem_region(phys, size);
> }

on x86 this prints out what I would expect with the correct value which
is:

> ******************MMIO*************************************
> mmio=0xf8e80000 phys=0xefa00000 size=0x00200000
> dword=54061131
> ***********************************************************

on mips however this crashes the kernel with a "Data bus error" the
exact output is below:

> ******************MMIO*************************************
> mmio=0xc0300000 phys=0x24000000 size=0x00200000
> Data bus error, epc == c0279a00, ra == c02799f4
> Oops[#1]:
> Cpu 0
> $ 0   : 00000000 10008400 c0340040 802e031c
> $ 4   : 802e031c 802e0000 00000001 8019f924
> $ 8   : 802e0000 0000020b 80320000 80320000
> $12   : 80320000 00000001 83031be6 8031c960
> $16   : 80086994 c0300000 24000000 00200000
> $20   : 802e0000 802e1ae4 c025a000 8008cde4
> $24   : 00000006 00000000                  
> $28   : 83030000 83031d20 c0288aec c02799f4
> Hi    : 00000051
> Lo    : eb851f00
> epc   : c0279a00     Tainted: P     
> ra    : c02799f4 Status: 10008403    KERNEL EXL IE 
> Cause : 1080001c
> PrId  : 00061200
> Modules linked in: tmman1700(P) mousedev usbhid usb_storage phStbFB(P) phStbFBRead(P) phStbVideoRenderer(P) phStbVe
> Process insmod (pid: 785, threadinfo=83030000, task=8109f8e8)
> Stack : c0288b28 c0300000 24000000 00200000 810b9c54 00000003 c0290000 810b9c00
>         c0288b28 c0279ae4 83031d48 00000002 00000000 00000000 810b9c00 c0288510
>         00000000 80373480 80177520 801774e4 810b9d1c c0288544 80177574 829f6066
>         810b9d1c 810b9c48 c0288544 801a3d40 810b9d1c 801a4538 81091c00 8011fc70
>         810b9d1c 810b9c48 c0288544 c0288544 801a475c 801a475c 801a2810 80168e08
>         ...
> Call Trace:[<c0279ae4>][<80177520>][<801774e4>][<80177574>][<801a3d40>][<801a4538>][<8011fc70>][<801a475c>][<801a4]
> 
> Code: 3c020004  34420040  02221021 <8c450000> 3c04c028  0200f809  24845164  3c04c028  0200f809 
> Segmentation fault

The data bus error occurs whenever I do a ioreadX on the ioremapped area. Any idea why this doesn't work on the mips?

Thanks,
Jon

Re: PCI resource unavailable on mips

[Ralf Baechle]

On Fri, Dec 14, 2007 at 04:12:15PM -0500, Jon Dufresne wrote:

> Hmm, I found more strange behavior with the bars that may or may not be
> related. I wrote a function that does another sanity check. It does an
> ioremap on one of the working bars, then reads one address for
> correctness. This is just to confirm that iomem is working. This is what
> the function looks like:
> 
> > void dump_mmio(struct pci_dev *dev)
> > {
> > 	unsigned long phys, size;
> > 	struct resource *mem_region;
> > 	void __iomem *mmio;
> > 	u32 dword;
> > 
> > 	phys = pci_resource_start(dev, 1);
> > 	size = pci_resource_len(dev, 1);
> > 
> > 	mem_region = request_mem_region(phys, size, MODULENAME);
> > 	BUG_ON(!mem_region);
> > 	mmio = ioremap_nocache(phys, size);
> > 	BUG_ON(!mmio);
> > 
> > 	printk("******************MMIO*************************************\n");
> > 	printk("mmio=0x%p phys=0x%08lx size=0x%08lx\n", mmio, phys, size);
> > 	dword = ioread32(mmio + PCI_MMIO_BASE + 0x40);
> > 	printk("dword=%x\n", dword);
> > 	printk("***********************************************************\n");
> > 
> > 	iounmap(mmio);
> > 	release_mem_region(phys, size);
> > }
> 
> on x86 this prints out what I would expect with the correct value which
> is:
> 
> > ******************MMIO*************************************
> > mmio=0xf8e80000 phys=0xefa00000 size=0x00200000
> > dword=54061131
> > ***********************************************************
> 
> on mips however this crashes the kernel with a "Data bus error" the
> exact output is below:
> 
> > ******************MMIO*************************************
> > mmio=0xc0300000 phys=0x24000000 size=0x00200000

The virtual address 0xc0300000 looks sensible and the physical address
0x24000000 is consistent with what you found in the BAR registers.  So that
all looks reasonable but that only means not obviously wrong.  So next I
wonder what the value of PCI_MMIO_BASE is ...

> > Cpu 0
> > $ 0   : 00000000 10008400 c0340040 802e031c
> > $ 4   : 802e031c 802e0000 00000001 8019f924
> > $ 8   : 802e0000 0000020b 80320000 80320000
> > $12   : 80320000 00000001 83031be6 8031c960
> > $16   : 80086994 c0300000 24000000 00200000
> > $20   : 802e0000 802e1ae4 c025a000 8008cde4
> > $24   : 00000006 00000000                  
> > $28   : 83030000 83031d20 c0288aec c02799f4
> > Hi    : 00000051
> > Lo    : eb851f00
> > epc   : c0279a00     Tainted: P     
> > ra    : c02799f4 Status: 10008403    KERNEL EXL IE 
> > Cause : 1080001c
> > PrId  : 00061200
> > Modules linked in: tmman1700(P) mousedev usbhid usb_storage phStbFB(P) phStbFBRead(P) phStbVideoRenderer(P) phStbVe
> > Process insmod (pid: 785, threadinfo=83030000, task=8109f8e8)
> > Stack : c0288b28 c0300000 24000000 00200000 810b9c54 00000003 c0290000 810b9c00
> >         c0288b28 c0279ae4 83031d48 00000002 00000000 00000000 810b9c00 c0288510
> >         00000000 80373480 80177520 801774e4 810b9d1c c0288544 80177574 829f6066
> >         810b9d1c 810b9c48 c0288544 801a3d40 810b9d1c 801a4538 81091c00 8011fc70
> >         810b9d1c 810b9c48 c0288544 c0288544 801a475c 801a475c 801a2810 80168e08
> >         ...
> > Call Trace:[<c0279ae4>][<80177520>][<801774e4>][<80177574>][<801a3d40>][<801a4538>][<8011fc70>][<801a475c>][<801a4]
> > 
> > Code: 3c020004  34420040  02221021 <8c450000> 3c04c028  0200f809  24845164  3c04c028  0200f809 
> > Segmentation fault
> 
> The data bus error occurs whenever I do a ioreadX on the ioremapped area. Any idea why this doesn't work on the mips?

A bus error is an exception which is signalled by agent external (often
called system controller) to the CPU core to signal a fatal error during a
read or write bus transaction, for example after a bus timeout or if the
address of the read/write isn't assigned to any device.  PCI master abort
also is often mapped to a bus error exception.

  Ralf

Re: PCI resource unavailable on mips

[Jon Dufresne]

I did a bit more work and investigation on this and it turns out I could
not read the mmio in kernel space because I had not done a
pci_enable_device_bars() on the device. I had never done this on x86 so
I didn't realize it was necessary.

> The virtual address 0xc0300000 looks sensible and the physical address
> 0x24000000 is consistent with what you found in the BAR registers.  So that
> all looks reasonable but that only means not obviously wrong.  So next I
> wonder what the value of PCI_MMIO_BASE is ...

The PCI_MMIO_BASE is a defined as:


> #define PCI_MMIO_BASE            (0x00040000)

This is define in the technical documentation as the offset to access
pci config space from the mmio. I am using this because I know what the
values should be so it provides a nice sanity check.


> A bus error is an exception which is signalled by agent external (often
> called system controller) to the CPU core to signal a fatal error during a
> read or write bus transaction, for example after a bus timeout or if the
> address of the read/write isn't assigned to any device.  PCI master abort
> also is often mapped to a bus error exception.

So after doing pci_enable_bars() I can now access this mmio region in
kernel space. However, if I try to mmap this into user space I still
receive the bus error. I am mapping this into user space using the
example for LDD which says to use the remap_pfn_range() function. I've
tested this on the x86 and it works as expected, however every time I
access the mmio from user space using the mips, I continue to get the
bus error I previously received in kernel space.

Any idea what I might be doing wrong? How can I access this from user
space?

Thanks,
Jon

Re: PCI resource unavailable on mips

[Sergei Shtylyov]

Hello.

Jon Dufresne wrote:

> I did a bit more work and investigation on this and it turns out I could
> not read the mmio in kernel space because I had not done a
> pci_enable_device_bars() on the device. I had never done this on x86 so
> I didn't realize it was necessary.

>>The virtual address 0xc0300000 looks sensible and the physical address
>>0x24000000 is consistent with what you found in the BAR registers.  So that
>>all looks reasonable but that only means not obviously wrong.  So next I
>>wonder what the value of PCI_MMIO_BASE is ...

> The PCI_MMIO_BASE is a defined as:

>>#define PCI_MMIO_BASE            (0x00040000)

> This is define in the technical documentation as the offset to access
> pci config space from the mmio.

    From what mmio? If it's for accessing a config. space why then you're 
using it as an offset from BAR?

> I am using this because I know what the
> values should be so it provides a nice sanity check.

> Any idea what I might be doing wrong? How can I access this from user
> space?

    Your example doesn't make sense to me so far.

> Thanks,
> Jon

WBR, Sergei

Re: PCI resource unavailable on mips

[Jon Dufresne]

>     Your example doesn't make sense to me so far.

Ok, I simplified my driver down to one small C file that does exactly
what I want, and that is it. Below is my driver under "driver.c" and the
user space program I am using to access it under "user-test.c".

When I insmod this driver under mips, it correctly prints out the PCI
config space by accessing it through the chip's mmio (which is provided
by BAR1), using the offset 0x00040000 as described in the technical
manual. This works correctly. This is the block of memory you see
printed out in the dmesg below.

When I run the user space program, however, I try to print out the same
data through the mmap. This causes a Bus Error which I put the output of
below.

I am very confused by this. I can read the memory in kernel space, but
not user space. This driver works as expected under x86, which only
confuses me further.

Any ideas what I am doing wrong?

I've been starring at this for quite some time. Please feel free to give
me any and all critiques on my code, it will only help towards solving
this problem.

Thanks for the help!

------driver.c----------
#include <linux/module.h>
#include <linux/fs.h>
#include <linux/cdev.h>
#include <linux/pci.h>

#define MODULE_NAME "tmexp"
#define PCI_DEVICE_ID_PHILIPS_PNX1700 0x5406

#define MMIO_PCI_BASE 0x00040000


static int pnx1700_mmap(struct file *filp, struct vm_area_struct *vma);

static int pnx1700_probe(struct pci_dev *dev, const struct pci_device_id
*id);
static void pnx1700_remove(struct pci_dev *dev);


struct pnx1700 {
	dev_t devno;
	struct cdev cdev;

	struct pci_dev *pci_dev;

	void __iomem *mmio;
};

struct pnx1700 tmexp;

static struct pci_device_id pnx1700_ids[] = {
	{ PCI_DEVICE(PCI_VENDOR_ID_PHILIPS, PCI_DEVICE_ID_PHILIPS_PNX1700) },
	{ 0, }
};

static struct pci_driver pnx1700_pci_driver = {
	.name = MODULE_NAME,
	.id_table = pnx1700_ids,
	.probe = pnx1700_probe,
	.remove = pnx1700_remove,
};

static struct file_operations pnx1700_fops = {
	.owner = THIS_MODULE,
	.mmap = pnx1700_mmap,
};

static int pnx1700_mmap(struct file *filp, struct vm_area_struct *vma)
{
	unsigned long off = vma->vm_pgoff << PAGE_SHIFT;
	unsigned long physical = pci_resource_start(tmexp.pci_dev, 1) + off;
	unsigned long vsize = vma->vm_end - vma->vm_start;
	unsigned long psize = pci_resource_len(tmexp.pci_dev, 1) - off;

	BUG_ON(vsize > psize);
	return io_remap_pfn_range(vma, vma->vm_start,
			physical >> PAGE_SHIFT,
			vsize, vma->vm_page_prot);
}

static void dump_pci_mmio(void __iomem *mmio)
{
	int i;
	u8 mem[16];
	for(i = 0; i < 0x100; i += 0x10) {
		memcpy_fromio(mem, mmio + MMIO_PCI_BASE + 0x40 + i, 16);
		printk("%02x: %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %
02x %02x %02x %02x %02x\n",
			i, mem[0], mem[1], mem[2], mem[3], mem[4], mem[5], mem[6], mem[7],
mem[8], mem[9], mem[10], mem[11], mem[12], mem[13], mem[14], mem[15]);
	}
}

static int pnx1700_probe(struct pci_dev *dev, const struct pci_device_id
*id)
{
	int err;
	struct resource *resource;
	unsigned long start = pci_resource_start(dev, 1);
	unsigned long len = pci_resource_len(dev, 1);

	tmexp.pci_dev = dev;

	err = pci_enable_device(dev);
	BUG_ON(err);
	err = pci_enable_device_bars(dev, 0x7);
	BUG_ON(err);

	resource = request_mem_region(start, len, MODULE_NAME);
	BUG_ON(!resource);
	tmexp.mmio = ioremap(start, len);
	BUG_ON(!tmexp.mmio);
	dump_pci_mmio(tmexp.mmio);

	return 0;
}

static void pnx1700_remove(struct pci_dev *dev)
{
	unsigned long start = pci_resource_start(dev, 1);
	unsigned long len = pci_resource_len(dev, 1);

	iounmap(tmexp.mmio);
	release_mem_region(start, len);
}

static int __init tmexp_init(void)
{
	int err;

	memset(&tmexp, 0, sizeof(struct pnx1700));

	err = pci_register_driver(&pnx1700_pci_driver);
	BUG_ON(err);

	alloc_chrdev_region(&tmexp.devno, 0, 1, MODULE_NAME);
	cdev_init(&tmexp.cdev, &pnx1700_fops);
	tmexp.cdev.owner = THIS_MODULE;
	err = cdev_add(&tmexp.cdev, tmexp.devno, 1);
	BUG_ON(err);
	return 0;
}

static void __exit tmexp_exit(void)
{
	cdev_del(&tmexp.cdev);
	unregister_chrdev_region(tmexp.devno, 1);
	pci_unregister_driver(&pnx1700_pci_driver);
}

module_init(tmexp_init);
module_exit(tmexp_exit);
----------------


-------user-test.c----------
#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>
#include <stdio.h>
#include <stdint.h>
#include <string.h>

#define PCI_MMIO_BASE            (0x00040000)

static void dump_mem(void const *src, int offset, int size);
static void sanity_check(void const *mmio);

int main(int argc, char *argv[])
{
	int fd;
	void *mmio;


	fd = open("/dev/pnx1700", O_RDWR);
	mmio = mmap(0, 0x00200000, PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);

	printf("stuff mapped ready to test\n");
	sanity_check(mmio);
	dump_mem(mmio, PCI_MMIO_BASE + 0x40, 0x100);

	munmap(mmio, 0x00200000);
	close(fd);
	return 0;
}

static void dump_mem(void const *src, int offset, int size)
{
	int i;
	unsigned char mem[16];

	for(i = 0; i < size; i += 0x10) {
		memcpy(mem, src + offset + i, 16);
		printf("%02x: %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %02x %
02x %02x %02x %02x %02x\n",
			i, mem[0], mem[1], mem[2], mem[3], mem[4], mem[5], mem[6], mem[7],
mem[8], mem[9], mem[10], mem[11], mem[12], mem[13], mem[14], mem[15]);
	}
}

static void sanity_check(void const *mmio)
{
	uint32_t dword;

	dword = *((uint32_t *)(mmio + PCI_MMIO_BASE + 0x40));
	printf("dword=0x%x\n", dword);
}
-------------


-----output-----
 # ./load_driver 
PCI: Enabling device 0000:00:0c.0 (0000 -> 0002)
00: 31 11 06 54 02 00 90 82 00 00 80 04 00 00 00 00
10: 08 00 00 20 00 00 00 24 00 00 00 1c 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 36 11 17 00
30: 00 00 00 00 40 00 00 00 00 00 00 00 00 01 09 18
40: 01 00 02 00 00 00 00 00 00 00 00 00 03 02 07 00
50: 06 18 00 00 00 00 00 00 00 00 00 00 00 00 00 00
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
10.41.1.162 # ./user-test 
stuff mapped reaData bus error, epc == 00400a24, ra == 004007f4
dy to test
Bus error
10.41.1.162 # dmesg
<4>PCI: Enabling device 0000:00:0c.0 (0000 -> 0002)
<4>00: 31 11 06 54 02 00 90 82 00 00 80 04 00 00 00 00
<4>10: 08 00 00 20 00 00 00 24 00 00 00 1c 00 00 00 00
<4>20: 00 00 00 00 00 00 00 00 00 00 00 00 36 11 17 00
<4>30: 00 00 00 00 40 00 00 00 00 00 00 00 00 01 09 18
<4>40: 01 00 02 00 00 00 00 00 00 00 00 00 03 02 07 00
<4>50: 06 18 00 00 00 00 00 00 00 00 00 00 00 00 00 00
<4>60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
<4>70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
<4>80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
<4>90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
<4>a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
<4>b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
<4>c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
<4>d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
<4>e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
<4>f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
<1>Data bus error, epc == 00400a24, ra == 004007f4
-----------

Re: PCI resource unavailable on mips

[Jon Dufresne]

> > Bar0:PHYS=e0000000 LEN=04000000
> > Bar1:PHYS=efa00000 LEN=00200000
> > Bar2:PHYS=e8000000 LEN=04000000
> 
> So, two 64MB BARs and a 2MB one?

That is right.

> Any PCI resource allocation errors in dmesg during the boot process? 
> Could be the kernel wasn't able to find a place to map all of the BARs.
> 

I went back and looked at the boot up messages and I found this:

PCI: Failed to allocate mem resource #2:4000000@24000000 for
0000:00:0c.0


That is my device. So it does appear that there is an allocation issue.
Any idea how to get around this?

Thanks,
Jon