Special handling of sysfs device resource files?

Special handling of sysfs device resource files?

[Ian Romanick]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm in the process of modifying X to be civilized in it's handling of
PCI devices on Linux.  As part of that, I've modified it to map the
/sys/bus/pci/device/*/resource[0-6] files instead of mucking about with
/dev/mem.

This seems to mostly work, but I am having one problem.  I map the
region by opening the file with O_RDWR, then mmap with
(PROT_READ|PROT_WRITE) and MAP_SHARED.  In all cases, the open and mmap
succeed.  However, for I/O BARs, the resulting pointer from mmap is
invalid.  Any access to it results in a segfault and GDB says it's "out
of range".

The base address of the BAR is page aligned, so its not a problem with
the alignment of mmap vs. the alignment of the BAR.  What else could it
be?  I'm pretty stumped.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (GNU/Linux)

iD8DBQFEPB7KX1gOwKyEAw8RAhG/AJ4x+Vjl8V9SNeyMhYe2txeAeKALKACePCwL
6s0kj4YhDY3/thVh6mvO5X4=
=g9eu
-----END PGP SIGNATURE-----

Re: Special handling of sysfs device resource files?

[Ian Romanick]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Ian Romanick wrote:
> I'm in the process of modifying X to be civilized in it's handling of
> PCI devices on Linux.  As part of that, I've modified it to map the
> /sys/bus/pci/device/*/resource[0-6] files instead of mucking about with
> /dev/mem.
> 
> This seems to mostly work, but I am having one problem.  I map the
> region by opening the file with O_RDWR, then mmap with
> (PROT_READ|PROT_WRITE) and MAP_SHARED.  In all cases, the open and mmap
> succeed.  However, for I/O BARs, the resulting pointer from mmap is
> invalid.  Any access to it results in a segfault and GDB says it's "out
> of range".

I was a little mistaken about this.  The BAR that causes the problem is
not I/O.  It *is* memory.

01:00.0 VGA compatible controller: Matrox Graphics, Inc. G400/G450 (rev
03) (prog-if 00 [VGA])
        Subsystem: Matrox Graphics, Inc. Millennium G400 16Mb SGRAM
        Flags: bus master, medium devsel, latency 64, IRQ 11
        Memory at cc000000 (32-bit, prefetchable) [size=32M]
        Memory at cfefc000 (32-bit, non-prefetchable) [size=16K]
        Memory at cf000000 (32-bit, non-prefetchable) [size=8M]
        Expansion ROM at cfee0000 [disabled] [size=64K]

When I open and mmap resource0 (the framebuffer) I get 0x2b9aa48ea000.
When I open and mmap resource1 (the card's registers) I get
0x2b9aa68ea000.  I can access the resource0 pointer all day long without
problems.  The firs access to the resource1 pointer results in a segfault.

> The base address of the BAR is page aligned, so its not a problem with
> the alignment of mmap vs. the alignment of the BAR.  What else could it
> be?  I'm pretty stumped.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (GNU/Linux)

iD8DBQFEPELpX1gOwKyEAw8RAjT+AJ0ZzDb49tr5WwKWE7eyKWdT7hRLUQCgkOSS
twDrsx8VrWG5xEf+hbbkFvg=
=im+D
-----END PGP SIGNATURE-----

Re: Special handling of sysfs device resource files?

[Paul Mackerras]

Ian Romanick writes:

> This seems to mostly work, but I am having one problem.  I map the
> region by opening the file with O_RDWR, then mmap with
> (PROT_READ|PROT_WRITE) and MAP_SHARED.  In all cases, the open and mmap
> succeed.  However, for I/O BARs, the resulting pointer from mmap is
> invalid.  Any access to it results in a segfault and GDB says it's "out
> of range".

On which architecture(s)?

Paul.

Re: Special handling of sysfs device resource files?

[Ian Romanick]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Paul Mackerras wrote:
> Ian Romanick writes:
> 
>>This seems to mostly work, but I am having one problem.  I map the
>>region by opening the file with O_RDWR, then mmap with
>>(PROT_READ|PROT_WRITE) and MAP_SHARED.  In all cases, the open and mmap
>>succeed.  However, for I/O BARs, the resulting pointer from mmap is
>>invalid.  Any access to it results in a segfault and GDB says it's "out
>>of range".
> 
> On which architecture(s)?

I've only tried on x86-64 (an Athlon64 3000+ to be exact) so far.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.1 (GNU/Linux)

iD8DBQFEPRd4X1gOwKyEAw8RAhdxAKCX9Y/ju2jK7q5Wdzmb9kfwt9T3lwCbBwrq
BIRJkEIGqSOrXtkDC3/2Fts=
=mePG
-----END PGP SIGNATURE-----

Re: Special handling of sysfs device resource files?

[Jesse Barnes]

On Tuesday, April 11, 2006 4:59 pm, Ian Romanick wrote:
> Ian Romanick wrote:
> > I'm in the process of modifying X to be civilized in it's handling
> > of PCI devices on Linux.  As part of that, I've modified it to map
> > the /sys/bus/pci/device/*/resource[0-6] files instead of mucking
> > about with /dev/mem.
> >
> > This seems to mostly work, but I am having one problem.  I map the
> > region by opening the file with O_RDWR, then mmap with
> > (PROT_READ|PROT_WRITE) and MAP_SHARED.  In all cases, the open and
> > mmap succeed.  However, for I/O BARs, the resulting pointer from
> > mmap is invalid.  Any access to it results in a segfault and GDB
> > says it's "out of range".
>
> I was a little mistaken about this.  The BAR that causes the problem
> is not I/O.  It *is* memory.
>
> 01:00.0 VGA compatible controller: Matrox Graphics, Inc. G400/G450
> (rev 03) (prog-if 00 [VGA])
>         Subsystem: Matrox Graphics, Inc. Millennium G400 16Mb SGRAM
>         Flags: bus master, medium devsel, latency 64, IRQ 11
>         Memory at cc000000 (32-bit, prefetchable) [size=32M]
>         Memory at cfefc000 (32-bit, non-prefetchable) [size=16K]
>         Memory at cf000000 (32-bit, non-prefetchable) [size=8M]
>         Expansion ROM at cfee0000 [disabled] [size=64K]
>
> When I open and mmap resource0 (the framebuffer) I get 0x2b9aa48ea000.
> When I open and mmap resource1 (the card's registers) I get
> 0x2b9aa68ea000.  I can access the resource0 pointer all day long
> without problems.  The firs access to the resource1 pointer results in
> a segfault.

Hm, that's strange.  On my via machine I can access all my VGA resources 
correctly, but that's not x86-64.  Maybe the x86-64 implementation of 
pci_mmap_page_range is doing the wrong thing for uncacheable, non-WC 
regions somehow?

Jesse

Re: Special handling of sysfs device resource files?

[Jesse Barnes]

[-- Attachment #1: Type: text/plain, Size: 1090 bytes --]

On Tuesday, April 11, 2006 4:59 pm, Ian Romanick wrote:
> I was a little mistaken about this.  The BAR that causes the problem
> is not I/O.  It *is* memory.
>
> 01:00.0 VGA compatible controller: Matrox Graphics, Inc. G400/G450
> (rev 03) (prog-if 00 [VGA])
>         Subsystem: Matrox Graphics, Inc. Millennium G400 16Mb SGRAM
>         Flags: bus master, medium devsel, latency 64, IRQ 11
>         Memory at cc000000 (32-bit, prefetchable) [size=32M]
>         Memory at cfefc000 (32-bit, non-prefetchable) [size=16K]
>         Memory at cf000000 (32-bit, non-prefetchable) [size=8M]
>         Expansion ROM at cfee0000 [disabled] [size=64K]
>
> When I open and mmap resource0 (the framebuffer) I get 0x2b9aa48ea000.
> When I open and mmap resource1 (the card's registers) I get
> 0x2b9aa68ea000.  I can access the resource0 pointer all day long
> without problems.  The firs access to the resource1 pointer results in
> a segfault.

Just tested on my x86-64 machine with this dumb little test program.  It 
seems to work ok, though I haven't tried writing data to the resource.

Jesse

[-- Attachment #2: mapdump.c --]
[-- Type: text/x-csrc, Size: 962 bytes --]

#include <errno.h>
#include <fcntl.h>
#include <stdint.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>

int main(int argc, char *argv[])
{
	size_t len;
	int fd, i;
	void *ptr;
	uint32_t *val;

	if (argc != 3) {
		fprintf(stderr, "usage: %s <file> <mapsize>\n",
			argv[0]);
		return -1;
	}

	len = atoi(argv[2]);

	fd = open(argv[1], O_RDONLY);
	if (fd == -1) {
		fprintf(stderr, "open failed: %s\n", strerror(errno));
		return errno;
	}

	ptr = mmap(NULL, len, PROT_READ, MAP_SHARED, fd, 0);
	if (ptr == MAP_FAILED) {
		fprintf(stderr, "mmap failed: %s\n", strerror(errno));
		return errno;
	}

	val = ptr;
	len = len / sizeof(uint32_t);
	for (i = 0; i < len; i += 4) {
		printf("%08x: 0x%08x 0x%08x 0x%08x 0x%08x\n", i * 4, val[i],
		       val[i+1], val[i+2], val[i+3]);
	}

	munmap(ptr, len); /* ignore any errors, we don't care */
	close(fd);
	return 0;
}