uncached user space mapping with mmap() ???

uncached user space mapping with mmap() ???

[Steven Scholz]

Hi there,

I am trying to access a hardware timer implemented in an FPGA from
user space. I implemented a simple mmap() functionality (taken from
Runbini's Linux Device Driver)

int simple_remap_mmap(struct file *filp, struct vm_area_struct *vma)
{
      unsigned long offset = vma->vm_pgoff << PAGE_SHIFT;

      if (offset >= __pa(high_memory) || (filp->f_flags & O_SYNC)) {
          vma->vm_flags |= VM_IO;
      }
      vma->vm_flags |= VM_RESERVED;

      if (remap_page_range(vma->vm_start, offset,
	vma->vm_end-vma->vm_start, vma->vm_page_prot))
	        return -EAGAIN;

      return 0;
}

I just learned that

 > getting an uncached user space mapping is architecture dependent.
 > On ARM, however, passing pgprot_noncached(vma->vm_page_prot) to
 > remap_page_range() will alter the page protections such that the
 > mapping will be uncached.

How could I do this on an MPC8xx?

Thanks a million!


--
Steven Scholz

imc Measurement & Control               imc Meßsysteme GmbH
Voltastr. 5                             Voltastr. 5
13355 Berlin                            13355 Berlin
Germany                                 Deutschland


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: uncached user space mapping with mmap() ???

[Dan Malek]

Steven Scholz wrote:

> How could I do this on an MPC8xx?

Open /dev/mem and just map the physical address of your
registers.  You don't even need your own driver.  If you
are mapping outside of the range of real memory, the mem
driver will automatically give you uncached access.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: uncached user space mapping with mmap() ???

[Steven Scholz]

Hi there,

thanks very much for all your help.
I used the stuff Stéphane suggested. But I have to ask again about
mapping of huge memory areas:

Let's say the peripheral has 32 MB address range staring at BASE. At
address GRAM 1MB of external SDRAM is located.

I use a simple test loop

	p = (unsigned short *)
		ioremap(base + GRAM, 32MB);

	START ();
	while (Retries--) {
		for (i = 0; i < Size; i++) {
			pData[i] = *p;
		}
	}
	STOP ();

When I map only the the GRAM I get a throughput of

IoremapTest...        8.0 s      => 2049.4 kW/s

But when I map the whole address range

	p = (unsigned short *)
		ioremap(BASE, imcdevif_iosize);

and move the pointer

	p += GRAM;

before entering the test loop I only get

IoremapTest...        8.4 s      => 1944.8 kW/s

How is that???

Is it always better to map only the small part I am going to use?

Thanks a million!

--
Steven Scholz

imc Measurement & Control               imc Meßsysteme GmbH
Voltastr. 5                             Voltastr. 5
13355 Berlin                            13355 Berlin
Germany                                 Deutschland


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: uncached user space mapping with mmap() ???

[Jon Masters]

Steven Scholz wrote:

| Is it always better to map only the small part I am going to use?

Of course it is always best to map only the small range that you are
going to use and there will not necessarily be only one TLB mapping (I
am here referring to your comments on ARM Linux Kernel List) etc. etc.
However here especially I am inclined to repeat what they said on the
other list - these figures are so close that they might differ for other
reasons and so on. Did you repeat this in a controlled way many times?

You will need to answer the questions Russell King posted in order for
us to work through this and figure out where the difference lies.

Cheers,

Jon.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: uncached user space mapping with mmap() ???

[Steven Scholz]

Jon Masters wrote:

> | Is it always better to map only the small part I am going to use?
>
> Of course it is always best to map only the small range that you are
> going to use and there will not necessarily be only one TLB mapping (I
> am here referring to your comments on ARM Linux Kernel List) etc. etc.
> However here especially I am inclined to repeat what they said on the
> other list - these figures are so close that they might differ for other
> reasons and so on. Did you repeat this in a controlled way many times?

Yes. The thruput is calculated after repeating the loop 1000 times
(Retries = 1000).

> You will need to answer the questions Russell King posted in order for
> us to work through this and figure out where the difference lies.
I am just about to do it.


--
Steven Scholz

imc Measurement & Control               imc Meßsysteme GmbH
Voltastr. 5                             Voltastr. 5
13355 Berlin                            13355 Berlin
Germany                                 Deutschland


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: uncached user space mapping with mmap() ???

[Jon Masters]

Steven Scholz wrote:

| Yes. The thruput is calculated after repeating the loop 1000 times
| (Retries = 1000).

That is not an accurate measurement though because you really need to
repeat this a few times on the host from a cold start under similar
conditions for people to actually see much difference in these figures -
within a single run you might well see a small delta between the two.

|> You will need to answer the questions Russell King posted in order for
|> us to work through this and figure out where the difference lies.

| I am just about to do it.

Ok.

Jon.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: uncached user space mapping with mmap() ???

[Fillod Stephane]

Hi Steven,

> I am trying to access a hardware timer implemented in an FPGA from
> user space. I implemented a simple mmap() functionality (taken from
> Runbini's Linux Device Driver)

Please refer to "11.13. Accessing peripherals from user space"
in the excellent FAQ from DENX's TWiki at :

http://www.denx.de/twiki/bin/view/PPCEmbedded/DeviceDrivers#Section_Accessin
gPeripheralsFromUserSpace


There's no need to implement kernel side stuff. /dev/mem is enough.


Regards,

--
Stéphane

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: uncached user space mapping with mmap() ???

[Steven Scholz]

Fillod Stephane wrote:

>>I am trying to access a hardware timer implemented in an FPGA from
>>user space. I implemented a simple mmap() functionality (taken from
>>Runbini's Linux Device Driver)
>
> Please refer to "11.13. Accessing peripherals from user space"
> in the excellent FAQ from DENX's TWiki at :
>
> http://www.denx.de/twiki/bin/view/PPCEmbedded/DeviceDrivers#Section_Accessin
> gPeripheralsFromUserSpace
>
> There's no need to implement kernel side stuff. /dev/mem is enough.

Oh! Nice. Thanks a million! It actually works (on my ARM board!)

So why will people need their own mmap() driver if this kind of things
can be done via /dev/mem ???

And: Can I mmap() 32MB of peripheral memory into user space?

--
Steven Scholz

imc Measurement & Control               imc Meßsysteme GmbH
Voltastr. 5                             Voltastr. 5
13355 Berlin                            13355 Berlin
Germany                                 Deutschland


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: uncached user space mapping with mmap() ???

[Eugene Surovegin]

On Thu, Mar 04, 2004 at 03:35:46PM +0100, Steven Scholz wrote:
> >There's no need to implement kernel side stuff. /dev/mem is enough.
>
> So why will people need their own mmap() driver if this kind of things
> can be done via /dev/mem ???

There are CPUs (e.g. PPC440) where you cannot use /dev/mem to
map I/O memory. In case of PPC440 it isn't in 4GB address space that
/dev/mem can access.

Eugene.

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: uncached user space mapping with mmap() ???

[Fillod Stephane]

Steven Scholz wrote:
>> Please refer to "11.13. Accessing peripherals from user space"
>> in the excellent FAQ from DENX's TWiki at :
>>
>>
http://www.denx.de/twiki/bin/view/PPCEmbedded/DeviceDrivers#Section_Accessin
>> gPeripheralsFromUserSpace
>>
>> There's no need to implement kernel side stuff. /dev/mem is enough.
>
>Oh! Nice. Thanks a million! It actually works (on my ARM board!)

Glad to hear it!

>So why will people need their own mmap() driver if this kind of things
>can be done via /dev/mem ???

For fun :)

Or weird(read clever) mappings.

>And: Can I mmap() 32MB of peripheral memory into user space?

I don't see any reason why you would not be able to.
The best answer is try it, and see for yourself.


Regards,

--
Stéphane

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: uncached user space mapping with mmap() ???

[Steven Scholz]

Fillod Stephane wrote:

>>And: Can I mmap() 32MB of peripheral memory into user space?
>
> I don't see any reason why you would not be able to.
Don't know. Maybe the number of entries in the translation tables are
limited...

--
Steven Scholz

imc Measurement & Control               imc Meßsysteme GmbH
Voltastr. 5                             Voltastr. 5
13355 Berlin                            13355 Berlin
Germany                                 Deutschland


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: uncached user space mapping with mmap() ???

[Richard Williams]

Hi,

>> How could I do this on an MPC8xx?
>
>Open /dev/mem and just map the physical address of your
>registers.  You don't even need your own driver.  If you
>are mapping outside of the range of real memory, the mem
>driver will automatically give you uncached access.

I just tried this using pread but it wont let me seek to memory locations >= 0x80000000. perror returns "Invalid argument". The seek offset is a signed long int which indicates that it was intended to be limited to less then 0x80000000.

How do you access 3rd and 4th GB memory?

Richard


-----Original Message-----
From: owner-linuxppc-embedded@lists.linuxppc.org
[mailto:owner-linuxppc-embedded@lists.linuxppc.org]On Behalf Of Dan
Malek
Sent: Friday, March 05, 2004 6:00 AM
To: Steven Scholz
Cc: LinuxPPC
Subject: Re: uncached user space mapping with mmap() ???



Steven Scholz wrote:

> How could I do this on an MPC8xx?

Open /dev/mem and just map the physical address of your
registers.  You don't even need your own driver.  If you
are mapping outside of the range of real memory, the mem
driver will automatically give you uncached access.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: uncached user space mapping with mmap() ???

[Dan Malek]

Richard Williams wrote:

> I just tried this using pread....

No, no.

Open /dev/mem and use mmap() to map your physical address:

   mmap(NULL, <page_size_bytes>, (PROT_READ|PROT_WRITE), MAP_SHARED, mem_fd, phys_addr);


Then, use the pointer returned from mmap() as the base of your
registers.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: uncached user space mapping with mmap() ???

[Fillod Stephane]

> When I map only the the GRAM I get a throughput of
>
> IoremapTest...        8.0 s      => 2049.4 kW/s

Talking about Retries count, the higher the better.
Getting 8 seconds is okay, but not less to have good average result.
Perform several times the IoremapTest to check for noise consistency.
To compare it with some other IoremapTest's, do it in identical (at best)
environement, for example at fresh bootup. But you must know all this
already.

To maximize thruput, you'll have to unroll the loop if your compiler
is not smart enough to do it. Here is an unroll 4 (provided Size%4=0):

	while (Retries--) {
		for (i = 0; i < Size; i+=4) {
			pData[i]   = *p;
			pData[i+1] = *p;
			pData[i+2] = *p;
			pData[i+3] = *p;
		}
	}

Unroll the loop 8 times or 16 times to have better result (minimize branch).
Then compare it to the spec of your bus bandwidth and memory bandwidth.

If you're chasing after high perf, it's always good to disassemble
the code to understand it (compile with -g and then objdump -S).

> But when I map the whole address range
>
>	p = (unsigned short *)
>		ioremap(BASE, imcdevif_iosize);
>
> and move the pointer
>
>	p += GRAM;
>
> before entering the test loop I only get
>
> IoremapTest...        8.4 s      => 1944.8 kW/s

This is only ~5% thruput variation afterall, so make sure you perform
serveral time the IoremapTest to check for consistency.

In the case you're moving the pointer *in* in the test loop,
the IoremapTest will also be measuring the minor page faults.

If this is an issue for you, you'll have to tweak the kernel
and use some hugetlb entries. Please ppc gurus, correct me if I'm wrong.

Note: if the external SDRAM is not changed while the CPU's accessing it,
you'd better access it throught the cache, and use prefetches.
This won't help the page faults, but you'll gain burst accesses,
and you may even access it directly through structures pointers, etc..
You'll need some cache invalidating functions before the copy/access.
For more on the topic, read any good paper on non-cache coherent memory.

> Is it always better to map only the small part I am going to use?

The bigger the mmap, the better, and the "lesser" entries in page table
there will be.
Note: this would make also a good victim for the Out-Of-Memory killer :)


Regards,
Stéphane

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: uncached user space mapping with mmap() ???

[Jon Masters]

Fillod Stephane wrote:

> The bigger the mmap, the better, and the "lesser" entries in page table
> there will be.

That's not true though. I went along with the rest of your mail, but the
above just does not make sense to me.

Are you somehow assuming you can have variable page sizes or will
necessarily be using BATs to map in large regions? If this is the case
then do bear in mind the fixed 4K page size on most platforms and the
fact that many architectures like 4xx do not have any BATs anyway.

Cheers,

Jon.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

RE: uncached user space mapping with mmap() ???

[Fillod Stephane]

Jon Masters wrote:
>> The bigger the mmap, the better, and the "lesser" entries in page table
>> there will be.
>
>That's not true though. I went along with the rest of your mail, but the
>above just does not make sense to me.

Indeed! I should have written:
"The bigger the mmap, the better, and the "lesser"  vma entries
there will be."

The idea is:  mmap(N*pagesize()) is better than   N times mmap(pagesize()),
provided you do need to address all this space.

>Are you somehow assuming you can have variable page sizes or will
>necessarily be using BATs to map in large regions? If this is the case
>then do bear in mind the fixed 4K page size on most platforms and the
>fact that many architectures like 4xx do not have any BATs anyway.

You're right. To have less entries in the page table, we would need
variable page sizes, since 4xx does not have BAT. Hence my remark
in form of question about ability of hugetlb. The answer must
be in the archive.

Thanks for the correction  :-)

Regards,
Stephane

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: uncached user space mapping with mmap() ???

[Jon Masters]

Fillod Stephane wrote:

<snip jcm being anal>

| Indeed! I should have written:
| "The bigger the mmap, the better, and the "lesser"  vma entries
| there will be."

Yes indeed. I shall try not to be such an anal pedant in future.

<snip>

|>Are you somehow assuming you can have variable page sizes or will
|>necessarily be using BATs to map in large regions? If this is the case
|>then do bear in mind the fixed 4K page size on most platforms and the
|>fact that many architectures like 4xx do not have any BATs anyway.

| You're right. To have less entries in the page table, we would need
| variable page sizes, since 4xx does not have BAT. Hence my remark
| in form of question about ability of hugetlb. The answer must
| be in the archive.

I would love to look at the hugetlb stuff but have not got around to it.
FWIW I looked at your Wiki example of uncached access the other day and
will probably send an update for a generic peekpoke utility unless
someone tells me there already is one generally available.

| Thanks for the correction  :-)

Coffee does this kind of thing so I apologise if I ever seem snappy.
Robert Love must hate me by now with the list of stuff for his book!

Jon.


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/