Speed of RAM reserved with memmap kernel command line option.

Speed of RAM reserved with memmap kernel command line option.

[Charles Buysschaert]

Hello all, 

We are writing a PCI-e "frame grabber-like" driver for linux 2.6 kernel.

For exchanging data with the board, we would like to be able to use a big piece of consecutive memory.  
-> we reserve 512 megs after the first Gig of RAM by setting "memmap=512M$1024M" in grub.conf.

In order to give access to that memory in user space, I use "remap_pfn_range" : 

int myboard_mmap(struct file *filep, struct vm_area_struct *vma) {
  unsigned int board;

  board = MINOR(filep->f_dentry->d_inode->i_rdev);
  vma->vm_flags |= VM_LOCKED;

  if(remap_pfn_range(vma, vma->vm_start, myboard_var[board].dmablkp>>PAGE_SHIFT, DMA_BUFFER_SIZE, vma->vm_page_prot)) {
    printk(KERN_ERR "myboard: myboard_mmap: failed to mmap\n");
    return -EAGAIN;
  }

  return 0;
}

(I have set myboard_var[board].dmablkp to the address I set in grub.conf: 0x40000000)

However when writing or reading that part of memory, speed is terrible. 
I compared the performances to the ones I can reach using a piece of memory got with malloc: 
For transfering 128 megs:
With malloc"ed" RAM : 23053us
With "mmap"ed RAM : 1913389us

I guess cache is not used. Does anyone know if there is a way to speed this up? How could we set that area to "cached"? 

We have seen this post:
http://www.linux-mips.org/archives/linux-mips/2006-02/msg00120.html

And we tried adding this to our driver: 

static inline pgprot_t pgprot_cached(pgprot_t _prot)

{
    unsigned long prot = pgprot_val(_prot);
    prot = (prot & ~_CACHE_MASK);
    return __pgprot(prot);
}

vma->vm_page_prot = pgprot_cached(vma->vm_page_prot);

But it did not help...

Thank you in advance! 

Best regards,

Charles

Re: Speed of RAM reserved with memmap kernel command line option.

[Matthias Schniedermeyer]

On 17.10.2011 07:51, Charles Buysschaert wrote:
> Hello all, 
> 
> We are writing a PCI-e "frame grabber-like" driver for linux 2.6 kernel.
...
> I guess cache is not used. Does anyone know if there is a way to speed this up? How could we set that area to "cached"? 

Smells to me like you are missing a mapping in MTRR ...
Documentation/x86/mtrr.txt

... or PAT.
Documentation/x86/pat.txt





Bis denn

-- 
Real Programmers consider "what you see is what you get" to be just as 
bad a concept in Text Editors as it is in women. No, the Real Programmer
wants a "you asked for it, you got it" text editor -- complicated, 
cryptic, powerful, unforgiving, dangerous.

RE: Speed of RAM reserved with memmap kernel command line option.

[Charles Buysschaert]

Hello Matthias, 

Thanks for your feedback. 

>Smells to me like you are missing a mapping in MTRR ...
>Documentation/x86/mtrr.txt

I checked and those definitely look correct:

This is the physical ram map at boot:

[    0.000000] user-defined physical RAM map:
[    0.000000]  user: 0000000000000000 - 0000000000096800 (usable)
[    0.000000]  user: 0000000000096800 - 00000000000a0000 (reserved)
[    0.000000]  user: 00000000000e4c00 - 0000000000100000 (reserved)
[    0.000000]  user: 0000000000100000 - 0000000040000000 (usable)
[    0.000000]  user: 0000000040000000 - 0000000060000000 (reserved)
[    0.000000]  user: 0000000060000000 - 00000000bf780000 (usable)
[    0.000000]  user: 00000000bf780000 - 00000000bf798000 (ACPI data)
[    0.000000]  user: 00000000bf798000 - 00000000bf7dc000 (ACPI NVS)
[    0.000000]  user: 00000000bf7dc000 - 00000000c0000000 (reserved)
[    0.000000]  user: 00000000fee00000 - 00000000fee01000 (reserved)
[    0.000000]  user: 00000000ffe00000 - 0000000100000000 (reserved)
[    0.000000]  user: 0000000100000000 - 00000001c0000000 (usable)

And this is the  MTRR contents.

$ cat /proc/mtrr
reg00: base=0x1c0000000 ( 7168MB), size= 1024MB, count=1: uncachable
reg01: base=0x000000000 (    0MB), size= 8192MB, count=1: write-back
reg02: base=0x0c0000000 ( 3072MB), size= 1024MB, count=1: uncachable
reg03: base=0x0bf800000 ( 3064MB), size=    8MB, count=1: uncachable

The reserved zone is indeed marked as 'reserved' 512Mb located at 1Go.
And this zone is
covered by the reg01 MTRR without being overriden by anyother, so it's 'write-back' which is good.

> ... or PAT.
> Documentation/x86/pat.txt
This is also what we tried to set by modifying the protection field given to the remap_pfn_range.

we dumped the kernel_page_tables file provided by "debugfs", and we can see the following entry:
0xFFFF880040000000 -> 0xFFFF880060000000	512M	RW	PCD	PSE	GLB	NX	pmd

I am not 100% sure it is the mapping of my "0000000040000000" physical address, but it looks like (is there a way to see which physical address it relates to?).
I can see the flag "PCD" Page Cache Disabled. 

Does the kernel automatically create a mapping for memory reserved with "memmap=" option? 
Would "aliasing" be the reason why my "remap_pfn_range" seems to ignore the vma->vm_page_prot we are setting? 
(
static inline pgprot_t pgprot_cached(pgprot_t _prot)
{
    unsigned long prot = pgprot_val(_prot);
    prot = (prot & ~_CACHE_MASK);
    return __pgprot(prot);
}
vma->vm_page_prot = pgprot_cached(vma->vm_page_prot);
remap_pfn_range(vma, vma->vm_start, myboard_var[board].dmablkp>>PAGE_SHIFT,DMA_BUFFER_SIZE, vma->vm_page_prot)
)

Thanks for your help,


Charles