BAT mapping exported to user-space

BAT mapping exported to user-space

[Linh Dang]

Hi all,

In our project, our application continuously does random-access to
various ASICs. The ASICs are mapped by a bat register:

- Forcing the application to use system-calls for accessing the ASICs
  would incur a too-big overhead.

- Adding _PAGE_USER to bat-mapping is ok but would export the mapping
  to every single userspace processes.

- I'm contemplating adding 2 bitmasks (1 read, 1 write) into the
  mm_struct:
  - each bit in the mask represent a BAT register that should be
    accessible from userspace.
  - in switch_mm, prior to the call to set_context(), the BATs's Vp
    bits would be updated based on the masks of the `next' mm.

My questions are:
- has it been discussed/comptemplate/done before?
- what issues (performance,style,...) do you think such a change would
  create?
- How the X server access the hw on PPC?


Thanx

--
Linh Dang

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: BAT mapping exported to user-space

[Dan Malek]

On Jul 26, 2004, at 1:20 PM, Linh Dang wrote:

> - has it been discussed/comptemplate/done before?

Of course.  Use mmap(), either on /dev/mem directly, or more
preferable through a simple device driver for your ASIC that
could also do simple system protection.

Please don't be making such memory mapping kernel
modifications.  It will only cause future trouble.

	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: BAT mapping exported to user-space

[Linh Dang]

Dan Malek <dan@embeddededge.com> wrote:

>
> On Jul 26, 2004, at 1:20 PM, Linh Dang wrote:
>
>> - has it been discussed/comptemplate/done before?
>
> Of course.  Use mmap(), either on /dev/mem directly, or more
> preferable through a simple device driver for your ASIC that
> could also do simple system protection.
>
> Please don't be making such memory mapping kernel
> modifications.  It will only cause future trouble.
>
> -- Dan


My understanding is:

- mmap(2) uses pages to map the device (my platform doesn't have large page.)
- mapping a large address range (hundreds-of-megabytes) is much more
  efficient with BATs than with pages.

So, do you mean:
        - mmap(2) can use BATs? or
        - the difference in performance is negligible between BATs and
          pages when mapping an address range of 200MBs?


Thanx
--
Linh Dang

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: BAT mapping exported to user-space

[Dan Malek]

On Jul 26, 2004, at 2:33 PM, Linh Dang wrote:

> My understanding is:
>
> - mmap(2) uses pages to map the device (my platform doesn't have large
> page.)

Yes, so?

> - mapping a large address range (hundreds-of-megabytes) is much more
>   efficient with BATs than with pages.

Depends upon your measurement of efficiency.

> So, do you mean:
>         - mmap(2) can use BATs? or

Not likely due to alignment and size constraints.

>         - the difference in performance is negligible between BATs and
>           pages when mapping an address range of 200MBs?

If you can measure or are affected by the difference, there are many
more things that will ensure your software will not work properly.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: BAT mapping exported to user-space

[Linh Dang]

Dan Malek <dan@embeddededge.com> wrote:
>
> On Jul 26, 2004, at 2:33 PM, Linh Dang wrote:
>
>> My understanding is:
>>
>> - mmap(2) uses pages to map the device (my platform doesn't have large
>> page.)
>
> Yes, so?
>
>> - mapping a large address range (hundreds-of-megabytes) is much more
>> efficient with BATs than with pages.
>
> Depends upon your measurement of efficiency.
>
>> So, do you mean:
>> - mmap(2) can use BATs? or
>
> Not likely due to alignment and size constraints.
>
>> - the difference in performance is negligible between BATs and
>> pages when mapping an address range of 200MBs?
>
> If you can measure or are affected by the difference, there are many
> more things that will ensure your software will not work properly.

What do you mean by the above sentence?

- 200MB would need 51200 ptes. that means doubling the current number
  of ptes on my system.

- If using block mapping doesn't help that much then why would X make
  all the effort to grab MTRRs on X86?

- why would the kernel use BATs to map its memory?

--
Linh Dang

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: BAT mapping exported to user-space

[Dan Malek]

On Jul 26, 2004, at 10:27 PM, Linh Dang wrote:

> What do you mean by the above sentence?

Exactly what I said.  If you are concerned about the difference in
performance between a BAT mapped space and page tables,
there are many other kernel behaviors that are going to cause
trouble for your software.

> - 200MB would need 51200 ptes. that means doubling the current number
>   of ptes on my system.

Doubling?  I don't think so.  How did you measure what you are
currently using?  The 51200 PTEs really isn't a lot.  Mapping huge
linear spaces with PTEs is actually quite efficient.  Small VM spaces
with holes are the killer.  Do you actually touch every byte within
that 200 MB space?

> - If using block mapping doesn't help that much then why would X make
>   all the effort to grab MTRRs on X86?

I dunno.  I've never done any performance or feature analysis of x86
page
tables to discuss this.

> - why would the kernel use BATs to map its memory?

It's convenient for some areas of memory.  Makes it trivial to write
some forms of IO mapping functions.  We can set up some early static
memory maps for kernel initialization.  Mainly, we don't pollute the
TLBs
during short interrupt or system calls, allowing the user applications
to
run without having to reload the TLB after such events.  Even though
the kernel may use BATs, it still maps everything with page tables and
makes extensive use of them for various memory mapping requirements.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: BAT mapping exported to user-space

[Dang, Linh [CAR:2X23:EXCH]]

Dan Malek <dan@embeddededge.com> wrote:

>
> On Jul 26, 2004, at 10:27 PM, Linh Dang wrote:
>
>> What do you mean by the above sentence?
>
> Exactly what I said.  If you are concerned about the difference in
> performance between a BAT mapped space and page tables,
> there are many other kernel behaviors that are going to cause
> trouble for your software.
>
>> - 200MB would need 51200 ptes. that means doubling the current number
>> of ptes on my system.
>
> Doubling?  I don't think so.  How did you measure what you are


# cat /proc/ppc_htab
PTE Hash Table Information
Size            : 256Kb
Buckets         : 4096
Address         : c02c0000
Entries         : 32768
User ptes       : 326
Kernel ptes     : 989
Percent full    : 4%
Reloads         : 29288
Preloads        : 17175
Searches        : 1791
Overflows       : 0
Evicts          : 0
Non-error misses: 24468
Error misses    : 0


That's about 1315 PTEs (before the applications start.)


> currently using?  The 51200 PTEs really isn't a lot.  Mapping huge

just 38 times what currently on my system.

> linear spaces with PTEs is actually quite efficient.  Small VM spaces
> with holes are the killer.  Do you actually touch every byte within
> that 200 MB space?

When the system is working full-steam it's pretty close too that.

>
>> - If using block mapping doesn't help that much then why would X make
>> all the effort to grab MTRRs on X86?
>
> I dunno.  I've never done any performance or feature analysis of x86
> page
> tables to discuss this.
>
>> - why would the kernel use BATs to map its memory?
>
> It's convenient for some areas of memory.  Makes it trivial to write
> some forms of IO mapping functions.  We can set up some early static
> memory maps for kernel initialization.  Mainly, we don't pollute the
> TLBs
> during short interrupt or system calls, allowing the user applications
> to
> run without having to reload the TLB after such events.  Even though
> the kernel may use BATs, it still maps everything with page tables and
> makes extensive use of them for various memory mapping requirements.

I though the kernel (2.6) only use pages for vmalloc.

--
Linh Dang

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: BAT mapping exported to user-space

[Dan Malek]

On Jul 27, 2004, at 1:34 PM, Dang, Linh [CAR:2X23:EXCH] wrote:

> # cat /proc/ppc_htab
> PTE Hash Table Information

You misunderstand.  This is the PPC Hash table, not Linux PTEs.
There are lots and lots and lots of Linux PTEs are are eventually
loaded into the PPC hash table for the processor to fetch.
What processor are you using?


> I though the kernel (2.6) only use pages for vmalloc.

There is lots of kernel space allocated through vmalloc(), plus
other specialized spaces depending upon the processor core
that are allocated through page tables.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: BAT mapping exported to user-space

[Linh Dang]

Dan Malek <dan@embeddededge.com> wrote:

>
> On Jul 27, 2004, at 1:34 PM, Dang, Linh [CAR:2X23:EXCH] wrote:
>
>>> cat /proc/ppc_htab
>> PTE Hash Table Information
>
> You misunderstand.  This is the PPC Hash table, not Linux PTEs.
> There are lots and lots and lots of Linux PTEs are are eventually
> loaded into the PPC hash table for the processor to fetch.

Thanx for pointing that out. I have to dig further. I guess it would
come down to how much pressure using pages for mapping our asics
(instead of BATs) would have on the TLBs.

Just to clarify things, I'm trying to sell Linux to management (we're
a VxWorks shop) and trying to tell them that Linux's
RT-performance/efficiency is adequate and Linux would bring in a lot
more robustness/stability.


> What processor are you using?

7447 at 1 GHz (the bottle of the system is the I/O to the
memory-mapped ASICs)

>
>> I though the kernel (2.6) only use pages for vmalloc.
>
> There is lots of kernel space allocated through vmalloc(), plus
> other specialized spaces depending upon the processor core
> that are allocated through page tables.

Hmm, i though most of kernel's objects (socket's buffer included) are
allocated thru the slab-allocator. slab, AFAIK, doesn't use neither
vmalloc nor pages.


--
Linh Dang

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: BAT mapping exported to user-space

[Dan Malek]

On Jul 27, 2004, at 3:32 PM, Linh Dang wrote:

> Thanx for pointing that out. I have to dig further. I guess it would
> come down to how much pressure using pages for mapping our asics
> (instead of BATs) would have on the TLBs.

As I pointed out in one of the very first messages, you can write
a custom device driver to implement various mapping schemes
based upon your requirements.  However, this could result in
something not portable to another kernel version.

> Just to clarify things, I'm trying to sell Linux to management (we're
> a VxWorks shop) and trying to tell them that Linux's
> RT-performance/efficiency is adequate and Linux would bring in a lot
> more robustness/stability.

I've been involved in too many of this discussions to realize this
is never an engineering trade off.  Linux can do this VxWorks can't,
VxWorks can do things Linux can't, and there is a whole bunch
of computing where they do equally well.   The discussion we've
had is irrelevant to the choice.  Find some features worthy of
discussion and use them to make the decision. You can solve
any OS challenge with Linux, you just have to find the people
with the knowledge and be willing to do it.


	-- Dan


** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/

Re: BAT mapping exported to user-space

[Oliver Korpilla]

Linh Dang wrote:
> Dan Malek <dan@embeddededge.com> wrote:
>
>>On Jul 27, 2004, at 1:34 PM, Dang, Linh [CAR:2X23:EXCH] wrote:
>>
>>>>cat /proc/ppc_htab
>>>
>>>PTE Hash Table Information
>>
>>You misunderstand. This is the PPC Hash table, not Linux PTEs. There
>>are lots and lots and lots of Linux PTEs are are eventually loaded
>>into the PPC hash table for the processor to fetch.
>
> Thanx for pointing that out. I have to dig further. I guess it would
> come down to how much pressure using pages for mapping our asics
> (instead of BATs) would have on the TLBs.

Maybe you want to have a look at the book "Understanding the Linux Virtual
Memory Manager". Though clearly not PowerPC-specific (as always this book, too,
focuses on the x86), it's a recommended buy for understanding how paging etc.
works with Linux. It does explain the non-architecture-specific part rather
thoroughly.

Then, together with the "Programming Environments Manual For 32-Bit
Implementations of the PowerPC Architecture" (from Motorola), this
should give you a good understanding of what's going on, and how Linux
is handling this stuff.

With kind regards,
Oliver Korpilla

** Sent via the linuxppc-embedded mail list. See http://lists.linuxppc.org/