Getting physical addresses of mmap'd pages from userspace

Getting physical addresses of mmap'd pages from userspace

[Tom Cooksey]

Is there any way to get the physical address of mlock()'d memory from userspace?

I want to avoid writing a kernel module to do it if possible, but I can't see any API which will give me the address? I'm not sure if doing mlock() will even pin the physical address (although I can't see why the physical location would change other than when the page is swapped out and back in again). Is there a syscall I've overlooked somewhere?

The only way I can see to do it would be to pass a kernel parameter in uBoot and tell the kernel not to use e.g. the top 2MB of physical memory. I think I could then mmap /dev/mem to access the physical memory from my application, but pass the physical address to my device which reads from the memory my application has written. Does anyone think this would work?


Cheers,

Tom

Re: Getting physical addresses of mmap'd pages from userspace

[Bart Van Assche]

On Fri, Oct 10, 2008 at 6:15 PM, Tom Cooksey
<thomas.cooksey@trolltech.com> wrote:
> Is there any way to get the physical address of mlock()'d memory from userspace?

Why do you need this information ?

Bart.

Re: Getting physical addresses of mmap'd pages from userspace

[Robert Schwebel]

On Fri, Oct 10, 2008 at 06:15:05PM +0200, Tom Cooksey wrote:
> Is there any way to get the physical address of mlock()'d memory from
> userspace?

What do you want to do? I don't really see a reason to do such strange
things any more these days.

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: Getting physical addresses of mmap'd pages from userspace

[Tom Cooksey]

On Friday 10 October 2008 21:12:13 Robert Schwebel wrote:
> On Fri, Oct 10, 2008 at 06:15:05PM +0200, Tom Cooksey wrote:
> > Is there any way to get the physical address of mlock()'d memory from
> > userspace?
> 
> What do you want to do? I don't really see a reason to do such strange
> things any more these days.

It's quite an annoying problem actually. Basically I have binary drivers for Imagination Technologies's PowerVR graphics drivers. We have tried very hard to get the source code for these drivers and have failed. Eventually ImgTec allowed us to sign an NDA to get the headers for one of their user-space libraries. This library allows us to direct the graphics hardware to render to a specific physical memory region. The problem is that there's no way to find out what the physical addresses are which we need pass to the graphics hardware (via the user-space library). Allthough the library allows us to allocate emory, what I want to do is then blit that memory in a different process. So a client process renders into an off-screen buffer and the server process blits that buffer to the framebuffe
 r. By allowing the server process to do the blit rather than the client process, we can get semi-transparent GL windows. 

The synchronisation we can do, it's the memory allocation I'm struggling with. If we ask the library to allocate the memory for us, we don't get the physical address to pass to the server process. Instead, we need to allocate memory ourselves and pass the physical address to the library. But like I say, I can't find a way to get the physical address from the kernel.

I realise getting round closed-source drivers isn't exactly encoraged, but sadly, we really have no choice. :-(



Cheers,

Tom

Re: Getting physical addresses of mmap'd pages from userspace

[Robert Schwebel]

Tom,

[Please configure your mail client to break likes @ column < 80]

On Mon, Oct 13, 2008 at 08:33:25AM +0200, Tom Cooksey wrote:
> On Friday 10 October 2008 21:12:13 Robert Schwebel wrote:
> > On Fri, Oct 10, 2008 at 06:15:05PM +0200, Tom Cooksey wrote:
> > > Is there any way to get the physical address of mlock()'d memory
> > > from userspace?
> >
> > What do you want to do? I don't really see a reason to do such
> > strange things any more these days.
>
> It's quite an annoying problem actually. Basically I have binary
> drivers for Imagination Technologies's PowerVR graphics drivers. We
> have tried very hard to get the source code for these drivers and have
> failed. Eventually ImgTec allowed us to sign an NDA to get the headers
> for one of their user-space libraries. This library allows us to
> direct the graphics hardware to render to a specific physical memory
> region. The problem is that there's no way to find out what the
> physical addresses are which we need pass to the graphics hardware
> (via the user-space library). Allthough the library allows us to
> allocate emory, what I want to do is then blit that memory in a
> different process. So a client process renders into an off-screen
> buffer and the server process blits that buffer to the framebuffer. By
> allowing the server process to do the blit rather than the client
> process, we can get semi-transparent GL windows.
>
> The synchronisation we can do, it's the memory allocation I'm
> struggling with. If we ask the library to allocate the memory for us,
> we don't get the physical address to pass to the server process.
> Instead, we need to allocate memory ourselves and pass the physical
> address to the library. But like I say, I can't find a way to get the
> physical address from the kernel.
>
> I realise getting round closed-source drivers isn't exactly encoraged,
> but sadly, we really have no choice. :-(

We are working on DirectFB [1] support for the PowerVR MBX & SGX chips
these days, so I know the problem :) Our hardware platform is i.MX31
with the MBX, MPC5121e also with MBX (both freescale) and a TI OMAP with
the SGX. The whole NDA and Licensing situation is annoying, but we have
to live with it; if you ask me as a community person, I'd say "vote with
your dollars". If manufacturers don't want to open their specs to allow
people writing driver, it must mean that they don't want to sell chips.
In reality, there are no real alternatives for accelerates chips.

Regarding the technical question, dok might have an idea. He's currently
working on DirectFB drivers ontop of the proprietary drivers in one of
our projects. Cc'ed him.

rsc

[1] Sidenote: we have Gtk+ and TTFNAQ [2] running on top of DirectFB,
    with 2D acceleration.

[2] The-Toolkit-Formerly-Known-As-Qtopia
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: Getting physical addresses of mmap'd pages from userspace

[Tom Cooksey]

On Monday 13 October 2008 09:00:17 Robert Schwebel wrote:
> Tom,
> 
> [Please configure your mail client to break likes @ column < 80]
Whoops - Sorry!

> On Mon, Oct 13, 2008 at 08:33:25AM +0200, Tom Cooksey wrote:
> > On Friday 10 October 2008 21:12:13 Robert Schwebel wrote:
> > > On Fri, Oct 10, 2008 at 06:15:05PM +0200, Tom Cooksey wrote:
> > > > Is there any way to get the physical address of mlock()'d memory
> > > > from userspace?
> > >
> > > What do you want to do? I don't really see a reason to do such
> > > strange things any more these days.
> >
> > It's quite an annoying problem actually. Basically I have binary
> > drivers for Imagination Technologies's PowerVR graphics drivers. We
> > have tried very hard to get the source code for these drivers and have
> > failed. Eventually ImgTec allowed us to sign an NDA to get the headers
> > for one of their user-space libraries. This library allows us to
> > direct the graphics hardware to render to a specific physical memory
> > region. The problem is that there's no way to find out what the
> > physical addresses are which we need pass to the graphics hardware
> > (via the user-space library). Allthough the library allows us to
> > allocate emory, what I want to do is then blit that memory in a
> > different process. So a client process renders into an off-screen
> > buffer and the server process blits that buffer to the framebuffer. By
> > allowing the server process to do the blit rather than the client
> > process, we can get semi-transparent GL windows.
> >
> > The synchronisation we can do, it's the memory allocation I'm
> > struggling with. If we ask the library to allocate the memory for us,
> > we don't get the physical address to pass to the server process.
> > Instead, we need to allocate memory ourselves and pass the physical
> > address to the library. But like I say, I can't find a way to get the
> > physical address from the kernel.
> >
> > I realise getting round closed-source drivers isn't exactly encoraged,
> > but sadly, we really have no choice. :-(
> 
> We are working on DirectFB [1] support for the PowerVR MBX & SGX chips
> these days, so I know the problem :) Our hardware platform is i.MX31
> with the MBX, MPC5121e also with MBX (both freescale) and a TI OMAP with
> the SGX. The whole NDA and Licensing situation is annoying, but we have
> to live with it; if you ask me as a community person, I'd say "vote with
> your dollars". If manufacturers don't want to open their specs to allow
> people writing driver, it must mean that they don't want to sell chips.
> In reality, there are no real alternatives for accelerates chips.

ImgTec has had the market pretty much to itself for a long time and as you say,
there's pretty much no alternatives for the moment. Hopefully that is about to
change as there are, I beleive, about 3 other companies with competing products
about to come to market. I've no idea if the other companies will open up their
drivers either though. It also doesn't help the current situation. :-(

> Regarding the technical question, dok might have an idea. He's currently
> working on DirectFB drivers ontop of the proprietary drivers in one of
> our projects. Cc'ed him.

> [1] Sidenote: we have Gtk+ and TTFNAQ [2] running on top of DirectFB,
>     with 2D acceleration.
> 
> [2] The-Toolkit-Formerly-Known-As-Qtopia

Gulp - yeah, I'm kinda the maintainer for the Qt/Embedded DirectFB driver now.
(Since the guy who wrote it left a month ago) :-|



Cheers,

Tom

Re: Getting physical addresses of mmap'd pages from userspace

[Thomas Petazzoni]

Le Mon, 13 Oct 2008 09:00:17 +0200,
Robert Schwebel <r.schwebel@pengutronix.de> a écrit :

> We are working on DirectFB [1] support for the PowerVR MBX & SGX chips
> these days, so I know the problem :) Our hardware platform is i.MX31
> with the MBX, MPC5121e also with MBX (both freescale) and a TI OMAP
> with the SGX.

This is interesting. Is this work available somewhere ? Is it
open-source, or does it rely on proprietary drivers ?

Thanks,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers and embedded Linux development,
consulting, training and support.
http://free-electrons.com

Re: Getting physical addresses of mmap'd pages from userspace

[Robert Schwebel]

On Mon, Oct 13, 2008 at 09:28:15AM +0200, Thomas Petazzoni wrote:
> > We are working on DirectFB [1] support for the PowerVR MBX & SGX
> > chips these days, so I know the problem :) Our hardware platform is
> > i.MX31 with the MBX, MPC5121e also with MBX (both freescale) and a
> > TI OMAP with the SGX.
>
> This is interesting. Is this work available somewhere? Is it
> open-source, or does it rely on proprietary drivers?

It will be available when we have something which is worth being
reviewed in public. As there is currently no possibility to get any
documentation about the cores without NDAs, an open *driver* is IMHO not
possible, so we'll have to rely on the proprietary drivers Freescale
deliver. Bad, but I don't see another chance by now.

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: Getting physical addresses of mmap'd pages from userspace

[Bill Gatliff]

Robert Schwebel wrote:
> In reality, there are no real alternatives for accelerates chips.

Would the Silicon Motion SM50x chips qualify as an alternative?

They can do the blitting, at least.  No OpenGL, tho.  So, I guess it depends on
your definition of "accelerated"...


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: Getting physical addresses of mmap'd pages from userspace

[Robert Schwebel]

On Mon, Oct 13, 2008 at 07:50:08AM -0500, Bill Gatliff wrote:
> Robert Schwebel wrote:
> > In reality, there are no real alternatives for accelerates chips.
>
> Would the Silicon Motion SM50x chips qualify as an alternative?
>
> They can do the blitting, at least.  No OpenGL, tho.  So, I guess it
> depends on your definition of "accelerated"...

Yup, from our perspective it would be fine. Unfortunately, the hardware
people in our project had (subjective, imho) concerns.

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: Getting physical addresses of mmap'd pages from userspace

[George G. Davis]

On Mon, Oct 13, 2008 at 03:23:03PM +0200, Robert Schwebel wrote:
> On Mon, Oct 13, 2008 at 07:50:08AM -0500, Bill Gatliff wrote:
> > Robert Schwebel wrote:
> > > In reality, there are no real alternatives for accelerates chips.
> >
> > Would the Silicon Motion SM50x chips qualify as an alternative?
> >
> > They can do the blitting, at least.  No OpenGL, tho.  So, I guess it
> > depends on your definition of "accelerated"...
> 
> Yup, from our perspective it would be fine. Unfortunately, the hardware
> people in our project had (subjective, imho) concerns.

It comes down to decisions of bill-of-materials and performance, the
MBX/SGX options mentioned already are part of the SoCs and have
high bandwidth connections to the processor.  External options increase
cost and won't match bandwidth of on chip options, but that's just my
ignorant opinion w/o looking at the low-level detail comparisons.
Unless you intend to encourage the SoC vendors to use the SM50x
instead?  : )  Of course, that won't help for the current and planned
generation(s) of devices...

--
Regards,
George
> 
> rsc
> -- 
>  Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
>  Pengutronix - Linux Solutions for Science and Industry
>    Handelsregister:  Amtsgericht Hildesheim, HRA 2686
>      Hannoversche Str. 2, 31134 Hildesheim, Germany
>    Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Getting physical addresses of mmap'd pages from userspace

[Robert Schwebel]

On Mon, Oct 13, 2008 at 11:58:20AM -0400, George G. Davis wrote:
> It comes down to decisions of bill-of-materials and performance, the
> MBX/SGX options mentioned already are part of the SoCs and have high
> bandwidth connections to the processor. External options increase cost
> and won't match bandwidth of on chip options, but that's just my
> ignorant opinion w/o looking at the low-level detail comparisons.
> Unless you intend to encourage the SoC vendors to use the SM50x
> instead?  : )  Of course, that won't help for the current and planned
> generation(s) of devices...

The BoM argument is probably true for high volume consumer or automotive
use cases. For industrial usage, long term maintainability and thus
availability of documentation and code is usually a more important
argument.

Just my 0.02 ct. I'm just a poor open source guy :)

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: Getting physical addresses of mmap'd pages from userspace

[Tom Cooksey]

On Monday 13 October 2008 18:09:27 Robert Schwebel wrote:
> On Mon, Oct 13, 2008 at 11:58:20AM -0400, George G. Davis wrote:
> > It comes down to decisions of bill-of-materials and performance, the
> > MBX/SGX options mentioned already are part of the SoCs and have high
> > bandwidth connections to the processor. External options increase cost
> > and won't match bandwidth of on chip options, but that's just my
> > ignorant opinion w/o looking at the low-level detail comparisons.
> > Unless you intend to encourage the SoC vendors to use the SM50x
> > instead?  : )  Of course, that won't help for the current and planned
> > generation(s) of devices...
> 
> The BoM argument is probably true for high volume consumer or automotive
> use cases. For industrial usage, long term maintainability and thus
> availability of documentation and code is usually a more important
> argument.
> 
> Just my 0.02 ct. I'm just a poor open source guy :)

<RANT>
What I don't understand is that I'm trying to do some pretty interesting & cool
stuff with their processors (of course I would say that!), which will probably
help them sell more units. Why then do they make it so difficult to work with
them? It feels like they're shooting themselves in the foot. Madness.
</RANT>

Cheers,

Tom

Re: Getting physical addresses of mmap'd pages from userspace

[Bill Gatliff]

Tom Cooksey wrote:

> <RANT>
> What I don't understand is that I'm trying to do some pretty interesting & cool
> stuff with their processors (of course I would say that!), which will probably
> help them sell more units. Why then do they make it so difficult to work with
> them? It feels like they're shooting themselves in the foot. Madness.
> </RANT>

Not to perpetuate this further, but I can't resist...  :)

That's because their product won't stand on its own; it needs vendor lock-in to
be successful.  There really isn't any other explanation for such behavior.

Think like a biologist.  If an organism does something, then the upside must be
better then the downside of NOT doing that something, or the organism wouldn't
waste scarce time and energy doing it--- no matter how ridiculous that something
might be.  Unusual markings, mating calls, mullet haircuts...

One would think that in the world of high-technology, there would be a huge
upside to making products easy to use, which would naturally require free
availability of documentation and code (among other things).  But vendors seem
to work contrary to that objective, which must mean that there's an even bigger
upside to NOT making a product easy to use.

Put another way, their revenue stream depends on making your life as painful as
possible, so that you won't want to risk repeating that pain by switching to a
competitor's product.  It's a "shock collar ^K^K^K electrically-enhanced
training aid", so to speak, and we're the dogs.  And not the
chihuahua-in-Paris-Hilton's-purse kind of dogs, either.

Here's more evidence to support my point: what exactly is the cost to release
documentation without an NDA?  About US$0, which is considerably less than the
expense of executing an NDA.  So why have the NDA?  Because that expense must be
an "investment" in something that nets a larger return to the vendor of the
documents in question.  What might that be?  Hmmm....


Just my US$0.02.  You can keep the change.  :)


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: Getting physical addresses of mmap'd pages from userspace

[Tom Cooksey]

On Tuesday 14 October 2008 17:47:06 Bill Gatliff wrote:
> Tom Cooksey wrote:
> 
> > <RANT>
> > What I don't understand is that I'm trying to do some pretty interesting & cool
> > stuff with their processors (of course I would say that!), which will probably
> > help them sell more units. Why then do they make it so difficult to work with
> > them? It feels like they're shooting themselves in the foot. Madness.
> > </RANT>
> 
> Not to perpetuate this further, but I can't resist...  :)
> 
> That's because their product won't stand on its own; it needs vendor lock-in to
> be successful.  There really isn't any other explanation for such behavior.
> 
> Think like a biologist.  If an organism does something, then the upside must be
> better then the downside of NOT doing that something, or the organism wouldn't
> waste scarce time and energy doing it--- no matter how ridiculous that something
> might be.  Unusual markings, mating calls, mullet haircuts...
> 
> One would think that in the world of high-technology, there would be a huge
> upside to making products easy to use, which would naturally require free
> availability of documentation and code (among other things).  But vendors seem
> to work contrary to that objective, which must mean that there's an even bigger
> upside to NOT making a product easy to use.
> 
> Put another way, their revenue stream depends on making your life as painful as
> possible, so that you won't want to risk repeating that pain by switching to a
> competitor's product.  It's a "shock collar ^K^K^K electrically-enhanced
> training aid", so to speak, and we're the dogs.  And not the
> chihuahua-in-Paris-Hilton's-purse kind of dogs, either.
> 
> Here's more evidence to support my point: what exactly is the cost to release
> documentation without an NDA?  About US$0, which is considerably less than the
> expense of executing an NDA.  So why have the NDA?  Because that expense must be
> an "investment" in something that nets a larger return to the vendor of the
> documents in question.  What might that be?  Hmmm....

I always assumed it's because releasing the source opens them up to patent
infringement law suits? Some companies are more paranoid than others.

Re: Getting physical addresses of mmap'd pages from userspace

[James Chapman]

Tom Cooksey wrote:
> On Tuesday 14 October 2008 17:47:06 Bill Gatliff wrote:
>> Tom Cooksey wrote:
>>
>>> <RANT>
>>> What I don't understand is that I'm trying to do some pretty interesting & cool
>>> stuff with their processors (of course I would say that!), which will probably
>>> help them sell more units. Why then do they make it so difficult to work with
>>> them? It feels like they're shooting themselves in the foot. Madness.
>>> </RANT>
>> Not to perpetuate this further, but I can't resist...  :)
>>
>> That's because their product won't stand on its own; it needs vendor lock-in to
>> be successful.  There really isn't any other explanation for such behavior.
>>
>> Think like a biologist.  If an organism does something, then the upside must be
>> better then the downside of NOT doing that something, or the organism wouldn't
>> waste scarce time and energy doing it--- no matter how ridiculous that something
>> might be.  Unusual markings, mating calls, mullet haircuts...
>>
>> One would think that in the world of high-technology, there would be a huge
>> upside to making products easy to use, which would naturally require free
>> availability of documentation and code (among other things).  But vendors seem
>> to work contrary to that objective, which must mean that there's an even bigger
>> upside to NOT making a product easy to use.
>>
>> Put another way, their revenue stream depends on making your life as painful as
>> possible, so that you won't want to risk repeating that pain by switching to a
>> competitor's product.  It's a "shock collar ^K^K^K electrically-enhanced
>> training aid", so to speak, and we're the dogs.  And not the
>> chihuahua-in-Paris-Hilton's-purse kind of dogs, either.
>>
>> Here's more evidence to support my point: what exactly is the cost to release
>> documentation without an NDA?  About US$0, which is considerably less than the
>> expense of executing an NDA.  So why have the NDA?  Because that expense must be
>> an "investment" in something that nets a larger return to the vendor of the
>> documents in question.  What might that be?  Hmmm....
> 
> I always assumed it's because releasing the source opens them up to patent
> infringement law suits? Some companies are more paranoid than others.

Some companies won't publish open software interface docs for their
devices because they think doing so will give their competitors an
advantage by giving away some info about how their device works
internally. They don't understand open source, period.

It's important to choose hardware devices carefully when developing an
Embedded Linux product; avoid any devices which don't have open source
drivers or docs whenever possible.

All device vendors understand money. If they start to lose market share
because Qt isn't supported by their devices then they might change their
approach. :)


-- 
James Chapman
Katalix Systems Ltd
http://www.katalix.com
Catalysts for your Embedded Linux software development

Re: Getting physical addresses of mmap'd pages from userspace

[Robert Schwebel]

On Tue, Oct 14, 2008 at 10:47:06AM -0500, Bill Gatliff wrote:
> One would think that in the world of high-technology, there would be a
> huge upside to making products easy to use, which would naturally
> require free availability of documentation and code (among other
> things).  But vendors seem to work contrary to that objective, which
> must mean that there's an even bigger upside to NOT making a product
> easy to use.

Your argumentation neglects the possibility that vendors just do it the
way they always did. Don't assume that some intelligent life form has
mandatorily taken an intentional decision :)

rsc
-- 
 Dipl.-Ing. Robert Schwebel | http://www.pengutronix.de
 Pengutronix - Linux Solutions for Science and Industry
   Handelsregister:  Amtsgericht Hildesheim, HRA 2686
     Hannoversche Str. 2, 31134 Hildesheim, Germany
   Phone: +49-5121-206917-0 |  Fax: +49-5121-206917-9

Re: Getting physical addresses of mmap'd pages from userspace

[Bill Gatliff]

Robert Schwebel wrote:
> Your argumentation neglects the possibility that vendors just do it the
> way they always did. Don't assume that some intelligent life form has
> mandatorily taken an intentional decision :)


D'oh!  Good point.  :)


b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: Getting physical addresses of mmap'd pages from userspace

[Gilad Ben-Yossef]

Tom Cooksey wrote:

> ...
>  This library allows us to direct the graphics hardware to render to a specific physical memory region. The problem is that there's no way to find out what the physical addresses are which we need pass to the graphics hardware (via the user-space library). Allthough the library allows us to allocate emory, what I want to do is then blit that memory in a different process. So a client process renders into an off-screen buffer and the server process blits that buffer to the framebuffer. By allowing the server process to do the blit rather than the client process, we can get semi-transparent GL windows. 
>
> The synchronisation we can do, it's the memory allocation I'm struggling with. If we ask the library to allocate the memory for us, we don't get the physical address to pass to the server process. Instead, we need to allocate memory ourselves and pass the physical address to the library. But like I say, I can't find a way to get the physical address from the kernel.
>
>   
You lost me on why you can't have the library allocate the memory - I 
understand the library needs the physical address to render into, but 
why does your server needs the physical address to blit to the frame 
buffer? I assume it access the frame buffer in host memory so needs the 
virtual address, not the physical one...

Anyways, ignoring that, it's pretty trivial to write  a character driver 
with an IOCTL that will get an mlocked virtual address of the processes, 
call virt_to_phys on it and return the result. Yes, it's an ugly as hell 
solution, but then again so is the problem (proprietary libraries etc.)

You do realize that the mlocked memory isn't contiguous in physical 
memory, right? unless your graphics engine has something like AGP GART 
or an IOMMU I think you might find that problematic.

Gilad


-- 
Gilad Ben-Yossef 
Chief Coffee Drinker

Codefidence Ltd.
The code is free, your time isn't.(TM)

Web:    http://codefidence.com
Email:  gilad@codefidence.com
Office: +972-8-9316883 ext. 201
Fax:    +972-8-9316885
Mobile: +972-52-8260388

	The Doctor: Don't worry, Reinette, just a nightmare. 
	Everyone has nightmares. Even monsters from under the 
	bed have nightmares, don't you, monster?
	Reinette: What do monsters have nightmares about?
	The Doctor: Me! 

Re: Getting physical addresses of mmap'd pages from userspace

[Tom Cooksey]

On Monday 13 October 2008 11:13:57 Gilad Ben-Yossef wrote:
> Tom Cooksey wrote:
> 
> 
> > This library allows us to direct the graphics hardware to render to a specific physical memory region. The problem is that there's no way to find out what the physical addresses are which we need pass to the graphics hardware (via the user-space library). Allthough the library allows us to allocate emory, what I want to do is then blit that memory in a different process. So a client process renders into an off-screen buffer and the server process blits that buffer to the framebuffer. By allowing the server process to do the blit rather than the client process, we can get semi-transparent GL windows. 
> >
> > The synchronisation we can do, it's the memory allocation I'm struggling with. If we ask the library to allocate the memory for us, we don't get the physical address to pass to the server process. Instead, we need to allocate memory ourselves and pass the physical address to the library. But like I say, I can't find a way to get the physical address from the kernel.
> >
> >   
> You lost me on why you can't have the library allocate the memory - I 
> understand the library needs the physical address to render into, but 
> why does your server needs the physical address to blit to the frame 
> buffer? I assume it access the frame buffer in host memory so needs the 
> virtual address, not the physical one...

The reason is that the hardware blitter is exposed via the user-space library,
but because the blit is done by hardware, it too needs the physical address of
both the source and the destination of the blit. The destination is easy - the
library provides an API to get the address of the framebuffer. It's the source
that's the problem.

> Anyways, ignoring that, it's pretty trivial to write  a character driver 
> with an IOCTL that will get an mlocked virtual address of the processes, 
> call virt_to_phys on it and return the result. Yes, it's an ugly as hell 
> solution, but then again so is the problem (proprietary libraries etc.)

I realise this is possible, but I'm trying to find a solution which doesn't
require me to write a kernel driver. If I can't find a solution, I will follow
your suggestion of using virt_to_phys - thanks!

> You do realize do that the mlocked memory isn't contiguous in physical 
> memory, right? unless your graphics engine has something like AGP GART 
> or an IOMMU I think you might find that problematic.

I had assumed that. Fortunatly the user-space library's API takes an array of
physical page addresses. The hardware actually has it's own MMU (I guess
that's an IOMMU?) with enough entries to address quite a lot of memory. I'm
assuming the library programs the GPU's MMU with the addresses I pass in.

I also think that I need to be careful with caching. The CPU (OMAP3) has both
an L1 and L2 cache. I doubt there's any cache coherency logic in the hardware
to make sure the CPU's cache is flushed before the GPU startsreading from it.

The last thing I need is to write to the memory I want to blit (using a 
software renderer running on the CPU), only to find the data I've written is
sitting around in the CPU cache and has not been flushed to RAM (in which case
I'll get garbage on the screen).

I guess I'm going to need a way to not only get the physical address of the
pages, but also set the cache policy or (ideally) have a way to flush the pages
to RAM.



Cheers,

Tom

Re: Getting physical addresses of mmap'd pages from userspace

[Bill Gatliff]

Tom Cooksey wrote:
> On Friday 10 October 2008 21:12:13 Robert Schwebel wrote:
>> On Fri, Oct 10, 2008 at 06:15:05PM +0200, Tom Cooksey wrote:
>>> Is there any way to get the physical address of mlock()'d memory from
>>> userspace?
>> What do you want to do? I don't really see a reason to do such strange
>> things any more these days.
> 
> It's quite an annoying problem actually. Basically I have binary drivers for Imagination Technologies's PowerVR graphics drivers. We have tried very hard to get the source code for these drivers and have failed. Eventually ImgTec allowed us to sign an NDA to get the headers for one of their user-space libraries. This library allows us to direct the graphics hardware to render to a specific physical memory region. The problem is that there's no way to find out what the physical addresses are which we need pass to the graphics hardware (via the user-space library). Allthough the library allows us to allocate emory, what I want to do is then blit that memory in a different process. So a client process renders into an off-screen buffer and the server process blits that buffer to the framebuf
 fer. By allowing the server process to do the blit rather than the client process, we can get semi-transparent GL windows. 
> 
> The synchronisation we can do, it's the memory allocation I'm struggling with. If we ask the library to allocate the memory for us, we don't get the physical address to pass to the server process. Instead, we need to allocate memory ourselves and pass the physical address to the library. But like I say, I can't find a way to get the physical address from the kernel.
> 
> I realise getting round closed-source drivers isn't exactly encoraged, but sadly, we really have no choice. :-(

Not that this would help, but what if the blitting process was working with a
shared memory area with the, er, main process?  Could the allocation be done in
the library, then?

Reading your post carefully, it sounds like either the library in question is
designed to run in kernel space (where you can get the physical address), isn't
a Linux library, or there's some more code somewhere that can do the address
translation--- you said that "This library allows us to direct the graphics
hardware to render to a specific physical memory region".  How are other users
of the library getting the parameters that the library needs?

What about writing a hacky little module that just does virtual-to-physical
translation, and returns the result?

Ugh, I think I need to shower.  :)



b.g.
-- 
Bill Gatliff
bgat@billgatliff.com

Re: Getting physical addresses of mmap'd pages from userspace

[Tom Cooksey]

On Monday 13 October 2008 14:48:06 Bill Gatliff wrote:
> Tom Cooksey wrote:
> > On Friday 10 October 2008 21:12:13 Robert Schwebel wrote:
> >> On Fri, Oct 10, 2008 at 06:15:05PM +0200, Tom Cooksey wrote:
> >>> Is there any way to get the physical address of mlock()'d memory from
> >>> userspace?
> >> What do you want to do? I don't really see a reason to do such strange
> >> things any more these days.
> > 
> > It's quite an annoying problem actually. Basically I have binary drivers for Imagination Technologies's PowerVR graphics drivers. We have tried very hard to get the source code for these drivers and have failed. Eventually ImgTec allowed us to sign an NDA to get the headers for one of their user-space libraries. This library allows us to direct the graphics hardware to render to a specific physical memory region. The problem is that there's no way to find out what the physical addresses are which we need pass to the graphics hardware (via the user-space library). Allthough the library allows us to allocate emory, what I want to do is then blit that memory in a different process. So a client process renders into an off-screen buffer and the server process blits that buffer to the frameb
 uffer. By allowing the server process to do the blit rather than the client process, we can get semi-transparent GL windows. 
> > 
> > The synchronisation we can do, it's the memory allocation I'm struggling with. If we ask the library to allocate the memory for us, we don't get the physical address to pass to the server process. Instead, we need to allocate memory ourselves and pass the physical address to the library. But like I say, I can't find a way to get the physical address from the kernel.
> > 
> > I realise getting round closed-source drivers isn't exactly encoraged, but sadly, we really have no choice. :-(
> 
> Not that this would help, but what if the blitting process was working with a
> shared memory area with the, er, main process?  Could the allocation be done in
> the library, then?
> 
> Reading your post carefully, it sounds like either the library in question is
> designed to run in kernel space (where you can get the physical address), isn't
> a Linux library, or there's some more code somewhere that can do the address
> translation--- you said that "This library allows us to direct the graphics
> hardware to render to a specific physical memory region".  How are other users
> of the library getting the parameters that the library needs?

The other user is more closed source OpenGL library code. The problem is that
the drivers are designed to blit from the client process, where the library
does the allocation & the blits (which means opaque-only blits). If we want
transparent windows or fancy window composition effects we have to blit from a
central server process and have clients render into off-screen buffers.

Also, you're right - The library seems to not be Linux-specific but targeted at
other OSes (Which ones I can't say without breaking the NDA I had to sign!).

> What about writing a hacky little module that just does virtual-to-physical
> translation, and returns the result?
> 
> Ugh, I think I need to shower.  :)

Yup, I think that's what I'm going to end up having to do. Even if I can get
the physical address, I need some way to flush the CPU cache to RAM before I
ask the GPU to blit. I doubt there's any way to do that from userspace. :-( I'm
not even sure of the kernel API to do it. Any ideas?


Cheers,

Tom

Re: Getting physical addresses of mmap'd pages from userspace

[Daniel THOMPSON]

Tom Cooksey wrote:
> Yup, I think that's what I'm going to end up having to do. Even if I can get
> the physical address, I need some way to flush the CPU cache to RAM before I
> ask the GPU to blit. I doubt there's any way to do that from userspace. :-( I'm
> not even sure of the kernel API to do it. Any ideas?

MIPS has it: http://www.linux-mips.org/wiki/Cacheflush_Syscall
and so does SH. I'm afraid I don't know about ARM though.

-- 
Daniel Thompson (STMicroelectronics) <daniel.thompson@st.com>
1000 Aztec West, Almondsbury, Bristol, BS32 4SQ. 01454 462659

If a car is a horseless carriage then is a motorcycle a horseless horse?

Re: Getting physical addresses of mmap'd pages from userspace

[George G. Davis]

On Mon, Oct 13, 2008 at 04:09:36PM +0100, Daniel THOMPSON wrote:
> Tom Cooksey wrote:
> > Yup, I think that's what I'm going to end up having to do. Even if I can get
> > the physical address, I need some way to flush the CPU cache to RAM before I
> > ask the GPU to blit. I doubt there's any way to do that from userspace. :-( I'm
> > not even sure of the kernel API to do it. Any ideas?
> 
> MIPS has it: http://www.linux-mips.org/wiki/Cacheflush_Syscall
> and so does SH. I'm afraid I don't know about ARM though.

ARM also implements sys_cacheflush(start, end /* exclusive */, 0) but
does not provide separate methods for I or D cache as in the MIPS docs
above, ARM sys_cacheflush'es both I+D caches.

--
Regards,
George
> 
> -- 
> Daniel Thompson (STMicroelectronics) <daniel.thompson@st.com>
> 1000 Aztec West, Almondsbury, Bristol, BS32 4SQ. 01454 462659
> 
> If a car is a horseless carriage then is a motorcycle a horseless horse?
> --
> To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Getting physical addresses of mmap'd pages from userspace

[Chris]

Tom Cooksey wrote:
> On Friday 10 October 2008 21:12:13 Robert Schwebel wrote:
>> On Fri, Oct 10, 2008 at 06:15:05PM +0200, Tom Cooksey wrote:
>>> Is there any way to get the physical address of mlock()'d memory from
>>> userspace?
>> What do you want to do? I don't really see a reason to do such strange
>> things any more these days.
> 
> It's quite an annoying problem actually. Basically I have binary drivers for Imagination Technologies's PowerVR graphics drivers. We have tried very hard to get the source code for these drivers and have failed. Eventually ImgTec allowed us to sign an NDA to get the headers for one of their user-space libraries. This library allows us to direct the graphics hardware to render to a specific physical memory region. The problem is that there's no way to find out what the physical addresses are which we need pass to the graphics hardware (via the user-space library). Allthough the library allows us to allocate emory, what I want to do is then blit that memory in a different process. So a client process renders into an off-screen buffer and the server process blits that buffer to the framebuf
 fer. By allowing the server process to do the blit rather than the client process, we can get semi-transparent GL windows. 
> 
> The synchronisation we can do, it's the memory allocation I'm struggling with. If we ask the library to allocate the memory for us, we don't get the physical address to pass to the server process. Instead, we need to allocate memory ourselves and pass the physical address to the library. But like I say, I can't find a way to get the physical address from the kernel.
> 
> I realise getting round closed-source drivers isn't exactly encoraged, but sadly, we really have no choice. :-(
> 
> 
> 
> Cheers,
> 
> Tom
> --
> To unsubscribe from this list: send the line "unsubscribe linux-embedded" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hi,

Could you reserve memory for the intermediate blit buffer at the end of 
system memory by passing a mem=xxx in the kernel command line and then 
use mmap /dev/mem to read it back? One advantage is that mmapping 
/dev/mem disables the processor cache on those pages (by calling
pgprot_noncached(), see drivers/char/mem.c).

Or, if you are going to write a small driver anyhoe, why not have it 
allocate memory using __get_free_pages and pass the physical address 
back via an ioctl or similar. Once again you can mmap /dev/mem to read 
the buffer from user space.

HTH,
Chris.

-- 
Chris Simmonds                   2net Limited
chris@2net.co.uk                 http://www.2net.co.uk/

Re: Getting physical addresses of mmap'd pages from userspace

[Tom Cooksey]

On Monday 13 October 2008 19:29:54 Chris wrote:
> Could you reserve memory for the intermediate blit buffer at the end of 
> system memory by passing a mem=xxx in the kernel command line and then 
> use mmap /dev/mem to read it back? One advantage is that mmapping 
> /dev/mem disables the processor cache on those pages (by calling
> pgprot_noncached(), see drivers/char/mem.c).

I'm not sure if disabling the processor cache completely is what I want to do.
Although if I think about it, I doubt I'll end up using our software rasterizer
too much. In which case, I guess it's only the GPU which will ever be rendering
so the cache problem goes away.

I think I might try a little test application and see if I can get away with
the mem= trick. It's just feels a shame to "reserve" memory & potentially not
use it when it's such a precious resource. I guess it's that or write a kernel
module.


Thanks for everyone's help.



Cheers,

Tom

RE: Getting physical addresses of mmap'd pages from userspace

[Daniel J Laird]

We have a 2D chip on board with our silicon.
It too had a driver in kernel space and a queue in user space that allowed us to post jobs to it like Blit/Fill etc.

We run DirectFB on the platform and we too needed to get Physical addresses etc.

In the end what we did was use Linux FB device to allocate our memory.

We allocate more than we need in the FB device when it is created.
The Double buffered FB comes first and what is left over we use as scratch memory.
This allows us to create surfaces etc in this 'scratch' memory.
When we need the physical address we use the offset from FB base in virtual memory (user space) and add that the physical base address of FB device.
This then allows us to post the job to the 2D hardware.

This approach has worked on 2 different chips now.  Maybe this might help for you?
Daniel Laird

-----Original Message-----
From: linux-embedded-owner@vger.kernel.org [mailto:linux-embedded-owner@vger.kernel.org] On Behalf Of Tom Cooksey
Sent: 2008 Oct 13 07:33
To: Robert Schwebel
Cc: linux-embedded mailing list
Subject: Re: Getting physical addresses of mmap'd pages from userspace

On Friday 10 October 2008 21:12:13 Robert Schwebel wrote:
> On Fri, Oct 10, 2008 at 06:15:05PM +0200, Tom Cooksey wrote:
> > Is there any way to get the physical address of mlock()'d memory from
> > userspace?
>
> What do you want to do? I don't really see a reason to do such strange
> things any more these days.

It's quite an annoying problem actually. Basically I have binary drivers for Imagination Technologies's PowerVR graphics drivers. We have tried very hard to get the source code for these drivers and have failed. Eventually ImgTec allowed us to sign an NDA to get the headers for one of their user-space libraries. This library allows us to direct the graphics hardware to render to a specific physical memory region. The problem is that there's no way to find out what the physical addresses are which we need pass to the graphics hardware (via the user-space library). Allthough the library allows us to allocate emory, what I want to do is then blit that memory in a different process. So a client process renders into an off-screen buffer and the server process blits that buffer to the framebuffe
 r. By allowing the server process to do the blit rather than the client process, we can get semi-transparent GL windows.

The synchronisation we can do, it's the memory allocation I'm struggling with. If we ask the library to allocate the memory for us, we don't get the physical address to pass to the server process. Instead, we need to allocate memory ourselves and pass the physical address to the library. But like I say, I can't find a way to get the physical address from the kernel.

I realise getting round closed-source drivers isn't exactly encoraged, but sadly, we really have no choice. :-(



Cheers,

Tom

Re: Getting physical addresses of mmap'd pages from userspace

[Tom Cooksey]

On Tuesday 14 October 2008 09:31:51 Daniel J Laird wrote:
> We have a 2D chip on board with our silicon.
> It too had a driver in kernel space and a queue in user space that allowed us to post jobs to it like Blit/Fill etc.
> 
> We run DirectFB on the platform and we too needed to get Physical addresses etc.
> 
> In the end what we did was use Linux FB device to allocate our memory.
> 
> We allocate more than we need in the FB device when it is created.
> The Double buffered FB comes first and what is left over we use as scratch memory.
> This allows us to create surfaces etc in this 'scratch' memory.
> When we need the physical address we use the offset from FB base in virtual memory (user space) and add that the physical base address of FB device.
> This then allows us to post the job to the 2D hardware.
> 
> This approach has worked on 2 different chips now.  Maybe this might help for you?

This is something I hadn't concidered - I'd forgotton that fbdev lets you query
the physical address of the framebuffer. I'm not sure it's possible to ask the
framebuffer device to use more memory however. The amount of memory is returned
in the fb_fix_screeninfo struct. I guess you must have implemented your own 
fbdev driver?

On a side note, I'm also tracking the DRM discussions on an in-kernel memory
manager for graphics. It seems a lot of what I'm trying to do is exactly what
the new memory manager is supposed to do. It sounds like this is also what you
needed? I'm asking because the DRM+GEM+modesetting API looks like a good API
for basic hardware as well as the more advanced desktop hardware it's intended
for. It would be really good if one day kernel graphics APIs could be unified.


Cheers,

Tom