howto use ioremap_wc?

howto use ioremap_wc?

[Brice Goglin]

Hello,

We're looking at using ioremap_wc() in myri10ge. No drivers seem to be
using it yet, so I'd like to get some clarification regarding ioremap_wc
failures, MTRR and so on.

What we currently do is mtrr_add() and then ioremap. Depending on the
mtrr_add() success, we use the "wc_fifo" or regular PIO with fences to
submit requests to the NIC. How are we supposed to switch this to
ioremap_wc?

Are we sure that if the arch supports _wc, it does not return success
when the underlying plain ioremap worked but setting up _wc() failed? If
so, why does it revert to ioremap_nocache when PAT isn't enabled?

Can we keep the mtrr_add() in the regular path? Or are we supposed to
drop it when the arch provides ioremap_wc and it did not fail?

thanks,
Brice

Re: howto use ioremap_wc?

[Arjan van de Ven]

On Sat, 31 May 2008 11:02:54 +0200
Brice Goglin <brice@myri.com> wrote:

> Hello,
> 
> We're looking at using ioremap_wc() in myri10ge. No drivers seem to be
> using it yet, so I'd like to get some clarification regarding
> ioremap_wc failures, MTRR and so on.
> 
> What we currently do is mtrr_add() and then ioremap. Depending on the
> mtrr_add() success, we use the "wc_fifo" or regular PIO with fences to
> submit requests to the NIC. 

Ok this leads to a question: since write combining is effectively an
extension (eg relaxation) to uncached, how much do you care if you
actually get uncached? Eg can you just use the "WC" function even for
the case where you get an uncached mapping ?

> How are we supposed to switch this to
> ioremap_wc?
> 
> Are we sure that if the arch supports _wc, it does not return success
> when the underlying plain ioremap worked but setting up _wc() failed?

why would it not return success? You asked for a mapping with a set of
guarantees, and you got something that adheres to those guarantees (and
then some ;(...

Would you expect ioremap_cache() to fail if you couldn't get a cachable
mapping? That would suck (hint: on PC's it will then fail 99.9% of the
time)....

> If so, why does it revert to ioremap_nocache when PAT isn't enabled?

Actually it would make sense for the ioremap_wc() implementation to try
to add an mtrr I suppose... it's better to be done there than trying to
do it in some driver....

-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: howto use ioremap_wc?

[Brice Goglin]

Arjan van de Ven wrote:
> Brice Goglin <brice@myri.com> wrote
>> We're looking at using ioremap_wc() in myri10ge. No drivers seem to be
>> using it yet, so I'd like to get some clarification regarding
>> ioremap_wc failures, MTRR and so on.
>>
>> What we currently do is mtrr_add() and then ioremap. Depending on the
>> mtrr_add() success, we use the "wc_fifo" or regular PIO with fences to
>> submit requests to the NIC. 
>>     
>
> Ok this leads to a question: since write combining is effectively an
> extension (eg relaxation) to uncached, how much do you care if you
> actually get uncached? Eg can you just use the "WC" function even for
> the case where you get an uncached mapping ?
>   

WC is strictly required for our "wcfifo" path, but this path is actually
not so important nowadays. It is disabled by default and might even be
removed in the future. So, no, myri10ge itself does not really need to
know whether the mapping is actually WC.

The only case I see where it would be helpful is to avoid calling
mtrr_add when WC got enabled through PAT. See below.

> Actually it would make sense for the ioremap_wc() implementation to try
> to add an mtrr I suppose... it's better to be done there than trying to
> do it in some driver....
>   

Agreed, that would be nice!

>From what we discussed here after your reply, our plan is now to just
replace
    ioremap+mtrr_add
with
    ioremap_wc+mtrr_add

If mtrr_add() is moved into ioremap_wc(), we'll remove it from myri10ge.
For now, when PAT is enabled, we may have PAT + MTRR both doing WC, but
I don't think it can break anything, right?

thanks,
Brice

Re: howto use ioremap_wc?

[Loic Prylli]

On 06/02/2008 12:27 PM, Brice Goglin wrote:
> Arjan van de Ven wrote:
>   
>> Ok this leads to a question: since write combining is effectively an
>> extension (eg relaxation) to uncached, how much do you care if you
>> actually get uncached? Eg can you just use the "WC" function even for
>> the case where you get an uncached mapping ?
>>   
>>     
>
> WC is strictly required for our "wcfifo" path,




To be more accurate about the above statement, that codepath will still 
work correctly even if the mapping ends-up being uncached, but that 
could lead to a 16X slowdown on some machines, since that path was 
really designed for the WC case.


Anyway that codepath does not really matter as Brice mentioned 
afterwards :-)



>  but this path is actually
> not so important nowadays. It is disabled by default and might even be
> removed in the future. So, no, myri10ge itself does not really need to
> know whether the mapping is actually WC.
>   




Loic

Re: howto use ioremap_wc?

[Arjan van de Ven]

On Mon, 02 Jun 2008 18:27:31 +0200
Brice Goglin <brice@myri.com> wrote:


> 
> Agreed, that would be nice!
> 
> >From what we discussed here after your reply, our plan is now to just
> replace
>     ioremap+mtrr_add

which doesn't work as ioremap() may/will force uncached ;)


> with
>     ioremap_wc+mtrr_add
> 
> If mtrr_add() is moved into ioremap_wc(), we'll remove it from
> myri10ge. For now, when PAT is enabled, we may have PAT + MTRR both
> doing WC, but I don't think it can break anything, right?

Both having WC is nicely consistent and the right thing happens...

-- 
If you want to reach me at my work email, use arjan@linux.intel.com
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org

Re: howto use ioremap_wc?

[Roland Dreier]

This thread reminds me of another question about interfaces for using
PAT.  Are there any plans for making pgprot_writecombine() a generic
interface that portable drivers can use to map PCI memory into userspace
with WC turned on?  (Falling back to uncached when WC is not available
for whatever reason is fine for the case I have in mind)

Maybe it's better to ask at a higher level: what's the right thing for a
driver to do in an mmap method when it wants to expose PCI memory to
userspace with WC turned on if possible?  I see that the PCI sysfs code
on x86 does it using non-exported (hence not available to a driver
buildable as a module) interfaces.

Thanks,
  Roland