seems ttm_bo_handle_move_mem not flush cache, if BO move SYS->VRAM?

seems ttm_bo_handle_move_mem not flush cache, if BO move SYS->VRAM?

[april]

hi all:

I have a question:

If a BO in VRAM (WC) evict to SYS memory(may be cached),  and user
process still can access it event it is in SYS memory (may be cached)
.
when this BO volidate to VRAM,  It seems "ttm_bo_handle_move_mem"  not
flush cache (If evict to SYS memory with cached).

 But flush cache is needed, If driver use DMA to handle BO
move(SYS->VRAM), should "ttm_bo_handle_move_mem" handle this? or
driver do it in "driver->move_notify" before "driver->move" called?

Thanks

Re: seems ttm_bo_handle_move_mem not flush cache, if BO move SYS->VRAM?

[Jerome Glisse]

On Tue, Nov 30, 2010 at 10:58 PM, april <aapril03@gmail.com> wrote:
> hi all:
>
> I have a question:
>
> If a BO in VRAM (WC) evict to SYS memory(may be cached),  and user
> process still can access it event it is in SYS memory (may be cached)
> .
> when this BO volidate to VRAM,  It seems "ttm_bo_handle_move_mem"  not
> flush cache (If evict to SYS memory with cached).
>
>  But flush cache is needed, If driver use DMA to handle BO
> move(SYS->VRAM), should "ttm_bo_handle_move_mem" handle this? or
> driver do it in "driver->move_notify" before "driver->move" called?
>
> Thanks

First not all cpu have instruction to explicitly flush cpu cache (at
least i don't think so), second most GPU gart, through which DMA goes,
can be CPU cache coherent, ie they will ask the CPU to flush its cache
and this is what TTM expect. The setup is done at TTM gart
registration where you say if your gart can be cache coherent or not
(AGP for instance) if the gart is not cache coherent then ttm will
change caching policy on bo page and this should trigger cache flush
too.

Cheers,
Jerome

Re: seems ttm_bo_handle_move_mem not flush cache, if BO move SYS->VRAM?

[Thomas Hellstrom]

On 12/01/2010 04:58 AM, april wrote:
> hi all:
>
> I have a question:
>
> If a BO in VRAM (WC) evict to SYS memory(may be cached),  and user
> process still can access it event it is in SYS memory (may be cached)
> .
> when this BO volidate to VRAM,  It seems "ttm_bo_handle_move_mem"  not
> flush cache (If evict to SYS memory with cached).
>
>   But flush cache is needed, If driver use DMA to handle BO
> move(SYS->VRAM), should "ttm_bo_handle_move_mem" handle this? or
> driver do it in "driver->move_notify" before "driver->move" called?
>
>    

April, in order to understand your proble, Can you be a bit more 
specific about.

1) Which cache should be flushed? Cpu cache or GPU cache.

2) Why should it be flushed? For CPU caches, TTM only flushes when 
transitioning the same page from cached to write-combined. If you use 
DMA from system memory to VRAM, and need to flush the system pages 
before performing the DMA operation, that should be done in the driver 
move function.

/Thomas


> Thanks
> _______________________________________________
> dri-devel mailing list
> dri-devel@lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/dri-devel
>    

Re: seems ttm_bo_handle_move_mem not flush cache, if BO move SYS->VRAM?

[april]

thanks Thomas

I want to flush CPU cache, before use DMA  from system memory to VRAM.

Seems I should do this in driver move function itself.

bty,
when BO changes from cached to write-combined, CPU cache will be
flushed, and all the pages in this BO will do this.
but, may be only a small part of page's cache flush is really
needed(CPU just write to a few pages in BO).

Is there any ways to solve this?

or there's no need to do this? (clflush those pages that not in cpu
cache not cost too much?)


Thank you

2010/12/2 Thomas Hellstrom <thomas@shipmail.org>:
> On 12/01/2010 04:58 AM, april wrote:
>>
>> hi all:
>>
>> I have a question:
>>
>> If a BO in VRAM (WC) evict to SYS memory(may be cached),  and user
>> process still can access it event it is in SYS memory (may be cached)
>> .
>> when this BO volidate to VRAM,  It seems "ttm_bo_handle_move_mem"  not
>> flush cache (If evict to SYS memory with cached).
>>
>>  But flush cache is needed, If driver use DMA to handle BO
>> move(SYS->VRAM), should "ttm_bo_handle_move_mem" handle this? or
>> driver do it in "driver->move_notify" before "driver->move" called?
>>
>>
>
> April, in order to understand your proble, Can you be a bit more specific
> about.
>
> 1) Which cache should be flushed? Cpu cache or GPU cache.
>
> 2) Why should it be flushed? For CPU caches, TTM only flushes when
> transitioning the same page from cached to write-combined. If you use DMA
> from system memory to VRAM, and need to flush the system pages before
> performing the DMA operation, that should be done in the driver move
> function.
>
> /Thomas
>
>
>> Thanks
>> _______________________________________________
>> dri-devel mailing list
>> dri-devel@lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/dri-devel
>>
>
>

Re: seems ttm_bo_handle_move_mem not flush cache, if BO move SYS->VRAM?

[Thomas Hellstrom]

On 12/02/2010 03:49 AM, april wrote:
> thanks Thomas
>
> I want to flush CPU cache, before use DMA  from system memory to VRAM.
>
> Seems I should do this in driver move function itself.
>    

Yes. That's the right way.

> bty,
> when BO changes from cached to write-combined, CPU cache will be
> flushed, and all the pages in this BO will do this.
> but, may be only a small part of page's cache flush is really
> needed(CPU just write to a few pages in BO).
>
> Is there any ways to solve this?
>    

Unfortunately not at the moment. It would require tracking of dirty 
pages in TTM and that is on my to-do list, but that also requries 
changes to the linux kernel VM system. At the moment I'm not completely 
sure how costly a clflush is for a cache line that's not in the CPU cache.

For BOs that are almost only written to, you should try to keep also 
system pages write-combined. In that case they are allocated 
write-combined from the TTM memory pool and the cache changing doesn't 
need to occur.

/Thomas

> or there's no need to do this? (clflush those pages that not in cpu
> cache not cost too much?)
>
>
> Thank you
>
> 2010/12/2 Thomas Hellstrom<thomas@shipmail.org>:
>    
>> On 12/01/2010 04:58 AM, april wrote:
>>      
>>> hi all:
>>>
>>> I have a question:
>>>
>>> If a BO in VRAM (WC) evict to SYS memory(may be cached),  and user
>>> process still can access it event it is in SYS memory (may be cached)
>>> .
>>> when this BO volidate to VRAM,  It seems "ttm_bo_handle_move_mem"  not
>>> flush cache (If evict to SYS memory with cached).
>>>
>>>   But flush cache is needed, If driver use DMA to handle BO
>>> move(SYS->VRAM), should "ttm_bo_handle_move_mem" handle this? or
>>> driver do it in "driver->move_notify" before "driver->move" called?
>>>
>>>
>>>        
>> April, in order to understand your proble, Can you be a bit more specific
>> about.
>>
>> 1) Which cache should be flushed? Cpu cache or GPU cache.
>>
>> 2) Why should it be flushed? For CPU caches, TTM only flushes when
>> transitioning the same page from cached to write-combined. If you use DMA
>> from system memory to VRAM, and need to flush the system pages before
>> performing the DMA operation, that should be done in the driver move
>> function.
>>
>> /Thomas
>>
>>
>>      
>>> Thanks
>>> _______________________________________________
>>> dri-devel mailing list
>>> dri-devel@lists.freedesktop.org
>>> http://lists.freedesktop.org/mailman/listinfo/dri-devel
>>>
>>>        
>>
>>      

Re: seems ttm_bo_handle_move_mem not flush cache, if BO move SYS->VRAM?

[Jerome Glisse]

On Thu, Dec 2, 2010 at 2:35 AM, Thomas Hellstrom <thomas@shipmail.org> wrote:
> On 12/02/2010 03:49 AM, april wrote:
>>
>> thanks Thomas
>>
>> I want to flush CPU cache, before use DMA  from system memory to VRAM.
>>
>> Seems I should do this in driver move function itself.
>>
>
> Yes. That's the right way.
>
>> bty,
>> when BO changes from cached to write-combined, CPU cache will be
>> flushed, and all the pages in this BO will do this.
>> but, may be only a small part of page's cache flush is really
>> needed(CPU just write to a few pages in BO).
>>
>> Is there any ways to solve this?
>>
>
> Unfortunately not at the moment. It would require tracking of dirty pages in
> TTM and that is on my to-do list, but that also requries changes to the
> linux kernel VM system. At the moment I'm not completely sure how costly a
> clflush is for a cache line that's not in the CPU cache.
>
> For BOs that are almost only written to, you should try to keep also system
> pages write-combined. In that case they are allocated write-combined from
> the TTM memory pool and the cache changing doesn't need to occur.
>
> /Thomas
>

Note that clflush is only available on cpu with sse2 iirc.

April also most of today GPU can do cache coherent DMA on PCI/PCIE (ie
no need to flush cache or write-combined page), may i ask what is the
GPU you are working on ?

Cheers,
Jerome