Re: [PATCH] drm/ttm: add minimum residency constraint for bo eviction

[Jerome Glisse <j.glisse@gmail.com>]

On Thu, Nov 29, 2012 at 09:41:34AM +0100, Thomas Hellstrom wrote:
> On 11/29/2012 12:24 AM, Jerome Glisse wrote:
> >On Wed, Nov 28, 2012 at 6:18 PM, Thomas Hellstrom <thomas@shipmail.org> wrote:
> >>On 11/28/2012 04:58 PM, j.glisse@gmail.com wrote:
> >>>From: Jerome Glisse <jglisse@redhat.com>
> >>>
> >>>This patch add a minimum residency time configurable for each memory
> >>>pool (VRAM, GTT, ...). Intention is to avoid having a lot of memory
> >>>eviction from VRAM up to a point where the GPU pretty much spend all
> >>>it's time moving things in and out.
> >>
> >>This patch seems odd to me.
> >>
> >>It seems the net effect is to refuse evictions from VRAM and make buffers go
> >>somewhere else, and that makes things faster?
> >>
> >>Why don't they go there in the first place instead of trying to force them
> >>into VRAM,
> >>when VRAM is full?
> >>
> >>/Thomas
> >It's mostly a side effect of cs and validating with each cs, if boA is
> >in cs1 and not in cs2 and boB is in cs1 but not in cs2 than boA could
> >be evicted by cs2 and boB moved in, if next cs ie cs3 is like cs1 then
> >boA move back again and boB is evicted, then you get cs4 which
> >reference boB but not boA, boA get evicted and boB move in ... So ttm
> >just spend its time doing eviction but he doing so because it's ask by
> >the driver to do so. Note that what is costly there is not the bo move
> >in itself but the page allocation.
> 
> Yes, this is the cause of the trashing, but that was not what I asked.
> 
> What your patch is doing is looking at the last recently used bo, to
> check if it has been
> resident for at least 500ms. Otherwise it refuses eviction for *all*
> buffers of that memory type.
> 
> This means new buffers can't fit in VRAM, they need to go somewhere
> else. Perhaps TT?
> 
> So my question was. If VRAM is full, instead of starting to evict,
> why not put new buffers in TT, so that
> 
> placement(GEM_DOMAIN_VRAM) = VRAM | TT  // Prefer VRAM but allow TT
> before starting to evict.
> busy_placement(GEM_DOMAIN_VRAM) = TT | VRAM // *If* we need to
> evict, prefer evicting TT, then evict VRAM)
> 
> This will more or less mimic carmack's algorithm by using TT as his
> "MRU scratch space".

Well not exactly, the ping-pong btw vram and tt is still very likely
to happen. If we always | GTT placement then some buffer that would
need to be in vram never goes there because some older buffer that
haven't been use in ages is present in vram. Only way to force those
buffer to be evicted is to ask for vram only (case i used i never
filled up vram + gtt so there was always room in either one).

So this way you still allow eviction of old vram buffer, and avoid
too much ping pong with heavily used buffer being moved in and out.

Anyway i did this as a quick hack to thinking it might interest
other. Proper solution as i said lies in not validating, to different
placement, buffer at each cs and use some worker thread to evict
things from vram and do things like compaction to minimize
fragmentation.

> And as a side note, your patch breaks
> ttm_bo_force_list_clean()
> which should be used at GPU memory exhaustion to avoid OOM due to
> fragmentation and for those drivers that
> implement VRAM cleanup on VT switch and / or suspend / hibernation.
> 
> /Thomas

Yeah, i did not paid much attention to the whole cleanup phase, was
only interested in dirty prototype.

Cheers,
Jerome