From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id A2629C433EF for ; Fri, 13 May 2022 11:24:33 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id 1AF0910E2A5; Fri, 13 May 2022 11:24:33 +0000 (UTC) Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) by gabe.freedesktop.org (Postfix) with ESMTPS id 5981E10E29F for ; Fri, 13 May 2022 11:24:31 +0000 (UTC) Received: by mail-ed1-x536.google.com with SMTP id z19so9580991edx.9 for ; Fri, 13 May 2022 04:24:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :cc:references:from:in-reply-to; bh=RtFlB8M+6XS2nBj7bk7hiutHv26sa7FTtb1PSd488Ps=; b=gvK1hXj8n+N0NSWwkPSpFUgAPBZDk8kCDQiOOp5w5MsMP3xFPN3xMjD7V5LjHu2vFp WI2iZ0+2m7PamWsuHS6nNdF2d+XFBG1hmxFTmjSLxsOyldgWlaGK9ra5HmIOgzQ4z3RX 1qJpQsUjUzxgn6K04FWV3L03nj6vrMtXlwr6aA8NOwC+uaR8ZHXZxCbTtmm3MtzbeqUn WbUsE4PDWyhThto9usj4IBNtPKmCCZ6ft/AGWaPOEgztLl0SAYj5qj6OT8d7p8LZFoId afTFkkWvU5CxtujjDmCFxzL2J16I+noRfQpV9PlWufeQT10h5BK3Fkq8Pa8XOnQnV+lh pwtw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to; bh=RtFlB8M+6XS2nBj7bk7hiutHv26sa7FTtb1PSd488Ps=; b=nd/dl9lbAQuL09ipNRvF0YRwORUvTneLs+MVSsYxnaSKlbJbnxpr571RyXTvlCGzUl n1MGPQibm15pzJWyDDhaJYdRSQKUbJ4lmYbmjqgQmmoh8Sl+ZS9xjsMIbagkti+0d4Nb D0AISOHAdBLi8TFzXDjLtP5DkPLAAbyEWm6zoGchJzdPL2l5FE/h6epU9YCqZnh2xcDX Id7+PcJa/L8myhz9SHOTjjb3dfGLJBK57QyZaTJFtgjITH3dhIVBiHSreJNwRZDnnIB/ TFVZLv17DJ4vxyfK5b834WkbPKkaFELcqAfYYneyKSa45yhvCJiRm0FfZsC8mu3WlIPy cxRA== X-Gm-Message-State: AOAM533fSlOF6PIxyNR8AR1QcOTdAQwe8tRVAl8bOJkLKiTbVUPGh0aV YYG2CscIFx3Oxk3Ywe4/aBgCmAceVE0= X-Google-Smtp-Source: ABdhPJyE6wyuIGZtWXIMO2l9tbV1rOT+P2IhzN6qyIN9ICApi8AoMLapcfTf/LB5m0wfRqtECc1C/w== X-Received: by 2002:a05:6402:51d3:b0:428:ce4a:69b with SMTP id r19-20020a05640251d300b00428ce4a069bmr15618615edd.72.1652441069773; Fri, 13 May 2022 04:24:29 -0700 (PDT) Received: from ?IPV6:2a02:908:1256:79a0:4107:8049:874f:2444? ([2a02:908:1256:79a0:4107:8049:874f:2444]) by smtp.gmail.com with ESMTPSA id v18-20020a1709063bd200b006f3ef214e57sm641783ejf.189.2022.05.13.04.24.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 13 May 2022 04:24:29 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------iR0vBbYBnuWj0mmQQJdZthWk" Message-ID: <63d547b5-c0a3-812c-e019-ed2a2010fe4b@gmail.com> Date: Fri, 13 May 2022 13:24:27 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: [PATCH 1/3] drm/amdgpu: add AMDGPU_GEM_CREATE_DISCARDABLE Content-Language: en-US To: =?UTF-8?B?TWFyZWsgT2zFocOhaw==?= References: <20220506112312.347519-1-christian.koenig@amd.com> <11d9492c-f727-f149-d473-9cda4bab2760@gmail.com> From: =?UTF-8?Q?Christian_K=c3=b6nig?= In-Reply-To: X-BeenThere: amd-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion list for AMD gfx List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: amd-gfx mailing list Errors-To: amd-gfx-bounces@lists.freedesktop.org Sender: "amd-gfx" This is a multi-part message in MIME format. --------------iR0vBbYBnuWj0mmQQJdZthWk Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Well the best placement is guaranteed as long as the application doesn't do any nonsense (e.g. trying to allocate a buffer larger than available VRAM). The VM_ALWAYS_VALID flag doesn't affect any of that handling. Regards, Christian. Am 13.05.22 um 00:17 schrieb Marek Olšák: > Would it be better to set the VM_ALWAYS_VALID flag to have a greater > guarantee that the best placement will be chosen? > > See, the main feature is getting the best placement, not being > discardable. The best placement is a hw design requirement due to > using memory for uses that are expected to have performance similar to > onchip SRAMs. We need to make sure the best placement is guaranteed if > it's VRAM. > > Marek > > On Thu., May 12, 2022, 03:26 Christian König, > wrote: > > Am 12.05.22 um 00:06 schrieb Marek Olšák: >> 3rd question: Is it worth using this on APUs? > > It makes memory management somewhat easier when we are really OOM. > > E.g. it should also work for GTT allocations and when the core > kernel says "Hey please free something up or I will start the > OOM-killer" it's something we can easily throw away. > > Not sure how many of those buffers we have, but marking everything > which is temporary with that flag is probably a good idea. > >> >> Thanks, >> Marek >> >> On Wed, May 11, 2022 at 5:58 PM Marek Olšák wrote: >> >> Will the kernel keep all discardable buffers in VRAM if VRAM >> is not overcommitted by discardable buffers, or will other >> buffers also affect the placement of discardable buffers? >> > > Regarding the eviction pressure the buffers will be handled like > any other buffer, but instead of preserving the content it is just > discarded on eviction. > >> >> Do evictions deallocate the buffer, or do they keep an >> allocation in GTT and only the copy is skipped? >> > > It really deallocates the backing store of the buffer, just keeps > a dummy page array around where all entries are NULL. > > There is a patch set on the mailing list to make this a little bit > more efficient, but even using the dummy page array should only > have a few bytes overhead. > > Regards, > Christian. > >> >> Thanks, >> Marek >> >> On Wed, May 11, 2022 at 3:08 AM Marek Olšák >> wrote: >> >> OK that sounds good. >> >> Marek >> >> On Wed, May 11, 2022 at 2:04 AM Christian König >> wrote: >> >> Hi Marek, >> >> Am 10.05.22 um 22:43 schrieb Marek Olšák: >>> A better flag name would be: >>> AMDGPU_GEM_CREATE_BEST_PLACEMENT_OR_DISCARD >> >> A bit long for my taste and I think the best >> placement is just a side effect. >> >>> >>> Marek >>> >>> On Tue, May 10, 2022 at 4:13 PM Marek Olšák >>> wrote: >>> >>> Does this really guarantee VRAM placement? The >>> code doesn't say anything about that. >>> >> >> Yes, see the code here: >> >>> >>> diff --git >>> a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c >>> b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c >>> index 8b7ee1142d9a..1944ef37a61e 100644 >>> --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c >>> +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c >>> @@ -567,6 +567,7 @@ int >>> amdgpu_bo_create(struct amdgpu_device *adev, >>>                 bp->domain; >>>         bo->allowed_domains = >>> bo->preferred_domains; >>>         if (bp->type != ttm_bo_type_kernel && >>> +           !(bp->flags & >>> AMDGPU_GEM_CREATE_DISCARDABLE) && >>>             bo->allowed_domains == >>> AMDGPU_GEM_DOMAIN_VRAM) >>> bo->allowed_domains |= AMDGPU_GEM_DOMAIN_GTT; >>> >> >> The only case where this could be circumvented is >> when you try to allocate more than physically >> available on an APU. >> >> E.g. you only have something like 32 MiB VRAM and >> request 64 MiB, then the GEM code will catch the >> error and fallback to GTT (IIRC). >> >> Regards, >> Christian. >> > --------------iR0vBbYBnuWj0mmQQJdZthWk Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit Well the best placement is guaranteed as long as the application doesn't do any nonsense (e.g. trying to allocate a buffer larger than available VRAM).

The VM_ALWAYS_VALID flag doesn't affect any of that handling.

Regards,
Christian.

Am 13.05.22 um 00:17 schrieb Marek Olšák:
Would it be better to set the VM_ALWAYS_VALID flag to have a greater guarantee that the best placement will be chosen?

See, the main feature is getting the best placement, not being discardable. The best placement is a hw design requirement due to using memory for uses that are expected to have performance similar to onchip SRAMs. We need to make sure the best placement is guaranteed if it's VRAM.

Marek

On Thu., May 12, 2022, 03:26 Christian König, <ckoenig.leichtzumerken@gmail.com> wrote:
Am 12.05.22 um 00:06 schrieb Marek Olšák:
3rd question: Is it worth using this on APUs?

It makes memory management somewhat easier when we are really OOM.

E.g. it should also work for GTT allocations and when the core kernel says "Hey please free something up or I will start the OOM-killer" it's something we can easily throw away.

Not sure how many of those buffers we have, but marking everything which is temporary with that flag is probably a good idea.


Thanks,
Marek

On Wed, May 11, 2022 at 5:58 PM Marek Olšák <maraeo@gmail.com> wrote:
Will the kernel keep all discardable buffers in VRAM if VRAM is not overcommitted by discardable buffers, or will other buffers also affect the placement of discardable buffers?

Regarding the eviction pressure the buffers will be handled like any other buffer, but instead of preserving the content it is just discarded on eviction.


Do evictions deallocate the buffer, or do they keep an allocation in GTT and only the copy is skipped?

It really deallocates the backing store of the buffer, just keeps a dummy page array around where all entries are NULL.

There is a patch set on the mailing list to make this a little bit more efficient, but even using the dummy page array should only have a few bytes overhead.

Regards,
Christian.


Thanks,
Marek

On Wed, May 11, 2022 at 3:08 AM Marek Olšák <maraeo@gmail.com> wrote:
OK that sounds good.

Marek

On Wed, May 11, 2022 at 2:04 AM Christian König <ckoenig.leichtzumerken@gmail.com> wrote:
Hi Marek,

Am 10.05.22 um 22:43 schrieb Marek Olšák:
A better flag name would be:
AMDGPU_GEM_CREATE_BEST_PLACEMENT_OR_DISCARD

A bit long for my taste and I think the best placement is just a side effect.


Marek

On Tue, May 10, 2022 at 4:13 PM Marek Olšák <maraeo@gmail.com> wrote:
Does this really guarantee VRAM placement? The code doesn't say anything about that.

Yes, see the code here:


diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
index 8b7ee1142d9a..1944ef37a61e 100644
--- a/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
+++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_object.c
@@ -567,6 +567,7 @@ int amdgpu_bo_create(struct amdgpu_device *adev,
                bp->domain;
        bo->allowed_domains = bo->preferred_domains;
        if (bp->type != ttm_bo_type_kernel &&
+           !(bp->flags & AMDGPU_GEM_CREATE_DISCARDABLE) &&
            bo->allowed_domains == AMDGPU_GEM_DOMAIN_VRAM)
                bo->allowed_domains |= AMDGPU_GEM_DOMAIN_GTT;

The only case where this could be circumvented is when you try to allocate more than physically available on an APU.

E.g. you only have something like 32 MiB VRAM and request 64 MiB, then the GEM code will catch the error and fallback to GTT (IIRC).

Regards,
Christian.


--------------iR0vBbYBnuWj0mmQQJdZthWk--