No GTT->VRAM unswapping with amdgpu?

No GTT->VRAM unswapping with amdgpu?

[Linus Lüssing]

Hi,

Initially noticed this in some games, which seem to be a bit lazy
with their VRAM allocations + freeing, that performance would drop
dramatically / become unusable once the VRAM is full and GTT gets
used. For instance in No Man's Sky after jumping a solar system
or on DCS World on several maps and in multiplayer. I'm using an
ATI/AMD Radeon RX 6650 XT, 8GB VRAM with an eGPU enclosure
(Razer Core X Chroma, Thunderbolt 3), connected via a 40Gbit/s
USB4 port. Which usually has great performance, until I hit the
VRAM limit.

I scripted some further tests and benchmarking around
memmtest_vulkan(*) and these were the results:

https://github.com/T-X/linux-amdgpu-radeon-vram-swapping-test/

I would have expected roughly the same speeds in all tests.
However tests 5) and 6) yielded significantly lower performance.

Which leads me to the conclusion that anything that gets allocated
on GTT stays there and is never unswapped from GTT / system memory
back to VRAM?

I also read that there was some rework with a new TTM allocator
in 2020. But dynamic (un)swapping via TTM seems currently unused?
Is this expected?

Regards, Linus

(*): https://github.com/GpuZelenograd/memtest_vulkan

Re: No GTT->VRAM unswapping with amdgpu?

[Alex Deucher]

On Thu, May 11, 2023 at 9:38 PM Linus Lüssing <linus.luessing@c0d3.blue> wrote:
>
> Hi,
>
> Initially noticed this in some games, which seem to be a bit lazy
> with their VRAM allocations + freeing, that performance would drop
> dramatically / become unusable once the VRAM is full and GTT gets
> used. For instance in No Man's Sky after jumping a solar system
> or on DCS World on several maps and in multiplayer. I'm using an
> ATI/AMD Radeon RX 6650 XT, 8GB VRAM with an eGPU enclosure
> (Razer Core X Chroma, Thunderbolt 3), connected via a 40Gbit/s
> USB4 port. Which usually has great performance, until I hit the
> VRAM limit.
>
> I scripted some further tests and benchmarking around
> memmtest_vulkan(*) and these were the results:
>
> https://github.com/T-X/linux-amdgpu-radeon-vram-swapping-test/
>
> I would have expected roughly the same speeds in all tests.
> However tests 5) and 6) yielded significantly lower performance.
>
> Which leads me to the conclusion that anything that gets allocated
> on GTT stays there and is never unswapped from GTT / system memory
> back to VRAM?

It will get swapped back to VRAM if it makes sense for performance.
The driver throttles swapping if there is too much contention to avoid
the overhead of swapping large amounts of memory back and forth
between vram and gtt for every command submission.

Alex

>
> I also read that there was some rework with a new TTM allocator
> in 2020. But dynamic (un)swapping via TTM seems currently unused?
> Is this expected?
>
> Regards, Linus
>
> (*): https://github.com/GpuZelenograd/memtest_vulkan

Re: No GTT->VRAM unswapping with amdgpu?

[Alex Deucher]

On Mon, May 15, 2023 at 10:42 AM Alex Deucher <alexdeucher@gmail.com> wrote:
>
> On Thu, May 11, 2023 at 9:38 PM Linus Lüssing <linus.luessing@c0d3.blue> wrote:
> >
> > Hi,
> >
> > Initially noticed this in some games, which seem to be a bit lazy
> > with their VRAM allocations + freeing, that performance would drop
> > dramatically / become unusable once the VRAM is full and GTT gets
> > used. For instance in No Man's Sky after jumping a solar system
> > or on DCS World on several maps and in multiplayer. I'm using an
> > ATI/AMD Radeon RX 6650 XT, 8GB VRAM with an eGPU enclosure
> > (Razer Core X Chroma, Thunderbolt 3), connected via a 40Gbit/s
> > USB4 port. Which usually has great performance, until I hit the
> > VRAM limit.
> >
> > I scripted some further tests and benchmarking around
> > memmtest_vulkan(*) and these were the results:
> >
> > https://github.com/T-X/linux-amdgpu-radeon-vram-swapping-test/
> >
> > I would have expected roughly the same speeds in all tests.
> > However tests 5) and 6) yielded significantly lower performance.
> >
> > Which leads me to the conclusion that anything that gets allocated
> > on GTT stays there and is never unswapped from GTT / system memory
> > back to VRAM?
>
> It will get swapped back to VRAM if it makes sense for performance.
> The driver throttles swapping if there is too much contention to avoid
> the overhead of swapping large amounts of memory back and forth
> between vram and gtt for every command submission.

RADV also dynamically prefers GTT if you are using an eGPU and memory
is contended:
https://cgit.freedesktop.org/mesa/mesa/commit/?id=95d06343c693aa12b4cda5cda31d81fae138b0ec

Alex

>
> Alex
>
> >
> > I also read that there was some rework with a new TTM allocator
> > in 2020. But dynamic (un)swapping via TTM seems currently unused?
> > Is this expected?
> >
> > Regards, Linus
> >
> > (*): https://github.com/GpuZelenograd/memtest_vulkan