[Intel-xe] [PATCH i-g-t v2] tests/xe/xe_noexec_ping_pong: Add a test to expose unnecessary rebinds

[Intel-xe] [PATCH i-g-t v2] tests/xe/xe_noexec_ping_pong: Add a test to expose unnecessary rebinds

[Thomas Hellström]

This test creates compute vms, binds a couple of bos and an engine each,
thus redying it for execution. However, VRAM memory is over-
committed and while there is still nothing to execute, an eviction
will trigger the VM's rebind worker to rebind the evicted bo, which
will in turn trigger another eviction and so on.

Since we don't have eviction stats yet we need to watch "top" for
the rebind kworkers using a lot of CPU while the test idles.

The correct driver behaviour should be not to rebind anything unless
there is work queued on one of the VM's compute engines.

v2:
- Use the ALIGN macro for aligning
- Reduce the idle pause duration to 10s. (Matthew Brost)
- Add an IGT_TEST_DESCRIPTION()
- Explicity use gt0 for testing.
- Skip if no VRAM available.

Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
---
 tests/meson.build              |   1 +
 tests/xe/xe_noexec_ping_pong.c | 106 +++++++++++++++++++++++++++++++++
 2 files changed, 107 insertions(+)
 create mode 100644 tests/xe/xe_noexec_ping_pong.c

diff --git a/tests/meson.build b/tests/meson.build
index 632e36e0..2e62ff23 100644
--- a/tests/meson.build
+++ b/tests/meson.build
@@ -256,6 +256,7 @@ xe_progs = [
 	'xe_huc_copy',
 	'xe_mmap',
 	'xe_mmio',
+	'xe_noexec_ping_pong',
 	'xe_pm',
 	'xe_prime_self_import',
 	'xe_query',
diff --git a/tests/xe/xe_noexec_ping_pong.c b/tests/xe/xe_noexec_ping_pong.c
new file mode 100644
index 00000000..dc14f8cf
--- /dev/null
+++ b/tests/xe/xe_noexec_ping_pong.c
@@ -0,0 +1,106 @@
+// SPDX-License-Identifier: MIT
+/*
+ * Copyright © 2023 Intel Corporation
+ */
+
+#include <unistd.h>
+
+#include "drmtest.h"
+#include "igt.h"
+
+#include "xe/xe_ioctl.h"
+#include "xe/xe_query.h"
+
+#define NUM_VMS 10
+#define NUM_BOS 1
+#define SECONDS_TO_WAIT 10
+
+/**
+ * TEST: Expose compute VM's unnecessary rebinds
+ * Category: Software building block
+ * Sub-category: compute
+ * Test category: functionality test
+ */
+
+/*
+ * This test creates compute vms, binds a couple of bos and an engine each,
+ * thus redying it for execution. However, VRAM memory is over-
+ * committed and while there is still nothing to execute, an eviction
+ * will trigger the VM's rebind worker to rebind the evicted bo, which
+ * will in turn trigger another eviction and so on.
+ *
+ * Since we don't have eviction stats yet we need to watch "top" for
+ * the rebind kworkers using a lot of CPU while the test idles.
+ *
+ * The correct driver behaviour should be not to rebind anything unless
+ * there is worked queued on one of the VM's compute engines.
+ */
+static void test_ping_pong(int fd, struct drm_xe_engine_class_instance *eci)
+{
+	size_t vram_size = xe_vram_size(fd, 0);
+	size_t align = xe_get_default_alignment(fd);
+	size_t bo_size = vram_size / NUM_VMS / NUM_BOS;
+	uint32_t vm[NUM_VMS];
+	uint32_t bo[NUM_VMS][NUM_BOS];
+	uint32_t engines[NUM_VMS];
+	unsigned int i, j;
+
+	igt_skip_on(!bo_size);
+
+	/* Align and make sure we overcommit vram with at least 10% */
+	bo_size = ALIGN(bo_size + bo_size / 10, align);
+
+	/*
+	 * This should not start ping-ponging memory between system and
+	 * VRAM. For now look at top to determine. TODO: Look at eviction
+	 * stats.
+	 */
+	for (i = 0; i < NUM_VMS; ++i) {
+		struct drm_xe_ext_engine_set_property ext = {
+			.base.next_extension = 0,
+			.base.name = XE_ENGINE_EXTENSION_SET_PROPERTY,
+			.property = XE_ENGINE_SET_PROPERTY_COMPUTE_MODE,
+			.value = 1,
+		};
+
+		vm[i] = xe_vm_create(fd, DRM_XE_VM_CREATE_COMPUTE_MODE, 0);
+		for (j = 0; j < NUM_BOS; ++j) {
+			igt_debug("Creating bo size %lu for vm %u\n",
+				  (unsigned long) bo_size,
+				  (unsigned int) vm[i]);
+
+			bo[i][j] = xe_bo_create_flags(fd, vm[i], bo_size,
+						      vram_memory(fd, 0));
+			xe_vm_bind(fd, vm[i], bo[i][j], 0, 0x40000 + j*bo_size,
+				   bo_size, NULL, 0);
+		}
+		engines[i] = xe_engine_create(fd, vm[i], eci,
+					      to_user_pointer(&ext));
+	}
+
+	igt_info("Now sleeping for %ds.\n", SECONDS_TO_WAIT);
+	igt_info("Watch \"top\" for high-cpu kworkers!\n");
+	sleep(SECONDS_TO_WAIT);
+
+	for (i = 0; i < NUM_VMS; ++i) {
+		xe_engine_destroy(fd, engines[i]);
+		for (j = 0; j < NUM_BOS; ++j)
+			gem_close(fd, bo[i][j]);
+		xe_vm_destroy(fd, vm[i]);
+	}
+}
+
+static int fd;
+
+IGT_TEST_DESCRIPTION("Expose compute VM's unnecessary rebinds");
+igt_simple_main
+{
+
+	fd = drm_open_driver(DRIVER_XE);
+	xe_device_get(fd);
+
+	test_ping_pong(fd, xe_hw_engine(fd, 0));
+
+	xe_device_put(fd);
+	close(fd);
+}
-- 
2.39.2

[Intel-xe] ✗ CI.Patch_applied: failure for tests/xe/xe_noexec_ping_pong: Add a test to expose unnecessary rebinds

[Patchwork]

== Series Details ==

Series: tests/xe/xe_noexec_ping_pong: Add a test to expose unnecessary rebinds
URL   : https://patchwork.freedesktop.org/series/115251/
State : failure

== Summary ==

=== Applying kernel patches on branch 'drm-xe-next' with base: ===
commit 8f6c3eaf3f9daab25b31e80c8ba277877fd10547
Author:     Mauro Carvalho Chehab <mchehab@kernel.org>
AuthorDate: Fri Mar 10 09:13:39 2023 +0100
Commit:     Lucas De Marchi <lucas.demarchi@intel.com>
CommitDate: Wed Mar 15 10:28:56 2023 -0700

    drm/xe/xe_uc_fw: Use firmware files from standard locations
    
    The GuC/HuC firmware files used by Xe drivers are the same as
    used by i915. Use the already-known location to find those
    firmware files, for a couple of reasons:
    
    1. Avoid having the same firmware placed on two different
       places on MODULE_FIRMWARE(), if both 915 and xe drivers
       are compiled;
    
    2. Having firmware files located on different locations may end
       creating bigger initramfs, as the same files will be copied
       twice my mkinitrd/dracut/...;
    
    3. this is the place where those firmware files are located at
       https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git
       Upstream doesn't expect them to have on other places;
    
    4. When built with display support, DMC firmware will be
       loaded from i915/ directory. It is very confusing to have
       some firmware files on a different place for the same driver.
    
    Cc: Matthew Brost <matthew.brost@intel.com>
    Cc: Lucas de Marchi <lucas.demarchi@intel.com>
    Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
    Cc: Thomas Hellstrom <thomas.hellstrom@linux.intel.com>
    Cc: Daniel Vetter <daniel@ffwll.ch>
    Cc: David Airlie <airlied@gmail.com>
    Signed-off-by: Mauro Carvalho Chehab <mchehab@kernel.org>
    [ Mostly agree with the direction of "use the firmware blobs from
      upstream at their current location for these platforms". Previous
      directory was not wrong as the plan was to have it handled in the
      upstream firmware repo. For future platforms the location can be
      changed if the support is only in xe ]
    Signed-off-by: Lucas De Marchi <lucas.demarchi@intel.com>
    Link: https://lore.kernel.org/r/20230310081338.3275583-1-mauro.chehab@linux.intel.com
=== git am output follows ===
error: tests/meson.build: does not exist in index
hint: Use 'git am --show-current-patch' to see the failed patch
Applying: tests/xe/xe_noexec_ping_pong: Add a test to expose unnecessary rebinds
Patch failed at 0001 tests/xe/xe_noexec_ping_pong: Add a test to expose unnecessary rebinds
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

Re: [Intel-xe] [PATCH i-g-t v2] tests/xe/xe_noexec_ping_pong: Add a test to expose unnecessary rebinds

[Matthew Brost]

On Thu, Mar 16, 2023 at 10:47:23AM +0100, Thomas Hellström wrote:
> This test creates compute vms, binds a couple of bos and an engine each,
> thus redying it for execution. However, VRAM memory is over-
> committed and while there is still nothing to execute, an eviction
> will trigger the VM's rebind worker to rebind the evicted bo, which
> will in turn trigger another eviction and so on.
> 
> Since we don't have eviction stats yet we need to watch "top" for
> the rebind kworkers using a lot of CPU while the test idles.
> 
> The correct driver behaviour should be not to rebind anything unless
> there is work queued on one of the VM's compute engines.
> 
> v2:
> - Use the ALIGN macro for aligning
> - Reduce the idle pause duration to 10s. (Matthew Brost)
> - Add an IGT_TEST_DESCRIPTION()
> - Explicity use gt0 for testing.
> - Skip if no VRAM available.
> 
> Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>

Reviewed-by: Matthew Brost <matthew.brost@intel.com>

> ---
>  tests/meson.build              |   1 +
>  tests/xe/xe_noexec_ping_pong.c | 106 +++++++++++++++++++++++++++++++++
>  2 files changed, 107 insertions(+)
>  create mode 100644 tests/xe/xe_noexec_ping_pong.c
> 
> diff --git a/tests/meson.build b/tests/meson.build
> index 632e36e0..2e62ff23 100644
> --- a/tests/meson.build
> +++ b/tests/meson.build
> @@ -256,6 +256,7 @@ xe_progs = [
>  	'xe_huc_copy',
>  	'xe_mmap',
>  	'xe_mmio',
> +	'xe_noexec_ping_pong',
>  	'xe_pm',
>  	'xe_prime_self_import',
>  	'xe_query',
> diff --git a/tests/xe/xe_noexec_ping_pong.c b/tests/xe/xe_noexec_ping_pong.c
> new file mode 100644
> index 00000000..dc14f8cf
> --- /dev/null
> +++ b/tests/xe/xe_noexec_ping_pong.c
> @@ -0,0 +1,106 @@
> +// SPDX-License-Identifier: MIT
> +/*
> + * Copyright © 2023 Intel Corporation
> + */
> +
> +#include <unistd.h>
> +
> +#include "drmtest.h"
> +#include "igt.h"
> +
> +#include "xe/xe_ioctl.h"
> +#include "xe/xe_query.h"
> +
> +#define NUM_VMS 10
> +#define NUM_BOS 1
> +#define SECONDS_TO_WAIT 10
> +
> +/**
> + * TEST: Expose compute VM's unnecessary rebinds
> + * Category: Software building block
> + * Sub-category: compute
> + * Test category: functionality test
> + */
> +
> +/*
> + * This test creates compute vms, binds a couple of bos and an engine each,
> + * thus redying it for execution. However, VRAM memory is over-
> + * committed and while there is still nothing to execute, an eviction
> + * will trigger the VM's rebind worker to rebind the evicted bo, which
> + * will in turn trigger another eviction and so on.
> + *
> + * Since we don't have eviction stats yet we need to watch "top" for
> + * the rebind kworkers using a lot of CPU while the test idles.
> + *
> + * The correct driver behaviour should be not to rebind anything unless
> + * there is worked queued on one of the VM's compute engines.
> + */
> +static void test_ping_pong(int fd, struct drm_xe_engine_class_instance *eci)
> +{
> +	size_t vram_size = xe_vram_size(fd, 0);
> +	size_t align = xe_get_default_alignment(fd);
> +	size_t bo_size = vram_size / NUM_VMS / NUM_BOS;
> +	uint32_t vm[NUM_VMS];
> +	uint32_t bo[NUM_VMS][NUM_BOS];
> +	uint32_t engines[NUM_VMS];
> +	unsigned int i, j;
> +
> +	igt_skip_on(!bo_size);
> +
> +	/* Align and make sure we overcommit vram with at least 10% */
> +	bo_size = ALIGN(bo_size + bo_size / 10, align);
> +
> +	/*
> +	 * This should not start ping-ponging memory between system and
> +	 * VRAM. For now look at top to determine. TODO: Look at eviction
> +	 * stats.
> +	 */
> +	for (i = 0; i < NUM_VMS; ++i) {
> +		struct drm_xe_ext_engine_set_property ext = {
> +			.base.next_extension = 0,
> +			.base.name = XE_ENGINE_EXTENSION_SET_PROPERTY,
> +			.property = XE_ENGINE_SET_PROPERTY_COMPUTE_MODE,
> +			.value = 1,
> +		};
> +
> +		vm[i] = xe_vm_create(fd, DRM_XE_VM_CREATE_COMPUTE_MODE, 0);
> +		for (j = 0; j < NUM_BOS; ++j) {
> +			igt_debug("Creating bo size %lu for vm %u\n",
> +				  (unsigned long) bo_size,
> +				  (unsigned int) vm[i]);
> +
> +			bo[i][j] = xe_bo_create_flags(fd, vm[i], bo_size,
> +						      vram_memory(fd, 0));
> +			xe_vm_bind(fd, vm[i], bo[i][j], 0, 0x40000 + j*bo_size,
> +				   bo_size, NULL, 0);
> +		}
> +		engines[i] = xe_engine_create(fd, vm[i], eci,
> +					      to_user_pointer(&ext));
> +	}
> +
> +	igt_info("Now sleeping for %ds.\n", SECONDS_TO_WAIT);
> +	igt_info("Watch \"top\" for high-cpu kworkers!\n");
> +	sleep(SECONDS_TO_WAIT);
> +
> +	for (i = 0; i < NUM_VMS; ++i) {
> +		xe_engine_destroy(fd, engines[i]);
> +		for (j = 0; j < NUM_BOS; ++j)
> +			gem_close(fd, bo[i][j]);
> +		xe_vm_destroy(fd, vm[i]);
> +	}
> +}
> +
> +static int fd;
> +
> +IGT_TEST_DESCRIPTION("Expose compute VM's unnecessary rebinds");
> +igt_simple_main
> +{
> +
> +	fd = drm_open_driver(DRIVER_XE);
> +	xe_device_get(fd);
> +
> +	test_ping_pong(fd, xe_hw_engine(fd, 0));
> +
> +	xe_device_put(fd);
> +	close(fd);
> +}
> -- 
> 2.39.2
>