[igt-dev] [PATCH i-g-t v5 0/9] D3Cold Tool & IGT

[igt-dev] [PATCH i-g-t v5 0/9] D3Cold Tool & IGT

[Anshuman Gupta]

This series adds support for D3Cold Tool and IGT.

v5 rev has addressed the review comment mentioned in
commit version log.

Anshuman Gupta (9):
  test/i915_pm_rpm: Add placement to gem-{mmap-type, execbuf}
  lib/igt_device: Get gfx PCI card root port
  lib/igt_pm: D3Cold runtime pm infrastructure
  tools: Add intel_pm_rpm tool
  tools/intel_pm_rpm: Add an option to setup d3cold
  i915_pm_rpm: Add D3Cold basic subtest
  i915_pm_rpm: Extend gem_exec_stress test with D3Cold
  i915_pm_rpm: Extend gem_execbuf test with D3Cold
  i915_pm_rpm: Extend gem-mmap-type test with D3Cold

 lib/igt_device.c         |  28 ++++
 lib/igt_device.h         |   1 +
 lib/igt_pm.c             | 317 +++++++++++++++++++++++++++++++++++++++
 lib/igt_pm.h             |  24 +++
 tests/i915/i915_pm_rpm.c | 199 ++++++++++++++++++++++--
 tools/intel_pm_rpm.c     | 229 ++++++++++++++++++++++++++++
 tools/meson.build        |   1 +
 7 files changed, 790 insertions(+), 9 deletions(-)
 create mode 100644 tools/intel_pm_rpm.c

-- 
2.26.2

[igt-dev] [PATCH i-g-t v5 1/9] test/i915_pm_rpm: Add placement to gem-{mmap-type, execbuf}

[Anshuman Gupta]

Add memory region placement to gem-{mmap-type, execbuf} subtest.
This will be useful to extend these sub-test for D3Cold.

Cc: Chris Wilson <chris.p.wilson@intel.com>
Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
---
 tests/i915/i915_pm_rpm.c | 23 +++++++++++++++--------
 1 file changed, 15 insertions(+), 8 deletions(-)

diff --git a/tests/i915/i915_pm_rpm.c b/tests/i915/i915_pm_rpm.c
index 461730e8a..d2bce5826 100644
--- a/tests/i915/i915_pm_rpm.c
+++ b/tests/i915/i915_pm_rpm.c
@@ -1069,7 +1069,8 @@ static void debugfs_forcewake_user_subtest(void)
 	igt_assert(wait_for_suspended());
 }
 
-static void gem_mmap_args(const struct mmap_offset *t)
+static void gem_mmap_args(const struct mmap_offset *t,
+			  struct drm_i915_gem_memory_class_instance *mem_regions)
 {
 	int i;
 	uint32_t handle;
@@ -1079,7 +1080,7 @@ static void gem_mmap_args(const struct mmap_offset *t)
 	/* Create, map and set data while the device is active. */
 	enable_one_screen_and_wait(&ms_data);
 
-	handle = gem_create(drm_fd, buf_size);
+	handle = gem_create_in_memory_region_list(drm_fd, buf_size, mem_regions, 1);
 
 	gem_buf = __gem_mmap_offset(drm_fd, handle, 0, buf_size,
 				    PROT_READ | PROT_WRITE, t->type);
@@ -1290,7 +1291,7 @@ static void submit_blt_cmd(uint32_t dst_handle, int dst_size,
 }
 
 /* Make sure we can submit a batch buffer and verify its result. */
-static void gem_execbuf_subtest(void)
+static void gem_execbuf_subtest(struct drm_i915_gem_memory_class_instance *mem_regions)
 {
 	int x, y;
 	uint32_t handle;
@@ -1308,7 +1309,7 @@ static void gem_execbuf_subtest(void)
 	/* Create and set data while the device is active. */
 	enable_one_screen_and_wait(&ms_data);
 
-	handle = gem_create(drm_fd, dst_size);
+	handle = gem_create_in_memory_region_list(drm_fd, dst_size, mem_regions, 1);
 
 	cpu_buf = malloc(dst_size);
 	igt_assert(cpu_buf);
@@ -2083,15 +2084,21 @@ igt_main_args("", long_options, help_str, opt_handler, NULL)
 	/* GEM */
 	igt_subtest_with_dynamic("gem-mmap-type") {
 		for_each_mmap_offset_type(drm_fd, t) {
-			igt_dynamic_f("%s", t->name)
-				gem_mmap_args(t);
+			for_each_memory_region(r, drm_fd) {
+				igt_dynamic_f("%s-%s", t->name, r->name)
+				gem_mmap_args(t, &r->ci);
+			}
 		}
 	}
 
 	igt_subtest("gem-pread")
 		gem_pread_subtest();
-	igt_subtest("gem-execbuf")
-		gem_execbuf_subtest();
+	igt_subtest_with_dynamic("gem-execbuf") {
+		for_each_memory_region(r, drm_fd) {
+			igt_dynamic_f("%s", r->name)
+				gem_execbuf_subtest(&r->ci);
+		}
+	}
 	igt_subtest("gem-idle")
 		gem_idle_subtest();
 	igt_subtest("gem-evict-pwrite") {
-- 
2.26.2

[igt-dev] [PATCH i-g-t v5 2/9] lib/igt_device: Get gfx PCI card root port

[Anshuman Gupta]

Utility library function to get the Gfx Card PCI topology
root port pci device using libpaciaccess.
root port will be used to get the real ACPI D state.

Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
---
 lib/igt_device.c | 28 ++++++++++++++++++++++++++++
 lib/igt_device.h |  1 +
 2 files changed, 29 insertions(+)

diff --git a/lib/igt_device.c b/lib/igt_device.c
index 9794a9928..fddfba72c 100644
--- a/lib/igt_device.c
+++ b/lib/igt_device.c
@@ -257,3 +257,31 @@ struct pci_device *igt_device_get_pci_device(int fd)
 
 	return pci_dev;
 }
+
+/**
+ * igt_device_get_pci_root_port:
+ * @fd: the device.
+ *
+ * Looks up the graphics pci device root port using libpciaccess.
+ *
+ * Returns:
+ * The root port pci_device.
+ */
+struct pci_device *
+igt_device_get_pci_root_port(int fd)
+{
+	struct pci_device *pci_dev, *prev;
+
+	pci_dev = __igt_device_get_pci_device(fd, 0);
+	igt_require(pci_dev);
+
+	while (pci_dev) {
+		prev = pci_dev;
+		pci_dev = pci_device_get_parent_bridge(pci_dev);
+	}
+
+	igt_debug("Root Port PCI device %04x:%02x:%02x.%01x\n",
+		  prev->domain, prev->bus, prev->dev, prev->func);
+
+	return prev;
+}
diff --git a/lib/igt_device.h b/lib/igt_device.h
index 1aaa840e2..800a0fcc3 100644
--- a/lib/igt_device.h
+++ b/lib/igt_device.h
@@ -34,5 +34,6 @@ void igt_device_drop_master(int fd);
 int igt_device_get_card_index(int fd);
 struct pci_device *igt_device_get_pci_device(int fd);
 struct pci_device *__igt_device_get_pci_device(int fd, unsigned int vf_id);
+struct pci_device *igt_device_get_pci_root_port(int fd);
 
 #endif /* __IGT_DEVICE_H__ */
-- 
2.26.2

[igt-dev] [PATCH i-g-t v5 3/9] lib/igt_pm: D3Cold runtime pm infrastructure

[Anshuman Gupta]

Enable gfx card pci devices runtime pm for all pci devices
and bridge under the topology of Gfx Card root port.

Added a library function to get the PCI root port ACPI
D state and to print the pci card devices runtime pm
status.

v2:
- Save pci dev power attrs to dynamically allocated array. [Rodrigo]
- Set autosuspend delay to 0 on supported devices. [Rodrigo]
- %s/else if/else in  igt_pm_get_acpi_real_d_state. [Kamil]
v3:
- Add comment for MAX_PCI_DEVICES. [Badal]
- Use static global arrary __pci_dev_pwrattr[]. [Rodrigo]
- Use pci_slot_match iter. [Badal]
- Destroy the iter. [Badal]
v4:
- Added igt_pm_enbale_pci_card_autosuspend() to avoid any
  control attr save/restore by exit handler. [Rodrigo]
v5:
- Code refactoring to avoid code duplication.
  Few function name changed. [Rodrigo]
  %s/igt_pm_enbale_pci_card_autosuspend/igt_pm_enable_runtime_pm.
  %s/__igt_pm_setup_pci_card_runtime_pm/igt_pm_setup_pci_dev_power_attrs.
  %s/igt_pm_setup_power_attr/igt_pm_write_power_attr.

Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
---
 lib/igt_pm.c | 317 +++++++++++++++++++++++++++++++++++++++++++++++++++
 lib/igt_pm.h |  24 ++++
 2 files changed, 341 insertions(+)

diff --git a/lib/igt_pm.c b/lib/igt_pm.c
index b409ec463..412b420b6 100644
--- a/lib/igt_pm.c
+++ b/lib/igt_pm.c
@@ -28,6 +28,7 @@
 #include <fcntl.h>
 #include <stdio.h>
 #include <limits.h>
+#include <pciaccess.h>
 #include <stdlib.h>
 #include <string.h>
 #include <unistd.h>
@@ -75,6 +76,8 @@ enum {
 #define MIN_POWER_STR		"min_power\n"
 /* Remember to fix this if adding longer strings */
 #define MAX_POLICY_STRLEN	strlen(MAX_PERFORMANCE_STR)
+/* Root Port bus can have max 32 dev and each dev can have max 8 func */
+#define MAX_PCI_DEVICES		256
 int8_t *__sata_pm_policies;
 int __scsi_host_cnt;
 
@@ -856,3 +859,317 @@ bool i915_output_is_lpsp_capable(int drm_fd, igt_output_t *output)
 
 	return strstr(buf, "LPSP: capable");
 }
+
+/**
+ * igt_pm_acpi_d3cold_supported:
+ * @pci_dev: root port pci_dev.
+ * Check ACPI D3Cold support.
+ *
+ * Returns:
+ * True if ACPI D3Cold supported otherwise false.
+ */
+bool igt_pm_acpi_d3cold_supported(struct pci_device *pci_dev)
+{
+	char name[PATH_MAX];
+	int dir, fd;
+
+	snprintf(name, PATH_MAX,
+		 "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/firmware_node",
+		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func);
+
+	dir = open(name, O_RDONLY);
+	igt_require(dir > 0);
+
+	/* BIOS need to enable ACPI D3Cold Support.*/
+	fd = openat(dir, "real_power_state", O_RDONLY);
+	if (fd < 0 && errno == ENOENT)
+		return false;
+
+	igt_require(fd > 0);
+
+	return true;
+}
+
+/**
+ * igt_pm_get_acpi_real_d_state:
+ * @pci_dev: root port pci_dev.
+ * Get ACPI D state for a given root port.
+ *
+ * Returns:
+ * igt_acpi_d_state state.
+ */
+enum igt_acpi_d_state
+igt_pm_get_acpi_real_d_state(struct pci_device *pci_dev)
+{
+	char name[PATH_MAX];
+	char acpi_d_state[64];
+	int fd, n_read;
+
+	snprintf(name, PATH_MAX,
+		 "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/firmware_node/real_power_state",
+		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func);
+
+	fd = open(name, O_RDONLY);
+	if (fd < 0)
+		return IGT_ACPI_UNKNOWN_STATE;
+
+	n_read = read(fd, acpi_d_state, sizeof(acpi_d_state) - 1);
+	igt_assert(n_read >= 0);
+	acpi_d_state[n_read] = '\0';
+	close(fd);
+
+	if (strncmp(acpi_d_state, "D0\n", n_read) == 0)
+		return IGT_ACPI_D0;
+	if  (strncmp(acpi_d_state, "D1\n", n_read) == 0)
+		return IGT_ACPI_D1;
+	if  (strncmp(acpi_d_state, "D2\n", n_read) == 0)
+		return IGT_ACPI_D2;
+	if  (strncmp(acpi_d_state, "D3hot\n", n_read) == 0)
+		return IGT_ACPI_D3Hot;
+	if  (strncmp(acpi_d_state, "D3cold\n", n_read) == 0)
+		return IGT_ACPI_D3Cold;
+
+	return IGT_ACPI_UNKNOWN_STATE;
+}
+
+static struct igt_pm_pci_dev_pwrattr  __pci_dev_pwrattr[MAX_PCI_DEVICES];
+
+static void __igt_pm_pci_card_exit_handler(int sig)
+{
+	igt_pm_restore_pci_card_runtime_pm();
+}
+
+static int igt_pm_get_power_attr_fd(struct pci_device *pci_dev, const char *attr)
+{
+	char name[PATH_MAX];
+	int fd;
+
+	snprintf(name, PATH_MAX, "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/power/%s",
+		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func, attr);
+
+	fd = open(name, O_RDWR);
+	igt_assert_f(fd >= 0, "Can't open %s\n", attr);
+
+	return fd;
+}
+
+static bool igt_pm_save_power_attr(int fd, char *attr, int len, bool autosuspend_delay)
+{
+	int size;
+
+	size = read(fd, attr, len - 1);
+
+	if (autosuspend_delay) {
+		if (size < 0 && errno == EIO)
+			return false;
+	} else {
+		igt_assert(size > 0);
+	}
+
+	attr[size] = '\0';
+	strchomp(attr);
+
+	return true;
+}
+
+static void igt_pm_write_power_attr(int fd, const char *val)
+{
+	int size, len;
+	char buf[6];
+
+	len = strlen(val);
+
+	size = write(fd, val, len);
+	if (size < 0 && errno == EIO)
+		return;
+
+	igt_assert_eq(size, len);
+	lseek(fd, 0, SEEK_SET);
+	size = read(fd, buf, ARRAY_SIZE(buf));
+	igt_assert_eq(size, len);
+	igt_assert(strncmp(buf, val, len) == 0);
+}
+
+static void
+igt_pm_setup_pci_dev_power_attrs(struct pci_device *pci_dev, struct igt_pm_pci_dev_pwrattr *pwrattr)
+{
+	int control_fd, delay_fd, control_size, delay_size;
+	char *control, *delay;
+
+	delay_fd = igt_pm_get_power_attr_fd(pci_dev, "autosuspend_delay_ms");
+	control_fd = igt_pm_get_power_attr_fd(pci_dev, "control");
+
+	if (!pwrattr)
+		goto write_power_attr;
+
+	control = pwrattr->control;
+	control_size =  sizeof(pwrattr->control);
+	delay = pwrattr->autosuspend_delay;
+	delay_size =  sizeof(pwrattr->autosuspend_delay);
+	pwrattr->pci_dev = pci_dev;
+	pwrattr->autosuspend_supported = true;
+
+	if (!igt_pm_save_power_attr(delay_fd, delay, delay_size, true)) {
+		pwrattr->autosuspend_supported = false;
+		igt_debug("PCI '%04x:%02x:%02x.%01x' doesn't support auto_suspend\n",
+			  pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func);
+	}
+
+	igt_pm_save_power_attr(control_fd, control, control_size, false);
+	igt_debug("PCI '%04x:%02x:%02x.%01x' Saved 'control, autosuspend_delay_ms' as '%s, %s'\n",
+		  pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func, control,
+		  pwrattr->autosuspend_supported ? delay : "NA");
+
+	igt_install_exit_handler(__igt_pm_pci_card_exit_handler);
+
+write_power_attr:
+	igt_pm_write_power_attr(delay_fd, "0\n");
+	igt_pm_write_power_attr(control_fd, "auto\n");
+
+	close(delay_fd);
+	close(control_fd);
+}
+
+static void
+igt_pm_setup_pci_card_power_attrs(struct pci_device *pci_dev, bool save_attrs)
+{
+	int primary, secondary, subordinate, ret;
+	struct pci_device_iterator *iter;
+	struct pci_slot_match match;
+	struct pci_device *dev;
+	int i = 0;
+
+	ret = pci_device_get_bridge_buses(pci_dev, &primary, &secondary, &subordinate);
+	igt_assert(!ret);
+
+	ret = pci_system_init();
+	igt_assert(!ret);
+
+	match.domain = pci_dev->domain;
+	match.bus = PCI_MATCH_ANY;
+	match.dev = PCI_MATCH_ANY;
+	match.func = PCI_MATCH_ANY;
+	iter = pci_slot_match_iterator_create(&match);
+	igt_assert(iter);
+
+	/* Setup power attrs for PCI root port */
+	igt_pm_setup_pci_dev_power_attrs(pci_dev, save_attrs ? &__pci_dev_pwrattr[i++] : NULL);
+	while ((dev = pci_device_next(iter)) != NULL) {
+		if (dev->bus >= secondary && dev->bus <= subordinate) {
+			igt_pm_setup_pci_dev_power_attrs(dev,
+							 save_attrs ? &__pci_dev_pwrattr[i++] : NULL);
+			if (save_attrs)
+				igt_assert(i <  MAX_PCI_DEVICES);
+		}
+	}
+
+	pci_iterator_destroy(iter);
+}
+
+/**
+ * igt_pm_enable_pci_card_runtime_pm:
+ * @pci_dev: root port pci_dev.
+ * Enable runtime PM for all PCI endpoints devices for a given root port by
+ * setting power/control attr to "auto" and setting autosuspend_delay_ms
+ * to zero.
+ */
+void igt_pm_enable_pci_card_runtime_pm(struct pci_device *pci_dev)
+{
+	igt_pm_setup_pci_card_power_attrs(pci_dev, false);
+	pci_system_cleanup();
+}
+
+/**
+ * igt_pm_setup_pci_card_runtime_pm:
+ * @pci_dev: root port pci_dev.
+ * Setup runtime PM for all PCI endpoints devices for a given root port by
+ * enabling runtime suspend and setting autosuspend_delay_ms to zero.
+ * It also saves and restore power control attribute for all PCI endpoints
+ * devices under given root port.
+ */
+void igt_pm_setup_pci_card_runtime_pm(struct pci_device *pci_dev)
+{
+	memset(__pci_dev_pwrattr, 0, sizeof(__pci_dev_pwrattr));
+	igt_pm_setup_pci_card_power_attrs(pci_dev, true);
+}
+
+static void
+igt_pm_restore_power_attr(struct pci_device *pci_dev, const char *attr, char *val, int len)
+{
+	int fd;
+
+	igt_debug("PCI '%04x:%02x:%02x.%01x' Restoring %s attr to '%s'\n",
+		  pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func, attr, val);
+
+	fd = igt_pm_get_power_attr_fd(pci_dev, attr);
+	igt_assert(write(fd, val, len) == len);
+
+	close(fd);
+}
+
+/**
+ * igt_pm_restore_pci_card_runtime_pm:
+ * Restore control and autosuspend_delay_ms power attribute for all
+ * PCI endpoints devices under gfx root port, which were saved earlier
+ * by igt_pm_setup_pci_card_runtime_pm().
+ */
+void igt_pm_restore_pci_card_runtime_pm(void)
+{
+	int i = 0;
+
+	for (i = 0; i < MAX_PCI_DEVICES; i++) {
+		if (!__pci_dev_pwrattr[i].pci_dev)
+			break;
+
+		igt_pm_restore_power_attr(__pci_dev_pwrattr[i].pci_dev, "control",
+					  __pci_dev_pwrattr[i].control,
+					  sizeof(__pci_dev_pwrattr[i].control));
+
+		if (!__pci_dev_pwrattr[i].autosuspend_supported)
+			continue;
+
+		igt_pm_restore_power_attr(__pci_dev_pwrattr[i].pci_dev, "autosuspend_delay_ms",
+					  __pci_dev_pwrattr[i].autosuspend_delay,
+					  sizeof(__pci_dev_pwrattr[i].autosuspend_delay));
+	}
+
+	memset(__pci_dev_pwrattr, 0, sizeof(__pci_dev_pwrattr));
+	pci_system_cleanup();
+}
+
+static void igt_pm_print_pci_dev_runtime_status(struct pci_device *pci_dev)
+{
+	char name[PATH_MAX], runtime_status[64];
+	int fd, n_read;
+
+	snprintf(name, PATH_MAX, "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/power/runtime_status",
+		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func);
+
+	fd = open(name, O_RDONLY);
+	igt_assert_f(fd >= 0, "Can't open runtime_status\n");
+
+	n_read = read(fd, runtime_status, sizeof(runtime_status) - 1);
+	igt_assert(n_read >= 0);
+	runtime_status[n_read] = '\0';
+	igt_info("runtime suspend status for PCI '%04x:%02x:%02x.%01x' %s\n",
+		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func, runtime_status);
+	close(fd);
+}
+
+/**
+ * igt_pm_print_pci_card_runtime_status:
+ * @pci_dev: root port pci_dev.
+ * Print runtime suspend status for all PCI endpoints devices for a given
+ * root port.
+ */
+void igt_pm_print_pci_card_runtime_status(void)
+{
+	int i = 0;
+
+	for (i = 0; i < MAX_PCI_DEVICES; i++) {
+		if (!__pci_dev_pwrattr[i].pci_dev)
+			break;
+
+		igt_pm_print_pci_dev_runtime_status(__pci_dev_pwrattr[i].pci_dev);
+	}
+}
diff --git a/lib/igt_pm.h b/lib/igt_pm.h
index 162d3ca3c..c53dae2c3 100644
--- a/lib/igt_pm.h
+++ b/lib/igt_pm.h
@@ -46,6 +46,23 @@ enum igt_runtime_pm_status {
 	IGT_RUNTIME_PM_STATUS_UNKNOWN,
 };
 
+/* PCI ACPI firmware node real state */
+enum igt_acpi_d_state {
+	IGT_ACPI_D0,
+	IGT_ACPI_D1,
+	IGT_ACPI_D2,
+	IGT_ACPI_D3Hot,
+	IGT_ACPI_D3Cold,
+	IGT_ACPI_UNKNOWN_STATE,
+};
+
+struct	igt_pm_pci_dev_pwrattr {
+	struct pci_device *pci_dev;
+	char control[64];
+	bool autosuspend_supported;
+	char autosuspend_delay[64];
+};
+
 bool igt_setup_runtime_pm(int device);
 void igt_disable_runtime_pm(void);
 void igt_restore_runtime_pm(void);
@@ -54,5 +71,12 @@ bool igt_wait_for_pm_status(enum igt_runtime_pm_status status);
 bool igt_pm_dmc_loaded(int debugfs);
 bool igt_pm_pc8_plus_residencies_enabled(int msr_fd);
 bool i915_output_is_lpsp_capable(int drm_fd, igt_output_t *output);
+bool igt_pm_acpi_d3cold_supported(struct pci_device *pci_dev);
+enum igt_acpi_d_state
+igt_pm_get_acpi_real_d_state(struct pci_device *pci_dev);
+void igt_pm_enable_pci_card_runtime_pm(struct pci_device *pci_dev);
+void igt_pm_setup_pci_card_runtime_pm(struct pci_device *pci_dev);
+void igt_pm_restore_pci_card_runtime_pm(void);
+void igt_pm_print_pci_card_runtime_status(void);
 
 #endif /* IGT_PM_H */
-- 
2.26.2

Re: [igt-dev] [PATCH i-g-t v5 3/9] lib/igt_pm: D3Cold runtime pm infrastructure

[Rodrigo Vivi]

On Fri, May 06, 2022 at 09:15:47PM +0530, Anshuman Gupta wrote:
> Enable gfx card pci devices runtime pm for all pci devices
> and bridge under the topology of Gfx Card root port.
> 
> Added a library function to get the PCI root port ACPI
> D state and to print the pci card devices runtime pm
> status.
> 
> v2:
> - Save pci dev power attrs to dynamically allocated array. [Rodrigo]
> - Set autosuspend delay to 0 on supported devices. [Rodrigo]
> - %s/else if/else in  igt_pm_get_acpi_real_d_state. [Kamil]
> v3:
> - Add comment for MAX_PCI_DEVICES. [Badal]
> - Use static global arrary __pci_dev_pwrattr[]. [Rodrigo]
> - Use pci_slot_match iter. [Badal]
> - Destroy the iter. [Badal]
> v4:
> - Added igt_pm_enbale_pci_card_autosuspend() to avoid any
>   control attr save/restore by exit handler. [Rodrigo]
> v5:
> - Code refactoring to avoid code duplication.
>   Few function name changed. [Rodrigo]
>   %s/igt_pm_enbale_pci_card_autosuspend/igt_pm_enable_runtime_pm.
>   %s/__igt_pm_setup_pci_card_runtime_pm/igt_pm_setup_pci_dev_power_attrs.
>   %s/igt_pm_setup_power_attr/igt_pm_write_power_attr.
> 
> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
> ---
>  lib/igt_pm.c | 317 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  lib/igt_pm.h |  24 ++++
>  2 files changed, 341 insertions(+)
> 
> diff --git a/lib/igt_pm.c b/lib/igt_pm.c
> index b409ec463..412b420b6 100644
> --- a/lib/igt_pm.c
> +++ b/lib/igt_pm.c
> @@ -28,6 +28,7 @@
>  #include <fcntl.h>
>  #include <stdio.h>
>  #include <limits.h>
> +#include <pciaccess.h>
>  #include <stdlib.h>
>  #include <string.h>
>  #include <unistd.h>
> @@ -75,6 +76,8 @@ enum {
>  #define MIN_POWER_STR		"min_power\n"
>  /* Remember to fix this if adding longer strings */
>  #define MAX_POLICY_STRLEN	strlen(MAX_PERFORMANCE_STR)
> +/* Root Port bus can have max 32 dev and each dev can have max 8 func */
> +#define MAX_PCI_DEVICES		256
>  int8_t *__sata_pm_policies;
>  int __scsi_host_cnt;
>  
> @@ -856,3 +859,317 @@ bool i915_output_is_lpsp_capable(int drm_fd, igt_output_t *output)
>  
>  	return strstr(buf, "LPSP: capable");
>  }
> +
> +/**
> + * igt_pm_acpi_d3cold_supported:
> + * @pci_dev: root port pci_dev.
> + * Check ACPI D3Cold support.
> + *
> + * Returns:
> + * True if ACPI D3Cold supported otherwise false.
> + */
> +bool igt_pm_acpi_d3cold_supported(struct pci_device *pci_dev)
> +{
> +	char name[PATH_MAX];
> +	int dir, fd;
> +
> +	snprintf(name, PATH_MAX,
> +		 "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/firmware_node",
> +		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func);
> +
> +	dir = open(name, O_RDONLY);
> +	igt_require(dir > 0);
> +
> +	/* BIOS need to enable ACPI D3Cold Support.*/
> +	fd = openat(dir, "real_power_state", O_RDONLY);
> +	if (fd < 0 && errno == ENOENT)
> +		return false;
> +
> +	igt_require(fd > 0);
> +
> +	return true;
> +}
> +
> +/**
> + * igt_pm_get_acpi_real_d_state:
> + * @pci_dev: root port pci_dev.
> + * Get ACPI D state for a given root port.
> + *
> + * Returns:
> + * igt_acpi_d_state state.
> + */
> +enum igt_acpi_d_state
> +igt_pm_get_acpi_real_d_state(struct pci_device *pci_dev)
> +{
> +	char name[PATH_MAX];
> +	char acpi_d_state[64];
> +	int fd, n_read;
> +
> +	snprintf(name, PATH_MAX,
> +		 "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/firmware_node/real_power_state",
> +		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func);
> +
> +	fd = open(name, O_RDONLY);
> +	if (fd < 0)
> +		return IGT_ACPI_UNKNOWN_STATE;
> +
> +	n_read = read(fd, acpi_d_state, sizeof(acpi_d_state) - 1);
> +	igt_assert(n_read >= 0);
> +	acpi_d_state[n_read] = '\0';
> +	close(fd);
> +
> +	if (strncmp(acpi_d_state, "D0\n", n_read) == 0)
> +		return IGT_ACPI_D0;
> +	if  (strncmp(acpi_d_state, "D1\n", n_read) == 0)
> +		return IGT_ACPI_D1;
> +	if  (strncmp(acpi_d_state, "D2\n", n_read) == 0)
> +		return IGT_ACPI_D2;
> +	if  (strncmp(acpi_d_state, "D3hot\n", n_read) == 0)
> +		return IGT_ACPI_D3Hot;
> +	if  (strncmp(acpi_d_state, "D3cold\n", n_read) == 0)
> +		return IGT_ACPI_D3Cold;
> +
> +	return IGT_ACPI_UNKNOWN_STATE;
> +}
> +
> +static struct igt_pm_pci_dev_pwrattr  __pci_dev_pwrattr[MAX_PCI_DEVICES];
> +
> +static void __igt_pm_pci_card_exit_handler(int sig)
> +{
> +	igt_pm_restore_pci_card_runtime_pm();
> +}
> +
> +static int igt_pm_get_power_attr_fd(struct pci_device *pci_dev, const char *attr)
> +{
> +	char name[PATH_MAX];
> +	int fd;
> +
> +	snprintf(name, PATH_MAX, "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/power/%s",
> +		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func, attr);
> +
> +	fd = open(name, O_RDWR);
> +	igt_assert_f(fd >= 0, "Can't open %s\n", attr);
> +
> +	return fd;
> +}
> +
> +static bool igt_pm_save_power_attr(int fd, char *attr, int len, bool autosuspend_delay)
> +{
> +	int size;
> +
> +	size = read(fd, attr, len - 1);
> +
> +	if (autosuspend_delay) {
> +		if (size < 0 && errno == EIO)
> +			return false;
> +	} else {
> +		igt_assert(size > 0);
> +	}
> +
> +	attr[size] = '\0';
> +	strchomp(attr);
> +
> +	return true;
> +}
> +
> +static void igt_pm_write_power_attr(int fd, const char *val)
> +{
> +	int size, len;
> +	char buf[6];
> +
> +	len = strlen(val);
> +
> +	size = write(fd, val, len);
> +	if (size < 0 && errno == EIO)
> +		return;
> +
> +	igt_assert_eq(size, len);
> +	lseek(fd, 0, SEEK_SET);
> +	size = read(fd, buf, ARRAY_SIZE(buf));
> +	igt_assert_eq(size, len);
> +	igt_assert(strncmp(buf, val, len) == 0);
> +}
> +
> +static void
> +igt_pm_setup_pci_dev_power_attrs(struct pci_device *pci_dev, struct igt_pm_pci_dev_pwrattr *pwrattr)
> +{
> +	int control_fd, delay_fd, control_size, delay_size;
> +	char *control, *delay;
> +
> +	delay_fd = igt_pm_get_power_attr_fd(pci_dev, "autosuspend_delay_ms");
> +	control_fd = igt_pm_get_power_attr_fd(pci_dev, "control");
> +
> +	if (!pwrattr)
> +		goto write_power_attr;
> +
> +	control = pwrattr->control;
> +	control_size =  sizeof(pwrattr->control);
> +	delay = pwrattr->autosuspend_delay;
> +	delay_size =  sizeof(pwrattr->autosuspend_delay);
> +	pwrattr->pci_dev = pci_dev;
> +	pwrattr->autosuspend_supported = true;
> +
> +	if (!igt_pm_save_power_attr(delay_fd, delay, delay_size, true)) {
> +		pwrattr->autosuspend_supported = false;
> +		igt_debug("PCI '%04x:%02x:%02x.%01x' doesn't support auto_suspend\n",
> +			  pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func);
> +	}
> +
> +	igt_pm_save_power_attr(control_fd, control, control_size, false);
> +	igt_debug("PCI '%04x:%02x:%02x.%01x' Saved 'control, autosuspend_delay_ms' as '%s, %s'\n",
> +		  pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func, control,
> +		  pwrattr->autosuspend_supported ? delay : "NA");
> +
> +	igt_install_exit_handler(__igt_pm_pci_card_exit_handler);
> +
> +write_power_attr:
> +	igt_pm_write_power_attr(delay_fd, "0\n");
> +	igt_pm_write_power_attr(control_fd, "auto\n");
> +
> +	close(delay_fd);
> +	close(control_fd);
> +}
> +
> +static void
> +igt_pm_setup_pci_card_power_attrs(struct pci_device *pci_dev, bool save_attrs)
> +{
> +	int primary, secondary, subordinate, ret;
> +	struct pci_device_iterator *iter;
> +	struct pci_slot_match match;
> +	struct pci_device *dev;
> +	int i = 0;
> +
> +	ret = pci_device_get_bridge_buses(pci_dev, &primary, &secondary, &subordinate);
> +	igt_assert(!ret);
> +
> +	ret = pci_system_init();
> +	igt_assert(!ret);
> +
> +	match.domain = pci_dev->domain;
> +	match.bus = PCI_MATCH_ANY;
> +	match.dev = PCI_MATCH_ANY;
> +	match.func = PCI_MATCH_ANY;
> +	iter = pci_slot_match_iterator_create(&match);
> +	igt_assert(iter);
> +
> +	/* Setup power attrs for PCI root port */
> +	igt_pm_setup_pci_dev_power_attrs(pci_dev, save_attrs ? &__pci_dev_pwrattr[i++] : NULL);
> +	while ((dev = pci_device_next(iter)) != NULL) {
> +		if (dev->bus >= secondary && dev->bus <= subordinate) {
> +			igt_pm_setup_pci_dev_power_attrs(dev,
> +							 save_attrs ? &__pci_dev_pwrattr[i++] : NULL);
> +			if (save_attrs)
> +				igt_assert(i <  MAX_PCI_DEVICES);
> +		}
> +	}
> +
> +	pci_iterator_destroy(iter);
> +}
> +
> +/**
> + * igt_pm_enable_pci_card_runtime_pm:
> + * @pci_dev: root port pci_dev.
> + * Enable runtime PM for all PCI endpoints devices for a given root port by
> + * setting power/control attr to "auto" and setting autosuspend_delay_ms
> + * to zero.
> + */
> +void igt_pm_enable_pci_card_runtime_pm(struct pci_device *pci_dev)
> +{
> +	igt_pm_setup_pci_card_power_attrs(pci_dev, false);
> +	pci_system_cleanup();
> +}
> +
> +/**
> + * igt_pm_setup_pci_card_runtime_pm:
> + * @pci_dev: root port pci_dev.
> + * Setup runtime PM for all PCI endpoints devices for a given root port by
> + * enabling runtime suspend and setting autosuspend_delay_ms to zero.

we know that we have bugs if we set i915's autosuspend delay to zero.
I believe we need to make this optional here and then read the i915 one
and use that as reference to set to all the endpoint devices here.

but that change is mostly in the other patch and change in this would
be minimal, so:

Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>

> + * It also saves and restore power control attribute for all PCI endpoints
> + * devices under given root port.
> + */
> +void igt_pm_setup_pci_card_runtime_pm(struct pci_device *pci_dev)
> +{
> +	memset(__pci_dev_pwrattr, 0, sizeof(__pci_dev_pwrattr));
> +	igt_pm_setup_pci_card_power_attrs(pci_dev, true);
> +}
> +
> +static void
> +igt_pm_restore_power_attr(struct pci_device *pci_dev, const char *attr, char *val, int len)
> +{
> +	int fd;
> +
> +	igt_debug("PCI '%04x:%02x:%02x.%01x' Restoring %s attr to '%s'\n",
> +		  pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func, attr, val);
> +
> +	fd = igt_pm_get_power_attr_fd(pci_dev, attr);
> +	igt_assert(write(fd, val, len) == len);
> +
> +	close(fd);
> +}
> +
> +/**
> + * igt_pm_restore_pci_card_runtime_pm:
> + * Restore control and autosuspend_delay_ms power attribute for all
> + * PCI endpoints devices under gfx root port, which were saved earlier
> + * by igt_pm_setup_pci_card_runtime_pm().
> + */
> +void igt_pm_restore_pci_card_runtime_pm(void)
> +{
> +	int i = 0;
> +
> +	for (i = 0; i < MAX_PCI_DEVICES; i++) {
> +		if (!__pci_dev_pwrattr[i].pci_dev)
> +			break;
> +
> +		igt_pm_restore_power_attr(__pci_dev_pwrattr[i].pci_dev, "control",
> +					  __pci_dev_pwrattr[i].control,
> +					  sizeof(__pci_dev_pwrattr[i].control));
> +
> +		if (!__pci_dev_pwrattr[i].autosuspend_supported)
> +			continue;
> +
> +		igt_pm_restore_power_attr(__pci_dev_pwrattr[i].pci_dev, "autosuspend_delay_ms",
> +					  __pci_dev_pwrattr[i].autosuspend_delay,
> +					  sizeof(__pci_dev_pwrattr[i].autosuspend_delay));
> +	}
> +
> +	memset(__pci_dev_pwrattr, 0, sizeof(__pci_dev_pwrattr));
> +	pci_system_cleanup();
> +}
> +
> +static void igt_pm_print_pci_dev_runtime_status(struct pci_device *pci_dev)
> +{
> +	char name[PATH_MAX], runtime_status[64];
> +	int fd, n_read;
> +
> +	snprintf(name, PATH_MAX, "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/power/runtime_status",
> +		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func);
> +
> +	fd = open(name, O_RDONLY);
> +	igt_assert_f(fd >= 0, "Can't open runtime_status\n");
> +
> +	n_read = read(fd, runtime_status, sizeof(runtime_status) - 1);
> +	igt_assert(n_read >= 0);
> +	runtime_status[n_read] = '\0';
> +	igt_info("runtime suspend status for PCI '%04x:%02x:%02x.%01x' %s\n",
> +		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func, runtime_status);
> +	close(fd);
> +}
> +
> +/**
> + * igt_pm_print_pci_card_runtime_status:
> + * @pci_dev: root port pci_dev.
> + * Print runtime suspend status for all PCI endpoints devices for a given
> + * root port.
> + */
> +void igt_pm_print_pci_card_runtime_status(void)
> +{
> +	int i = 0;
> +
> +	for (i = 0; i < MAX_PCI_DEVICES; i++) {
> +		if (!__pci_dev_pwrattr[i].pci_dev)
> +			break;
> +
> +		igt_pm_print_pci_dev_runtime_status(__pci_dev_pwrattr[i].pci_dev);
> +	}
> +}
> diff --git a/lib/igt_pm.h b/lib/igt_pm.h
> index 162d3ca3c..c53dae2c3 100644
> --- a/lib/igt_pm.h
> +++ b/lib/igt_pm.h
> @@ -46,6 +46,23 @@ enum igt_runtime_pm_status {
>  	IGT_RUNTIME_PM_STATUS_UNKNOWN,
>  };
>  
> +/* PCI ACPI firmware node real state */
> +enum igt_acpi_d_state {
> +	IGT_ACPI_D0,
> +	IGT_ACPI_D1,
> +	IGT_ACPI_D2,
> +	IGT_ACPI_D3Hot,
> +	IGT_ACPI_D3Cold,
> +	IGT_ACPI_UNKNOWN_STATE,
> +};
> +
> +struct	igt_pm_pci_dev_pwrattr {
> +	struct pci_device *pci_dev;
> +	char control[64];
> +	bool autosuspend_supported;
> +	char autosuspend_delay[64];
> +};
> +
>  bool igt_setup_runtime_pm(int device);
>  void igt_disable_runtime_pm(void);
>  void igt_restore_runtime_pm(void);
> @@ -54,5 +71,12 @@ bool igt_wait_for_pm_status(enum igt_runtime_pm_status status);
>  bool igt_pm_dmc_loaded(int debugfs);
>  bool igt_pm_pc8_plus_residencies_enabled(int msr_fd);
>  bool i915_output_is_lpsp_capable(int drm_fd, igt_output_t *output);
> +bool igt_pm_acpi_d3cold_supported(struct pci_device *pci_dev);
> +enum igt_acpi_d_state
> +igt_pm_get_acpi_real_d_state(struct pci_device *pci_dev);
> +void igt_pm_enable_pci_card_runtime_pm(struct pci_device *pci_dev);
> +void igt_pm_setup_pci_card_runtime_pm(struct pci_device *pci_dev);
> +void igt_pm_restore_pci_card_runtime_pm(void);
> +void igt_pm_print_pci_card_runtime_status(void);
>  
>  #endif /* IGT_PM_H */
> -- 
> 2.26.2
> 

Re: [igt-dev] [PATCH i-g-t v5 3/9] lib/igt_pm: D3Cold runtime pm infrastructure

[Gupta, Anshuman]

> -----Original Message-----
> From: Vivi, Rodrigo <rodrigo.vivi@intel.com>
> Sent: Monday, May 9, 2022 2:12 PM
> To: Gupta, Anshuman <anshuman.gupta@intel.com>
> Cc: igt-dev@lists.freedesktop.org; Nilawar, Badal <badal.nilawar@intel.com>;
> Ewins, Jon <jon.ewins@intel.com>; kamil.konieczny@linux.intel.com; Latvala,
> Petri <petri.latvala@intel.com>; Tangudu, Tilak <tilak.tangudu@intel.com>
> Subject: Re: [PATCH i-g-t v5 3/9] lib/igt_pm: D3Cold runtime pm infrastructure
> 
> On Fri, May 06, 2022 at 09:15:47PM +0530, Anshuman Gupta wrote:
> > Enable gfx card pci devices runtime pm for all pci devices and bridge
> > under the topology of Gfx Card root port.
> >
> > Added a library function to get the PCI root port ACPI D state and to
> > print the pci card devices runtime pm status.
> >
> > v2:
> > - Save pci dev power attrs to dynamically allocated array. [Rodrigo]
> > - Set autosuspend delay to 0 on supported devices. [Rodrigo]
> > - %s/else if/else in  igt_pm_get_acpi_real_d_state. [Kamil]
> > v3:
> > - Add comment for MAX_PCI_DEVICES. [Badal]
> > - Use static global arrary __pci_dev_pwrattr[]. [Rodrigo]
> > - Use pci_slot_match iter. [Badal]
> > - Destroy the iter. [Badal]
> > v4:
> > - Added igt_pm_enbale_pci_card_autosuspend() to avoid any
> >   control attr save/restore by exit handler. [Rodrigo]
> > v5:
> > - Code refactoring to avoid code duplication.
> >   Few function name changed. [Rodrigo]
> >   %s/igt_pm_enbale_pci_card_autosuspend/igt_pm_enable_runtime_pm.
> >
> %s/__igt_pm_setup_pci_card_runtime_pm/igt_pm_setup_pci_dev_power_attrs
> .
> >   %s/igt_pm_setup_power_attr/igt_pm_write_power_attr.
> >
> > Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> > Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
> > ---
> >  lib/igt_pm.c | 317
> > +++++++++++++++++++++++++++++++++++++++++++++++++++
> >  lib/igt_pm.h |  24 ++++
> >  2 files changed, 341 insertions(+)
> >
> > diff --git a/lib/igt_pm.c b/lib/igt_pm.c index b409ec463..412b420b6
> > 100644
> > --- a/lib/igt_pm.c
> > +++ b/lib/igt_pm.c
> > @@ -28,6 +28,7 @@
> >  #include <fcntl.h>
> >  #include <stdio.h>
> >  #include <limits.h>
> > +#include <pciaccess.h>
> >  #include <stdlib.h>
> >  #include <string.h>
> >  #include <unistd.h>
> > @@ -75,6 +76,8 @@ enum {
> >  #define MIN_POWER_STR		"min_power\n"
> >  /* Remember to fix this if adding longer strings */
> >  #define MAX_POLICY_STRLEN	strlen(MAX_PERFORMANCE_STR)
> > +/* Root Port bus can have max 32 dev and each dev can have max 8 func */
> > +#define MAX_PCI_DEVICES		256
> >  int8_t *__sata_pm_policies;
> >  int __scsi_host_cnt;
> >
> > @@ -856,3 +859,317 @@ bool i915_output_is_lpsp_capable(int drm_fd,
> > igt_output_t *output)
> >
> >  	return strstr(buf, "LPSP: capable");  }
> > +
> > +/**
> > + * igt_pm_acpi_d3cold_supported:
> > + * @pci_dev: root port pci_dev.
> > + * Check ACPI D3Cold support.
> > + *
> > + * Returns:
> > + * True if ACPI D3Cold supported otherwise false.
> > + */
> > +bool igt_pm_acpi_d3cold_supported(struct pci_device *pci_dev) {
> > +	char name[PATH_MAX];
> > +	int dir, fd;
> > +
> > +	snprintf(name, PATH_MAX,
> > +
> "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/firmware_node",
> > +		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func);
> > +
> > +	dir = open(name, O_RDONLY);
> > +	igt_require(dir > 0);
> > +
> > +	/* BIOS need to enable ACPI D3Cold Support.*/
> > +	fd = openat(dir, "real_power_state", O_RDONLY);
> > +	if (fd < 0 && errno == ENOENT)
> > +		return false;
> > +
> > +	igt_require(fd > 0);
> > +
> > +	return true;
> > +}
> > +
> > +/**
> > + * igt_pm_get_acpi_real_d_state:
> > + * @pci_dev: root port pci_dev.
> > + * Get ACPI D state for a given root port.
> > + *
> > + * Returns:
> > + * igt_acpi_d_state state.
> > + */
> > +enum igt_acpi_d_state
> > +igt_pm_get_acpi_real_d_state(struct pci_device *pci_dev) {
> > +	char name[PATH_MAX];
> > +	char acpi_d_state[64];
> > +	int fd, n_read;
> > +
> > +	snprintf(name, PATH_MAX,
> > +
> "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/firmware_node/real_power_stat
> e",
> > +		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func);
> > +
> > +	fd = open(name, O_RDONLY);
> > +	if (fd < 0)
> > +		return IGT_ACPI_UNKNOWN_STATE;
> > +
> > +	n_read = read(fd, acpi_d_state, sizeof(acpi_d_state) - 1);
> > +	igt_assert(n_read >= 0);
> > +	acpi_d_state[n_read] = '\0';
> > +	close(fd);
> > +
> > +	if (strncmp(acpi_d_state, "D0\n", n_read) == 0)
> > +		return IGT_ACPI_D0;
> > +	if  (strncmp(acpi_d_state, "D1\n", n_read) == 0)
> > +		return IGT_ACPI_D1;
> > +	if  (strncmp(acpi_d_state, "D2\n", n_read) == 0)
> > +		return IGT_ACPI_D2;
> > +	if  (strncmp(acpi_d_state, "D3hot\n", n_read) == 0)
> > +		return IGT_ACPI_D3Hot;
> > +	if  (strncmp(acpi_d_state, "D3cold\n", n_read) == 0)
> > +		return IGT_ACPI_D3Cold;
> > +
> > +	return IGT_ACPI_UNKNOWN_STATE;
> > +}
> > +
> > +static struct igt_pm_pci_dev_pwrattr
> > +__pci_dev_pwrattr[MAX_PCI_DEVICES];
> > +
> > +static void __igt_pm_pci_card_exit_handler(int sig) {
> > +	igt_pm_restore_pci_card_runtime_pm();
> > +}
> > +
> > +static int igt_pm_get_power_attr_fd(struct pci_device *pci_dev, const
> > +char *attr) {
> > +	char name[PATH_MAX];
> > +	int fd;
> > +
> > +	snprintf(name, PATH_MAX,
> "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/power/%s",
> > +		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func,
> attr);
> > +
> > +	fd = open(name, O_RDWR);
> > +	igt_assert_f(fd >= 0, "Can't open %s\n", attr);
> > +
> > +	return fd;
> > +}
> > +
> > +static bool igt_pm_save_power_attr(int fd, char *attr, int len, bool
> > +autosuspend_delay) {
> > +	int size;
> > +
> > +	size = read(fd, attr, len - 1);
> > +
> > +	if (autosuspend_delay) {
> > +		if (size < 0 && errno == EIO)
> > +			return false;
> > +	} else {
> > +		igt_assert(size > 0);
> > +	}
> > +
> > +	attr[size] = '\0';
> > +	strchomp(attr);
> > +
> > +	return true;
> > +}
> > +
> > +static void igt_pm_write_power_attr(int fd, const char *val) {
> > +	int size, len;
> > +	char buf[6];
> > +
> > +	len = strlen(val);
> > +
> > +	size = write(fd, val, len);
> > +	if (size < 0 && errno == EIO)
> > +		return;
> > +
> > +	igt_assert_eq(size, len);
> > +	lseek(fd, 0, SEEK_SET);
> > +	size = read(fd, buf, ARRAY_SIZE(buf));
> > +	igt_assert_eq(size, len);
> > +	igt_assert(strncmp(buf, val, len) == 0); }
> > +
> > +static void
> > +igt_pm_setup_pci_dev_power_attrs(struct pci_device *pci_dev, struct
> > +igt_pm_pci_dev_pwrattr *pwrattr) {
> > +	int control_fd, delay_fd, control_size, delay_size;
> > +	char *control, *delay;
> > +
> > +	delay_fd = igt_pm_get_power_attr_fd(pci_dev,
> "autosuspend_delay_ms");
> > +	control_fd = igt_pm_get_power_attr_fd(pci_dev, "control");
> > +
> > +	if (!pwrattr)
> > +		goto write_power_attr;
> > +
> > +	control = pwrattr->control;
> > +	control_size =  sizeof(pwrattr->control);
> > +	delay = pwrattr->autosuspend_delay;
> > +	delay_size =  sizeof(pwrattr->autosuspend_delay);
> > +	pwrattr->pci_dev = pci_dev;
> > +	pwrattr->autosuspend_supported = true;
> > +
> > +	if (!igt_pm_save_power_attr(delay_fd, delay, delay_size, true)) {
> > +		pwrattr->autosuspend_supported = false;
> > +		igt_debug("PCI '%04x:%02x:%02x.%01x' doesn't support
> auto_suspend\n",
> > +			  pci_dev->domain, pci_dev->bus, pci_dev->dev,
> pci_dev->func);
> > +	}
> > +
> > +	igt_pm_save_power_attr(control_fd, control, control_size, false);
> > +	igt_debug("PCI '%04x:%02x:%02x.%01x' Saved 'control,
> autosuspend_delay_ms' as '%s, %s'\n",
> > +		  pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func,
> control,
> > +		  pwrattr->autosuspend_supported ? delay : "NA");
> > +
> > +	igt_install_exit_handler(__igt_pm_pci_card_exit_handler);
> > +
> > +write_power_attr:
> > +	igt_pm_write_power_attr(delay_fd, "0\n");
> > +	igt_pm_write_power_attr(control_fd, "auto\n");
> > +
> > +	close(delay_fd);
> > +	close(control_fd);
> > +}
> > +
> > +static void
> > +igt_pm_setup_pci_card_power_attrs(struct pci_device *pci_dev, bool
> > +save_attrs) {
> > +	int primary, secondary, subordinate, ret;
> > +	struct pci_device_iterator *iter;
> > +	struct pci_slot_match match;
> > +	struct pci_device *dev;
> > +	int i = 0;
> > +
> > +	ret = pci_device_get_bridge_buses(pci_dev, &primary, &secondary,
> &subordinate);
> > +	igt_assert(!ret);
> > +
> > +	ret = pci_system_init();
> > +	igt_assert(!ret);
> > +
> > +	match.domain = pci_dev->domain;
> > +	match.bus = PCI_MATCH_ANY;
> > +	match.dev = PCI_MATCH_ANY;
> > +	match.func = PCI_MATCH_ANY;
> > +	iter = pci_slot_match_iterator_create(&match);
> > +	igt_assert(iter);
> > +
> > +	/* Setup power attrs for PCI root port */
> > +	igt_pm_setup_pci_dev_power_attrs(pci_dev, save_attrs ?
> &__pci_dev_pwrattr[i++] : NULL);
> > +	while ((dev = pci_device_next(iter)) != NULL) {
> > +		if (dev->bus >= secondary && dev->bus <= subordinate) {
> > +			igt_pm_setup_pci_dev_power_attrs(dev,
> > +							 save_attrs ?
> &__pci_dev_pwrattr[i++] : NULL);
> > +			if (save_attrs)
> > +				igt_assert(i <  MAX_PCI_DEVICES);
> > +		}
> > +	}
> > +
> > +	pci_iterator_destroy(iter);
> > +}
> > +
> > +/**
> > + * igt_pm_enable_pci_card_runtime_pm:
> > + * @pci_dev: root port pci_dev.
> > + * Enable runtime PM for all PCI endpoints devices for a given root
> > +port by
> > + * setting power/control attr to "auto" and setting
> > +autosuspend_delay_ms
> > + * to zero.
> > + */
> > +void igt_pm_enable_pci_card_runtime_pm(struct pci_device *pci_dev) {
> > +	igt_pm_setup_pci_card_power_attrs(pci_dev, false);
> > +	pci_system_cleanup();
> > +}
> > +
> > +/**
> > + * igt_pm_setup_pci_card_runtime_pm:
> > + * @pci_dev: root port pci_dev.
> > + * Setup runtime PM for all PCI endpoints devices for a given root
> > +port by
> > + * enabling runtime suspend and setting autosuspend_delay_ms to zero.
> 
> we know that we have bugs if we set i915's autosuspend delay to zero.
> I believe we need to make this optional here and then read the i915 one and use
> that as reference to set to all the endpoint devices here.
AFAIK igt sets i915 autosuspend_ms_delay to zero for any runtime pm based test, not sure about any issue about zero ms delay but I agree here this can be optional.
We can have two options.
1. Removal of autosuspend delay setup completely. Let it be default.
(Most of devices has 100ms barring i915)
2. Setting a uniform delay of 1 second to all devices in pci topology tree. 
I can change above based upon your opinion.
Br,
Anshuman Gupta.
> 
> but that change is mostly in the other patch and change in this would be
> minimal, so:
> 
> Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
> 
> > + * It also saves and restore power control attribute for all PCI
> > +endpoints
> > + * devices under given root port.
> > + */
> > +void igt_pm_setup_pci_card_runtime_pm(struct pci_device *pci_dev) {
> > +	memset(__pci_dev_pwrattr, 0, sizeof(__pci_dev_pwrattr));
> > +	igt_pm_setup_pci_card_power_attrs(pci_dev, true); }
> > +
> > +static void
> > +igt_pm_restore_power_attr(struct pci_device *pci_dev, const char
> > +*attr, char *val, int len) {
> > +	int fd;
> > +
> > +	igt_debug("PCI '%04x:%02x:%02x.%01x' Restoring %s attr to '%s'\n",
> > +		  pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func,
> attr,
> > +val);
> > +
> > +	fd = igt_pm_get_power_attr_fd(pci_dev, attr);
> > +	igt_assert(write(fd, val, len) == len);
> > +
> > +	close(fd);
> > +}
> > +
> > +/**
> > + * igt_pm_restore_pci_card_runtime_pm:
> > + * Restore control and autosuspend_delay_ms power attribute for all
> > + * PCI endpoints devices under gfx root port, which were saved
> > +earlier
> > + * by igt_pm_setup_pci_card_runtime_pm().
> > + */
> > +void igt_pm_restore_pci_card_runtime_pm(void)
> > +{
> > +	int i = 0;
> > +
> > +	for (i = 0; i < MAX_PCI_DEVICES; i++) {
> > +		if (!__pci_dev_pwrattr[i].pci_dev)
> > +			break;
> > +
> > +		igt_pm_restore_power_attr(__pci_dev_pwrattr[i].pci_dev,
> "control",
> > +					  __pci_dev_pwrattr[i].control,
> > +					  sizeof(__pci_dev_pwrattr[i].control));
> > +
> > +		if (!__pci_dev_pwrattr[i].autosuspend_supported)
> > +			continue;
> > +
> > +		igt_pm_restore_power_attr(__pci_dev_pwrattr[i].pci_dev,
> "autosuspend_delay_ms",
> > +
> __pci_dev_pwrattr[i].autosuspend_delay,
> > +
> sizeof(__pci_dev_pwrattr[i].autosuspend_delay));
> > +	}
> > +
> > +	memset(__pci_dev_pwrattr, 0, sizeof(__pci_dev_pwrattr));
> > +	pci_system_cleanup();
> > +}
> > +
> > +static void igt_pm_print_pci_dev_runtime_status(struct pci_device
> > +*pci_dev) {
> > +	char name[PATH_MAX], runtime_status[64];
> > +	int fd, n_read;
> > +
> > +	snprintf(name, PATH_MAX,
> "/sys/bus/pci/devices/%04x:%02x:%02x.%01x/power/runtime_status",
> > +		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func);
> > +
> > +	fd = open(name, O_RDONLY);
> > +	igt_assert_f(fd >= 0, "Can't open runtime_status\n");
> > +
> > +	n_read = read(fd, runtime_status, sizeof(runtime_status) - 1);
> > +	igt_assert(n_read >= 0);
> > +	runtime_status[n_read] = '\0';
> > +	igt_info("runtime suspend status for PCI '%04x:%02x:%02x.%01x' %s\n",
> > +		 pci_dev->domain, pci_dev->bus, pci_dev->dev, pci_dev->func,
> runtime_status);
> > +	close(fd);
> > +}
> > +
> > +/**
> > + * igt_pm_print_pci_card_runtime_status:
> > + * @pci_dev: root port pci_dev.
> > + * Print runtime suspend status for all PCI endpoints devices for a
> > +given
> > + * root port.
> > + */
> > +void igt_pm_print_pci_card_runtime_status(void)
> > +{
> > +	int i = 0;
> > +
> > +	for (i = 0; i < MAX_PCI_DEVICES; i++) {
> > +		if (!__pci_dev_pwrattr[i].pci_dev)
> > +			break;
> > +
> > +
> 	igt_pm_print_pci_dev_runtime_status(__pci_dev_pwrattr[i].pci_dev);
> > +	}
> > +}
> > diff --git a/lib/igt_pm.h b/lib/igt_pm.h index 162d3ca3c..c53dae2c3
> > 100644
> > --- a/lib/igt_pm.h
> > +++ b/lib/igt_pm.h
> > @@ -46,6 +46,23 @@ enum igt_runtime_pm_status {
> >  	IGT_RUNTIME_PM_STATUS_UNKNOWN,
> >  };
> >
> > +/* PCI ACPI firmware node real state */ enum igt_acpi_d_state {
> > +	IGT_ACPI_D0,
> > +	IGT_ACPI_D1,
> > +	IGT_ACPI_D2,
> > +	IGT_ACPI_D3Hot,
> > +	IGT_ACPI_D3Cold,
> > +	IGT_ACPI_UNKNOWN_STATE,
> > +};
> > +
> > +struct	igt_pm_pci_dev_pwrattr {
> > +	struct pci_device *pci_dev;
> > +	char control[64];
> > +	bool autosuspend_supported;
> > +	char autosuspend_delay[64];
> > +};
> > +
> >  bool igt_setup_runtime_pm(int device);  void
> > igt_disable_runtime_pm(void);  void igt_restore_runtime_pm(void); @@
> > -54,5 +71,12 @@ bool igt_wait_for_pm_status(enum igt_runtime_pm_status
> > status);  bool igt_pm_dmc_loaded(int debugfs);  bool
> > igt_pm_pc8_plus_residencies_enabled(int msr_fd);  bool
> > i915_output_is_lpsp_capable(int drm_fd, igt_output_t *output);
> > +bool igt_pm_acpi_d3cold_supported(struct pci_device *pci_dev); enum
> > +igt_acpi_d_state igt_pm_get_acpi_real_d_state(struct pci_device
> > +*pci_dev); void igt_pm_enable_pci_card_runtime_pm(struct pci_device
> > +*pci_dev); void igt_pm_setup_pci_card_runtime_pm(struct pci_device
> > +*pci_dev); void igt_pm_restore_pci_card_runtime_pm(void);
> > +void igt_pm_print_pci_card_runtime_status(void);
> >
> >  #endif /* IGT_PM_H */
> > --
> > 2.26.2
> >

[igt-dev] [PATCH i-g-t v5 4/9] tools: Add intel_pm_rpm tool

[Anshuman Gupta]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 6875 bytes --]

intel_pm_rpm tool is a debug tool. It can be use to setup
and prepare the gfx card to go to D3Cold.
It also provide the debug option to disable all display and
prepare device to enter to runtime suspend.

v2:
- Removed IS_DGFX() condition.
v3:
- Change naming convention from setup-d3cold to force-d3cold-wait
  disable-display to disable-display-wait. [Rodrigo]

Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
---
 tools/intel_pm_rpm.c | 210 +++++++++++++++++++++++++++++++++++++++++++
 tools/meson.build    |   1 +
 2 files changed, 211 insertions(+)
 create mode 100644 tools/intel_pm_rpm.c

diff --git a/tools/intel_pm_rpm.c b/tools/intel_pm_rpm.c
new file mode 100644
index 000000000..909a2a34d
--- /dev/null
+++ b/tools/intel_pm_rpm.c
@@ -0,0 +1,210 @@
+/*
+ * Copyright © 2022 Intel Corporation
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a
+ * copy of this software and associated documentation files (the "Software"),
+ * to deal in the Software without restriction, including without limitation
+ * the rights to use, copy, modify, merge, publish, distribute, sublicense,
+ * and/or sell copies of the Software, and to permit persons to whom the
+ * Software is furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice (including the next
+ * paragraph) shall be included in all copies or substantial portions of the
+ * Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
+ * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+ * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+ * IN THE SOFTWARE.
+ *
+ */
+
+#include <errno.h>
+#include <getopt.h>
+#include <glib.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <stdbool.h>
+#include "igt.h"
+#include "igt_device.h"
+#include "igt_device_scan.h"
+#include "igt_pm.h"
+
+typedef struct {
+	int drm_fd;
+	int debugfs_fd;
+	uint32_t devid;
+	drmModeResPtr res;
+	igt_display_t display;
+} data_t;
+
+const char *help_str =
+	"  --disable-display-wait\t\tDisable all screens and try to go into runtime pm.\n"
+	"  --force-d3cold-wait\t\tForce dgfx gfx card to enter runtime D3Cold.\n"
+	"  --help\t\tProvide help. Provide card name with IGT_DEVICE=drm:/dev/dri/card*.";
+static struct option long_options[] = {
+	{"disable-display-wait", 0, 0, 'd'},
+	{"force-d3cold-wait", 0, 0, 'f'},
+	{"help", 0, 0, 'h'},
+	{ 0, 0, 0, 0 }
+};
+
+const char *optstr = "dfh";
+
+static void usage(const char *name)
+{
+	igt_info("Usage: %s [options]\n", name);
+	igt_info("%s\n", help_str);
+}
+
+static void disable_all_displays(data_t *data)
+{
+	igt_output_t *output;
+
+	for (int i = 0; i < data->display.n_outputs; i++) {
+		output = &data->display.outputs[i];
+		igt_output_set_pipe(output, PIPE_NONE);
+		igt_display_commit(&data->display);
+	}
+}
+
+static void force_gfx_card_d3cold(data_t *data)
+{
+	struct pci_device *root;
+	int d_state;
+
+	root = igt_device_get_pci_root_port(data->drm_fd);
+
+	if (!igt_pm_acpi_d3cold_supported(root)) {
+		igt_info("D3Cold isn't supported on Root port %04x:%02x:%02x.%01x\n",
+			 root->domain, root->bus, root->dev, root->func);
+		return;
+	}
+
+	disable_all_displays(data);
+	igt_pm_setup_pci_card_runtime_pm(root);
+	sleep(1);
+	d_state = igt_pm_get_acpi_real_d_state(root);
+
+	if (d_state == IGT_ACPI_D3Cold) {
+		igt_info("D3Cold achieved for root port %04x:%02x:%02x.%01x\n",
+			 root->domain, root->bus, root->dev, root->func);
+	} else {
+		igt_pm_print_pci_card_runtime_status();
+		igt_info("D3Cold not achieved yet. Please monitor %04x:%02x:%02x.%01x real_power_state\n",
+			 root->domain, root->bus, root->dev, root->func);
+	}
+
+	igt_info("Hit CTRL-C to exit\n");
+	while (1)
+		sleep(600);
+}
+
+int main(int argc, char *argv[])
+{
+	bool disable_display = false, force_d3cold = false;
+	struct igt_device_card card;
+	char *env_device = NULL;
+	int c, option_index = 0;
+	data_t data = {};
+	int ret = 0;
+
+	if (argc <= 1) {
+		usage(argv[0]);
+		return EXIT_SUCCESS;
+	}
+
+	env_device = getenv("IGT_DEVICE");
+	igt_devices_scan(false);
+
+	if (env_device) {
+		if (!igt_device_card_match(env_device, &card)) {
+			igt_warn("No device found for the env_device\n");
+			ret = EXIT_FAILURE;
+			goto exit;
+		}
+	} else {
+		if (!igt_device_find_first_i915_discrete_card(&card)) {
+			igt_warn("No discrete gpu found\n");
+			ret = EXIT_FAILURE;
+			goto exit;
+		}
+	}
+
+	while ((c = getopt_long(argc, argv, optstr,
+				long_options, &option_index)) != -1) {
+		switch (c) {
+		case 'd':
+			disable_display = true;
+			break;
+		case 'f':
+			force_d3cold = true;
+			break;
+		default:
+		case 'h':
+			usage(argv[0]);
+			ret = EXIT_SUCCESS;
+			goto exit;
+		}
+	}
+
+	data.drm_fd = igt_open_card(&card);
+	if (data.drm_fd  >= 0) {
+		igt_info("Device %s successfully opened\n", card.card);
+	} else {
+		igt_warn("Cannot open card %s device\n", card.card);
+		ret = EXIT_FAILURE;
+		goto exit;
+	}
+
+	data.debugfs_fd = igt_debugfs_dir(data.drm_fd);
+	data.devid = intel_get_drm_devid(data.drm_fd);
+	igt_setup_runtime_pm(data.drm_fd);
+
+	data.res = drmModeGetResources(data.drm_fd);
+	if (data.res) {
+		kmstest_set_vt_graphics_mode();
+		igt_display_require(&data.display, data.drm_fd);
+
+		/* i915 disables RPM in case DMC is not loaded on kms supported cards */
+		if (!igt_pm_dmc_loaded(data.debugfs_fd)) {
+			igt_warn("dmc fw is not loaded, no runtime pm\n");
+			ret = EXIT_FAILURE;
+			goto exit;
+		}
+	}
+
+	if (disable_display) {
+		disable_all_displays(&data);
+		if (!igt_wait_for_pm_status(IGT_RUNTIME_PM_STATUS_SUSPENDED)) {
+			__igt_debugfs_dump(data.drm_fd, "i915_runtime_pm_status", IGT_LOG_INFO);
+			ret = EXIT_FAILURE;
+			goto exit;
+		}
+
+		igt_info("Device runtime suspended, Useful for debugging.\n"
+			 "Hit CTRL-C to exit\n");
+		/* Don't return useful for debugging */
+		while (1)
+			sleep(600);
+	}
+
+	if (force_d3cold)
+		force_gfx_card_d3cold(&data);
+
+exit:
+	igt_restore_runtime_pm();
+
+	if (data.res)
+		igt_display_fini(&data.display);
+
+	close(data.debugfs_fd);
+	close(data.drm_fd);
+	igt_devices_free();
+
+	return ret;
+}
diff --git a/tools/meson.build b/tools/meson.build
index 771d0b9e3..24d0ea714 100644
--- a/tools/meson.build
+++ b/tools/meson.build
@@ -28,6 +28,7 @@ tools_progs = [
 	'intel_lid',
 	'intel_opregion_decode',
 	'intel_panel_fitter',
+	'intel_pm_rpm',
 	'intel_reg_checker',
 	'intel_residency',
 	'intel_stepping',
-- 
2.26.2

[igt-dev] [PATCH i-g-t v5 5/9] tools/intel_pm_rpm: Add an option to setup d3cold

[Anshuman Gupta]

Few PCI devices under DGFX card tree don't have any kernel driver.
These devices will block D3cold state of gfx root port.
Adding support to setup d3cold by enabling runtime PM for all PCI
devices under the gfx root port.
It will not save/restore pci devices power attributes.
It will be useful to tune D3Cold after boot.

v2:
- Change naming convention from configure_autosuspend to
  setup-d3cold. [Rodrigo].

Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
---
 tools/intel_pm_rpm.c | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/tools/intel_pm_rpm.c b/tools/intel_pm_rpm.c
index 909a2a34d..2703ac305 100644
--- a/tools/intel_pm_rpm.c
+++ b/tools/intel_pm_rpm.c
@@ -45,15 +45,17 @@ typedef struct {
 const char *help_str =
 	"  --disable-display-wait\t\tDisable all screens and try to go into runtime pm.\n"
 	"  --force-d3cold-wait\t\tForce dgfx gfx card to enter runtime D3Cold.\n"
+	"  --setup-d3cold\t\tPrepare dgfx gfx card to enter to D3Cold.\n"
 	"  --help\t\tProvide help. Provide card name with IGT_DEVICE=drm:/dev/dri/card*.";
 static struct option long_options[] = {
 	{"disable-display-wait", 0, 0, 'd'},
 	{"force-d3cold-wait", 0, 0, 'f'},
+	{"setup-d3cold", 0, 0, 's'},
 	{"help", 0, 0, 'h'},
 	{ 0, 0, 0, 0 }
 };
 
-const char *optstr = "dfh";
+const char *optstr = "dfsh";
 
 static void usage(const char *name)
 {
@@ -72,6 +74,17 @@ static void disable_all_displays(data_t *data)
 	}
 }
 
+static void
+setup_gfx_card_d3cold(data_t *data)
+{
+	struct pci_device *root;
+
+	root = igt_device_get_pci_root_port(data->drm_fd);
+	igt_pm_enable_pci_card_runtime_pm(root);
+	igt_info("Enabled pci devs runtime pm under Root port %04x:%02x:%02x.%01x\n",
+		 root->domain, root->bus, root->dev, root->func);
+}
+
 static void force_gfx_card_d3cold(data_t *data)
 {
 	struct pci_device *root;
@@ -106,7 +119,7 @@ static void force_gfx_card_d3cold(data_t *data)
 
 int main(int argc, char *argv[])
 {
-	bool disable_display = false, force_d3cold = false;
+	bool disable_display = false, force_d3cold = false, setup_d3cold = false;
 	struct igt_device_card card;
 	char *env_device = NULL;
 	int c, option_index = 0;
@@ -144,6 +157,9 @@ int main(int argc, char *argv[])
 		case 'f':
 			force_d3cold = true;
 			break;
+		case 's':
+			setup_d3cold = true;
+			break;
 		default:
 		case 'h':
 			usage(argv[0]);
@@ -196,6 +212,9 @@ int main(int argc, char *argv[])
 	if (force_d3cold)
 		force_gfx_card_d3cold(&data);
 
+	if (setup_d3cold)
+		setup_gfx_card_d3cold(&data);
+
 exit:
 	igt_restore_runtime_pm();
 
-- 
2.26.2

Re: [igt-dev] [PATCH i-g-t v5 5/9] tools/intel_pm_rpm: Add an option to setup d3cold

[Rodrigo Vivi]

On Fri, May 06, 2022 at 09:15:49PM +0530, Anshuman Gupta wrote:
> Few PCI devices under DGFX card tree don't have any kernel driver.
> These devices will block D3cold state of gfx root port.
> Adding support to setup d3cold by enabling runtime PM for all PCI
> devices under the gfx root port.
> It will not save/restore pci devices power attributes.
> It will be useful to tune D3Cold after boot.
> 
> v2:
> - Change naming convention from configure_autosuspend to
>   setup-d3cold. [Rodrigo].
> 
> Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
> Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
> ---
>  tools/intel_pm_rpm.c | 23 +++++++++++++++++++++--
>  1 file changed, 21 insertions(+), 2 deletions(-)
> 
> diff --git a/tools/intel_pm_rpm.c b/tools/intel_pm_rpm.c
> index 909a2a34d..2703ac305 100644
> --- a/tools/intel_pm_rpm.c
> +++ b/tools/intel_pm_rpm.c
> @@ -45,15 +45,17 @@ typedef struct {
>  const char *help_str =
>  	"  --disable-display-wait\t\tDisable all screens and try to go into runtime pm.\n"
>  	"  --force-d3cold-wait\t\tForce dgfx gfx card to enter runtime D3Cold.\n"
> +	"  --setup-d3cold\t\tPrepare dgfx gfx card to enter to D3Cold.\n"

thanks
so, as I told in the other patch I prefer this to use i915 autosuspend delay as reference
for the others or at least avoid touching the i915 one.

with that taken care:

Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>



>  	"  --help\t\tProvide help. Provide card name with IGT_DEVICE=drm:/dev/dri/card*.";
>  static struct option long_options[] = {
>  	{"disable-display-wait", 0, 0, 'd'},
>  	{"force-d3cold-wait", 0, 0, 'f'},
> +	{"setup-d3cold", 0, 0, 's'},
>  	{"help", 0, 0, 'h'},
>  	{ 0, 0, 0, 0 }
>  };
>  
> -const char *optstr = "dfh";
> +const char *optstr = "dfsh";
>  
>  static void usage(const char *name)
>  {
> @@ -72,6 +74,17 @@ static void disable_all_displays(data_t *data)
>  	}
>  }
>  
> +static void
> +setup_gfx_card_d3cold(data_t *data)
> +{
> +	struct pci_device *root;
> +
> +	root = igt_device_get_pci_root_port(data->drm_fd);
> +	igt_pm_enable_pci_card_runtime_pm(root);
> +	igt_info("Enabled pci devs runtime pm under Root port %04x:%02x:%02x.%01x\n",
> +		 root->domain, root->bus, root->dev, root->func);
> +}
> +
>  static void force_gfx_card_d3cold(data_t *data)
>  {
>  	struct pci_device *root;
> @@ -106,7 +119,7 @@ static void force_gfx_card_d3cold(data_t *data)
>  
>  int main(int argc, char *argv[])
>  {
> -	bool disable_display = false, force_d3cold = false;
> +	bool disable_display = false, force_d3cold = false, setup_d3cold = false;
>  	struct igt_device_card card;
>  	char *env_device = NULL;
>  	int c, option_index = 0;
> @@ -144,6 +157,9 @@ int main(int argc, char *argv[])
>  		case 'f':
>  			force_d3cold = true;
>  			break;
> +		case 's':
> +			setup_d3cold = true;
> +			break;
>  		default:
>  		case 'h':
>  			usage(argv[0]);
> @@ -196,6 +212,9 @@ int main(int argc, char *argv[])
>  	if (force_d3cold)
>  		force_gfx_card_d3cold(&data);
>  
> +	if (setup_d3cold)
> +		setup_gfx_card_d3cold(&data);
> +
>  exit:
>  	igt_restore_runtime_pm();
>  
> -- 
> 2.26.2
> 

[igt-dev] [PATCH i-g-t v5 6/9] i915_pm_rpm: Add D3Cold basic subtest

[Anshuman Gupta]

Add support for D3Cold basic subtest.
It setup and prepares the gfx PCI card for D3Cold
and checks the ACPI D3Cold state.

v2:
- Removed IS_DGFX() condition.
- Added igt_describe(). [Kamil]

Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
---
 tests/i915/i915_pm_rpm.c | 26 ++++++++++++++++++++++++++
 1 file changed, 26 insertions(+)

diff --git a/tests/i915/i915_pm_rpm.c b/tests/i915/i915_pm_rpm.c
index d2bce5826..31f9712a8 100644
--- a/tests/i915/i915_pm_rpm.c
+++ b/tests/i915/i915_pm_rpm.c
@@ -1534,6 +1534,28 @@ __noreturn static void stay_subtest(void)
 		sleep(600);
 }
 
+static void d3cold_basic_subtest(void)
+{
+	struct pci_device *root;
+	bool result;
+
+	root = igt_device_get_pci_root_port(drm_fd);
+	igt_require(igt_pm_acpi_d3cold_supported(root));
+	igt_pm_setup_pci_card_runtime_pm(root);
+	disable_all_screens_and_wait(&ms_data);
+
+	result = igt_wait(igt_pm_get_acpi_real_d_state(root) == IGT_ACPI_D3Cold, 10000, 500);
+
+	if (!result) {
+		igt_info("D3Cold not achieved for root port %04x:%02x:%02x.%01x\n",
+			 root->domain, root->bus, root->dev, root->func);
+		igt_pm_print_pci_card_runtime_status();
+	}
+
+	igt_assert(result);
+	igt_pm_restore_pci_card_runtime_pm();
+}
+
 static void system_suspend_subtest(int state, int debug)
 {
 	disable_all_screens_and_wait(&ms_data);
@@ -2065,6 +2087,10 @@ igt_main_args("", long_options, help_str, opt_handler, NULL)
 		igt_subtest("stay")
 			stay_subtest();
 
+	igt_describe("Validate gfx root port d3cold state");
+	igt_subtest("d3cold-basic")
+		d3cold_basic_subtest();
+
 	/* Essential things */
 	igt_subtest("drm-resources-equal")
 		drm_resources_equal_subtest();
-- 
2.26.2

[igt-dev] [PATCH i-g-t v5 7/9] i915_pm_rpm: Extend gem_exec_stress test with D3Cold

[Anshuman Gupta]

Added d3cold dynamic subtest to gem_exec_stress with device class
memory region. It test both D3Cold-{VRAM_SR, Off} by using
d3cold_sr_lmem_threshold i915_params debugfs.

v2:
- Two different subtest d3cold-off and d3cold-vrsr. [Rodrigo]
- commit header modification.
v3:
- Move {get, set}_d3cold_sr_lmem_threshold in igt_fixture. [Rodrigo]
- Skip the d3cold-vram-sr test when lmem_threshold is not available.

Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
---
 tests/i915/i915_pm_rpm.c | 111 +++++++++++++++++++++++++++++++++------
 1 file changed, 96 insertions(+), 15 deletions(-)

diff --git a/tests/i915/i915_pm_rpm.c b/tests/i915/i915_pm_rpm.c
index 31f9712a8..b16d61643 100644
--- a/tests/i915/i915_pm_rpm.c
+++ b/tests/i915/i915_pm_rpm.c
@@ -85,6 +85,7 @@ enum plane_type {
 #define WAIT_PC8_RES	2
 #define WAIT_EXTRA	4
 #define USE_DPMS	8
+#define WAIT_D3COLD	16
 
 int drm_fd, msr_fd, pc8_status_fd;
 int debugfs;
@@ -215,6 +216,21 @@ static bool wait_for_suspended(void)
 	}
 }
 
+static bool wait_for_d3cold(struct pci_device *root)
+{
+	bool d3colded;
+
+	d3colded = igt_wait(igt_pm_get_acpi_real_d_state(root) == IGT_ACPI_D3Cold, 10000, 500);
+
+	if (!d3colded) {
+		igt_info("D3Cold not achieved for root port %04x:%02x:%02x.%01x\n",
+			 root->domain, root->bus, root->dev, root->func);
+		igt_pm_print_pci_card_runtime_status();
+	}
+
+	return d3colded;
+}
+
 static bool wait_for_active(void)
 {
 	if (has_pc8 && !has_runtime_pm)
@@ -744,6 +760,38 @@ static void test_i2c(struct mode_set_data *data)
 			"There is an EDID mismatch between i2c and DRM!\n");
 }
 
+static int get_d3cold_sr_lmem_threshold(int dir)
+{
+	int i915_params, lmem_threshold, ret;
+
+	i915_params = openat(dir, "i915_params", O_RDONLY);
+	igt_assert(i915_params > 0);
+	ret = igt_sysfs_scanf(i915_params, "d3cold_sr_lmem_threshold", "%d", &lmem_threshold);
+	if (ret < 0) {
+		close(i915_params);
+		return ret;
+	}
+
+	close(i915_params);
+	return lmem_threshold;
+}
+
+static int set_d3cold_sr_lmem_threshold(int dir, int val)
+{
+	int i915_params, ret;
+
+	i915_params = openat(dir, "i915_params", O_RDONLY);
+	igt_assert(i915_params > 0);
+	ret = igt_sysfs_printf(i915_params, "d3cold_sr_lmem_threshold", "%d", val);
+	if (ret < 0) {
+		close(i915_params);
+		return ret;
+	}
+
+	close(i915_params);
+	return ret;
+}
+
 static void setup_pc8(void)
 {
 	has_pc8 = false;
@@ -1069,6 +1117,17 @@ static void debugfs_forcewake_user_subtest(void)
 	igt_assert(wait_for_suspended());
 }
 
+static struct pci_device *setup_d3cold_and_get_root_port(void)
+{
+	struct pci_device *root;
+
+	root = igt_device_get_pci_root_port(drm_fd);
+	igt_require(igt_pm_acpi_d3cold_supported(root));
+	igt_pm_setup_pci_card_runtime_pm(root);
+
+	return root;
+}
+
 static void gem_mmap_args(const struct mmap_offset *t,
 			  struct drm_i915_gem_memory_class_instance *mem_regions)
 {
@@ -1392,6 +1451,7 @@ gem_execbuf_stress_subtest(int rounds, int wait_flags,
 	int i;
 	int batch_size = 4 * sizeof(uint32_t);
 	uint32_t batch_buf[batch_size];
+	struct pci_device *root;
 	uint32_t handle;
 	struct drm_i915_gem_execbuffer2 execbuf = {};
 	struct drm_i915_gem_exec_object2 objs[1] = {{}};
@@ -1408,6 +1468,9 @@ gem_execbuf_stress_subtest(int rounds, int wait_flags,
 	batch_buf[i++] = MI_NOOP;
 	igt_assert(i * sizeof(uint32_t) == batch_size);
 
+	if (wait_flags & WAIT_D3COLD)
+		root = setup_d3cold_and_get_root_port();
+
 	disable_all_screens_and_wait(&ms_data);
 
 	/* PC8 test is only applicable to igfx  */
@@ -1433,6 +1496,9 @@ gem_execbuf_stress_subtest(int rounds, int wait_flags,
 			/* clean up idle work */
 			igt_drop_caches_set(drm_fd, DROP_IDLE);
 			igt_assert(wait_for_suspended());
+			if (wait_flags & WAIT_D3COLD)
+				igt_assert(wait_for_d3cold(root));
+
 		}
 		if (wait_flags & WAIT_PC8_RES)
 			igt_assert(pc8_plus_residency_changed(30));
@@ -1440,6 +1506,9 @@ gem_execbuf_stress_subtest(int rounds, int wait_flags,
 			sleep(5);
 	}
 
+	if (wait_flags & WAIT_D3COLD)
+		igt_pm_restore_pci_card_runtime_pm();
+
 	gem_close(drm_fd, handle);
 }
 
@@ -1537,22 +1606,10 @@ __noreturn static void stay_subtest(void)
 static void d3cold_basic_subtest(void)
 {
 	struct pci_device *root;
-	bool result;
 
-	root = igt_device_get_pci_root_port(drm_fd);
-	igt_require(igt_pm_acpi_d3cold_supported(root));
-	igt_pm_setup_pci_card_runtime_pm(root);
+	root = setup_d3cold_and_get_root_port();
 	disable_all_screens_and_wait(&ms_data);
-
-	result = igt_wait(igt_pm_get_acpi_real_d_state(root) == IGT_ACPI_D3Cold, 10000, 500);
-
-	if (!result) {
-		igt_info("D3Cold not achieved for root port %04x:%02x:%02x.%01x\n",
-			 root->domain, root->bus, root->dev, root->func);
-		igt_pm_print_pci_card_runtime_status();
-	}
-
-	igt_assert(result);
+	igt_assert(wait_for_d3cold(root));
 	igt_pm_restore_pci_card_runtime_pm();
 }
 
@@ -2071,6 +2128,8 @@ static struct option long_options[] = {
 
 igt_main_args("", long_options, help_str, opt_handler, NULL)
 {
+	int lmem_threshold;
+
 	igt_subtest("basic-rte") {
 		igt_assert(setup_environment(false));
 		basic_subtest();
@@ -2081,6 +2140,7 @@ igt_main_args("", long_options, help_str, opt_handler, NULL)
 	 * not properly configured. */
 	igt_fixture {
 		igt_require(setup_environment(false));
+		lmem_threshold = get_d3cold_sr_lmem_threshold(debugfs);
 	}
 
 	if (stay)
@@ -2211,6 +2271,24 @@ igt_main_args("", long_options, help_str, opt_handler, NULL)
 				gem_execbuf_stress_subtest(rounds, WAIT_STATUS, &r->ci);
 			igt_dynamic_f("%s-%s", "extra-wait", r->name)
 				gem_execbuf_stress_subtest(rounds, WAIT_STATUS | WAIT_EXTRA, &r->ci);
+
+			if (r->ci.memory_class == I915_MEMORY_CLASS_SYSTEM)
+				continue;
+
+			igt_dynamic_f("%s-%s", "d3cold-off", r->name) {
+				if (lmem_threshold >= 0)
+					set_d3cold_sr_lmem_threshold(debugfs, 300);
+
+				gem_execbuf_stress_subtest(rounds, WAIT_STATUS | WAIT_D3COLD,
+							   &r->ci);
+			}
+
+			igt_dynamic_f("%s-%s", "d3cold-vrsr", r->name) {
+				igt_require(lmem_threshold >= 0);
+				set_d3cold_sr_lmem_threshold(debugfs, 0);
+				gem_execbuf_stress_subtest(rounds, WAIT_STATUS | WAIT_D3COLD,
+							   &r->ci);
+			}
 		}
 	}
 
@@ -2227,8 +2305,11 @@ igt_main_args("", long_options, help_str, opt_handler, NULL)
 		pm_test_caching();
 	}
 
-	igt_fixture
+	igt_fixture {
+		if (lmem_threshold >= 0)
+			set_d3cold_sr_lmem_threshold(debugfs, lmem_threshold);
 		teardown_environment(false);
+	}
 
 	igt_subtest("module-reload") {
 		igt_debug("Reload w/o display\n");
-- 
2.26.2

[igt-dev] [PATCH i-g-t v5 8/9] i915_pm_rpm: Extend gem_execbuf test with D3Cold

[Anshuman Gupta]

Added d3cold dynamic subtest to gem_execbuf with device class
memory region. It test both D3Cold-{VRAM_SR, Off} by using
d3cold_sr_lmem_threshold i915_params debugfs.

v2:
- Two different subtest d3cold-off and d3cold-vrsr. [Rodrigo]
v3:
- Skip the d3cold-vram-sr test when lmem_threshold is not available.

Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
---
 tests/i915/i915_pm_rpm.c | 34 ++++++++++++++++++++++++++++++++--
 1 file changed, 32 insertions(+), 2 deletions(-)

diff --git a/tests/i915/i915_pm_rpm.c b/tests/i915/i915_pm_rpm.c
index b16d61643..a062fde88 100644
--- a/tests/i915/i915_pm_rpm.c
+++ b/tests/i915/i915_pm_rpm.c
@@ -1350,10 +1350,11 @@ static void submit_blt_cmd(uint32_t dst_handle, int dst_size,
 }
 
 /* Make sure we can submit a batch buffer and verify its result. */
-static void gem_execbuf_subtest(struct drm_i915_gem_memory_class_instance *mem_regions)
+static void gem_execbuf_subtest(struct drm_i915_gem_memory_class_instance *mem_regions, bool d3cold)
 {
 	int x, y;
 	uint32_t handle;
+	struct pci_device *root;
 	int bpp = 4;
 	int pitch = 128 * bpp;
 	int dst_size = 128 * 128 * bpp; /* 128x128 square */
@@ -1375,6 +1376,9 @@ static void gem_execbuf_subtest(struct drm_i915_gem_memory_class_instance *mem_r
 	memset(cpu_buf, 0, dst_size);
 	gem_write(drm_fd, handle, 0, cpu_buf, dst_size);
 
+	if (d3cold)
+		root = setup_d3cold_and_get_root_port();
+
 	/* Now suspend and try it. */
 	disable_all_screens_and_wait(&ms_data);
 
@@ -1383,8 +1387,15 @@ static void gem_execbuf_subtest(struct drm_i915_gem_memory_class_instance *mem_r
 		       &presumed_offset);
 	igt_assert(wait_for_suspended());
 
+	if (d3cold)
+		igt_assert(wait_for_d3cold(root));
+
 	gem_read(drm_fd, handle, 0, cpu_buf, dst_size);
 	igt_assert(wait_for_suspended());
+
+	if (d3cold)
+		igt_assert(wait_for_d3cold(root));
+
 	for (y = 0; y < 128; y++) {
 		for (x = 0; x < 128; x++) {
 			uint32_t px = cpu_buf[y * 128 + x];
@@ -1423,6 +1434,9 @@ static void gem_execbuf_subtest(struct drm_i915_gem_memory_class_instance *mem_r
 
 	disable_all_screens_and_wait(&ms_data);
 
+	if (d3cold)
+		igt_assert(wait_for_d3cold(root));
+
 	memset(cpu_buf, 0, dst_size);
 	gem_read(drm_fd, handle, 0, cpu_buf, dst_size);
 	for (y = 0; y < 128; y++) {
@@ -2182,7 +2196,23 @@ igt_main_args("", long_options, help_str, opt_handler, NULL)
 	igt_subtest_with_dynamic("gem-execbuf") {
 		for_each_memory_region(r, drm_fd) {
 			igt_dynamic_f("%s", r->name)
-				gem_execbuf_subtest(&r->ci);
+				gem_execbuf_subtest(&r->ci, false);
+
+			if (r->ci.memory_class == I915_MEMORY_CLASS_SYSTEM)
+				continue;
+
+			igt_dynamic_f("%s-%s", "d3cold-off", r->name) {
+				if (lmem_threshold >= 0)
+					set_d3cold_sr_lmem_threshold(debugfs, 300);
+
+				gem_execbuf_subtest(&r->ci, true);
+			}
+
+			igt_dynamic_f("%s-%s", "d3cold-vrsr", r->name) {
+				igt_require(lmem_threshold >= 0);
+				set_d3cold_sr_lmem_threshold(debugfs, 0);
+				gem_execbuf_subtest(&r->ci, true);
+			}
 		}
 	}
 	igt_subtest("gem-idle")
-- 
2.26.2

[igt-dev] [PATCH i-g-t v5 9/9] i915_pm_rpm: Extend gem-mmap-type test with D3Cold

[Anshuman Gupta]

Added d3cold dynamic subtest to gem-mmap-type with device class
memory region. It test both D3Cold-{VRAM_SR, Off} by using
d3cold_sr_lmem_threshold i915_params debugfs.

v2:
- Two different subtest d3cold-off and d3cold-vrsr. [Rodrigo]
v3:
- Skip the d3cold-vram-sr test when lmem_threshold is not available.

Cc: Rodrigo Vivi <rodrigo.vivi@intel.com>
Reviewed-by: Rodrigo Vivi <rodrigo.vivi@intel.com>
Signed-off-by: Anshuman Gupta <anshuman.gupta@intel.com>
---
 tests/i915/i915_pm_rpm.c | 41 ++++++++++++++++++++++++++++++++++++++--
 1 file changed, 39 insertions(+), 2 deletions(-)

diff --git a/tests/i915/i915_pm_rpm.c b/tests/i915/i915_pm_rpm.c
index a062fde88..3a4e2901b 100644
--- a/tests/i915/i915_pm_rpm.c
+++ b/tests/i915/i915_pm_rpm.c
@@ -1129,11 +1129,12 @@ static struct pci_device *setup_d3cold_and_get_root_port(void)
 }
 
 static void gem_mmap_args(const struct mmap_offset *t,
-			  struct drm_i915_gem_memory_class_instance *mem_regions)
+			  struct drm_i915_gem_memory_class_instance *mem_regions, bool d3cold)
 {
 	int i;
 	uint32_t handle;
 	int buf_size = 8192;
+	struct pci_device *root;
 	uint8_t *gem_buf;
 
 	/* Create, map and set data while the device is active. */
@@ -1151,17 +1152,29 @@ static void gem_mmap_args(const struct mmap_offset *t,
 	for (i = 0; i < buf_size; i++)
 		igt_assert(gem_buf[i] == (i & 0xFF));
 
+	if (d3cold)
+		root = setup_d3cold_and_get_root_port();
+
 	/* Now suspend, read and modify. */
 	disable_all_screens_and_wait(&ms_data);
 
+	if (d3cold)
+		igt_assert(wait_for_d3cold(root));
+
 	for (i = 0; i < buf_size; i++)
 		igt_assert(gem_buf[i] == (i & 0xFF));
 	igt_assert(wait_for_suspended());
 
+	if (d3cold)
+		igt_assert(wait_for_d3cold(root));
+
 	for (i = 0; i < buf_size; i++)
 		gem_buf[i] = (~i & 0xFF);
 	igt_assert(wait_for_suspended());
 
+	if (d3cold)
+		igt_assert(wait_for_d3cold(root));
+
 	/* Now resume and see if it's still there. */
 	enable_one_screen_and_wait(&ms_data);
 	for (i = 0; i < buf_size; i++)
@@ -1173,12 +1186,18 @@ static void gem_mmap_args(const struct mmap_offset *t,
 	 * suspended. */
 	disable_all_screens_and_wait(&ms_data);
 
+	if (d3cold)
+		igt_assert(wait_for_d3cold(root));
+
 	gem_buf = __gem_mmap_offset(drm_fd, handle, 0, buf_size,
 				    PROT_READ | PROT_WRITE, t->type);
 	igt_require(gem_buf);
 
 	igt_assert(wait_for_suspended());
 
+	if (d3cold)
+		igt_assert(wait_for_d3cold(root));
+
 	for (i = 0; i < buf_size; i++)
 		gem_buf[i] = i & 0xFF;
 
@@ -1187,6 +1206,9 @@ static void gem_mmap_args(const struct mmap_offset *t,
 
 	igt_assert(wait_for_suspended());
 
+	if (d3cold)
+		igt_assert(wait_for_d3cold(root));
+
 	/* Resume and check if it's still there. */
 	enable_one_screen_and_wait(&ms_data);
 	for (i = 0; i < buf_size; i++)
@@ -2186,7 +2208,22 @@ igt_main_args("", long_options, help_str, opt_handler, NULL)
 		for_each_mmap_offset_type(drm_fd, t) {
 			for_each_memory_region(r, drm_fd) {
 				igt_dynamic_f("%s-%s", t->name, r->name)
-				gem_mmap_args(t, &r->ci);
+				gem_mmap_args(t, &r->ci, false);
+
+				if (r->ci.memory_class == I915_MEMORY_CLASS_SYSTEM)
+					continue;
+
+				igt_dynamic_f("%s-%s-%s", t->name, "d3cold-off", r->name) {
+					if (lmem_threshold >= 0)
+						set_d3cold_sr_lmem_threshold(debugfs, 300);
+					gem_mmap_args(t, &r->ci, true);
+				}
+
+				igt_dynamic_f("%s-%s-%s", t->name, "d3cold-vrsr", r->name) {
+					igt_require(lmem_threshold >= 0);
+					set_d3cold_sr_lmem_threshold(debugfs, 0);
+					gem_mmap_args(t, &r->ci, true);
+				}
 			}
 		}
 	}
-- 
2.26.2

[igt-dev] ✓ Fi.CI.BAT: success for D3Cold Tool & IGT (rev5)

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 3111 bytes --]

== Series Details ==

Series: D3Cold Tool & IGT (rev5)
URL   : https://patchwork.freedesktop.org/series/102780/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_11617 -> IGTPW_7055
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/index.html

Participating hosts (45 -> 44)
------------------------------

  Additional (2): bat-rpls-1 fi-snb-2520m 
  Missing    (3): bat-adlm-1 fi-bxt-dsi fi-bsw-cyan 

Known issues
------------

  Here are the changes found in IGTPW_7055 that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@gem_lmem_swapping@basic:
    - fi-snb-2520m:       NOTRUN -> [SKIP][1] ([fdo#109271]) +24 similar issues
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/fi-snb-2520m/igt@gem_lmem_swapping@basic.html

  * igt@kms_chamelium@hdmi-hpd-fast:
    - fi-snb-2520m:       NOTRUN -> [SKIP][2] ([fdo#109271] / [fdo#111827]) +8 similar issues
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/fi-snb-2520m/igt@kms_chamelium@hdmi-hpd-fast.html

  
#### Possible fixes ####

  * igt@i915_selftest@live@gem_contexts:
    - {bat-dg2-8}:        [DMESG-FAIL][3] ([i915#5885]) -> [PASS][4]
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11617/bat-dg2-8/igt@i915_selftest@live@gem_contexts.html
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/bat-dg2-8/igt@i915_selftest@live@gem_contexts.html

  
#### Warnings ####

  * igt@runner@aborted:
    - bat-dg1-6:          [FAIL][5] ([i915#4312] / [i915#5257]) -> [FAIL][6] ([i915#5616])
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11617/bat-dg1-6/igt@runner@aborted.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/bat-dg1-6/igt@runner@aborted.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
  [fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
  [i915#4312]: https://gitlab.freedesktop.org/drm/intel/issues/4312
  [i915#4897]: https://gitlab.freedesktop.org/drm/intel/issues/4897
  [i915#5257]: https://gitlab.freedesktop.org/drm/intel/issues/5257
  [i915#5616]: https://gitlab.freedesktop.org/drm/intel/issues/5616
  [i915#5879]: https://gitlab.freedesktop.org/drm/intel/issues/5879
  [i915#5885]: https://gitlab.freedesktop.org/drm/intel/issues/5885


Build changes
-------------

  * CI: CI-20190529 -> None
  * IGT: IGT_6468 -> IGTPW_7055

  CI-20190529: 20190529
  CI_DRM_11617: d96cea3d7ffb524248fcc8db433c579cf262eaea @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_7055: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/index.html
  IGT_6468: cffa5fffe9acddf49565b4caeeb5e3355ff2ea44 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git


Testlist changes
----------------

+igt@i915_pm_rpm@d3cold-basic

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/index.html

[-- Attachment #2: Type: text/html, Size: 3721 bytes --]

[igt-dev] ✓ Fi.CI.IGT: success for D3Cold Tool & IGT (rev5)

[Patchwork]

[-- Attachment #1: Type: text/plain, Size: 3267 bytes --]

== Series Details ==

Series: D3Cold Tool & IGT (rev5)
URL   : https://patchwork.freedesktop.org/series/102780/
State : success

== Summary ==

CI Bug Log - changes from CI_DRM_11617_full -> IGTPW_7055_full
====================================================

Summary
-------

  **SUCCESS**

  No regressions found.

  External URL: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/index.html

Participating hosts (11 -> 7)
------------------------------

  Missing    (4): pig-skl-6260u pig-kbl-iris shard-tglu pig-glk-j5005 

New tests
---------

  New tests have been introduced between CI_DRM_11617_full and IGTPW_7055_full:

### New IGT tests (1) ###

  * igt@i915_pm_rpm@d3cold-basic:
    - Statuses : 1 skip(s)
    - Exec time: [0.0] s

  

Known issues
------------

  Here are the changes found in IGTPW_7055_full that come from known issues:

### IGT changes ###

#### Issues hit ####

  * igt@gem_ctx_persistence@engines-cleanup:
    - shard-snb:          NOTRUN -> [SKIP][1] ([fdo#109271] / [i915#1099])
   [1]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/shard-snb2/igt@gem_ctx_persistence@engines-cleanup.html

  * igt@i915_hangman@engine-engine-hang:
    - shard-snb:          NOTRUN -> [SKIP][2] ([fdo#109271]) +109 similar issues
   [2]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/shard-snb4/igt@i915_hangman@engine-engine-hang.html

  * igt@kms_chamelium@hdmi-hpd-enable-disable-mode:
    - shard-snb:          NOTRUN -> [SKIP][3] ([fdo#109271] / [fdo#111827]) +4 similar issues
   [3]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/shard-snb6/igt@kms_chamelium@hdmi-hpd-enable-disable-mode.html

  * igt@syncobj_timeline@transfer-timeline-point:
    - shard-snb:          NOTRUN -> [DMESG-FAIL][4] ([i915#5098])
   [4]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/shard-snb6/igt@syncobj_timeline@transfer-timeline-point.html

  
#### Possible fixes ####

  * igt@gem_mmap_gtt@coherency:
    - shard-snb:          [SKIP][5] ([fdo#109271]) -> [PASS][6]
   [5]: https://intel-gfx-ci.01.org/tree/drm-tip/CI_DRM_11617/shard-snb6/igt@gem_mmap_gtt@coherency.html
   [6]: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/shard-snb5/igt@gem_mmap_gtt@coherency.html

  
  {name}: This element is suppressed. This means it is ignored when computing
          the status of the difference (SUCCESS, WARNING, or FAILURE).

  [fdo#109271]: https://bugs.freedesktop.org/show_bug.cgi?id=109271
  [fdo#111827]: https://bugs.freedesktop.org/show_bug.cgi?id=111827
  [i915#1099]: https://gitlab.freedesktop.org/drm/intel/issues/1099
  [i915#5098]: https://gitlab.freedesktop.org/drm/intel/issues/5098


Build changes
-------------

  * CI: CI-20190529 -> None
  * IGT: IGT_6468 -> IGTPW_7055
  * Piglit: piglit_4509 -> None

  CI-20190529: 20190529
  CI_DRM_11617: d96cea3d7ffb524248fcc8db433c579cf262eaea @ git://anongit.freedesktop.org/gfx-ci/linux
  IGTPW_7055: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/index.html
  IGT_6468: cffa5fffe9acddf49565b4caeeb5e3355ff2ea44 @ https://gitlab.freedesktop.org/drm/igt-gpu-tools.git
  piglit_4509: fdc5a4ca11124ab8413c7988896eec4c97336694 @ git://anongit.freedesktop.org/piglit

== Logs ==

For more details see: https://intel-gfx-ci.01.org/tree/drm-tip/IGTPW_7055/index.html

[-- Attachment #2: Type: text/html, Size: 4222 bytes --]