[PATCH 0/7] s390/dasd: add dasd autoquiesce feature

[PATCH 0/7] s390/dasd: add dasd autoquiesce feature

[Stefan Haberland]

Hi Jens,

please apply the following patchset that introduces an
autoquiesce feature for DASD devices.

Quiesce and resume are functions that tell Linux to stop/resume
issuing I/Os to a specific DASD.
The DASD driver allows a manual quiesce/resume via ioctl.

Autoquiesce will define an amount of triggers that will lead to
an automatic quiesce if a certain event occurs.
There is no automatic resume.

The autoquiesce feature is useful for devices in a copy relation
where different events should lead to an automatic quiesce of a
device. This prevents IO errors, unnecessary load and gives the
controlling agent time to sort out the situation and trigger a swap
of copy pairs if needed before it will resume device operation.

Stefan Haberland (7):
  s390/dasd: remove unused DASD EER defines
  s390/dasd: add autoquiesce feature
  s390/dasd: add aq_mask sysfs attribute
  s390/dasd: add aq_requeue sysfs attribute
  s390/dasd: add aq_timeouts autoquiesce trigger
  s390/dasd: add autoquiesce event for start IO error
  s390/dasd: fix hanging blockdevice after request requeue

 arch/s390/include/uapi/asm/dasd.h |   2 +
 drivers/s390/block/dasd.c         |  75 +++++++++++++-----
 drivers/s390/block/dasd_devmap.c  | 126 ++++++++++++++++++++++++++++++
 drivers/s390/block/dasd_eckd.c    |   1 +
 drivers/s390/block/dasd_eer.c     |   1 +
 drivers/s390/block/dasd_int.h     |  32 ++++----
 6 files changed, 204 insertions(+), 33 deletions(-)

-- 
2.37.2

[PATCH 1/7] s390/dasd: remove unused DASD EER defines

[Stefan Haberland]

Remove definitions that have never been used.

Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>
---
 drivers/s390/block/dasd_int.h | 12 +-----------
 1 file changed, 1 insertion(+), 11 deletions(-)

diff --git a/drivers/s390/block/dasd_int.h b/drivers/s390/block/dasd_int.h
index 97adc8a7ae6b..47cad999fc2a 100644
--- a/drivers/s390/block/dasd_int.h
+++ b/drivers/s390/block/dasd_int.h
@@ -444,17 +444,7 @@ struct dasd_discipline {
 
 extern struct dasd_discipline *dasd_diag_discipline_pointer;
 
-/*
- * Notification numbers for extended error reporting notifications:
- * The DASD_EER_DISABLE notification is sent before a dasd_device (and it's
- * eer pointer) is freed. The error reporting module needs to do all necessary
- * cleanup steps.
- * The DASD_EER_TRIGGER notification sends the actual error reports (triggers).
- */
-#define DASD_EER_DISABLE 0
-#define DASD_EER_TRIGGER 1
-
-/* Trigger IDs for extended error reporting DASD_EER_TRIGGER notification */
+/* Trigger IDs for extended error reporting */
 #define DASD_EER_FATALERROR  1
 #define DASD_EER_NOPATH      2
 #define DASD_EER_STATECHANGE 3
-- 
2.37.2

Re: [PATCH 1/7] s390/dasd: remove unused DASD EER defines

[Halil Pasic]

On Wed,  5 Apr 2023 16:20:11 +0200
Stefan Haberland <sth@linux.ibm.com> wrote:

> Remove definitions that have never been used.
> 
> Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
> Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>

Reviewed-by: Halil Pasic <pasic@linux.ibm.com>

[PATCH 2/7] s390/dasd: add autoquiesce feature

[Stefan Haberland]

Add the internal logic to check for autoquiesce triggers and handle
them.

Quiesce and resume are functions that tell Linux to stop/resume
issuing I/Os to a specific DASD.
The DASD driver allows a manual quiesce/resume via ioctl.

Autoquiesce will define an amount of triggers that will lead to
an automatic quiesce if a certain event occurs.
There is no automatic resume.

All events will be reported via DASD Extended Error Reporting (EER)
if configured.

Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>
---
 arch/s390/include/uapi/asm/dasd.h |  2 ++
 drivers/s390/block/dasd.c         | 60 ++++++++++++++++++++++---------
 drivers/s390/block/dasd_eer.c     |  1 +
 drivers/s390/block/dasd_int.h     |  2 ++
 4 files changed, 48 insertions(+), 17 deletions(-)

diff --git a/arch/s390/include/uapi/asm/dasd.h b/arch/s390/include/uapi/asm/dasd.h
index 93d1ccd3304c..9c49c3d67cd5 100644
--- a/arch/s390/include/uapi/asm/dasd.h
+++ b/arch/s390/include/uapi/asm/dasd.h
@@ -78,6 +78,7 @@ typedef struct dasd_information2_t {
  * 0x040: give access to raw eckd data
  * 0x080: enable discard support
  * 0x100: enable autodisable for IFCC errors (default)
+ * 0x200: enable requeue of all requests on autoquiesce
  */
 #define DASD_FEATURE_READONLY	      0x001
 #define DASD_FEATURE_USEDIAG	      0x002
@@ -88,6 +89,7 @@ typedef struct dasd_information2_t {
 #define DASD_FEATURE_USERAW	      0x040
 #define DASD_FEATURE_DISCARD	      0x080
 #define DASD_FEATURE_PATH_AUTODISABLE 0x100
+#define DASD_FEATURE_REQUEUEQUIESCE   0x200
 #define DASD_FEATURE_DEFAULT	      DASD_FEATURE_PATH_AUTODISABLE
 
 #define DASD_PARTN_BITS 2
diff --git a/drivers/s390/block/dasd.c b/drivers/s390/block/dasd.c
index a9c2a8d76c45..8186473b9aa7 100644
--- a/drivers/s390/block/dasd.c
+++ b/drivers/s390/block/dasd.c
@@ -73,7 +73,8 @@ static void dasd_profile_init(struct dasd_profile *, struct dentry *);
 static void dasd_profile_exit(struct dasd_profile *);
 static void dasd_hosts_init(struct dentry *, struct dasd_device *);
 static void dasd_hosts_exit(struct dasd_device *);
-
+static int dasd_handle_autoquiesce(struct dasd_device *, struct dasd_ccw_req *,
+				   unsigned int);
 /*
  * SECTION: Operations on the device structure.
  */
@@ -2325,7 +2326,7 @@ static int _dasd_sleep_on(struct dasd_ccw_req *maincqr, int interruptible)
 		/* Non-temporary stop condition will trigger fail fast */
 		if (device->stopped & ~DASD_STOPPED_PENDING &&
 		    test_bit(DASD_CQR_FLAGS_FAILFAST, &cqr->flags) &&
-		    (!dasd_eer_enabled(device))) {
+		    !dasd_eer_enabled(device) && device->aq_mask == 0) {
 			cqr->status = DASD_CQR_FAILED;
 			cqr->intrc = -ENOLINK;
 			continue;
@@ -2801,20 +2802,18 @@ static void __dasd_process_block_ccw_queue(struct dasd_block *block,
 			dasd_log_sense(cqr, &cqr->irb);
 		}
 
-		/* First of all call extended error reporting. */
-		if (dasd_eer_enabled(base) &&
-		    cqr->status == DASD_CQR_FAILED) {
-			dasd_eer_write(base, cqr, DASD_EER_FATALERROR);
-
-			/* restart request  */
+		/*
+		 * First call extended error reporting and check for autoquiesce
+		 */
+		spin_lock_irqsave(get_ccwdev_lock(base->cdev), flags);
+		if (cqr->status == DASD_CQR_FAILED &&
+		    dasd_handle_autoquiesce(base, cqr, DASD_EER_FATALERROR)) {
 			cqr->status = DASD_CQR_FILLED;
 			cqr->retries = 255;
-			spin_lock_irqsave(get_ccwdev_lock(base->cdev), flags);
-			dasd_device_set_stop_bits(base, DASD_STOPPED_QUIESCE);
-			spin_unlock_irqrestore(get_ccwdev_lock(base->cdev),
-					       flags);
+			spin_unlock_irqrestore(get_ccwdev_lock(base->cdev), flags);
 			goto restart;
 		}
+		spin_unlock_irqrestore(get_ccwdev_lock(base->cdev), flags);
 
 		/* Process finished ERP request. */
 		if (cqr->refers) {
@@ -2856,7 +2855,7 @@ static void __dasd_block_start_head(struct dasd_block *block)
 		/* Non-temporary stop condition will trigger fail fast */
 		if (block->base->stopped & ~DASD_STOPPED_PENDING &&
 		    test_bit(DASD_CQR_FLAGS_FAILFAST, &cqr->flags) &&
-		    (!dasd_eer_enabled(block->base))) {
+		    !dasd_eer_enabled(block->base) && block->base->aq_mask == 0) {
 			cqr->status = DASD_CQR_FAILED;
 			cqr->intrc = -ENOLINK;
 			dasd_schedule_block_bh(block);
@@ -3670,8 +3669,8 @@ int dasd_generic_last_path_gone(struct dasd_device *device)
 	dev_warn(&device->cdev->dev, "No operational channel path is left "
 		 "for the device\n");
 	DBF_DEV_EVENT(DBF_WARNING, device, "%s", "last path gone");
-	/* First of all call extended error reporting. */
-	dasd_eer_write(device, NULL, DASD_EER_NOPATH);
+	/* First call extended error reporting and check for autoquiesce. */
+	dasd_handle_autoquiesce(device, NULL, DASD_EER_NOPATH);
 
 	if (device->state < DASD_STATE_BASIC)
 		return 0;
@@ -3803,7 +3802,8 @@ void dasd_generic_path_event(struct ccw_device *cdev, int *path_event)
 			 "No verified channel paths remain for the device\n");
 		DBF_DEV_EVENT(DBF_WARNING, device,
 			      "%s", "last verified path gone");
-		dasd_eer_write(device, NULL, DASD_EER_NOPATH);
+		/* First call extended error reporting and check for autoquiesce. */
+		dasd_handle_autoquiesce(device, NULL, DASD_EER_NOPATH);
 		dasd_device_set_stop_bits(device,
 					  DASD_STOPPED_DC_WAIT);
 	}
@@ -3825,7 +3825,8 @@ EXPORT_SYMBOL_GPL(dasd_generic_verify_path);
 void dasd_generic_space_exhaust(struct dasd_device *device,
 				struct dasd_ccw_req *cqr)
 {
-	dasd_eer_write(device, NULL, DASD_EER_NOSPC);
+	/* First call extended error reporting and check for autoquiesce. */
+	dasd_handle_autoquiesce(device, NULL, DASD_EER_NOSPC);
 
 	if (device->state < DASD_STATE_BASIC)
 		return;
@@ -3958,6 +3959,31 @@ void dasd_schedule_requeue(struct dasd_device *device)
 }
 EXPORT_SYMBOL(dasd_schedule_requeue);
 
+static int dasd_handle_autoquiesce(struct dasd_device *device,
+				   struct dasd_ccw_req *cqr,
+				   unsigned int reason)
+{
+	/* in any case write eer message with reason */
+	if (dasd_eer_enabled(device))
+		dasd_eer_write(device, cqr, reason);
+
+	if (!test_bit(reason, &device->aq_mask))
+		return 0;
+
+	/* notify eer about autoquiesce */
+	if (dasd_eer_enabled(device))
+		dasd_eer_write(device, NULL, DASD_EER_AUTOQUIESCE);
+
+	pr_info("%s: The DASD has been put in the quiesce state\n",
+		dev_name(&device->cdev->dev));
+	dasd_device_set_stop_bits(device, DASD_STOPPED_QUIESCE);
+
+	if (device->features & DASD_FEATURE_REQUEUEQUIESCE)
+		dasd_schedule_requeue(device);
+
+	return 1;
+}
+
 static struct dasd_ccw_req *dasd_generic_build_rdc(struct dasd_device *device,
 						   int rdc_buffer_size,
 						   int magic)
diff --git a/drivers/s390/block/dasd_eer.c b/drivers/s390/block/dasd_eer.c
index a4cc772208a6..c956de711cf7 100644
--- a/drivers/s390/block/dasd_eer.c
+++ b/drivers/s390/block/dasd_eer.c
@@ -387,6 +387,7 @@ void dasd_eer_write(struct dasd_device *device, struct dasd_ccw_req *cqr,
 		break;
 	case DASD_EER_NOPATH:
 	case DASD_EER_NOSPC:
+	case DASD_EER_AUTOQUIESCE:
 		dasd_eer_write_standard_trigger(device, NULL, id);
 		break;
 	case DASD_EER_STATECHANGE:
diff --git a/drivers/s390/block/dasd_int.h b/drivers/s390/block/dasd_int.h
index 47cad999fc2a..d0ff4528d489 100644
--- a/drivers/s390/block/dasd_int.h
+++ b/drivers/s390/block/dasd_int.h
@@ -450,6 +450,7 @@ extern struct dasd_discipline *dasd_diag_discipline_pointer;
 #define DASD_EER_STATECHANGE 3
 #define DASD_EER_PPRCSUSPEND 4
 #define DASD_EER_NOSPC	     5
+#define DASD_EER_AUTOQUIESCE 31
 
 /* DASD path handling */
 
@@ -627,6 +628,7 @@ struct dasd_device {
 	struct dasd_format_entry format_entry;
 	struct kset *paths_info;
 	struct dasd_copy_relation *copy;
+	unsigned long aq_mask;
 };
 
 struct dasd_block {
-- 
2.37.2

Re: [PATCH 2/7] s390/dasd: add autoquiesce feature

[Halil Pasic]

On Wed,  5 Apr 2023 16:20:12 +0200
Stefan Haberland <sth@linux.ibm.com> wrote:

> Add the internal logic to check for autoquiesce triggers and handle
> them.
> 
> Quiesce and resume are functions that tell Linux to stop/resume
> issuing I/Os to a specific DASD.
> The DASD driver allows a manual quiesce/resume via ioctl.
> 
> Autoquiesce will define an amount of triggers that will lead to
> an automatic quiesce if a certain event occurs.
> There is no automatic resume.
> 
> All events will be reported via DASD Extended Error Reporting (EER)
> if configured.
> 
> Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
> Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>

Reviewed-by: Halil Pasic <pasic@linux.ibm.com>

[..]

[PATCH 3/7] s390/dasd: add aq_mask sysfs attribute

[Stefan Haberland]

Add sysfs attribute that controls the DASD autoquiesce feature.
The autoquiesce is disabled when 0 is echoed to the attribute.

A value greater than 0 will enable the feature.
The aq_mask attribute will accept an unsigned integer and the value
will be interpreted as bitmask defining the trigger events that will
lead to an automatic quiesce.

The following autoquiesce triggers will currently be available:

DASD_EER_FATALERROR  1 - any final I/O error
DASD_EER_NOPATH      2 - no remaining paths for the device
DASD_EER_STATECHANGE 3 - a state change interrupt occurred
DASD_EER_PPRCSUSPEND 4 - the device is PPRC suspended
DASD_EER_NOSPC       5 - there is no space remaining on an ESE device
DASD_EER_TIMEOUT     6 - a certain amount of timeouts occurred
DASD_EER_STARTIO     7 - the IO start function encountered an error

The currently supported maximum value is 255.

Bit 31 is reserved for internal usage.
Bit 0 is not used.

Example:

 - deactivate autoquiesce
   $ echo 0 > /sys/bus/ccw/0.0.1234/aq_mask

 - enable autoquiesce for FATALERROR, NOPATH and TIMEOUT
   (0000 0000 0000 0000  0000 0000 0100 0110 => 70)
   $ echo 70 > /sys/bus/ccw/0.0.1234/aq_mask

Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>
---
 drivers/s390/block/dasd_devmap.c | 43 ++++++++++++++++++++++++++++++++
 drivers/s390/block/dasd_int.h    | 23 +++++++++++------
 2 files changed, 59 insertions(+), 7 deletions(-)

diff --git a/drivers/s390/block/dasd_devmap.c b/drivers/s390/block/dasd_devmap.c
index df17f0f9cb0f..c7c948c55761 100644
--- a/drivers/s390/block/dasd_devmap.c
+++ b/drivers/s390/block/dasd_devmap.c
@@ -50,6 +50,7 @@ struct dasd_devmap {
         unsigned short features;
 	struct dasd_device *device;
 	struct dasd_copy_relation *copy;
+	unsigned int aq_mask;
 };
 
 /*
@@ -1475,6 +1476,47 @@ dasd_eer_store(struct device *dev, struct device_attribute *attr,
 
 static DEVICE_ATTR(eer_enabled, 0644, dasd_eer_show, dasd_eer_store);
 
+/*
+ * aq_mask controls if the DASD should be quiesced on certain triggers
+ * The aq_mask attribute is interpreted as bitmap of the DASD_EER_* triggers.
+ */
+static ssize_t dasd_aq_mask_show(struct device *dev, struct device_attribute *attr,
+				 char *buf)
+{
+	struct dasd_devmap *devmap;
+	unsigned int aq_mask = 0;
+
+	devmap = dasd_find_busid(dev_name(dev));
+	if (!IS_ERR(devmap))
+		aq_mask = devmap->aq_mask;
+
+	return sysfs_emit(buf, "%d\n", aq_mask);
+}
+
+static ssize_t dasd_aq_mask_store(struct device *dev, struct device_attribute *attr,
+				  const char *buf, size_t count)
+{
+	struct dasd_devmap *devmap;
+	unsigned int val;
+
+	if (kstrtouint(buf, 0, &val) || val > DASD_EER_VALID)
+		return -EINVAL;
+
+	devmap = dasd_devmap_from_cdev(to_ccwdev(dev));
+	if (IS_ERR(devmap))
+		return PTR_ERR(devmap);
+
+	spin_lock(&dasd_devmap_lock);
+	devmap->aq_mask = val;
+	if (devmap->device)
+		devmap->device->aq_mask = devmap->aq_mask;
+	spin_unlock(&dasd_devmap_lock);
+
+	return count;
+}
+
+static DEVICE_ATTR(aq_mask, 0644, dasd_aq_mask_show, dasd_aq_mask_store);
+
 /*
  * expiration time for default requests
  */
@@ -2324,6 +2366,7 @@ static struct attribute * dasd_attrs[] = {
 	&dev_attr_copy_pair.attr,
 	&dev_attr_copy_role.attr,
 	&dev_attr_ping.attr,
+	&dev_attr_aq_mask.attr,
 	NULL,
 };
 
diff --git a/drivers/s390/block/dasd_int.h b/drivers/s390/block/dasd_int.h
index d0ff4528d489..c4633a4aeeb1 100644
--- a/drivers/s390/block/dasd_int.h
+++ b/drivers/s390/block/dasd_int.h
@@ -444,13 +444,22 @@ struct dasd_discipline {
 
 extern struct dasd_discipline *dasd_diag_discipline_pointer;
 
-/* Trigger IDs for extended error reporting */
-#define DASD_EER_FATALERROR  1
-#define DASD_EER_NOPATH      2
-#define DASD_EER_STATECHANGE 3
-#define DASD_EER_PPRCSUSPEND 4
-#define DASD_EER_NOSPC	     5
-#define DASD_EER_AUTOQUIESCE 31
+/* Trigger IDs for extended error reporting DASD EER and autoquiesce */
+enum eer_trigger {
+	DASD_EER_FATALERROR = 1,
+	DASD_EER_NOPATH,
+	DASD_EER_STATECHANGE,
+	DASD_EER_PPRCSUSPEND,
+	DASD_EER_NOSPC,
+	DASD_EER_TIMEOUTS,
+	DASD_EER_STARTIO,
+
+	/* enum end marker, only add new trigger above */
+	DASD_EER_MAX,
+	DASD_EER_AUTOQUIESCE = 31, /* internal only */
+};
+
+#define DASD_EER_VALID ((1U << DASD_EER_MAX) - 1)
 
 /* DASD path handling */
 
-- 
2.37.2

Re: [PATCH 3/7] s390/dasd: add aq_mask sysfs attribute

[Halil Pasic]

On Wed,  5 Apr 2023 16:20:13 +0200
Stefan Haberland <sth@linux.ibm.com> wrote:

> Add sysfs attribute that controls the DASD autoquiesce feature.
> The autoquiesce is disabled when 0 is echoed to the attribute.
> 
> A value greater than 0 will enable the feature.
> The aq_mask attribute will accept an unsigned integer and the value
> will be interpreted as bitmask defining the trigger events that will
> lead to an automatic quiesce.
> 
> The following autoquiesce triggers will currently be available:
> 
> DASD_EER_FATALERROR  1 - any final I/O error
> DASD_EER_NOPATH      2 - no remaining paths for the device
> DASD_EER_STATECHANGE 3 - a state change interrupt occurred
> DASD_EER_PPRCSUSPEND 4 - the device is PPRC suspended
> DASD_EER_NOSPC       5 - there is no space remaining on an ESE device
> DASD_EER_TIMEOUT     6 - a certain amount of timeouts occurred
> DASD_EER_STARTIO     7 - the IO start function encountered an error
> 
> The currently supported maximum value is 255.
> 
> Bit 31 is reserved for internal usage.
> Bit 0 is not used.
> 
> Example:
> 
>  - deactivate autoquiesce
>    $ echo 0 > /sys/bus/ccw/0.0.1234/aq_mask
> 
>  - enable autoquiesce for FATALERROR, NOPATH and TIMEOUT
>    (0000 0000 0000 0000  0000 0000 0100 0110 => 70)
>    $ echo 70 > /sys/bus/ccw/0.0.1234/aq_mask
> 
> Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
> Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>

Reviewed-by: Halil Pasic <pasic@linux.ibm.com>

[..]

[PATCH 4/7] s390/dasd: add aq_requeue sysfs attribute

[Stefan Haberland]

Add a sysfs attribute to control if all IO requests will be requeued to
the blocklayer in case of an autoquiesce event or not.

A value of 1 means that in case of an autoquiesce event all IO requests
will be requeued to the blocklayer.

A value of 0 means that the device will only be stopped.

Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>
---
 drivers/s390/block/dasd_devmap.c | 36 ++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/drivers/s390/block/dasd_devmap.c b/drivers/s390/block/dasd_devmap.c
index c7c948c55761..95c7959c7949 100644
--- a/drivers/s390/block/dasd_devmap.c
+++ b/drivers/s390/block/dasd_devmap.c
@@ -1517,6 +1517,41 @@ static ssize_t dasd_aq_mask_store(struct device *dev, struct device_attribute *a
 
 static DEVICE_ATTR(aq_mask, 0644, dasd_aq_mask_show, dasd_aq_mask_store);
 
+/*
+ * aq_requeue controls if requests are returned to the blocklayer on quiesce
+ * or if requests are only not started
+ */
+static ssize_t dasd_aqr_show(struct device *dev, struct device_attribute *attr,
+			     char *buf)
+{
+	struct dasd_devmap *devmap;
+	int flag;
+
+	devmap = dasd_find_busid(dev_name(dev));
+	if (!IS_ERR(devmap))
+		flag = (devmap->features & DASD_FEATURE_REQUEUEQUIESCE) != 0;
+	else
+		flag = (DASD_FEATURE_DEFAULT &
+			DASD_FEATURE_REQUEUEQUIESCE) != 0;
+	return sysfs_emit(buf, "%d\n", flag);
+}
+
+static ssize_t dasd_aqr_store(struct device *dev, struct device_attribute *attr,
+			      const char *buf, size_t count)
+{
+	bool val;
+	int rc;
+
+	if (kstrtobool(buf, &val))
+		return -EINVAL;
+
+	rc = dasd_set_feature(to_ccwdev(dev), DASD_FEATURE_REQUEUEQUIESCE, val);
+
+	return rc ? : count;
+}
+
+static DEVICE_ATTR(aq_requeue, 0644, dasd_aqr_show, dasd_aqr_store);
+
 /*
  * expiration time for default requests
  */
@@ -2367,6 +2402,7 @@ static struct attribute * dasd_attrs[] = {
 	&dev_attr_copy_role.attr,
 	&dev_attr_ping.attr,
 	&dev_attr_aq_mask.attr,
+	&dev_attr_aq_requeue.attr,
 	NULL,
 };
 
-- 
2.37.2

Re: [PATCH 4/7] s390/dasd: add aq_requeue sysfs attribute

[Halil Pasic]

On Wed,  5 Apr 2023 16:20:14 +0200
Stefan Haberland <sth@linux.ibm.com> wrote:

> Add a sysfs attribute to control if all IO requests will be requeued to
> the blocklayer in case of an autoquiesce event or not.
> 
> A value of 1 means that in case of an autoquiesce event all IO requests
> will be requeued to the blocklayer.
> 
> A value of 0 means that the device will only be stopped.
> 
> Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
> Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>

Reviewed-by: Halil Pasic <pasic@linux.ibm.com>

[..]

[PATCH 5/7] s390/dasd: add aq_timeouts autoquiesce trigger

[Stefan Haberland]

Add a sysfs attribute aq_timeouts that controls after how many
timeouts a autoquiesce event might be triggered.

The default value is 32768 which is the maximum number of retries
for the DASD device driver DASD_RETRIES_MAX. This means that the
timeout trigger will never happen.

The default value for DASD retries is 255.
Setting the value to below 255 will trigger the timeout autoquiesce
event before an IO error is generated.

Also add the check for the configured amount of timeouts and trigger
an autoquiesce event if exceeded.

Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>
---
 drivers/s390/block/dasd.c        | 11 ++++++++
 drivers/s390/block/dasd_devmap.c | 47 ++++++++++++++++++++++++++++++++
 drivers/s390/block/dasd_eckd.c   |  1 +
 drivers/s390/block/dasd_int.h    |  1 +
 4 files changed, 60 insertions(+)

diff --git a/drivers/s390/block/dasd.c b/drivers/s390/block/dasd.c
index 8186473b9aa7..1bfd0e17a4dc 100644
--- a/drivers/s390/block/dasd.c
+++ b/drivers/s390/block/dasd.c
@@ -1953,6 +1953,16 @@ static void __dasd_device_process_final_queue(struct dasd_device *device,
 	}
 }
 
+/*
+ * check if device should be autoquiesced due to too many timeouts
+ */
+static void __dasd_device_check_autoquiesce_timeout(struct dasd_device *device,
+						    struct dasd_ccw_req *cqr)
+{
+	if ((device->default_retries - cqr->retries) >= device->aq_timeouts)
+		dasd_handle_autoquiesce(device, cqr, DASD_EER_TIMEOUTS);
+}
+
 /*
  * Take a look at the first request on the ccw queue and check
  * if it reached its expire time. If so, terminate the IO.
@@ -1987,6 +1997,7 @@ static void __dasd_device_check_expire(struct dasd_device *device)
 				"remaining\n", cqr, (cqr->expires/HZ),
 				cqr->retries);
 		}
+		__dasd_device_check_autoquiesce_timeout(device, cqr);
 	}
 }
 
diff --git a/drivers/s390/block/dasd_devmap.c b/drivers/s390/block/dasd_devmap.c
index 95c7959c7949..620fab01b710 100644
--- a/drivers/s390/block/dasd_devmap.c
+++ b/drivers/s390/block/dasd_devmap.c
@@ -1552,6 +1552,52 @@ static ssize_t dasd_aqr_store(struct device *dev, struct device_attribute *attr,
 
 static DEVICE_ATTR(aq_requeue, 0644, dasd_aqr_show, dasd_aqr_store);
 
+/*
+ * aq_timeouts controls how much retries have to time out until
+ * a device gets autoquiesced
+ */
+static ssize_t
+dasd_aq_timeouts_show(struct device *dev, struct device_attribute *attr,
+		      char *buf)
+{
+	struct dasd_device *device;
+	int len;
+
+	device = dasd_device_from_cdev(to_ccwdev(dev));
+	if (IS_ERR(device))
+		return -ENODEV;
+	len = sysfs_emit(buf, "%u\n", device->aq_timeouts);
+	dasd_put_device(device);
+	return len;
+}
+
+static ssize_t
+dasd_aq_timeouts_store(struct device *dev, struct device_attribute *attr,
+		       const char *buf, size_t count)
+{
+	struct dasd_device *device;
+	unsigned int val;
+
+	device = dasd_device_from_cdev(to_ccwdev(dev));
+	if (IS_ERR(device))
+		return -ENODEV;
+
+	if ((kstrtouint(buf, 10, &val) != 0) ||
+	    val > DASD_RETRIES_MAX || val == 0) {
+		dasd_put_device(device);
+		return -EINVAL;
+	}
+
+	if (val)
+		device->aq_timeouts = val;
+
+	dasd_put_device(device);
+	return count;
+}
+
+static DEVICE_ATTR(aq_timeouts, 0644, dasd_aq_timeouts_show,
+		   dasd_aq_timeouts_store);
+
 /*
  * expiration time for default requests
  */
@@ -2403,6 +2449,7 @@ static struct attribute * dasd_attrs[] = {
 	&dev_attr_ping.attr,
 	&dev_attr_aq_mask.attr,
 	&dev_attr_aq_requeue.attr,
+	&dev_attr_aq_timeouts.attr,
 	NULL,
 };
 
diff --git a/drivers/s390/block/dasd_eckd.c b/drivers/s390/block/dasd_eckd.c
index 1a69f97e88fb..ade1369fe5ed 100644
--- a/drivers/s390/block/dasd_eckd.c
+++ b/drivers/s390/block/dasd_eckd.c
@@ -2109,6 +2109,7 @@ dasd_eckd_check_characteristics(struct dasd_device *device)
 	device->default_retries = DASD_RETRIES;
 	device->path_thrhld = DASD_ECKD_PATH_THRHLD;
 	device->path_interval = DASD_ECKD_PATH_INTERVAL;
+	device->aq_timeouts = DASD_RETRIES_MAX;
 
 	if (private->conf.gneq) {
 		value = 1;
diff --git a/drivers/s390/block/dasd_int.h b/drivers/s390/block/dasd_int.h
index c4633a4aeeb1..33f812f0e515 100644
--- a/drivers/s390/block/dasd_int.h
+++ b/drivers/s390/block/dasd_int.h
@@ -638,6 +638,7 @@ struct dasd_device {
 	struct kset *paths_info;
 	struct dasd_copy_relation *copy;
 	unsigned long aq_mask;
+	unsigned int aq_timeouts;
 };
 
 struct dasd_block {
-- 
2.37.2

Re: [PATCH 5/7] s390/dasd: add aq_timeouts autoquiesce trigger

[Halil Pasic]

On Wed,  5 Apr 2023 16:20:15 +0200
Stefan Haberland <sth@linux.ibm.com> wrote:

> Add a sysfs attribute aq_timeouts that controls after how many
> timeouts a autoquiesce event might be triggered.
> 
> The default value is 32768 which is the maximum number of retries
> for the DASD device driver DASD_RETRIES_MAX. This means that the
> timeout trigger will never happen.
> 
> The default value for DASD retries is 255.
> Setting the value to below 255 will trigger the timeout autoquiesce
> event before an IO error is generated.
> 
> Also add the check for the configured amount of timeouts and trigger
> an autoquiesce event if exceeded.
> 
> Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
> Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>

Reviewed-by: Halil Pasic <pasic@linux.ibm.com>

[..]

[PATCH 6/7] s390/dasd: add autoquiesce event for start IO error

[Stefan Haberland]

Add a check for errors in the start_io function that signal a not
working device. Trigger an autoquiesce event in that case.

Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>
---
 drivers/s390/block/dasd.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/s390/block/dasd.c b/drivers/s390/block/dasd.c
index 1bfd0e17a4dc..3696931f8015 100644
--- a/drivers/s390/block/dasd.c
+++ b/drivers/s390/block/dasd.c
@@ -1452,6 +1452,8 @@ int dasd_start_IO(struct dasd_ccw_req *cqr)
 	case -ENODEV:
 		DBF_DEV_EVENT(DBF_WARNING, device, "%s",
 			      "start_IO: -ENODEV device gone, retry");
+		/* this is equivalent to CC=3 for SSCH report this to EER */
+		dasd_handle_autoquiesce(device, cqr, DASD_EER_STARTIO);
 		break;
 	case -EIO:
 		DBF_DEV_EVENT(DBF_WARNING, device, "%s",
-- 
2.37.2

Re: [PATCH 6/7] s390/dasd: add autoquiesce event for start IO error

[Halil Pasic]

On Wed,  5 Apr 2023 16:20:16 +0200
Stefan Haberland <sth@linux.ibm.com> wrote:

> Add a check for errors in the start_io function that signal a not
> working device. Trigger an autoquiesce event in that case.
> 
> Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
> Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>

Reviewed-by: Halil Pasic <pasic@linux.ibm.com>

[..]

[PATCH 7/7] s390/dasd: fix hanging blockdevice after request requeue

[Stefan Haberland]

The DASD driver does not kick the requeue list when requeuing IO requests
to the blocklayer. This might lead to hanging blockdevice when there is
no other trigger for this.

Fix by automatically kick the requeue list when requeuing DASD requests
to the blocklayer.

Fixes: e443343e509a ("s390/dasd: blk-mq conversion")
CC: stable@vger.kernel.org # 4.14+
Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>
---
 drivers/s390/block/dasd.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/s390/block/dasd.c b/drivers/s390/block/dasd.c
index 3696931f8015..9fbfce735d56 100644
--- a/drivers/s390/block/dasd.c
+++ b/drivers/s390/block/dasd.c
@@ -2953,7 +2953,7 @@ static int _dasd_requeue_request(struct dasd_ccw_req *cqr)
 		return 0;
 	spin_lock_irq(&cqr->dq->lock);
 	req = (struct request *) cqr->callback_data;
-	blk_mq_requeue_request(req, false);
+	blk_mq_requeue_request(req, true);
 	spin_unlock_irq(&cqr->dq->lock);
 
 	return 0;
-- 
2.37.2

Re: [PATCH 7/7] s390/dasd: fix hanging blockdevice after request requeue

[Halil Pasic]

On Wed,  5 Apr 2023 16:20:17 +0200
Stefan Haberland <sth@linux.ibm.com> wrote:

> The DASD driver does not kick the requeue list when requeuing IO requests
> to the blocklayer. This might lead to hanging blockdevice when there is
> no other trigger for this.
> 
> Fix by automatically kick the requeue list when requeuing DASD requests
> to the blocklayer.
> 
> Fixes: e443343e509a ("s390/dasd: blk-mq conversion")
> CC: stable@vger.kernel.org # 4.14+
> Signed-off-by: Stefan Haberland <sth@linux.ibm.com>
> Reviewed-by: Jan Hoeppner <hoeppner@linux.ibm.com>

Reviewed-by: Halil Pasic <pasic@linux.ibm.com>

[..]

Re: [PATCH 0/7] s390/dasd: add dasd autoquiesce feature

[Jens Axboe]

On Wed, 05 Apr 2023 16:20:10 +0200, Stefan Haberland wrote:
> please apply the following patchset that introduces an
> autoquiesce feature for DASD devices.
> 
> Quiesce and resume are functions that tell Linux to stop/resume
> issuing I/Os to a specific DASD.
> The DASD driver allows a manual quiesce/resume via ioctl.
> 
> [...]

Applied, thanks!

[1/7] s390/dasd: remove unused DASD EER defines
      commit: 861d53dbed4cad8cf1bbef692111f2215e02c38e
[2/7] s390/dasd: add autoquiesce feature
      commit: 1cee2975bbabd89df1097c354867192106b058ea
[3/7] s390/dasd: add aq_mask sysfs attribute
      commit: 9558a8e9d4a681e67b3abe9cabf3e3d8825af57e
[4/7] s390/dasd: add aq_requeue sysfs attribute
      commit: bdac94e29564bab9f24c2700f16ff11f31af7c11
[5/7] s390/dasd: add aq_timeouts autoquiesce trigger
      commit: 0c1a14748133024a33aa8ffd763ca7f5c03bb27e
[6/7] s390/dasd: add autoquiesce event for start IO error
      commit: d9ee2bee4a63844cd9d1e0d00b1e3c49eacd1c2f
[7/7] s390/dasd: fix hanging blockdevice after request requeue
      commit: d8898ee50edecacdf0141f26fd90acf43d7e9cd7

Best regards,
-- 
Jens Axboe