Re: [SCSI][REGRESSION][BISECTED] Disk errors loop forever in 2.6.29

Re: [SCSI][REGRESSION][BISECTED] Disk errors loop forever in 2.6.29

[Sitsofe Wheeler]

> From: Alan Stern <stern@rowland.harvard.edu>
>
> On Thu, 19 Feb 2009, Sitsofe Wheeler wrote:
> 
> > Hi,
> > 
> > There appears to be a regression from 2.6.28 in how disk errors are
> > handled in 2.6.29rc5 - rather than trying and eventually giving up, it
> > appears to try (and report) forever. 
> 
> See this thread and patch:
> 
>     http://marc.info/?l=linux-kernel&m=123490148422684&w=2

The patch there (actually I downloaded it from http://patchwork.kernel.org/patch/7989/ )
did not make any diference. I fear my disk will soon have torn itself to bits but until then I
can trigger the error at will so I can test any patches that are suggested...



      

Re: [SCSI][REGRESSION][BISECTED] Disk errors loop forever in 2.6.29

[James Bottomley]

On Thu, 2009-02-19 at 08:52 -0800, Sitsofe Wheeler wrote:
> > From: Alan Stern <stern@rowland.harvard.edu>
> >
> > On Thu, 19 Feb 2009, Sitsofe Wheeler wrote:
> > 
> > > Hi,
> > > 
> > > There appears to be a regression from 2.6.28 in how disk errors are
> > > handled in 2.6.29rc5 - rather than trying and eventually giving up, it
> > > appears to try (and report) forever. 
> > 
> > See this thread and patch:
> > 
> >     http://marc.info/?l=linux-kernel&m=123490148422684&w=2
> 
> The patch there (actually I downloaded it from http://patchwork.kernel.org/patch/7989/ )
> did not make any diference. I fear my disk will soon have torn itself to bits but until then I
> can trigger the error at will so I can test any patches that are suggested...

Can you try this patch ... it was something I meant to get into 2.6.29
but forgot about.  The key problem that you seem to be hitting is that
the requeue evades the timeout check.  Moving the timeout check to block
should fix that.

James

---

>From 5546538f37a1f4319ec4dbdb6f2e7261ce986e61 Mon Sep 17 00:00:00 2001
From: James Bottomley <James.Bottomley@HansenPartnership.com>
Date: Tue, 16 Dec 2008 17:00:44 -0500
Subject: block: move SCSI timeout check into block

We can eliminate the SCSI command timed out check entirely if the block
layer does this for us.  The way to do this in block is to check how
long the request has been outstanding if a requeue is requested and
ending it if we've gone over retries * timeout.

This will also eliminate many cases in SCSI where we evade the command
timeout for various reasons (like initial success converted to requeue)

Signed-off-by: James Bottomley <James.Bottomley@HansenPartnership.com>
---
 block/blk-core.c |   10 +++++++++-
 1 files changed, 9 insertions(+), 1 deletions(-)

diff --git a/block/blk-core.c b/block/blk-core.c
index 29bcfac..3928ec8 100644
--- a/block/blk-core.c
+++ b/block/blk-core.c
@@ -937,6 +937,8 @@ EXPORT_SYMBOL(blk_start_queueing);
  */
 void blk_requeue_request(struct request_queue *q, struct request *rq)
 {
+	unsigned long wait_for = (rq->retries + 1) * rq->timeout;
+
 	blk_delete_timer(rq);
 	blk_clear_rq_complete(rq);
 	trace_block_rq_requeue(q, rq);
@@ -944,7 +946,13 @@ void blk_requeue_request(struct request_queue *q, struct request *rq)
 	if (blk_rq_tagged(rq))
 		blk_queue_end_tag(q, rq);
 
-	elv_requeue_request(q, rq);
+	if (time_before(rq->start_time + wait_for, jiffies)) {
+		printk(KERN_ERR "%s: timing out command, waited %lus\n",
+		       rq->rq_disk ? rq->rq_disk->disk_name : "?",
+		       wait_for/HZ);
+		blk_end_request(rq, -EIO, blk_rq_bytes(rq));
+	} else
+		elv_requeue_request(q, rq);
 }
 EXPORT_SYMBOL(blk_requeue_request);
 
-- 
1.5.6.6

Re: [SCSI][REGRESSION][BISECTED] Disk errors loop forever in 2.6.29

[Sitsofe Wheeler]

> From: James Bottomley <James.Bottomley@HansenPartnership.com>
>
> Can you try this patch ... it was something I meant to get into 2.6.29
> but forgot about.  The key problem that you seem to be hitting is that
> the requeue evades the timeout check.  Moving the timeout check to block
> should fix that.

Even by itself this patch works although the behaviour is now different
to that of 2.6.28.  Whereas in 2.6.28 it would timeout very quickly
(after about 1 or 2 seconds) it now times out after 30 seconds
(initially I hadn't realised that it would be so much longer). The
other difference is that the failure becomes cached - trying to reread
the same area using dd instantly fails from now on without any more disk
IO/kernel error messages until echo 1 > /proc/sys/vm/drop_caches is
issued. Before it would always go straight to the disk for the unread
section and the kernel spew would be repeated before the failure.

I guess if this is new intended behaviour then all is well - it's just
different and slower/faster :)

Tested-by: Sitsofe Wheeler <sitsofe@yahoo.com>



      

Re: [SCSI][REGRESSION][BISECTED] Disk errors loop forever in 2.6.29

[James Bottomley]

On Thu, 2009-02-19 at 13:11 -0800, Sitsofe Wheeler wrote:
> > From: James Bottomley <James.Bottomley@HansenPartnership.com>
> >
> > Can you try this patch ... it was something I meant to get into 2.6.29
> > but forgot about.  The key problem that you seem to be hitting is that
> > the requeue evades the timeout check.  Moving the timeout check to block
> > should fix that.
> 
> Even by itself this patch works although the behaviour is now different
> to that of 2.6.28.  Whereas in 2.6.28 it would timeout very quickly
> (after about 1 or 2 seconds) it now times out after 30 seconds
> (initially I hadn't realised that it would be so much longer). The
> other difference is that the failure becomes cached - trying to reread
> the same area using dd instantly fails from now on without any more disk
> IO/kernel error messages until echo 1 > /proc/sys/vm/drop_caches is
> issued. Before it would always go straight to the disk for the unread
> section and the kernel spew would be repeated before the failure.

Well, the block patch is more of a catch all:  without it apparently the
scsi_io_completion() rewrite has several cases where it will apparently
loop forever.

Your specific behaviour change seems to be a bug in the way the rewrite
is handling ABORTED_COMMAND.  Could you try the patch below (with or
without the block patch, it shouldn't matter) and see if it gets you
back to 2.6.28 behaviour?

Thanks,

James

---

diff --git a/drivers/scsi/scsi_lib.c b/drivers/scsi/scsi_lib.c
index c034841..4b13e36 100644
--- a/drivers/scsi/scsi_lib.c
+++ b/drivers/scsi/scsi_lib.c
@@ -850,12 +850,11 @@ void scsi_io_completion(struct scsi_cmnd *cmd, unsigned int good_bytes)
 				action = ACTION_FAIL;
 			break;
 		case ABORTED_COMMAND:
+			action = ACTION_FAIL;
 			if (sshdr.asc == 0x10) { /* DIF */
 				description = "Target Data Integrity Failure";
-				action = ACTION_FAIL;
 				error = -EILSEQ;
-			} else
-				action = ACTION_RETRY;
+			}
 			break;
 		case NOT_READY:
 			/* If the device is in the process of becoming

Re: [SCSI][REGRESSION][BISECTED] Disk errors loop forever in 2.6.29

[Sitsofe Wheeler]

> From: James Bottomley <James.Bottomley@HansenPartnership.com>

> 
> Your specific behaviour change seems to be a bug in the way the rewrite
> is handling ABORTED_COMMAND.  Could you try the patch below (with or
> without the block patch, it shouldn't matter) and see if it gets you
> back to 2.6.28 behaviour?

Yup that patch makes things timeout quickly just like 2.6.28. The whole 
"it didn't used to cache failures" piece in the previous email seems to
have been a fabrication inside my own head - I checked and 2.6.28 caches
the failure until the cache is dropped too.

It might be worth turning your original patch into some sort of warning
stack trace trigger so that people understand it's just there for 
unwedging and isn't expected behaviour (it is useful for breaking out of
near lockups though).

Tested-by: Sitsofe Wheeler <sitsofe@yahoo.com>