[Bug 79901] New: Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

[Bug 79901] New: Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=79901

            Bug ID: 79901
           Summary: Extremely slow boot on Promise VTrak E610f due to
                    sd_mod RSOC usage
           Product: IO/Storage
           Version: 2.5
    Kernel Version: 3.14.7
          Hardware: x86-64
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: SCSI
          Assignee: linux-scsi@vger.kernel.org
          Reporter: rraptorr@nails.eu.org
        Regression: No

Recently I've started upgrading all of my machines to kernel 3.14 (from Debian
wheezy backports to be precise). Mostly there were not problems, but I've
stumbled upon weird behavior on Fibre Channel servers (QLogic cards inside HP
blades) using Promise VTrak E610f arrays.

As soon as SCSI subsystem tries to detect partitions a lot of SCSI errors are
reported. The system stalls (but initramfs is responsive) for about 20-30
minutes (depending on number of arrays and LUNs). After that time, disk
detection finishes and system continues booting as usual. Everything works
perfectly afterwards.

I've spent some time fiddling with qla2xxx driver versions, SCSI scanning
options and anything else I could think of. Finally, I was able to find
culprit. The problem lies in sd_mod usage of scsi_report_opcode(). This
function is used to determine if the disk supports WRITE SAME command. It does
so by issuing REPORT SUPPORTED OPERATION CODES command. Unfortunately, it seems
Promise VTrak E610f really, really does not like RSOC. As soon as RSOC is
issued the array stalls for a while, then kernel tries to abort the command and
finally it must reset the port. Fortunately the array starts working again
after the reset. I've also verified this behavior with sg_opcodes utility.

Commit that introduced RSOC usage in sd_mod:
http://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/commit/?id=98dcc2946adbe4349ef1ef9b99873b912831edd4
Removing it fixes the issue.

I'm not sure what is the correct way to fix this as I'm not very familiar with
SCSI spec. If RSOC support cannot be reliably determined then probably some
kind of blacklist should be introduced.

As a workaround, I've modified qla2xxx driver to set 'no_write_same' flag.
While not directly related it forces sd_mod not to issue RSOC and it is easier
for me to ship Debian package with modified single driver (I'd prefer not to
manage my own kernel packages).

-- 
You are receiving this mail because:
You are the assignee for the bug.

Re: [Bug 79901] New: Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

[Christoph Hellwig]

Sounds like we need to skip REPORT SUPPORTED OPERATION CODES on this
array.

For this it needs a new BLIST_NORSOC flag added to
include/scsi/scsi_devinfo.h which sets sdev->no_report_opcodes in
scsi_add_lun, and an entry for your array in the big black list array in
drivers/scsi/scsi_devinfo.c.  Does this sound easy enough that you could
prepare a patch for it?

Re: [Bug 79901] New: Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

[Janusz Dziemidowicz]

2014-07-10 15:54 GMT+02:00 Christoph Hellwig <hch@infradead.org>:
> Sounds like we need to skip REPORT SUPPORTED OPERATION CODES on this
> array.
>
> For this it needs a new BLIST_NORSOC flag added to
> include/scsi/scsi_devinfo.h which sets sdev->no_report_opcodes in
> scsi_add_lun, and an entry for your array in the big black list array in
> drivers/scsi/scsi_devinfo.c.  Does this sound easy enough that you could
> prepare a patch for it?

Sure, I'll give it a try. It will probably take a couple of days as
currently I have only one server that is available for experiments for
a few hours a day.

-- 
Janusz Dziemidowicz

[Bug 79901] Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=79901

--- Comment #1 from Janusz Dziemidowicz <rraptorr@nails.eu.org> ---
Created attachment 144101
  --> https://bugzilla.kernel.org/attachment.cgi?id=144101&action=edit
Patch adding BLIST_NO_RSOC scsi scan flag

As discussed on the list, attached simple patch that blacklist RSOC on Promise
VTrak E610f

-- 
You are receiving this mail because:
You are the assignee for the bug.

Re: [Bug 79901] Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

[Christoph Hellwig]

Thanks, this one looks good to me.  Can someone else give me a second
review on the attached patch so I can merged it?

---
>From 95bc83fa0d6e9a6152cd6fce79ff87c0994e83ed Mon Sep 17 00:00:00 2001
From: Janusz Dziemidowicz <rraptorr@nails.eu.org>
Date: Thu, 24 Jul 2014 15:48:46 +0200
Subject: [PATCH] Do not issue SCSI RSOC command to Promise Vtrak E610f

Some devices don't like REPORT SUPPORTED OPERATION CODES and will
simply timeout causing sd_mod init to take a very very long time.
Introduce BLIST_NO_RSOC scsi scan flag, that stops RSOC from being
issued. Add it to Promise Vtrak E610f entry in scsi scan
blacklist. Fixes bug #79901 reported at
https://bugzilla.kernel.org/show_bug.cgi?id=79901

Fixes: 98dcc2946adb ("SCSI: sd: Update WRITE SAME heuristics")

Signed-off-by: Janusz Dziemidowicz <rraptorr@nails.eu.org>
---
 drivers/scsi/scsi_devinfo.c | 1 +
 drivers/scsi/scsi_scan.c    | 6 ++++++
 include/scsi/scsi_devinfo.h | 1 +
 3 files changed, 8 insertions(+)

diff --git a/drivers/scsi/scsi_devinfo.c b/drivers/scsi/scsi_devinfo.c
index f969aca..49014a1 100644
--- a/drivers/scsi/scsi_devinfo.c
+++ b/drivers/scsi/scsi_devinfo.c
@@ -222,6 +222,7 @@ static struct {
 	{"PIONEER", "CD-ROM DRM-602X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
 	{"PIONEER", "CD-ROM DRM-604X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
 	{"PIONEER", "CD-ROM DRM-624X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
+	{"Promise", "VTrak E610f", NULL, BLIST_SPARSELUN | BLIST_NO_RSOC},
 	{"Promise", "", NULL, BLIST_SPARSELUN},
 	{"QUANTUM", "XP34301", "1071", BLIST_NOTQ},
 	{"REGAL", "CDC-4X", NULL, BLIST_MAX5LUN | BLIST_SINGLELUN},
diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
index e02b3aa..70919fc 100644
--- a/drivers/scsi/scsi_scan.c
+++ b/drivers/scsi/scsi_scan.c
@@ -922,6 +922,12 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result,
 	if (*bflags & BLIST_USE_10_BYTE_MS)
 		sdev->use_10_for_ms = 1;
 
+	/* some devices don't like REPORT SUPPORTED OPERATION CODES
+	 * and will simply timeout causing sd_mod init to take a very
+	 * very long time */
+	if (*bflags & BLIST_NO_RSOC)
+		sdev->no_report_opcodes = 1;
+
 	/* set the device running here so that slave configure
 	 * may do I/O */
 	ret = scsi_device_set_state(sdev, SDEV_RUNNING);
diff --git a/include/scsi/scsi_devinfo.h b/include/scsi/scsi_devinfo.h
index 447d2d7..1d5ba9e 100644
--- a/include/scsi/scsi_devinfo.h
+++ b/include/scsi/scsi_devinfo.h
@@ -32,4 +32,5 @@
 #define BLIST_ATTACH_PQ3	0x1000000 /* Scan: Attach to PQ3 devices */
 #define BLIST_NO_DIF		0x2000000 /* Disable T10 PI (DIF) */
 #define BLIST_SKIP_VPD_PAGES	0x4000000 /* Ignore SBC-3 VPD pages */
+#define BLIST_NO_RSOC		0x8000000 /* don't try REPORT SUPPORTED OPERATION CODES */
 #endif
-- 
1.9.3

[PATCH] Do not issue SCSI RSOC command to Promise Vtrak E610f sd_mod RSOC usage

[Christoph Hellwig]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit, Size: 3547 bytes --]

ping?  ÍI'd really like to get a review on this one and merge it for
3.17

On Thu, Jul 24, 2014 at 08:28:38AM -0700, Christoph Hellwig wrote:
> Thanks, this one looks good to me.  Can someone else give me a second
> review on the attached patch so I can merged it?
> 
> ---
> >From 95bc83fa0d6e9a6152cd6fce79ff87c0994e83ed Mon Sep 17 00:00:00 2001
> From: Janusz Dziemidowicz <rraptorr@nails.eu.org>
> Date: Thu, 24 Jul 2014 15:48:46 +0200
> Subject: [PATCH] Do not issue SCSI RSOC command to Promise Vtrak E610f
> 
> Some devices don't like REPORT SUPPORTED OPERATION CODES and will
> simply timeout causing sd_mod init to take a very very long time.
> Introduce BLIST_NO_RSOC scsi scan flag, that stops RSOC from being
> issued. Add it to Promise Vtrak E610f entry in scsi scan
> blacklist. Fixes bug #79901 reported at
> https://bugzilla.kernel.org/show_bug.cgi?id=79901
> 
> Fixes: 98dcc2946adb ("SCSI: sd: Update WRITE SAME heuristics")
> 
> Signed-off-by: Janusz Dziemidowicz <rraptorr@nails.eu.org>
> ---
>  drivers/scsi/scsi_devinfo.c | 1 +
>  drivers/scsi/scsi_scan.c    | 6 ++++++
>  include/scsi/scsi_devinfo.h | 1 +
>  3 files changed, 8 insertions(+)
> 
> diff --git a/drivers/scsi/scsi_devinfo.c b/drivers/scsi/scsi_devinfo.c
> index f969aca..49014a1 100644
> --- a/drivers/scsi/scsi_devinfo.c
> +++ b/drivers/scsi/scsi_devinfo.c
> @@ -222,6 +222,7 @@ static struct {
>  	{"PIONEER", "CD-ROM DRM-602X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
>  	{"PIONEER", "CD-ROM DRM-604X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
>  	{"PIONEER", "CD-ROM DRM-624X", NULL, BLIST_FORCELUN | BLIST_SINGLELUN},
> +	{"Promise", "VTrak E610f", NULL, BLIST_SPARSELUN | BLIST_NO_RSOC},
>  	{"Promise", "", NULL, BLIST_SPARSELUN},
>  	{"QUANTUM", "XP34301", "1071", BLIST_NOTQ},
>  	{"REGAL", "CDC-4X", NULL, BLIST_MAX5LUN | BLIST_SINGLELUN},
> diff --git a/drivers/scsi/scsi_scan.c b/drivers/scsi/scsi_scan.c
> index e02b3aa..70919fc 100644
> --- a/drivers/scsi/scsi_scan.c
> +++ b/drivers/scsi/scsi_scan.c
> @@ -922,6 +922,12 @@ static int scsi_add_lun(struct scsi_device *sdev, unsigned char *inq_result,
>  	if (*bflags & BLIST_USE_10_BYTE_MS)
>  		sdev->use_10_for_ms = 1;
>  
> +	/* some devices don't like REPORT SUPPORTED OPERATION CODES
> +	 * and will simply timeout causing sd_mod init to take a very
> +	 * very long time */
> +	if (*bflags & BLIST_NO_RSOC)
> +		sdev->no_report_opcodes = 1;
> +
>  	/* set the device running here so that slave configure
>  	 * may do I/O */
>  	ret = scsi_device_set_state(sdev, SDEV_RUNNING);
> diff --git a/include/scsi/scsi_devinfo.h b/include/scsi/scsi_devinfo.h
> index 447d2d7..1d5ba9e 100644
> --- a/include/scsi/scsi_devinfo.h
> +++ b/include/scsi/scsi_devinfo.h
> @@ -32,4 +32,5 @@
>  #define BLIST_ATTACH_PQ3	0x1000000 /* Scan: Attach to PQ3 devices */
>  #define BLIST_NO_DIF		0x2000000 /* Disable T10 PI (DIF) */
>  #define BLIST_SKIP_VPD_PAGES	0x4000000 /* Ignore SBC-3 VPD pages */
> +#define BLIST_NO_RSOC		0x8000000 /* don't try REPORT SUPPORTED OPERATION CODES */
>  #endif
> -- 
> 1.9.3
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
---end quoted text---
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Do not issue SCSI RSOC command to Promise Vtrak E610f sd_mod RSOC usage

[Hannes Reinecke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit; format=flowed, Size: 0 bytes --]

Re: [PATCH] Do not issue SCSI RSOC command to Promise Vtrak E610f sd_mod RSOC usage

[Martin K. Petersen]

>>>>> "Christoph" == Christoph Hellwig <hch@infradead.org> writes:

Christoph> ping?  ÍI'd really like to get a review on this one and merge
Christoph> it for 3.17

Looks good to me.

Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>

-- 
Martin K. Petersen	Oracle Linux Engineering
--
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH] Do not issue SCSI RSOC command to Promise Vtrak E610f sd_mod RSOC usage

[Hannes Reinecke]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=unknown-8bit; format=flowed, Size: 0 bytes --]

[Bug 79901] [BISECTED]Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=79901

Alan <alan@lxorguk.ukuu.org.uk> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |alan@lxorguk.ukuu.org.uk
            Summary|Extremely slow boot on      |[BISECTED]Extremely slow
                   |Promise VTrak E610f due to  |boot on Promise VTrak E610f
                   |sd_mod RSOC usage           |due to sd_mod RSOC usage

--- Comment #2 from Alan <alan@lxorguk.ukuu.org.uk> ---
See Documentation/SubmittingPatches - we need a valid email/Signed-off-by for
submissions

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 79901] [BISECTED]Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=79901

--- Comment #3 from Janusz Dziemidowicz <rraptorr@nails.eu.org> ---
I've read that document before creating the patch. Now, I've read it again. I
must say I'm at a loss. In the patch attached to this bug entry I've already
included Signed-off-by line.
Can anyone point precisely what is wrong here?

-- 
You are receiving this mail because:
You are the assignee for the bug.

Re: [Bug 79901] [BISECTED]Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

[Christoph Hellwig]

On Tue, Jul 29, 2014 at 07:07:51PM +0000, bugzilla-daemon@bugzilla.kernel.org wrote:
> https://bugzilla.kernel.org/show_bug.cgi?id=79901
> 
> --- Comment #3 from Janusz Dziemidowicz <rraptorr@nails.eu.org> ---
> I've read that document before creating the patch. Now, I've read it again. I
> must say I'm at a loss. In the patch attached to this bug entry I've already
> included Signed-off-by line.
> Can anyone point precisely what is wrong here?

Everything is fine and I already applied your patch.  Thanks a lot for
debugging the issue and coming up with the patch!

[Bug 79901] [BISECTED]Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=79901

--- Comment #4 from Alan <alan@lxorguk.ukuu.org.uk> ---
It needs to go via email. If you send it to linux-scsi@vger.kernel.org then the
right things should happen (feel free to cc me as well so I can help keep an
eye on it)

-- 
You are receiving this mail because:
You are the assignee for the bug.

[Bug 79901] [BISECTED]Extremely slow boot on Promise VTrak E610f due to sd_mod RSOC usage

[bugzilla-daemon]

https://bugzilla.kernel.org/show_bug.cgi?id=79901

Alan <alan@lxorguk.ukuu.org.uk> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
         Regression|No                          |Yes

-- 
You are receiving this mail because:
You are the assignee for the bug.