storvsc loops with No Sense messages

storvsc loops with No Sense messages

[Olaf Hering]

KY,

while attempting to install the upcoming openSuSE 12.3 in a hyper-v
guest (1 vcpu, 512MB) hosted on Windows Server 2012 I see a hanging
guest after formating the root partition, when the installer starts to
write files to it.  After booting the install kernel with
'ignore_loglevel' I see a flood of the "No Sense" messages shown below.
I can reproduce it every time with 12.3.

The sglist init patch does not fix it for me.

Hannes did many storage related backports for SLES11 SP3, and sometimes
the installer hangs at similar points during a fresh SP3 install.

The same procedure works fine on Windows Server 2008.

Any ideas what 2012 does differently? It happens with a plain 3.7.n
kernel. I will try 3.8-rc now, and see if it happens as well.

Olaf

...
[    0.000000] Linux version 3.7.5-5.home_olh_12_3-vanilla (geeko@buildhost) (gcc version 4.7.2 20130108 [gcc-4_7-branch revision 195012] (SUSE Linux) ) #1 SMP Mon Jan 28 12:39:46 UTC 2013 (d9a9e6d)
[    0.000000] Command line: quiet panic=9 sysrq=yes start_shell splash=silent vga=0x317 video=1024x768 console=ttyS0,115200 console=tty1 ignore_loglevel initrd=sl/12.3/initrd-x86_64 BOOT_IMAGE=sl/12.3/linux-x86_64
...
[    7.938806] scsi2 : storvsc_host_t
[    7.940709] scsi 2:0:0:0: Direct-Access     Msft     Virtual Disk     1.0  PQ: 0 ANSI: 4
[    7.943952] sd 2:0:0:0: Attached scsi generic sg1 type 0

[    7.946440] sd 2:0:0:0: [sda] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
[    7.949485] sd 2:0:0:0: [sda] Write Protect is off
[    7.951231] sd 2:0:0:0: [sda] Mode Sense: 0f 00 00 00
[    7.953105] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[    7.955269] scsi3 : storvsc_host_t
[    7.957027] scsi scan: INQUIRY result too short (5), using 36
[    7.958786] scsi scan: INQUIRY result too short (5), using 36
[    7.961292] input: Microsoft Vmbus HID-compliant Mouse as /devices/virtual/input/input3
[    7.965400] hid-generic 0006:045E:0621.0001: input: <UNKNOWN> HID v0.01 Mouse [Microsoft Vmbus HID-compliant Mouse] on
[    7.971872] hv_netvsc: hv_netvsc channel opened successfully
[    7.974574] hv_netvsc vmbus_0_11: Device MAC 00:15:5d:02:51:02 link state up
[    7.985086]  sda: sda1 sda2
[    7.986725] sd 2:0:0:0: [sda] Attached SCSI disk
...
[  156.205185] Adding 746492k swap on /dev/sda1.  Priority:-1 extents:1 across:746492k
...
[  156.478186] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: acl,user_xattr
[  161.459523] hv_storvsc vmbus_0_1: cmd 0x41 scsi status 0x2 srb status 0x6
[  161.462157] sd 2:0:0:0: [sda]
[  161.463135] Sense Key : No Sense [current]
[  161.464983] sd 2:0:0:0: [sda]
[  161.465899] Add. Sense: No additional sense information
[  161.468211] hv_storvsc vmbus_0_1: cmd 0x41 scsi status 0x2 srb status 0x6
[  161.475766] sd 2:0:0:0: [sda]
[  161.476728] Sense Key : No Sense [current]
[  161.478284] sd 2:0:0:0: [sda]
[  161.479441] Add. Sense: No additional sense information
[  161.481311] hv_storvsc vmbus_0_1: cmd 0x41 scsi status 0x2 srb status 0x6
[  161.484477] sd 2:0:0:0: [sda]
[  161.485422] Sense Key : No Sense [current]
[  161.486977] sd 2:0:0:0: [sda]
[  161.487920] Add. Sense: No additional sense information

... and so it goes on until the VM is powered off.

RE: storvsc loops with No Sense messages

[KY Srinivasan]

> -----Original Message-----
> From: Olaf Hering [mailto:olaf@aepfle.de]
> Sent: Tuesday, January 29, 2013 10:28 AM
> To: KY Srinivasan
> Cc: linux-scsi@vger.kernel.org
> Subject: storvsc loops with No Sense messages
> 
> 
> KY,
> 
> while attempting to install the upcoming openSuSE 12.3 in a hyper-v
> guest (1 vcpu, 512MB) hosted on Windows Server 2012 I see a hanging
> guest after formating the root partition, when the installer starts to
> write files to it.  After booting the install kernel with
> 'ignore_loglevel' I see a flood of the "No Sense" messages shown below.
> I can reproduce it every time with 12.3.
> 
> The sglist init patch does not fix it for me.
> 
> Hannes did many storage related backports for SLES11 SP3, and sometimes
> the installer hangs at similar points during a fresh SP3 install.
> 
> The same procedure works fine on Windows Server 2008.
> 
> Any ideas what 2012 does differently? It happens with a plain 3.7.n
> kernel. I will try 3.8-rc now, and see if it happens as well.
> 
> Olaf
> 
> ...
> [    0.000000] Linux version 3.7.5-5.home_olh_12_3-vanilla (geeko@buildhost)
> (gcc version 4.7.2 20130108 [gcc-4_7-branch revision 195012] (SUSE Linux) ) #1
> SMP Mon Jan 28 12:39:46 UTC 2013 (d9a9e6d)
> [    0.000000] Command line: quiet panic=9 sysrq=yes start_shell splash=silent
> vga=0x317 video=1024x768 console=ttyS0,115200 console=tty1 ignore_loglevel
> initrd=sl/12.3/initrd-x86_64 BOOT_IMAGE=sl/12.3/linux-x86_64
> ...
> [    7.938806] scsi2 : storvsc_host_t
> [    7.940709] scsi 2:0:0:0: Direct-Access     Msft     Virtual Disk     1.0  PQ: 0 ANSI: 4
> [    7.943952] sd 2:0:0:0: Attached scsi generic sg1 type 0
> 
> [    7.946440] sd 2:0:0:0: [sda] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
> [    7.949485] sd 2:0:0:0: [sda] Write Protect is off
> [    7.951231] sd 2:0:0:0: [sda] Mode Sense: 0f 00 00 00
> [    7.953105] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't
> support DPO or FUA
> [    7.955269] scsi3 : storvsc_host_t
> [    7.957027] scsi scan: INQUIRY result too short (5), using 36
> [    7.958786] scsi scan: INQUIRY result too short (5), using 36
> [    7.961292] input: Microsoft Vmbus HID-compliant Mouse as
> /devices/virtual/input/input3
> [    7.965400] hid-generic 0006:045E:0621.0001: input: <UNKNOWN> HID v0.01
> Mouse [Microsoft Vmbus HID-compliant Mouse] on
> [    7.971872] hv_netvsc: hv_netvsc channel opened successfully
> [    7.974574] hv_netvsc vmbus_0_11: Device MAC 00:15:5d:02:51:02 link state up
> [    7.985086]  sda: sda1 sda2
> [    7.986725] sd 2:0:0:0: [sda] Attached SCSI disk
> ...
> [  156.205185] Adding 746492k swap on /dev/sda1.  Priority:-1 extents:1
> across:746492k
> ...
> [  156.478186] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts:
> acl,user_xattr
> [  161.459523] hv_storvsc vmbus_0_1: cmd 0x41 scsi status 0x2 srb status 0x6
> [  161.462157] sd 2:0:0:0: [sda]
> [  161.463135] Sense Key : No Sense [current]
> [  161.464983] sd 2:0:0:0: [sda]
> [  161.465899] Add. Sense: No additional sense information
> [  161.468211] hv_storvsc vmbus_0_1: cmd 0x41 scsi status 0x2 srb status 0x6
> [  161.475766] sd 2:0:0:0: [sda]
> [  161.476728] Sense Key : No Sense [current]
> [  161.478284] sd 2:0:0:0: [sda]
> [  161.479441] Add. Sense: No additional sense information
> [  161.481311] hv_storvsc vmbus_0_1: cmd 0x41 scsi status 0x2 srb status 0x6
> [  161.484477] sd 2:0:0:0: [sda]
> [  161.485422] Sense Key : No Sense [current]
> [  161.486977] sd 2:0:0:0: [sda]
> [  161.487920] Add. Sense: No additional sense information
> 
> ... and so it goes on until the VM is powered off.

Olaf,

I have not seen this. The SRB status indicates it is an illegal request; the command is WRITE_SAME. I will take a look.

K. Y
> 
> 

Re: storvsc loops with No Sense messages

[Olaf Hering]

On Tue, Jan 29, KY Srinivasan wrote:

> I have not seen this. The SRB status indicates it is an illegal request; the command is WRITE_SAME. I will take a look.

I can ignore it like SET_WINDOW, and the installation proceeds.
Just a few messages in dmesg:

...
[  107.448281] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts: acl,user_xattr
[  111.020383] sd 2:0:0:0: [sda] Unhandled error code
[  111.020913] sd 2:0:0:0: [sda]
[  111.021264] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[  111.021791] sd 2:0:0:0: [sda] CDB:
[  111.022186] Write Same(10): 41 00 00 16 e3 a8 00 0f d8 00
[  111.024400] end_request: I/O error, dev sda, sector 1500072
[  111.025790] sda2: WRITE SAME failed. Manually zeroing.
[  112.936342] sd 2:0:0:0: [sda] Unhandled error code
[  112.937588] sd 2:0:0:0: [sda]
[  112.938429] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[  112.939642] sd 2:0:0:0: [sda] CDB:
[  112.940550] Write Same(10): 41 00 00 16 f3 80 00 0f e0 00
[  112.945081] end_request: I/O error, dev sda, sector 1504128
[  112.946627] sda2: WRITE SAME failed. Manually zeroing.
[  113.495880] ISO 9660 Extensions: Microsoft Joliet Level 3
[  113.498959] ISO 9660 Extensions: RRIP_1991A
[  114.728274] sd 2:0:0:0: [sda] Unhandled error code
[  114.729540] sd 2:0:0:0: [sda]
[  114.730419] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[  114.731670] sd 2:0:0:0: [sda] CDB:
[  114.732644] Write Same(10): 41 00 00 17 03 60 00 0f e0 00
[  114.737219] end_request: I/O error, dev sda, sector 1508192
[  114.739027] sda2: WRITE SAME failed. Manually zeroing.
[  116.512217] sd 2:0:0:0: [sda] Unhandled error code
[  116.513206] sd 2:0:0:0: [sda]
[  116.513947] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
[  116.515187] sd 2:0:0:0: [sda] CDB:
[  116.515983] Write Same(10): 41 00 00 17 13 40 00 0f e0 00
[  116.519547] end_request: I/O error, dev sda, sector 1512256
[  116.521011] sda2: WRITE SAME failed. Manually zeroing.
...

Olaf

RE: storvsc loops with No Sense messages

[KY Srinivasan]

Was the installation done on a vhd or VHDX. I checked with Hyper-V guys and they say they never supported WRITE_SAME. I am wondering how it worked on ws2008.

K. Y

> -----Original Message-----
> From: Olaf Hering [mailto:olaf@aepfle.de]
> Sent: Tuesday, January 29, 2013 1:56 PM
> To: KY Srinivasan
> Cc: linux-scsi@vger.kernel.org
> Subject: Re: storvsc loops with No Sense messages
> 
> On Tue, Jan 29, KY Srinivasan wrote:
> 
> > I have not seen this. The SRB status indicates it is an illegal request; the
> command is WRITE_SAME. I will take a look.
> 
> I can ignore it like SET_WINDOW, and the installation proceeds.
> Just a few messages in dmesg:
> 
> ...
> [  107.448281] EXT4-fs (sda2): mounted filesystem with ordered data mode. Opts:
> acl,user_xattr
> [  111.020383] sd 2:0:0:0: [sda] Unhandled error code
> [  111.020913] sd 2:0:0:0: [sda]
> [  111.021264] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
> [  111.021791] sd 2:0:0:0: [sda] CDB:
> [  111.022186] Write Same(10): 41 00 00 16 e3 a8 00 0f d8 00
> [  111.024400] end_request: I/O error, dev sda, sector 1500072
> [  111.025790] sda2: WRITE SAME failed. Manually zeroing.
> [  112.936342] sd 2:0:0:0: [sda] Unhandled error code
> [  112.937588] sd 2:0:0:0: [sda]
> [  112.938429] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
> [  112.939642] sd 2:0:0:0: [sda] CDB:
> [  112.940550] Write Same(10): 41 00 00 16 f3 80 00 0f e0 00
> [  112.945081] end_request: I/O error, dev sda, sector 1504128
> [  112.946627] sda2: WRITE SAME failed. Manually zeroing.
> [  113.495880] ISO 9660 Extensions: Microsoft Joliet Level 3
> [  113.498959] ISO 9660 Extensions: RRIP_1991A
> [  114.728274] sd 2:0:0:0: [sda] Unhandled error code
> [  114.729540] sd 2:0:0:0: [sda]
> [  114.730419] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
> [  114.731670] sd 2:0:0:0: [sda] CDB:
> [  114.732644] Write Same(10): 41 00 00 17 03 60 00 0f e0 00
> [  114.737219] end_request: I/O error, dev sda, sector 1508192
> [  114.739027] sda2: WRITE SAME failed. Manually zeroing.
> [  116.512217] sd 2:0:0:0: [sda] Unhandled error code
> [  116.513206] sd 2:0:0:0: [sda]
> [  116.513947] Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK
> [  116.515187] sd 2:0:0:0: [sda] CDB:
> [  116.515983] Write Same(10): 41 00 00 17 13 40 00 0f e0 00
> [  116.519547] end_request: I/O error, dev sda, sector 1512256
> [  116.521011] sda2: WRITE SAME failed. Manually zeroing.
> ...
> 
> Olaf
> 

Re: storvsc loops with No Sense messages

[Olaf Hering]

On Tue, Jan 29, KY Srinivasan wrote:

> Was the installation done on a vhd or VHDX. I checked with Hyper-V
> guys and they say they never supported WRITE_SAME. I am wondering how
> it worked on ws2008.

Its a vhdx image, both fixed and dynamically size have the same issue.
I will see what happens with ws2008, at least the install proceeds
without issues. 


Also the unpatched kernel works fine in the installed system.

Olaf

Re: storvsc loops with No Sense messages

[Olaf Hering]

On Tue, Jan 29, Olaf Hering wrote:

> On Tue, Jan 29, KY Srinivasan wrote:
> 
> > Was the installation done on a vhd or VHDX. I checked with Hyper-V
> > guys and they say they never supported WRITE_SAME. I am wondering how
> > it worked on ws2008.
> 
> Its a vhdx image, both fixed and dynamically size have the same issue.
> I will see what happens with ws2008, at least the install proceeds
> without issues. 

I see that WRITE_SAME is also used when running on ws2008, but
appearently it does not cause any issues.
This triggers only when the default ext4 is used, not when I force ext3
for the root filesystem.
So to me it looks like ws2012 has issues with WRITE_SAME handling.


Is there a way to not use WRITE_SAME at all? While browsing the code its
not clear if there is a conditional for this command.


Olaf

Re: storvsc loops with No Sense messages

[Olaf Hering]

On Wed, Jan 30, Olaf Hering wrote:

> Is there a way to not use WRITE_SAME at all? While browsing the code its
> not clear if there is a conditional for this command.

It seems scsi_device->no_write_same may avoid this command, I will
test this patch:

# Subject: [PATCH] scsi: storvsc: avoid usage of WRITE_SAME

Set scsi_device->no_write_same because the host does not support it.
Also blacklist WRITE_SAME to avoid (and log) accident usage.

Signed-off-by: Olaf Hering <olaf@aepfle.de>
---
 drivers/scsi/storvsc_drv.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
index 0144078..21f788a 100644
--- a/drivers/scsi/storvsc_drv.c
+++ b/drivers/scsi/storvsc_drv.c
@@ -1155,6 +1155,8 @@ static int storvsc_device_configure(struct scsi_device *sdevice)
 
 	blk_queue_bounce_limit(sdevice->request_queue, BLK_BOUNCE_ANY);
 
+	sdevice->no_write_same = 1;
+
 	return 0;
 }
 
@@ -1241,6 +1243,7 @@ static bool storvsc_scsi_cmd_ok(struct scsi_cmnd *scmnd)
 	 * smartd sends this command and the host does not handle
 	 * this. So, don't send it.
 	 */
+	case WRITE_SAME:
 	case SET_WINDOW:
 		scmnd->result = ILLEGAL_REQUEST << 16;
 		allowed = false;
-- 
1.8.1.1

RE: storvsc loops with No Sense messages

[KY Srinivasan]

> -----Original Message-----
> From: Olaf Hering [mailto:olaf@aepfle.de]
> Sent: Wednesday, January 30, 2013 8:35 AM
> To: KY Srinivasan
> Cc: linux-scsi@vger.kernel.org
> Subject: Re: storvsc loops with No Sense messages
> 
> On Wed, Jan 30, Olaf Hering wrote:
> 
> > Is there a way to not use WRITE_SAME at all? While browsing the code its
> > not clear if there is a conditional for this command.
> 
> It seems scsi_device->no_write_same may avoid this command, I will
> test this patch:
> 
> # Subject: [PATCH] scsi: storvsc: avoid usage of WRITE_SAME
> 
> Set scsi_device->no_write_same because the host does not support it.
> Also blacklist WRITE_SAME to avoid (and log) accident usage.
>
Olaf,

Thanks for looking into this.

Acked-by:  K. Y. Srinivasan <kys@microsoft.com>
 
> Signed-off-by: Olaf Hering <olaf@aepfle.de>
> ---
>  drivers/scsi/storvsc_drv.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/scsi/storvsc_drv.c b/drivers/scsi/storvsc_drv.c
> index 0144078..21f788a 100644
> --- a/drivers/scsi/storvsc_drv.c
> +++ b/drivers/scsi/storvsc_drv.c
> @@ -1155,6 +1155,8 @@ static int storvsc_device_configure(struct scsi_device
> *sdevice)
> 
>  	blk_queue_bounce_limit(sdevice->request_queue, BLK_BOUNCE_ANY);
> 
> +	sdevice->no_write_same = 1;
> +
>  	return 0;
>  }
> 
> @@ -1241,6 +1243,7 @@ static bool storvsc_scsi_cmd_ok(struct scsi_cmnd
> *scmnd)
>  	 * smartd sends this command and the host does not handle
>  	 * this. So, don't send it.
>  	 */
> +	case WRITE_SAME:
>  	case SET_WINDOW:
>  		scmnd->result = ILLEGAL_REQUEST << 16;
>  		allowed = false;
> --
> 1.8.1.1
> 
> 

RE: storvsc loops with No Sense messages

[KY Srinivasan]

> -----Original Message-----
> From: Olaf Hering [mailto:olaf@aepfle.de]
> Sent: Wednesday, January 30, 2013 5:00 AM
> To: KY Srinivasan
> Cc: linux-scsi@vger.kernel.org
> Subject: Re: storvsc loops with No Sense messages
> 
> On Tue, Jan 29, Olaf Hering wrote:
> 
> > On Tue, Jan 29, KY Srinivasan wrote:
> >
> > > Was the installation done on a vhd or VHDX. I checked with Hyper-V
> > > guys and they say they never supported WRITE_SAME. I am wondering how
> > > it worked on ws2008.
> >
> > Its a vhdx image, both fixed and dynamically size have the same issue.
> > I will see what happens with ws2008, at least the install proceeds
> > without issues.
> 
> I see that WRITE_SAME is also used when running on ws2008, but
> appearently it does not cause any issues.
> This triggers only when the default ext4 is used, not when I force ext3
> for the root filesystem.

Talking to the MSFT storage guys, WRITE_SAME has never been supported - not
on ws2008 or on ws2012. So I am not sure why or how this works on ws2008.

K. Y
> So to me it looks like ws2012 has issues with WRITE_SAME handling.
> 
> 
> Is there a way to not use WRITE_SAME at all? While browsing the code its
> not clear if there is a conditional for this command.
> 
> 
> Olaf
> 

Re: storvsc loops with No Sense messages

[Jari Turkia]

> On Wed, Jan 30, Olaf Hering wrote:
>> Is there a way to not use WRITE_SAME at all? While browsing the code its
>> not clear if there is a conditional for this command.
>It seems scsi_device->no_write_same may avoid this command, I will
>test this patch:
># Subject: [PATCH] scsi: storvsc: avoid usage of WRITE_SAME
>Set scsi_device->no_write_same because the host does not support it.
>Also blacklist WRITE_SAME to avoid (and log) accident usage.
>Signed-off-by: Olaf Hering <olaf@aepfle.de>

Hopefully Olaf will find time to test the patch and push it forward. I did test it and it solved the issue perfectly!

Most of the current Linux-distros won't work under Windows Server 2012 or Windows 8 Hyper-V.

Regards,
Jari Turkia