iSCSI regression with linux 3.9 and 4.0

iSCSI regression with linux 3.9 and 4.0

[Christian Hesse]

[-- Attachment #1: Type: text/plain, Size: 5113 bytes --]

Hello everybody!

I reported this issue at LKML [0] but received no answer. Hopefully
linux-scsi is a better place...

Beginning with linux 3.19 I see an iSCSI regressen. This works perfectly with
linux 3.18.x (tested with 3.18.6) and before. Effected kernels I tested are
3.19.0, 3.19.2 and 4.0rc4.r199.gb314aca.

The logs tell the story:

Feb 19 11:26:49 thebe kernel: scsi host6: iSCSI Initiator over TCP/IP
Feb 19 11:26:49 thebe kernel: scsi 6:0:0:0: Direct-Access     QNAP     iSCSI Storage    4.0  PQ: 0 ANSI: 5
Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] 1073741824 512-byte logical blocks: (549 GB/512 GiB)
Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Write Protect is off
Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Mode Sense: 2f 00 00 00
Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Feb 19 11:26:49 thebe kernel:  sdb: unknown partition table
Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Attached SCSI disk
Feb 19 11:26:49 thebe iscsid[10804]: Connection1:0 to [target: iqn.2004-04.com.qnap:ts-859:iscsi.xxxxxxx.c40a18, portal: xx.xx.xx.xx,3260] through [iface: default] is operational now
Feb 19 11:26:57 thebe kernel:  sdb: unknown partition table
Feb 19 11:28:20 thebe kernel: EXT4-fs (dm-8): mounting with "discard" option, but the device does not support discard
Feb 19 11:28:20 thebe kernel: EXT4-fs (dm-8): mounted filesystem with ordered data mode. Opts: (null)
Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] UNKNOWN Result: hostbyte=0x00 driverbyte=0x08
Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] Sense Key : 0x5 [current]
Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] ASC=0x24 ASCQ=0x0
Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] CDB: 
Feb 19 11:28:24 thebe kernel: cdb[0]=0x2a: 2a 00 34 5b 07 ff 00 2f 88 00
Feb 19 11:28:24 thebe kernel: blk_update_request: critical target error, dev sdb, sector 878381055
Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108749056)
Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749056
Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749057
Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749058
Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749059
Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749060
Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749061
Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749062
Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749063
Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749064
Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749065
Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108749312)
Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108749568)
Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108749824)
Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108750080)
Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108750336)
Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb] UNKNOWN Result: hostbyte=0x00 driverbyte=0x08
Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb] Sense Key : 0x5 [current]
Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb] ASC=0x24 ASCQ=0x0
Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb] CDB: 
Feb 19 11:29:10 thebe kernel: cdb[0]=0x2a: 2a 00 20 44 89 17 00 20 50 00
Feb 19 11:29:10 thebe kernel: blk_update_request: critical target error, dev sdb, sector 541362455
Feb 19 11:29:10 thebe kernel: Buffer I/O error on dev dm-8, logical block 66621731, lost sync page write
Feb 19 11:29:10 thebe kernel: Aborting journal on device dm-8-8.
Feb 19 11:29:10 thebe kernel: EXT4-fs error (device dm-8): ext4_journal_check_start:56: Detected aborted journal
Feb 19 11:29:10 thebe kernel: EXT4-fs (dm-8): Remounting filesystem read-only
Feb 19 11:29:20 thebe kernel: EXT4-fs error (device dm-8): ext4_put_super:780: Couldn't clean up the journal

[0] https://lkml.org/lkml/2015/2/19/91
-- 
main(a){char*c=/*    Schoene Gruesse                         */"B?IJj;MEH"
"CX:;",b;for(a/*    Chris           get my mail address:    */=0;b=c[a++];)
putchar(b-1/(/*               gcc -o sig sig.c && ./sig    */b/42*2-3)*42);}

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: iSCSI regression with linux 3.9 and 4.0

[Ewan Milne]

On Fri, 2015-03-20 at 13:57 +0100, Christian Hesse wrote:
> Hello everybody!
> 
> I reported this issue at LKML [0] but received no answer. Hopefully
> linux-scsi is a better place...
> 
> Beginning with linux 3.19 I see an iSCSI regressen. This works perfectly with
> linux 3.18.x (tested with 3.18.6) and before. Effected kernels I tested are
> 3.19.0, 3.19.2 and 4.0rc4.r199.gb314aca.
> 
> The logs tell the story:
> 
> Feb 19 11:26:49 thebe kernel: scsi host6: iSCSI Initiator over TCP/IP
> Feb 19 11:26:49 thebe kernel: scsi 6:0:0:0: Direct-Access     QNAP     iSCSI Storage    4.0  PQ: 0 ANSI: 5
> Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] 1073741824 512-byte logical blocks: (549 GB/512 GiB)
> Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Write Protect is off
> Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Mode Sense: 2f 00 00 00
> Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
> Feb 19 11:26:49 thebe kernel:  sdb: unknown partition table
> Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Attached SCSI disk
> Feb 19 11:26:49 thebe iscsid[10804]: Connection1:0 to [target: iqn.2004-04.com.qnap:ts-859:iscsi.xxxxxxx.c40a18, portal: xx.xx.xx.xx,3260] through [iface: default] is operational now
> Feb 19 11:26:57 thebe kernel:  sdb: unknown partition table
> Feb 19 11:28:20 thebe kernel: EXT4-fs (dm-8): mounting with "discard" option, but the device does not support discard
> Feb 19 11:28:20 thebe kernel: EXT4-fs (dm-8): mounted filesystem with ordered data mode. Opts: (null)
> Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] UNKNOWN Result: hostbyte=0x00 driverbyte=0x08
> Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] Sense Key : 0x5 [current]
> Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] ASC=0x24 ASCQ=0x0
> Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] CDB: 
> Feb 19 11:28:24 thebe kernel: cdb[0]=0x2a: 2a 00 34 5b 07 ff 00 2f 88 00
> Feb 19 11:28:24 thebe kernel: blk_update_request: critical target error, dev sdb, sector 878381055
> Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108749056)
> Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749056
> Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749057
> Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749058
> Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749059
> Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749060
> Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749061
> Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749062
> Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749063
> Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749064
> Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block 108749065
> Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108749312)
> Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108749568)
> Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108749824)
> Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108750080)
> Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592 starting block 108750336)
> Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb] UNKNOWN Result: hostbyte=0x00 driverbyte=0x08
> Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb] Sense Key : 0x5 [current]
> Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb] ASC=0x24 ASCQ=0x0
> Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb] CDB: 
> Feb 19 11:29:10 thebe kernel: cdb[0]=0x2a: 2a 00 20 44 89 17 00 20 50 00
> Feb 19 11:29:10 thebe kernel: blk_update_request: critical target error, dev sdb, sector 541362455
> Feb 19 11:29:10 thebe kernel: Buffer I/O error on dev dm-8, logical block 66621731, lost sync page write
> Feb 19 11:29:10 thebe kernel: Aborting journal on device dm-8-8.
> Feb 19 11:29:10 thebe kernel: EXT4-fs error (device dm-8): ext4_journal_check_start:56: Detected aborted journal
> Feb 19 11:29:10 thebe kernel: EXT4-fs (dm-8): Remounting filesystem read-only
> Feb 19 11:29:20 thebe kernel: EXT4-fs error (device dm-8): ext4_put_super:780: Couldn't clean up the journal
> 
> [0] https://lkml.org/lkml/2015/2/19/91

Sense key 0x5 ASC/ASCQ 0x24 0x00 is ILLEGAL REQUEST, INVALID FIELD IN
CDB.  The CDB was 2A 00 34 5B 07 FF 00 2F 88 00, which is a WRITE_10
to LBA 878381055 with a length of 12168 blocks (a little less than 6MB).
It looks like this is within the reported capacity of the device, and
there are no other bits set in the CDB.

Looks like you could get this error if RWWP (reject without write
protection) is set in the control mode page.  I don't see any messages
about the protection type, though.  What does sysfs report?

-Ewan

Re: iSCSI regression with linux 3.9 and 4.0

[Christian Hesse]

[-- Attachment #1: Type: text/plain, Size: 6427 bytes --]

Ewan Milne <emilne@redhat.com> on Fri, 2015/03/20 09:51:
> On Fri, 2015-03-20 at 13:57 +0100, Christian Hesse wrote:
> > Hello everybody!
> > 
> > I reported this issue at LKML [0] but received no answer. Hopefully
> > linux-scsi is a better place...
> > 
> > Beginning with linux 3.19 I see an iSCSI regressen. This works perfectly
> > with linux 3.18.x (tested with 3.18.6) and before. Effected kernels I
> > tested are 3.19.0, 3.19.2 and 4.0rc4.r199.gb314aca.
> > 
> > The logs tell the story:
> > 
> > Feb 19 11:26:49 thebe kernel: scsi host6: iSCSI Initiator over TCP/IP
> > Feb 19 11:26:49 thebe kernel: scsi 6:0:0:0: Direct-Access     QNAP
> > iSCSI Storage    4.0  PQ: 0 ANSI: 5 Feb 19 11:26:49 thebe kernel: sd
> > 6:0:0:0: [sdb] 1073741824 512-byte logical blocks: (549 GB/512 GiB) Feb
> > 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Write Protect is off Feb 19
> > 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Mode Sense: 2f 00 00 00 Feb 19
> > 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Write cache: disabled, read
> > cache: enabled, doesn't support DPO or FUA Feb 19 11:26:49 thebe kernel:
> > sdb: unknown partition table Feb 19 11:26:49 thebe kernel: sd 6:0:0:0:
> > [sdb] Attached SCSI disk Feb 19 11:26:49 thebe iscsid[10804]:
> > Connection1:0 to [target:
> > iqn.2004-04.com.qnap:ts-859:iscsi.xxxxxxx.c40a18, portal:
> > xx.xx.xx.xx,3260] through [iface: default] is operational now Feb 19
> > 11:26:57 thebe kernel:  sdb: unknown partition table Feb 19 11:28:20
> > thebe kernel: EXT4-fs (dm-8): mounting with "discard" option, but the
> > device does not support discard Feb 19 11:28:20 thebe kernel: EXT4-fs
> > (dm-8): mounted filesystem with ordered data mode. Opts: (null) Feb 19
> > 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] UNKNOWN Result: hostbyte=0x00
> > driverbyte=0x08 Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] Sense
> > Key : 0x5 [current] Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb]
> > ASC=0x24 ASCQ=0x0 Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] CDB:
> > Feb 19 11:28:24 thebe kernel: cdb[0]=0x2a: 2a 00 34 5b 07 ff 00 2f 88 00
> > Feb 19 11:28:24 thebe kernel: blk_update_request: critical target error,
> > dev sdb, sector 878381055 Feb 19 11:28:24 thebe kernel: EXT4-fs warning
> > (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503
> > (offset 8388608 size 7278592 starting block 108749056) Feb 19 11:28:24
> > thebe kernel: Buffer I/O error on device dm-8, logical block 108749056
> > Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical
> > block 108749057 Feb 19 11:28:24 thebe kernel: Buffer I/O error on device
> > dm-8, logical block 108749058 Feb 19 11:28:24 thebe kernel: Buffer I/O
> > error on device dm-8, logical block 108749059 Feb 19 11:28:24 thebe
> > kernel: Buffer I/O error on device dm-8, logical block 108749060 Feb 19
> > 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block
> > 108749061 Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8,
> > logical block 108749062 Feb 19 11:28:24 thebe kernel: Buffer I/O error on
> > device dm-8, logical block 108749063 Feb 19 11:28:24 thebe kernel: Buffer
> > I/O error on device dm-8, logical block 108749064 Feb 19 11:28:24 thebe
> > kernel: Buffer I/O error on device dm-8, logical block 108749065 Feb 19
> > 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317:
> > I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592
> > starting block 108749312) Feb 19 11:28:24 thebe kernel: EXT4-fs warning
> > (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503
> > (offset 8388608 size 7278592 starting block 108749568) Feb 19 11:28:24
> > thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error
> > -121 writing to inode 33196503 (offset 8388608 size 7278592 starting
> > block 108749824) Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device
> > dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset
> > 8388608 size 7278592 starting block 108750080) Feb 19 11:28:24 thebe
> > kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121
> > writing to inode 33196503 (offset 8388608 size 7278592 starting block
> > 108750336) Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb] UNKNOWN
> > Result: hostbyte=0x00 driverbyte=0x08 Feb 19 11:29:10 thebe kernel: sd
> > 6:0:0:0: [sdb] Sense Key : 0x5 [current] Feb 19 11:29:10 thebe kernel: sd
> > 6:0:0:0: [sdb] ASC=0x24 ASCQ=0x0 Feb 19 11:29:10 thebe kernel: sd
> > 6:0:0:0: [sdb] CDB: Feb 19 11:29:10 thebe kernel: cdb[0]=0x2a: 2a 00 20
> > 44 89 17 00 20 50 00 Feb 19 11:29:10 thebe kernel: blk_update_request:
> > critical target error, dev sdb, sector 541362455 Feb 19 11:29:10 thebe
> > kernel: Buffer I/O error on dev dm-8, logical block 66621731, lost sync
> > page write Feb 19 11:29:10 thebe kernel: Aborting journal on device
> > dm-8-8. Feb 19 11:29:10 thebe kernel: EXT4-fs error (device dm-8):
> > ext4_journal_check_start:56: Detected aborted journal Feb 19 11:29:10
> > thebe kernel: EXT4-fs (dm-8): Remounting filesystem read-only Feb 19
> > 11:29:20 thebe kernel: EXT4-fs error (device dm-8): ext4_put_super:780:
> > Couldn't clean up the journal
> > 
> > [0] https://lkml.org/lkml/2015/2/19/91
> 
> Sense key 0x5 ASC/ASCQ 0x24 0x00 is ILLEGAL REQUEST, INVALID FIELD IN
> CDB.  The CDB was 2A 00 34 5B 07 FF 00 2F 88 00, which is a WRITE_10
> to LBA 878381055 with a length of 12168 blocks (a little less than 6MB).
> It looks like this is within the reported capacity of the device, and
> there are no other bits set in the CDB.
> 
> Looks like you could get this error if RWWP (reject without write
> protection) is set in the control mode page.  I don't see any messages
> about the protection type, though.  What does sysfs report?

Is that what you are interested in?

# cat protection_mode protection_type 
none
0

In case it matters: The iSCSI device is LUKS encrypted, that is why device
mapper shows up.

I removed the discard option from filesystem's default mount option, but
that brings no difference except the message is not printed.
-- 
main(a){char*c=/*    Schoene Gruesse                         */"B?IJj;MEH"
"CX:;",b;for(a/*    Chris           get my mail address:    */=0;b=c[a++];)
putchar(b-1/(/*               gcc -o sig sig.c && ./sig    */b/42*2-3)*42);}

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: iSCSI regression with linux 3.9 and 4.0

[Ewan Milne]

On Fri, 2015-03-20 at 15:31 +0100, Christian Hesse wrote:
> Ewan Milne <emilne@redhat.com> on Fri, 2015/03/20 09:51:
> > On Fri, 2015-03-20 at 13:57 +0100, Christian Hesse wrote:
> > > Hello everybody!
> > > 
> > > I reported this issue at LKML [0] but received no answer. Hopefully
> > > linux-scsi is a better place...
> > > 
> > > Beginning with linux 3.19 I see an iSCSI regressen. This works perfectly
> > > with linux 3.18.x (tested with 3.18.6) and before. Effected kernels I
> > > tested are 3.19.0, 3.19.2 and 4.0rc4.r199.gb314aca.
> > > 
> > > The logs tell the story:
> > > 
> > > Feb 19 11:26:49 thebe kernel: scsi host6: iSCSI Initiator over TCP/IP
> > > Feb 19 11:26:49 thebe kernel: scsi 6:0:0:0: Direct-Access     QNAP
> > > iSCSI Storage    4.0  PQ: 0 ANSI: 5 Feb 19 11:26:49 thebe kernel: sd
> > > 6:0:0:0: [sdb] 1073741824 512-byte logical blocks: (549 GB/512 GiB) Feb
> > > 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Write Protect is off Feb 19
> > > 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Mode Sense: 2f 00 00 00 Feb 19
> > > 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Write cache: disabled, read
> > > cache: enabled, doesn't support DPO or FUA Feb 19 11:26:49 thebe kernel:
> > > sdb: unknown partition table Feb 19 11:26:49 thebe kernel: sd 6:0:0:0:
> > > [sdb] Attached SCSI disk Feb 19 11:26:49 thebe iscsid[10804]:
> > > Connection1:0 to [target:
> > > iqn.2004-04.com.qnap:ts-859:iscsi.xxxxxxx.c40a18, portal:
> > > xx.xx.xx.xx,3260] through [iface: default] is operational now Feb 19
> > > 11:26:57 thebe kernel:  sdb: unknown partition table Feb 19 11:28:20
> > > thebe kernel: EXT4-fs (dm-8): mounting with "discard" option, but the
> > > device does not support discard Feb 19 11:28:20 thebe kernel: EXT4-fs
> > > (dm-8): mounted filesystem with ordered data mode. Opts: (null) Feb 19
> > > 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] UNKNOWN Result: hostbyte=0x00
> > > driverbyte=0x08 Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] Sense
> > > Key : 0x5 [current] Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb]
> > > ASC=0x24 ASCQ=0x0 Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] CDB:
> > > Feb 19 11:28:24 thebe kernel: cdb[0]=0x2a: 2a 00 34 5b 07 ff 00 2f 88 00
> > > Feb 19 11:28:24 thebe kernel: blk_update_request: critical target error,
> > > dev sdb, sector 878381055 Feb 19 11:28:24 thebe kernel: EXT4-fs warning
> > > (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503
> > > (offset 8388608 size 7278592 starting block 108749056) Feb 19 11:28:24
> > > thebe kernel: Buffer I/O error on device dm-8, logical block 108749056
> > > Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical
> > > block 108749057 Feb 19 11:28:24 thebe kernel: Buffer I/O error on device
> > > dm-8, logical block 108749058 Feb 19 11:28:24 thebe kernel: Buffer I/O
> > > error on device dm-8, logical block 108749059 Feb 19 11:28:24 thebe
> > > kernel: Buffer I/O error on device dm-8, logical block 108749060 Feb 19
> > > 11:28:24 thebe kernel: Buffer I/O error on device dm-8, logical block
> > > 108749061 Feb 19 11:28:24 thebe kernel: Buffer I/O error on device dm-8,
> > > logical block 108749062 Feb 19 11:28:24 thebe kernel: Buffer I/O error on
> > > device dm-8, logical block 108749063 Feb 19 11:28:24 thebe kernel: Buffer
> > > I/O error on device dm-8, logical block 108749064 Feb 19 11:28:24 thebe
> > > kernel: Buffer I/O error on device dm-8, logical block 108749065 Feb 19
> > > 11:28:24 thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317:
> > > I/O error -121 writing to inode 33196503 (offset 8388608 size 7278592
> > > starting block 108749312) Feb 19 11:28:24 thebe kernel: EXT4-fs warning
> > > (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503
> > > (offset 8388608 size 7278592 starting block 108749568) Feb 19 11:28:24
> > > thebe kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error
> > > -121 writing to inode 33196503 (offset 8388608 size 7278592 starting
> > > block 108749824) Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device
> > > dm-8): ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset
> > > 8388608 size 7278592 starting block 108750080) Feb 19 11:28:24 thebe
> > > kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121
> > > writing to inode 33196503 (offset 8388608 size 7278592 starting block
> > > 108750336) Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb] UNKNOWN
> > > Result: hostbyte=0x00 driverbyte=0x08 Feb 19 11:29:10 thebe kernel: sd
> > > 6:0:0:0: [sdb] Sense Key : 0x5 [current] Feb 19 11:29:10 thebe kernel: sd
> > > 6:0:0:0: [sdb] ASC=0x24 ASCQ=0x0 Feb 19 11:29:10 thebe kernel: sd
> > > 6:0:0:0: [sdb] CDB: Feb 19 11:29:10 thebe kernel: cdb[0]=0x2a: 2a 00 20
> > > 44 89 17 00 20 50 00 Feb 19 11:29:10 thebe kernel: blk_update_request:
> > > critical target error, dev sdb, sector 541362455 Feb 19 11:29:10 thebe
> > > kernel: Buffer I/O error on dev dm-8, logical block 66621731, lost sync
> > > page write Feb 19 11:29:10 thebe kernel: Aborting journal on device
> > > dm-8-8. Feb 19 11:29:10 thebe kernel: EXT4-fs error (device dm-8):
> > > ext4_journal_check_start:56: Detected aborted journal Feb 19 11:29:10
> > > thebe kernel: EXT4-fs (dm-8): Remounting filesystem read-only Feb 19
> > > 11:29:20 thebe kernel: EXT4-fs error (device dm-8): ext4_put_super:780:
> > > Couldn't clean up the journal
> > > 
> > > [0] https://lkml.org/lkml/2015/2/19/91
> > 
> > Sense key 0x5 ASC/ASCQ 0x24 0x00 is ILLEGAL REQUEST, INVALID FIELD IN
> > CDB.  The CDB was 2A 00 34 5B 07 FF 00 2F 88 00, which is a WRITE_10
> > to LBA 878381055 with a length of 12168 blocks (a little less than 6MB).
> > It looks like this is within the reported capacity of the device, and
> > there are no other bits set in the CDB.
> > 
> > Looks like you could get this error if RWWP (reject without write
> > protection) is set in the control mode page.  I don't see any messages
> > about the protection type, though.  What does sysfs report?
> 
> Is that what you are interested in?
> 
> # cat protection_mode protection_type 
> none
> 0
> 
> In case it matters: The iSCSI device is LUKS encrypted, that is why device
> mapper shows up.
> 
> I removed the discard option from filesystem's default mount option, but
> that brings no difference except the message is not printed.

It is most likely the device that is returning the error, there is a
place in the iSCSI Initiator that generates an ILLEGAL REQUEST sense,
but it is not the same ASC/ASCQ.

There was this change:

commit bcdb247c6b6a1f3e72b9b787b73f47dd509d17ec
Author: Martin K. Petersen <martin.petersen@oracle.com>
Date:   Tue Jun 3 18:45:51 2014 -0400

    sd: Limit transfer length
    
    Until now the per-command transfer length has exclusively been gated by
    the max_sectors parameter in the scsi_host template. Given that the size
    of this parameter has been bumped to an unsigned int we have to be
    careful not to exceed the target device's capabilities.
    
    If the if the device specifies a Maximum Transfer Length in the Block
    Limits VPD we'll use that value. Otherwise we'll use 0xffffffff for
    devices that have use_16_for_rw set and 0xffff for the rest. We then
    combine the chosen disk limit with max_sectors in the host template. The
    smaller of the two will be used to set the max_hw_sectors queue limit.
    
    Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
    Reviewed-by: Ewan D. Milne <emilne@redhat.com>
    Signed-off-by: Christoph Hellwig <hch@lst.de>

What is the value of max_sectors_kb and queue_max_sectors_kb in sysfs
for the device?  Is it different than what is reported on 3.18?

Does your target support the Block Limits VPD (page B0)?  (i.e. can
you run "sg_inq /dev/sda -p bl" from the sg3_utils package?)

-Ewan

Re: iSCSI regression with linux 3.9 and 4.0

[Ewan Milne]

On Fri, 2015-03-20 at 11:04 -0400, Ewan Milne wrote:
> Does your target support the Block Limits VPD (page B0)?  (i.e. can
> you run "sg_inq /dev/sda -p bl" from the sg3_utils package?)
> 

I meant /dev/sdb, sorry.

Re: iSCSI regression with linux 3.9 and 4.0

[Christian Hesse]

[-- Attachment #1: Type: text/plain, Size: 4458 bytes --]

Ewan Milne <emilne@redhat.com> on Fri, 2015/03/20 11:04:
> On Fri, 2015-03-20 at 15:31 +0100, Christian Hesse wrote:
> > Ewan Milne <emilne@redhat.com> on Fri, 2015/03/20 09:51:
> > > On Fri, 2015-03-20 at 13:57 +0100, Christian Hesse wrote:
> > > > Hello everybody!
> > > > 
> > > > I reported this issue at LKML [0] but received no answer. Hopefully
> > > > linux-scsi is a better place...
> > > > 
> > > > Beginning with linux 3.19 I see an iSCSI regressen. This works
> > > > perfectly with linux 3.18.x (tested with 3.18.6) and before. Effected
> > > > kernels I tested are 3.19.0, 3.19.2 and 4.0rc4.r199.gb314aca.
> > > > 
> > > > The logs tell the story:
> > > > 
> > > > [snip log]
> > > > 
> > > > [0] https://lkml.org/lkml/2015/2/19/91
> > > 
> > > Sense key 0x5 ASC/ASCQ 0x24 0x00 is ILLEGAL REQUEST, INVALID FIELD IN
> > > CDB.  The CDB was 2A 00 34 5B 07 FF 00 2F 88 00, which is a WRITE_10
> > > to LBA 878381055 with a length of 12168 blocks (a little less than 6MB).
> > > It looks like this is within the reported capacity of the device, and
> > > there are no other bits set in the CDB.
> > > 
> > > Looks like you could get this error if RWWP (reject without write
> > > protection) is set in the control mode page.  I don't see any messages
> > > about the protection type, though.  What does sysfs report?
> > 
> > Is that what you are interested in?
> > 
> > # cat protection_mode protection_type 
> > none
> > 0
> > 
> > In case it matters: The iSCSI device is LUKS encrypted, that is why device
> > mapper shows up.
> > 
> > I removed the discard option from filesystem's default mount option, but
> > that brings no difference except the message is not printed.
> 
> It is most likely the device that is returning the error, there is a
> place in the iSCSI Initiator that generates an ILLEGAL REQUEST sense,
> but it is not the same ASC/ASCQ.
> 
> There was this change:
> 
> commit bcdb247c6b6a1f3e72b9b787b73f47dd509d17ec
> Author: Martin K. Petersen <martin.petersen@oracle.com>
> Date:   Tue Jun 3 18:45:51 2014 -0400
> 
>     sd: Limit transfer length
>     
>     Until now the per-command transfer length has exclusively been gated by
>     the max_sectors parameter in the scsi_host template. Given that the size
>     of this parameter has been bumped to an unsigned int we have to be
>     careful not to exceed the target device's capabilities.
>     
>     If the if the device specifies a Maximum Transfer Length in the Block
>     Limits VPD we'll use that value. Otherwise we'll use 0xffffffff for
>     devices that have use_16_for_rw set and 0xffff for the rest. We then
>     combine the chosen disk limit with max_sectors in the host template. The
>     smaller of the two will be used to set the max_hw_sectors queue limit.
>     
>     Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
>     Reviewed-by: Ewan D. Milne <emilne@redhat.com>
>     Signed-off-by: Christoph Hellwig <hch@lst.de>
> 
> What is the value of max_sectors_kb and queue_max_sectors_kb in sysfs
> for the device?  Is it different than what is reported on 3.18?

I found 'max_sectors_kb' which is inside in directory called 'queue'. Is that
the value you asked for?

for 4.0 git:

# cat max_sectors_kb
32767

for 3.18.6:

# cat max_sectors_kb
512

> Does your target support the Block Limits VPD (page B0)?  (i.e. can
> you run "sg_inq /dev/sda -p bl" from the sg3_utils package?)

This does not differ for different kernels. I think this is expected.

# sg_inq /dev/sdb -p bl
VPD INQUIRY: Block limits page (SBC)
  Maximum compare and write length: 1 blocks
  Optimal transfer length granularity: 1 blocks
  Maximum transfer length: 4294967295 blocks
  Optimal transfer length: 4294967295 blocks
  Maximum prefetch, xdread, xdwrite transfer length: 0 blocks
  Maximum unmap LBA count: 8388607
  Maximum unmap block descriptor count: 1
  Optimal unmap granularity: 16383
  Unmap granularity alignment valid: 0
  Unmap granularity alignment: 0
  Maximum write same length: 0xffffffff blocks
  Maximum atomic transfer length: 0
  Atomic alignment: 0
  Atomic transfer length granularity: 0
-- 
main(a){char*c=/*    Schoene Gruesse                         */"B?IJj;MEH"
"CX:;",b;for(a/*    Chris           get my mail address:    */=0;b=c[a++];)
putchar(b-1/(/*               gcc -o sig sig.c && ./sig    */b/42*2-3)*42);}

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: iSCSI regression with linux 3.9 and 4.0

[Ewan Milne]

On Fri, 2015-03-20 at 16:24 +0100, Christian Hesse wrote:
> Ewan Milne <emilne@redhat.com> on Fri, 2015/03/20 11:04:
> > On Fri, 2015-03-20 at 15:31 +0100, Christian Hesse wrote:
> > > Ewan Milne <emilne@redhat.com> on Fri, 2015/03/20 09:51:
> > > > On Fri, 2015-03-20 at 13:57 +0100, Christian Hesse wrote:
> > > > > Hello everybody!
> > > > > 
> > > > > I reported this issue at LKML [0] but received no answer. Hopefully
> > > > > linux-scsi is a better place...
> > > > > 
> > > > > Beginning with linux 3.19 I see an iSCSI regressen. This works
> > > > > perfectly with linux 3.18.x (tested with 3.18.6) and before. Effected
> > > > > kernels I tested are 3.19.0, 3.19.2 and 4.0rc4.r199.gb314aca.
> > > > > 
> > > > > The logs tell the story:
> > > > > 
> > > > > [snip log]
> > > > > 
> > > > > [0] https://lkml.org/lkml/2015/2/19/91
> > > > 
> > > > Sense key 0x5 ASC/ASCQ 0x24 0x00 is ILLEGAL REQUEST, INVALID FIELD IN
> > > > CDB.  The CDB was 2A 00 34 5B 07 FF 00 2F 88 00, which is a WRITE_10
> > > > to LBA 878381055 with a length of 12168 blocks (a little less than 6MB).
> > > > It looks like this is within the reported capacity of the device, and
> > > > there are no other bits set in the CDB.
> > > > 
> > > > Looks like you could get this error if RWWP (reject without write
> > > > protection) is set in the control mode page.  I don't see any messages
> > > > about the protection type, though.  What does sysfs report?
> > > 
> > > Is that what you are interested in?
> > > 
> > > # cat protection_mode protection_type 
> > > none
> > > 0
> > > 
> > > In case it matters: The iSCSI device is LUKS encrypted, that is why device
> > > mapper shows up.
> > > 
> > > I removed the discard option from filesystem's default mount option, but
> > > that brings no difference except the message is not printed.
> > 
> > It is most likely the device that is returning the error, there is a
> > place in the iSCSI Initiator that generates an ILLEGAL REQUEST sense,
> > but it is not the same ASC/ASCQ.
> > 
> > There was this change:
> > 
> > commit bcdb247c6b6a1f3e72b9b787b73f47dd509d17ec
> > Author: Martin K. Petersen <martin.petersen@oracle.com>
> > Date:   Tue Jun 3 18:45:51 2014 -0400
> > 
> >     sd: Limit transfer length
> >     
> >     Until now the per-command transfer length has exclusively been gated by
> >     the max_sectors parameter in the scsi_host template. Given that the size
> >     of this parameter has been bumped to an unsigned int we have to be
> >     careful not to exceed the target device's capabilities.
> >     
> >     If the if the device specifies a Maximum Transfer Length in the Block
> >     Limits VPD we'll use that value. Otherwise we'll use 0xffffffff for
> >     devices that have use_16_for_rw set and 0xffff for the rest. We then
> >     combine the chosen disk limit with max_sectors in the host template. The
> >     smaller of the two will be used to set the max_hw_sectors queue limit.
> >     
> >     Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
> >     Reviewed-by: Ewan D. Milne <emilne@redhat.com>
> >     Signed-off-by: Christoph Hellwig <hch@lst.de>
> > 
> > What is the value of max_sectors_kb and queue_max_sectors_kb in sysfs
> > for the device?  Is it different than what is reported on 3.18?
> 
> I found 'max_sectors_kb' which is inside in directory called 'queue'. Is that
> the value you asked for?
> 
> for 4.0 git:
> 
> # cat max_sectors_kb
> 32767

If you change max_sectors_kb to a lower value (e.g. 512) can you get the
device to work?

There is a max_hw_sectors_kb value but you can't change it.  Is it
32768 also for 4.0?

Your device reports a maximum transfer length of 2^32-1 blocks but
I suspect that it might not be actually able to do that.  I don't see
what else would be causing the error.  Maybe there is a transport
limitation that is getting in the way?

-Ewan

> 
> for 3.18.6:
> 
> # cat max_sectors_kb
> 512
> 
> > Does your target support the Block Limits VPD (page B0)?  (i.e. can
> > you run "sg_inq /dev/sda -p bl" from the sg3_utils package?)
> 
> This does not differ for different kernels. I think this is expected.
> 
> # sg_inq /dev/sdb -p bl
> VPD INQUIRY: Block limits page (SBC)
>   Maximum compare and write length: 1 blocks
>   Optimal transfer length granularity: 1 blocks
>   Maximum transfer length: 4294967295 blocks
>   Optimal transfer length: 4294967295 blocks
>   Maximum prefetch, xdread, xdwrite transfer length: 0 blocks
>   Maximum unmap LBA count: 8388607
>   Maximum unmap block descriptor count: 1
>   Optimal unmap granularity: 16383
>   Unmap granularity alignment valid: 0
>   Unmap granularity alignment: 0
>   Maximum write same length: 0xffffffff blocks
>   Maximum atomic transfer length: 0
>   Atomic alignment: 0
>   Atomic transfer length granularity: 0

Re: iSCSI regression with linux 3.9 and 4.0

[Christian Hesse]

[-- Attachment #1: Type: text/plain, Size: 1237 bytes --]

Ewan Milne <emilne@redhat.com> on Fri, 2015/03/20 11:46:
> On Fri, 2015-03-20 at 16:24 +0100, Christian Hesse wrote:
> > I found 'max_sectors_kb' which is inside in directory called 'queue'. Is
> > that the value you asked for?
> > 
> > for 4.0 git:
> > 
> > # cat max_sectors_kb
> > 32767
> 
> If you change max_sectors_kb to a lower value (e.g. 512) can you get the
> device to work?

I will check that on monday. Sitting behind a slow dial up connection right
now...
 
> There is a max_hw_sectors_kb value but you can't change it.  Is it
> 32768 also for 4.0?

# cat max_hw_sectors_kb
32767

It's 3276*7*, not 32768.

> Your device reports a maximum transfer length of 2^32-1 blocks but
> I suspect that it might not be actually able to do that.  I don't see
> what else would be causing the error.  Maybe there is a transport
> limitation that is getting in the way?

What kind of transport limitation can that be? Network MTU and friends should
no be an issue, no?
-- 
main(a){char*c=/*    Schoene Gruesse                         */"B?IJj;MEH"
"CX:;",b;for(a/*    Chris           get my mail address:    */=0;b=c[a++];)
putchar(b-1/(/*               gcc -o sig sig.c && ./sig    */b/42*2-3)*42);}

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: iSCSI regression with linux 3.9 and 4.0

[Christian Hesse]

[-- Attachment #1: Type: text/plain, Size: 727 bytes --]

Christian Hesse <list@eworm.de> on Fri, 2015/03/20 18:59:
> Ewan Milne <emilne@redhat.com> on Fri, 2015/03/20 11:46:
> > On Fri, 2015-03-20 at 16:24 +0100, Christian Hesse wrote:
> > > I found 'max_sectors_kb' which is inside in directory called 'queue'. Is
> > > that the value you asked for?
> > > 
> > > for 4.0 git:
> > > 
> > > # cat max_sectors_kb
> > > 32767
> > 
> > If you change max_sectors_kb to a lower value (e.g. 512) can you get the
> > device to work?
> 
> I will check that on monday. Sitting behind a slow dial up connection right
> now...

No, it still fails the same way.

After the error it is back to a value of 32767. Is this because of a tried
reconnect?
-- 
Best regards,
Chris

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: iSCSI regression with linux 3.9 and 4.0

[Christian Hesse]

[-- Attachment #1: Type: text/plain, Size: 8947 bytes --]

Ewan Milne <emilne@redhat.com> on Fri, 2015/03/20 11:04:
> On Fri, 2015-03-20 at 15:31 +0100, Christian Hesse wrote:
> > Ewan Milne <emilne@redhat.com> on Fri, 2015/03/20 09:51:
> > > On Fri, 2015-03-20 at 13:57 +0100, Christian Hesse wrote:
> > > > Hello everybody!
> > > > 
> > > > I reported this issue at LKML [0] but received no answer. Hopefully
> > > > linux-scsi is a better place...
> > > > 
> > > > Beginning with linux 3.19 I see an iSCSI regressen. This works
> > > > perfectly with linux 3.18.x (tested with 3.18.6) and before. Effected
> > > > kernels I tested are 3.19.0, 3.19.2 and 4.0rc4.r199.gb314aca.
> > > > 
> > > > The logs tell the story:
> > > > 
> > > > Feb 19 11:26:49 thebe kernel: scsi host6: iSCSI Initiator over TCP/IP
> > > > Feb 19 11:26:49 thebe kernel: scsi 6:0:0:0: Direct-Access     QNAP
> > > > iSCSI Storage    4.0  PQ: 0 ANSI: 5 Feb 19 11:26:49 thebe kernel: sd
> > > > 6:0:0:0: [sdb] 1073741824 512-byte logical blocks: (549 GB/512 GiB)
> > > > Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Write Protect is off
> > > > Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Mode Sense: 2f 00 00
> > > > 00 Feb 19 11:26:49 thebe kernel: sd 6:0:0:0: [sdb] Write cache:
> > > > disabled, read cache: enabled, doesn't support DPO or FUA Feb 19
> > > > 11:26:49 thebe kernel: sdb: unknown partition table Feb 19 11:26:49
> > > > thebe kernel: sd 6:0:0:0: [sdb] Attached SCSI disk Feb 19 11:26:49
> > > > thebe iscsid[10804]: Connection1:0 to [target:
> > > > iqn.2004-04.com.qnap:ts-859:iscsi.xxxxxxx.c40a18, portal:
> > > > xx.xx.xx.xx,3260] through [iface: default] is operational now Feb 19
> > > > 11:26:57 thebe kernel:  sdb: unknown partition table Feb 19 11:28:20
> > > > thebe kernel: EXT4-fs (dm-8): mounting with "discard" option, but the
> > > > device does not support discard Feb 19 11:28:20 thebe kernel: EXT4-fs
> > > > (dm-8): mounted filesystem with ordered data mode. Opts: (null) Feb 19
> > > > 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] UNKNOWN Result: hostbyte=0x00
> > > > driverbyte=0x08 Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] Sense
> > > > Key : 0x5 [current] Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb]
> > > > ASC=0x24 ASCQ=0x0 Feb 19 11:28:24 thebe kernel: sd 6:0:0:0: [sdb] CDB:
> > > > Feb 19 11:28:24 thebe kernel: cdb[0]=0x2a: 2a 00 34 5b 07 ff 00 2f 88
> > > > 00 Feb 19 11:28:24 thebe kernel: blk_update_request: critical target
> > > > error, dev sdb, sector 878381055 Feb 19 11:28:24 thebe kernel:
> > > > EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error -121
> > > > writing to inode 33196503 (offset 8388608 size 7278592 starting block
> > > > 108749056) Feb 19 11:28:24 thebe kernel: Buffer I/O error on device
> > > > dm-8, logical block 108749056 Feb 19 11:28:24 thebe kernel: Buffer
> > > > I/O error on device dm-8, logical block 108749057 Feb 19 11:28:24
> > > > thebe kernel: Buffer I/O error on device dm-8, logical block
> > > > 108749058 Feb 19 11:28:24 thebe kernel: Buffer I/O error on device
> > > > dm-8, logical block 108749059 Feb 19 11:28:24 thebe kernel: Buffer
> > > > I/O error on device dm-8, logical block 108749060 Feb 19 11:28:24
> > > > thebe kernel: Buffer I/O error on device dm-8, logical block
> > > > 108749061 Feb 19 11:28:24 thebe kernel: Buffer I/O error on device
> > > > dm-8, logical block 108749062 Feb 19 11:28:24 thebe kernel: Buffer
> > > > I/O error on device dm-8, logical block 108749063 Feb 19 11:28:24
> > > > thebe kernel: Buffer I/O error on device dm-8, logical block
> > > > 108749064 Feb 19 11:28:24 thebe kernel: Buffer I/O error on device
> > > > dm-8, logical block 108749065 Feb 19 11:28:24 thebe kernel: EXT4-fs
> > > > warning (device dm-8): ext4_end_bio:317: I/O error -121 writing to
> > > > inode 33196503 (offset 8388608 size 7278592 starting block 108749312)
> > > > Feb 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8):
> > > > ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset
> > > > 8388608 size 7278592 starting block 108749568) Feb 19 11:28:24 thebe
> > > > kernel: EXT4-fs warning (device dm-8): ext4_end_bio:317: I/O error
> > > > -121 writing to inode 33196503 (offset 8388608 size 7278592 starting
> > > > block 108749824) Feb 19 11:28:24 thebe kernel: EXT4-fs warning
> > > > (device dm-8): ext4_end_bio:317: I/O error -121 writing to inode
> > > > 33196503 (offset 8388608 size 7278592 starting block 108750080) Feb
> > > > 19 11:28:24 thebe kernel: EXT4-fs warning (device dm-8):
> > > > ext4_end_bio:317: I/O error -121 writing to inode 33196503 (offset
> > > > 8388608 size 7278592 starting block 108750336) Feb 19 11:29:10 thebe
> > > > kernel: sd 6:0:0:0: [sdb] UNKNOWN Result: hostbyte=0x00
> > > > driverbyte=0x08 Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb] Sense
> > > > Key : 0x5 [current] Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb]
> > > > ASC=0x24 ASCQ=0x0 Feb 19 11:29:10 thebe kernel: sd 6:0:0:0: [sdb]
> > > > CDB: Feb 19 11:29:10 thebe kernel: cdb[0]=0x2a: 2a 00 20 44 89 17 00
> > > > 20 50 00 Feb 19 11:29:10 thebe kernel: blk_update_request: critical
> > > > target error, dev sdb, sector 541362455 Feb 19 11:29:10 thebe kernel:
> > > > Buffer I/O error on dev dm-8, logical block 66621731, lost sync page
> > > > write Feb 19 11:29:10 thebe kernel: Aborting journal on device
> > > > dm-8-8. Feb 19 11:29:10 thebe kernel: EXT4-fs error (device dm-8):
> > > > ext4_journal_check_start:56: Detected aborted journal Feb 19 11:29:10
> > > > thebe kernel: EXT4-fs (dm-8): Remounting filesystem read-only Feb 19
> > > > 11:29:20 thebe kernel: EXT4-fs error (device dm-8):
> > > > ext4_put_super:780: Couldn't clean up the journal
> > > > 
> > > > [0] https://lkml.org/lkml/2015/2/19/91
> > > 
> > > Sense key 0x5 ASC/ASCQ 0x24 0x00 is ILLEGAL REQUEST, INVALID FIELD IN
> > > CDB.  The CDB was 2A 00 34 5B 07 FF 00 2F 88 00, which is a WRITE_10
> > > to LBA 878381055 with a length of 12168 blocks (a little less than 6MB).
> > > It looks like this is within the reported capacity of the device, and
> > > there are no other bits set in the CDB.
> > > 
> > > Looks like you could get this error if RWWP (reject without write
> > > protection) is set in the control mode page.  I don't see any messages
> > > about the protection type, though.  What does sysfs report?
> > 
> > Is that what you are interested in?
> > 
> > # cat protection_mode protection_type 
> > none
> > 0
> > 
> > In case it matters: The iSCSI device is LUKS encrypted, that is why device
> > mapper shows up.
> > 
> > I removed the discard option from filesystem's default mount option, but
> > that brings no difference except the message is not printed.
> 
> It is most likely the device that is returning the error, there is a
> place in the iSCSI Initiator that generates an ILLEGAL REQUEST sense,
> but it is not the same ASC/ASCQ.
> 
> There was this change:
> 
> commit bcdb247c6b6a1f3e72b9b787b73f47dd509d17ec
> Author: Martin K. Petersen <martin.petersen@oracle.com>
> Date:   Tue Jun 3 18:45:51 2014 -0400
> 
>     sd: Limit transfer length
>     
>     Until now the per-command transfer length has exclusively been gated by
>     the max_sectors parameter in the scsi_host template. Given that the size
>     of this parameter has been bumped to an unsigned int we have to be
>     careful not to exceed the target device's capabilities.
>     
>     If the if the device specifies a Maximum Transfer Length in the Block
>     Limits VPD we'll use that value. Otherwise we'll use 0xffffffff for
>     devices that have use_16_for_rw set and 0xffff for the rest. We then
>     combine the chosen disk limit with max_sectors in the host template. The
>     smaller of the two will be used to set the max_hw_sectors queue limit.
>     
>     Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
>     Reviewed-by: Ewan D. Milne <emilne@redhat.com>
>     Signed-off-by: Christoph Hellwig <hch@lst.de>
> 
> What is the value of max_sectors_kb and queue_max_sectors_kb in sysfs
> for the device?  Is it different than what is reported on 3.18?

I reverted these two commits:

commit 3a9794d32984b67a6d8992226918618f0e51e5d5
Author: Brian King <brking@linux.vnet.ibm.com>
Date:   Thu Jan 29 15:54:40 2015 -0600

    sd: Fix max transfer length for 4k disks

commit bcdb247c6b6a1f3e72b9b787b73f47dd509d17ec
Author: Martin K. Petersen <martin.petersen@oracle.com>
Date:   Tue Jun 3 18:45:51 2014 -0400

    sd: Limit transfer length
 
iSCSI device still fails, max_sectors_kb is still set to 32767. Looks like
there is another change that introduced the regression.
-- 
main(a){char*c=/*    Schoene Gruesse                         */"B?IJj;MEH"
"CX:;",b;for(a/*    Chris           get my mail address:    */=0;b=c[a++];)
putchar(b-1/(/*               gcc -o sig sig.c && ./sig    */b/42*2-3)*42);}

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 473 bytes --]

Re: iSCSI regression with linux 3.9 and 4.0

[Ewan Milne]

On Mon, 2015-03-23 at 09:53 +0100, Christian Hesse wrote:
> 
> I reverted these two commits:
> 
> commit 3a9794d32984b67a6d8992226918618f0e51e5d5
> Author: Brian King <brking@linux.vnet.ibm.com>
> Date:   Thu Jan 29 15:54:40 2015 -0600
> 
>     sd: Fix max transfer length for 4k disks
> 
> commit bcdb247c6b6a1f3e72b9b787b73f47dd509d17ec
> Author: Martin K. Petersen <martin.petersen@oracle.com>
> Date:   Tue Jun 3 18:45:51 2014 -0400
> 
>     sd: Limit transfer length
>  
> iSCSI device still fails, max_sectors_kb is still set to 32767. Looks like
> there is another change that introduced the regression.

OK, as Mike Christie suggested, see if you can get a wireshark/tcpdump
trace with both the 3.18 (working) and 3.19 (failing) kernels.

-Ewan

Re: iSCSI regression with linux 3.9 and 4.0

[Mike Christie]

On 03/20/2015 07:57 AM, Christian Hesse wrote:
> Hello everybody!
> 
> I reported this issue at LKML [0] but received no answer. Hopefully
> linux-scsi is a better place...
> 
> Beginning with linux 3.19 I see an iSCSI regressen. This works perfectly with
> linux 3.18.x (tested with 3.18.6) and before. Effected kernels I tested are
> 3.19.0, 3.19.2 and 4.0rc4.r199.gb314aca.
> 

Could you take a wrireshark/tcpdump trace when running 3.18 and 3.19? We
can then see what bits are different and narrow down the patch/change.