From: Tejas Rao <raot@bnl.gov>
To: dm-devel@redhat.com
Cc: Rao Tejas <raot@bnl.gov>
Subject: Add. Sense: Data synchronization mark error
Date: Thu, 14 Jan 2016 15:43:49 -0500 [thread overview]
Message-ID: <56980885.2040506@bnl.gov> (raw)
We are using linux raid on top of multipath devices (each jbod disk has
two paths).
Usually medium erros are handled as below. See this bug for a similar
problem but fixed in RHEL6.
https://bugzilla.redhat.com/show_bug.cgi?id=516170
Jan 12 02:15:59 kernel: sd 8:0:21:0: [sdcf] Unhandled sense code
Jan 12 02:15:59 kernel: sd 8:0:21:0: [sdcf] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
Jan 12 02:15:59 kernel: sd 8:0:21:0: [sdcf] Sense Key : Medium Error
[current] [descriptor]
Jan 12 02:15:59 kernel: Descriptor sense data with sense descriptors
(in hex):
Jan 12 02:15:59 kernel: 72 03 11 00 00 00 00 34 00 0a 80 00 00
00 00 01
Jan 12 02:15:59 kernel: cd e3 86 90 01 0a 00 00 00 00 00 00 81
03 01 00
Jan 12 02:15:59 kernel: 02 06 00 00 80 00 ff 00 03 02 00 86 80
0e 00 00
Jan 12 02:15:59 kernel: 00 00 00 00 00 00 00 00 00 00 00 00
Jan 12 02:15:59 kernel: sd 8:0:21:0: [sdcf] Add. Sense: Unrecovered
read error
Jan 12 02:15:59 kernel: sd 8:0:21:0: [sdcf] CDB: Read(16): 88 00 00 00
00 01 cd e3 86 00 00 00 01 00 00 00
Jan 12 02:16:02 kernel: sd 7:0:21:0: [sdx] Unhandled sense code
Jan 12 02:16:02 kernel: sd 7:0:21:0: [sdx] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
Jan 12 02:16:02 kernel: sd 7:0:21:0: [sdx] Sense Key : Medium Error
[current] [descriptor]
Jan 12 02:16:02 kernel: Descriptor sense data with sense descriptors
(in hex):
Jan 12 02:16:02 kernel: 72 03 11 00 00 00 00 34 00 0a 80 00 00
00 00 01
Jan 12 02:16:02 kernel: cd e3 86 90 01 0a 00 00 00 00 00 00 81
03 01 00
Jan 12 02:16:02 kernel: 02 06 00 00 80 00 ff 00 03 02 00 86 80
0e 00 00
Jan 12 02:16:02 kernel: 00 00 00 00 00 00 00 00 00 00 00 00
Jan 12 02:16:02 kernel: sd 7:0:21:0: [sdx] Add. Sense: Unrecovered read
error
Jan 12 02:16:02 kernel: sd 7:0:21:0: [sdx] CDB: Read(16): 88 00 00 00
00 01 cd e3 86 90 00 00 00 70 00 00
Jan 12 02:16:03 kernel: md/raid:md3: read error corrected (8 sectors at
7749205728 on dm-22)
Jan 12 02:16:03 kernel: md/raid:md3: read error corrected (8 sectors at
7749205736 on dm-22)
Jan 12 02:16:03 kernel: md/raid:md3: read error corrected (8 sectors at
7749205744 on dm-22)
Jan 12 02:16:03 kernel: md/raid:md3: read error corrected (8 sectors at
7749205752 on dm-22)
This is all fine and dandy.
We had a case as below and this continued repeatedly for 4 hours until I
logged in and manually failed both paths sdx and sdcf. - (sdx and sdcf
are the same drive). The filesystem running on md3 was hung.
Jan 12 02:38:48 kernel: sd 7:0:21:0: [sdx] Unhandled sense code
Jan 12 02:38:48 kernel: sd 7:0:21:0: [sdx] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
Jan 12 02:38:48 kernel: sd 7:0:21:0: [sdx] Sense Key : Medium Error
[current] [descriptor]
Jan 12 02:38:48 kernel: Descriptor sense data with sense descriptors (in
hex):
Jan 12 02:38:48 kernel: 72 03 16 00 00 00 00 34 00 0a 80 00 00 00
00 01
Jan 12 02:38:48 kernel: d1 53 bd 98 01 0a 00 00 00 00 00 00 86 01
00 00
Jan 12 02:38:48 kernel: 02 06 00 00 80 00 ff 00 03 02 00 80 80 0e
00 00
Jan 12 02:38:48 kernel: 00 00 00 00 00 00 00 00 00 00 00 00
Jan 12 02:38:48 kernel: sd 7:0:21:0: [sdx] Add. Sense: Data
synchronization mark error
Jan 12 02:38:48 kernel: sd 7:0:21:0: [sdx] CDB: Read(16): 88 00 00 00 00
01 d1 53 bd 98 00 00 00 68 00 00
Jan 12 02:38:48 kernel: device-mapper: multipath: Failing path 65:112.
Jan 12 02:38:48 multipathd: 65:112: mark as failed
Jan 12 02:38:48 multipathd: mpathab: remaining active paths: 1
Jan 12 02:38:52 multipathd: mpathab: sdx - directio checker reports path
is up
Jan 12 02:38:52 multipathd: 65:112: reinstated
Jan 12 02:38:52 multipathd: mpathab: remaining active paths: 2
Jan 12 02:39:04 multipathd: 69:48: mark as failed
Jan 12 02:39:04 multipathd: mpathab: remaining active paths: 1
Jan 12 02:39:04 kernel: sd 8:0:21:0: [sdcf] Unhandled sense code
Jan 12 02:39:04 kernel: sd 8:0:21:0: [sdcf] Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
Jan 12 02:39:04 kernel: sd 8:0:21:0: [sdcf] Sense Key : Medium Error
[current] [descriptor]
Jan 12 02:39:04 kernel: Descriptor sense data with sense descriptors (in
hex):
Jan 12 02:39:04 kernel: 72 03 16 00 00 00 00 34 00 0a 80 00 00 00
00 01
Jan 12 02:39:04 kernel: d1 53 bd 98 01 0a 00 00 00 00 00 00 86 01
00 00
Jan 12 02:39:04 kernel: 02 06 00 00 80 00 ff 00 03 02 00 80 80 0e
00 00
Jan 12 02:39:04 kernel: 00 00 00 00 00 00 00 00 00 00 00 00
Jan 12 02:39:04 kernel: sd 8:0:21:0: [sdcf] Add. Sense: Data
synchronization mark error
Jan 12 02:39:04 kernel: sd 8:0:21:0: [sdcf] CDB: Read(16): 88 00 00 00
00 01 d1 53 bd 98 00 00 00 68 00 00
Jan 12 02:39:04 kernel: device-mapper: multipath: Failing path 69:48.
Jan 12 02:39:05 multipathd: mpathab: sdcf - directio checker reports
path is up
Jan 12 02:39:05 multipathd: 69:48: reinstated
Jan 12 02:39:05 multipathd: mpathab: remaining active paths: 2
[root@ ~]# rpm -qa | grep multi
device-mapper-multipath-libs-0.4.9-72.el6_5.3.x86_64
device-mapper-multipath-0.4.9-72.el6_5.3.x86_64
[root@ ~]# uname -a
Linux 2.6.32-431.23.3.el6.x86_64 #1 SMP Wed Jul 16 06:12:23 EDT 2014
x86_64 x86_64 x86_64 GNU/Linux
Anyone know if there is a workaround or patch for this? What does "Add.
Sense: Data synchronization mark error" mean?
Thanks
next reply other threads:[~2016-01-14 20:43 UTC|newest]
Thread overview: 2+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-01-14 20:43 Tejas Rao [this message]
-- strict thread matches above, loose matches on Subject: below --
2016-01-15 0:02 Add. Sense: Data synchronization mark error Tejas Rao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=56980885.2040506@bnl.gov \
--to=raot@bnl.gov \
--cc=dm-devel@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.