Re: 'Device not ready' issue on mpt2sas since 3.1.10

Re: 'Device not ready' issue on mpt2sas since 3.1.10

[Matthias Prager]

Hello linux-scsi and linux-raid,

I did some further research regarding my problem.
It appears to me the fault does not lie with the mpt2sas driver (not
that I can definitely exclude it), but with the md implementation.

I reproduced what I think is the same issue on a different machine (also
running Vmware ESXi 5 and an LSI 9211-8i in IR mode) with a different
set of hard-drives of the same model. Using systemrescuecd
(2.8.1-beta003) and booting the 64bit 3.4.4 kernel, I issued the
following commands:

1) 'hdparm -y /dev/sda' (to put the hard-drive to sleep)
2) 'mdadm --create /dev/md1 --metadata 1.2 --level=mirror
--raid-devices=2 --name=test1 /dev/sda missing'
3) 'fdisk -l /dev/md127' (for some reason /proc/mdstat indicates the md
is being created as md127)

2) gave me this feedback:
------
mdadm: super1.x cannot open /dev/sda: Device or resource busy
mdadm: /dev/sda is not suitable for this array.
mdadm: create aborted
-------
Even though it says creating aborted it still created md127.

And 3) lead to these lines in dmesg:
-------
[  604.838640] sd 2:0:0:0: [sda] Device not ready
[  604.838645] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[  604.838655] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
[  604.838663] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
initializing command required
[  604.838668] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
20 00
[  604.838680] end_request: I/O error, dev sda, sector 2048
[  604.838688] Buffer I/O error on device md127, logical block 0
[  604.838695] Buffer I/O error on device md127, logical block 1
[  604.838699] Buffer I/O error on device md127, logical block 2
[  604.838702] Buffer I/O error on device md127, logical block 3
[  604.838783] sd 2:0:0:0: [sda] Device not ready
[  604.838785] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[  604.838789] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
[  604.838793] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
initializing command required
[  604.838797] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
08 00
[  604.838805] end_request: I/O error, dev sda, sector 2048
[  604.838808] Buffer I/O error on device md127, logical block 0
[  604.838983] sd 2:0:0:0: [sda] Device not ready
[  604.838986] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[  604.838989] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
[  604.838993] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
initializing command required
[  604.838998] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 57 54 65 d8 00 00
08 00
[  604.839006] end_request: I/O error, dev sda, sector 1465148888
[  604.839009] Buffer I/O error on device md127, logical block 183143355
[  604.839087] sd 2:0:0:0: [sda] Device not ready
[  604.839090] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[  604.839093] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
[  604.839097] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
initializing command required
[  604.839102] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 57 54 65 d8 00 00
08 00
[  604.839110] end_request: I/O error, dev sda, sector 1465148888
[  604.839113] Buffer I/O error on device md127, logical block 183143355
[  604.839271] sd 2:0:0:0: [sda] Device not ready
[  604.839274] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[  604.839278] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
[  604.839282] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
initializing command required
[  604.839286] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
20 00
[  604.839321] end_request: I/O error, dev sda, sector 2048
[  604.839324] Buffer I/O error on device md127, logical block 0
[  604.839330] Buffer I/O error on device md127, logical block 1
[  604.840494] sd 2:0:0:0: [sda] Device not ready
[  604.840497] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
driverbyte=DRIVER_SENSE
[  604.840504] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
[  604.840512] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
initializing command required
[  604.840516] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
08 00
[  604.840526] end_request: I/O error, dev sda, sector 2048
------

This excludes hardware-errors (different physical machine and devices)
as cause and also ext4 which the other system was using as filesystem.
Maybe Neil Brown (who scripts/get_maintainer.pl identified as the
maintainer of the md-code) can make bits and pieces of this. It may well
be this is the same problem but a different error-path - I don't know.

I will try to make the scenario more generic, but I don't have a
non-virtual machine to spare atm. Also please do let me know if I'm
posting this to the wrong lists (linux-scsi and linux-raid) or if there
is anything which might not be helpful with the way I'm reporting this.

Regards,
Matthias Prager

Re: 'Device not ready' issue on mpt2sas since 3.1.10

[Robert Trace]

> I did some further research regarding my problem.
> It appears to me the fault does not lie with the mpt2sas driver (not
> that I can definitely exclude it), but with the md implementation.

I'm actually discovering some of the same issues (LSI 9211-8i w/ SATA
disks), but I've come to a slightly different conclusion.

I noticed that when my SATA disks are on a SATA controller and they spin
down (or are spun down via hdparm -y), then they response to TUR (TEST
UNIT READY) commands with an OK.  Any I/O sent to these disks simply
wait while the disks spin up and then complete as usual.

However, my SATA disks on the SAS controller respond to TUR with the
sense error "Not Ready/Initializing command required".  Any I/O sent to
these disks immediately fails.  You saw this in your logging:

> [  604.838640] sd 2:0:0:0: [sda] Device not ready
> [  604.838645] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> [  604.838655] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
> [  604.838663] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
> initializing command required
> [  604.838668] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
> 20 00
> [  604.838680] end_request: I/O error, dev sda, sector 2048
> [  604.838688] Buffer I/O error on device md127, logical block 0
> [  604.838695] Buffer I/O error on device md127, logical block 1
> [  604.838699] Buffer I/O error on device md127, logical block 2
> [  604.838702] Buffer I/O error on device md127, logical block 3

Sending an explicit START UNIT command to these sleeping disks will wake
them up and then they behave normally.  (BTW, you can issue TURs and
START UNITs via the sg_turs and sg_start commands).

I've reproduced this behavior on the raw disks themselves, no MD layer
involved (although the freak-out by my MD layer is what alerted me to
this issue too... Having your entire array punted the first time you
access it is a little scary :-).  I'm also on raw hardware and I've seen
this behavior on kernels 3.0.33 through 3.4.4.

So, SATA disks respond differently depending on the controller they're
on.  I don't know if this is a SCSI thing, a SAS thing or a
firmware/driver thing for the 9211.

Now, whether or not the MD layer should be assembling arrays from
"failed" disks is, I think, a separate issue.

-- Rob

Re: 'Device not ready' issue on mpt2sas since 3.1.10

[Darrick J. Wong]

On Mon, Jul 09, 2012 at 03:37:09PM -0400, Robert Trace wrote:
> > I did some further research regarding my problem.
> > It appears to me the fault does not lie with the mpt2sas driver (not
> > that I can definitely exclude it), but with the md implementation.
> 
> I'm actually discovering some of the same issues (LSI 9211-8i w/ SATA
> disks), but I've come to a slightly different conclusion.
> 
> I noticed that when my SATA disks are on a SATA controller and they spin
> down (or are spun down via hdparm -y), then they response to TUR (TEST
> UNIT READY) commands with an OK.  Any I/O sent to these disks simply
> wait while the disks spin up and then complete as usual.
> 
> However, my SATA disks on the SAS controller respond to TUR with the
> sense error "Not Ready/Initializing command required".  Any I/O sent to
> these disks immediately fails.  You saw this in your logging:
> 
> > [  604.838640] sd 2:0:0:0: [sda] Device not ready
> > [  604.838645] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
> > driverbyte=DRIVER_SENSE
> > [  604.838655] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
> > [  604.838663] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
> > initializing command required
> > [  604.838668] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
> > 20 00
> > [  604.838680] end_request: I/O error, dev sda, sector 2048
> > [  604.838688] Buffer I/O error on device md127, logical block 0
> > [  604.838695] Buffer I/O error on device md127, logical block 1
> > [  604.838699] Buffer I/O error on device md127, logical block 2
> > [  604.838702] Buffer I/O error on device md127, logical block 3
> 
> Sending an explicit START UNIT command to these sleeping disks will wake
> them up and then they behave normally.  (BTW, you can issue TURs and
> START UNITs via the sg_turs and sg_start commands).
> 
> I've reproduced this behavior on the raw disks themselves, no MD layer
> involved (although the freak-out by my MD layer is what alerted me to
> this issue too... Having your entire array punted the first time you
> access it is a little scary :-).  I'm also on raw hardware and I've seen
> this behavior on kernels 3.0.33 through 3.4.4.
> 
> So, SATA disks respond differently depending on the controller they're
> on.  I don't know if this is a SCSI thing, a SAS thing or a
> firmware/driver thing for the 9211.

I suspect that /sys/devices/<bunch of sas topology here>/manage_start_stop = 0
for the SATA devices hanging off the SAS controller.  Setting that sysfs
attribute to 1 is supposed to enable the SCSI layer to send TUR when it sees
"LU not ready", as well as spin down the drives at suspend/poweroff time.

--D
> 
> Now, whether or not the MD layer should be assembling arrays from
> "failed" disks is, I think, a separate issue.
> 
> -- Rob
> --
> To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Re: 'Device not ready' issue on mpt2sas since 3.1.10

[Robert Trace]

On 07/09/2012 04:45 PM, Darrick J. Wong wrote:
>
> I suspect that /sys/devices/<bunch of sas topology here>/manage_start_stop = 0
> for the SATA devices hanging off the SAS controller.

Yep, looks like you're right.  For my system:

# cat /sys/block/sd?/device/scsi_disk/*/manage_start_stop
1
1
1
1
1
0
0
0
0
0
0
0
0

Those first 5 disks are SATA disks on SATA controllers.  The last 8
disks are SATA disks on the SAS controller.

> Setting that sysfs
> attribute to 1 is supposed to enable the SCSI layer to send TUR when it sees
> "LU not ready", as well as spin down the drives at suspend/poweroff time.

Setting it to 1 doesn't seem to have made any difference, however.

# cat /sys/block/sdm/device/scsi_disk/14\:0\:7\:0/manage_start_stop
0
# echo 1 > /sys/block/sdm/device/scsi_disk/14\:0\:7\:/manage_start_stop
# cat /sys/block/sdm/device/scsi_disk/14\:0\:7\:0/manage_start_stop
1
# hdparm -y /dev/sdm

/dev/sdm:
 issuing standby command
# hdparm -C /dev/sdm

/dev/sdm:
 drive state is:  standby
# dd if=/dev/sdm of=/dev/null bs=512 count=1
dd: reading `/dev/sdm': Input/output error
0+0 records in
0+0 records out
0 bytes (0 B) copied, 0.00117802 s, 0.0 kB/s

... and on the scsi logging side, I see the read(10) to the disk which
immediately returns "Not Ready" and the I/O failure bubbles up the
chain.  And afterwards, the disk is still asleep.

# hdparm -C /dev/sdm

/dev/sdm:
 drive state is:  standby

Also, TURs don't appear to actually wake the disk up (should they?).
The only thing I've found that'll wake the disk up is an explicit START
UNIT command.

-- Rob

Re: 'Device not ready' issue on mpt2sas since 3.1.10

[Matthias Prager]

Am 10.07.2012 00:24, schrieb Robert Trace:
> 
> Also, TURs don't appear to actually wake the disk up (should they?).
> The only thing I've found that'll wake the disk up is an explicit START
> UNIT command.

I haven't checked the scsi logging side, but about the only commands
that wake up the disks are 'smartctl -a /dev/sda' and 'sg_start'
(smartcl maybe issuing a START UNIT command on it's own).

Matthias

Re: 'Device not ready' issue on mpt2sas since 3.1.10

[Darrick J. Wong]

On Mon, Jul 09, 2012 at 06:24:15PM -0400, Robert Trace wrote:
> On 07/09/2012 04:45 PM, Darrick J. Wong wrote:
> >
> > I suspect that /sys/devices/<bunch of sas topology here>/manage_start_stop = 0
> > for the SATA devices hanging off the SAS controller.
> 
> Yep, looks like you're right.  For my system:
> 
> # cat /sys/block/sd?/device/scsi_disk/*/manage_start_stop
> 1
> 1
> 1
> 1
> 1
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 0
> 
> Those first 5 disks are SATA disks on SATA controllers.  The last 8
> disks are SATA disks on the SAS controller.
> 
> > Setting that sysfs
> > attribute to 1 is supposed to enable the SCSI layer to send TUR when it sees
> > "LU not ready", as well as spin down the drives at suspend/poweroff time.
> 
> Setting it to 1 doesn't seem to have made any difference, however.
> 
> # cat /sys/block/sdm/device/scsi_disk/14\:0\:7\:0/manage_start_stop
> 0
> # echo 1 > /sys/block/sdm/device/scsi_disk/14\:0\:7\:/manage_start_stop
> # cat /sys/block/sdm/device/scsi_disk/14\:0\:7\:0/manage_start_stop
> 1
> # hdparm -y /dev/sdm
> 
> /dev/sdm:
>  issuing standby command
> # hdparm -C /dev/sdm
> 
> /dev/sdm:
>  drive state is:  standby
> # dd if=/dev/sdm of=/dev/null bs=512 count=1
> dd: reading `/dev/sdm': Input/output error
> 0+0 records in
> 0+0 records out
> 0 bytes (0 B) copied, 0.00117802 s, 0.0 kB/s
> 
> ... and on the scsi logging side, I see the read(10) to the disk which
> immediately returns "Not Ready" and the I/O failure bubbles up the
> chain.  And afterwards, the disk is still asleep.
> 
> # hdparm -C /dev/sdm
> 
> /dev/sdm:
>  drive state is:  standby
> 
> Also, TURs don't appear to actually wake the disk up (should they?).
> The only thing I've found that'll wake the disk up is an explicit START
> UNIT command.

Sorry, I misspoke, manage_start_stop=1 sends START UNIT, not TUR.  Also, it
only manages spindown/up at suspend/resume time, hence the behavior you see.
The relevant source code is sd_start_stop_device() in drivers/scsi/sd.c.

--D
> 
> -- Rob
> 

Re: 'Device not ready' issue on mpt2sas since 3.1.10

[Matthias Prager]

Am 09.07.2012 21:37, schrieb Robert Trace:
>> I did some further research regarding my problem.
>> It appears to me the fault does not lie with the mpt2sas driver (not
>> that I can definitely exclude it), but with the md implementation.
> 
> I'm actually discovering some of the same issues (LSI 9211-8i w/ SATA
> disks), but I've come to a slightly different conclusion.
> 
> I noticed that when my SATA disks are on a SATA controller and they spin
> down (or are spun down via hdparm -y), then they response to TUR (TEST
> UNIT READY) commands with an OK.  Any I/O sent to these disks simply
> wait while the disks spin up and then complete as usual.
> 
> However, my SATA disks on the SAS controller respond to TUR with the
> sense error "Not Ready/Initializing command required".  Any I/O sent to
> these disks immediately fails.  You saw this in your logging:
> 
>> [  604.838640] sd 2:0:0:0: [sda] Device not ready
>> [  604.838645] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
>> driverbyte=DRIVER_SENSE
>> [  604.838655] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
>> [  604.838663] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
>> initializing command required
>> [  604.838668] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
>> 20 00
>> [  604.838680] end_request: I/O error, dev sda, sector 2048
>> [  604.838688] Buffer I/O error on device md127, logical block 0
>> [  604.838695] Buffer I/O error on device md127, logical block 1
>> [  604.838699] Buffer I/O error on device md127, logical block 2
>> [  604.838702] Buffer I/O error on device md127, logical block 3
> 
> Sending an explicit START UNIT command to these sleeping disks will wake
> them up and then they behave normally.  (BTW, you can issue TURs and
> START UNITs via the sg_turs and sg_start commands).
Thanks for these pointers.

> 
> I've reproduced this behavior on the raw disks themselves, no MD layer
> involved (although the freak-out by my MD layer is what alerted me to
> this issue too... Having your entire array punted the first time you
> access it is a little scary :-).  I'm also on raw hardware and I've seen
> this behavior on kernels 3.0.33 through 3.4.4.
This is interesting - are you sure about 3.0.33? I'm running this kernel
atm for it gives me no trouble (as opposed to >=3.1.10). The SATA disks
are spun up when I access data on them.

> 
> So, SATA disks respond differently depending on the controller they're
> on.  I don't know if this is a SCSI thing, a SAS thing or a
> firmware/driver thing for the 9211.
> 
> Now, whether or not the MD layer should be assembling arrays from
> "failed" disks is, I think, a separate issue.
I realize now in my cases the MD layer behaved correctly.

> 
> -- Rob
> 

Re: 'Device not ready' issue on mpt2sas since 3.1.10

[NeilBrown]

[-- Attachment #1: Type: text/plain, Size: 7177 bytes --]

On Mon, 09 Jul 2012 16:40:15 +0200 Matthias Prager <linux@matthiasprager.de>
wrote:

> Hello linux-scsi and linux-raid,
> 
> I did some further research regarding my problem.
> It appears to me the fault does not lie with the mpt2sas driver (not
> that I can definitely exclude it), but with the md implementation.
> 
> I reproduced what I think is the same issue on a different machine (also
> running Vmware ESXi 5 and an LSI 9211-8i in IR mode) with a different
> set of hard-drives of the same model. Using systemrescuecd
> (2.8.1-beta003) and booting the 64bit 3.4.4 kernel, I issued the
> following commands:
> 
> 1) 'hdparm -y /dev/sda' (to put the hard-drive to sleep)
> 2) 'mdadm --create /dev/md1 --metadata 1.2 --level=mirror
> --raid-devices=2 --name=test1 /dev/sda missing'
> 3) 'fdisk -l /dev/md127' (for some reason /proc/mdstat indicates the md
> is being created as md127)
> 
> 2) gave me this feedback:
> ------
> mdadm: super1.x cannot open /dev/sda: Device or resource busy
> mdadm: /dev/sda is not suitable for this array.
> mdadm: create aborted
> -------
> Even though it says creating aborted it still created md127.

One of my pet peeves in when people interpret the observations wrongly and
then report their interpretation instead of their observation.  However
sometimes it is very hard to separate the two.  You comment above looks
perfectly reasonable and looks like a clean observation and not and
interpretation.  Yet it is an interpretation :-)

The observation would be
   "Even though it says creating abort, md127 was still created".

You see, it wasn't this mdadm that created md127 - it certainly shouldn't
have as you asked it to create md1.

I don't know the exact sequence of events, but something - possibly relating
to the error messages reported below - caused udev to notice /dev/sda.
udev then ran "mdadm -I /dev/sda" and as it had some metadata on it, it
created an array with it.  As the name information in that metadata was
probably "test1" or similar, rather than "1", mdadm didn't know what number
was wanted for the array, so it chose a free high number - 127.

This metadata must have been left over from an earlier experiment.

So it might have been something like.

- you run mdadm (call this mdadm-1).
- mdadm tries to open sda
- driver notices that device is asleep, and wakes it up
- the waking up of the device causes a CHANGE uevent to udev
- this cause udev to run a new mdadm - mdadm-2
- mdadm-2 reads the metadata, sees old metadata, assembled sda in a new md127
- mdadm-1 gets scheduled again, tries to get O_EXCL access to sda and fails, 
  because sda is now part of md127

Clearly undesirable behaviour.  I'm not sure which bit is "wrong".

NeilBrown


> 
> And 3) lead to these lines in dmesg:
> -------
> [  604.838640] sd 2:0:0:0: [sda] Device not ready
> [  604.838645] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> [  604.838655] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
> [  604.838663] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
> initializing command required
> [  604.838668] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
> 20 00
> [  604.838680] end_request: I/O error, dev sda, sector 2048
> [  604.838688] Buffer I/O error on device md127, logical block 0
> [  604.838695] Buffer I/O error on device md127, logical block 1
> [  604.838699] Buffer I/O error on device md127, logical block 2
> [  604.838702] Buffer I/O error on device md127, logical block 3
> [  604.838783] sd 2:0:0:0: [sda] Device not ready
> [  604.838785] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> [  604.838789] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
> [  604.838793] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
> initializing command required
> [  604.838797] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
> 08 00
> [  604.838805] end_request: I/O error, dev sda, sector 2048
> [  604.838808] Buffer I/O error on device md127, logical block 0
> [  604.838983] sd 2:0:0:0: [sda] Device not ready
> [  604.838986] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> [  604.838989] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
> [  604.838993] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
> initializing command required
> [  604.838998] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 57 54 65 d8 00 00
> 08 00
> [  604.839006] end_request: I/O error, dev sda, sector 1465148888
> [  604.839009] Buffer I/O error on device md127, logical block 183143355
> [  604.839087] sd 2:0:0:0: [sda] Device not ready
> [  604.839090] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> [  604.839093] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
> [  604.839097] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
> initializing command required
> [  604.839102] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 57 54 65 d8 00 00
> 08 00
> [  604.839110] end_request: I/O error, dev sda, sector 1465148888
> [  604.839113] Buffer I/O error on device md127, logical block 183143355
> [  604.839271] sd 2:0:0:0: [sda] Device not ready
> [  604.839274] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> [  604.839278] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
> [  604.839282] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
> initializing command required
> [  604.839286] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
> 20 00
> [  604.839321] end_request: I/O error, dev sda, sector 2048
> [  604.839324] Buffer I/O error on device md127, logical block 0
> [  604.839330] Buffer I/O error on device md127, logical block 1
> [  604.840494] sd 2:0:0:0: [sda] Device not ready
> [  604.840497] sd 2:0:0:0: [sda]  Result: hostbyte=DID_OK
> driverbyte=DRIVER_SENSE
> [  604.840504] sd 2:0:0:0: [sda]  Sense Key : Not Ready [current]
> [  604.840512] sd 2:0:0:0: [sda]  Add. Sense: Logical unit not ready,
> initializing command required
> [  604.840516] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 00 08 00 00 00
> 08 00
> [  604.840526] end_request: I/O error, dev sda, sector 2048
> ------
> 
> This excludes hardware-errors (different physical machine and devices)
> as cause and also ext4 which the other system was using as filesystem.
> Maybe Neil Brown (who scripts/get_maintainer.pl identified as the
> maintainer of the md-code) can make bits and pieces of this. It may well
> be this is the same problem but a different error-path - I don't know.
> 
> I will try to make the scenario more generic, but I don't have a
> non-virtual machine to spare atm. Also please do let me know if I'm
> posting this to the wrong lists (linux-scsi and linux-raid) or if there
> is anything which might not be helpful with the way I'm reporting this.
> 
> Regards,
> Matthias Prager
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]

Re: 'Device not ready' issue on mpt2sas since 3.1.10

[Matthias Prager]

Am 10.07.2012 00:08, schrieb NeilBrown:
> On Mon, 09 Jul 2012 16:40:15 +0200 Matthias Prager <linux@matthiasprager.de>
> wrote:
> 
>> Even though it says creating aborted it still created md127.
> 
> One of my pet peeves in when people interpret the observations wrongly and
> then report their interpretation instead of their observation.  However
> sometimes it is very hard to separate the two.  You comment above looks
> perfectly reasonable and looks like a clean observation and not and
> interpretation.  Yet it is an interpretation :-)
> 
> The observation would be
>    "Even though it says creating abort, md127 was still created".
> 
> You see, it wasn't this mdadm that created md127 - it certainly shouldn't
> have as you asked it to create md1.
Sry - I jumped to conclusions without knowing what was actually going on.

> 
> I don't know the exact sequence of events, but something - possibly relating
> to the error messages reported below - caused udev to notice /dev/sda.
> udev then ran "mdadm -I /dev/sda" and as it had some metadata on it, it
> created an array with it.  As the name information in that metadata was
> probably "test1" or similar, rather than "1", mdadm didn't know what number
> was wanted for the array, so it chose a free high number - 127.
> 
> This metadata must have been left over from an earlier experiment.
That is correct (as am just realizing now). There is metadata of an
raid1 array left on the disk even though it was used (for a short time)
with zfs on freebsd before doing these experiments.

> 
> So it might have been something like.
> 
> - you run mdadm (call this mdadm-1).
> - mdadm tries to open sda
> - driver notices that device is asleep, and wakes it up
> - the waking up of the device causes a CHANGE uevent to udev
> - this cause udev to run a new mdadm - mdadm-2
> - mdadm-2 reads the metadata, sees old metadata, assembled sda in a new md127
> - mdadm-1 gets scheduled again, tries to get O_EXCL access to sda and fails, 
>   because sda is now part of md127
> 
> Clearly undesirable behaviour.  I'm not sure which bit is "wrong".
As it turns out mdadm is doing everything right. md127 is actually
already present (though inactive) at boot-time. So mdadm is absolutly
correct in saying sda is busy and refusing to do anything further.

> 
> NeilBrown
> 

The real problem seems to be located in some layer below md, which is
not waking up the disk for any i/o (at all - not even for fdisk -l).

Matthias