* SES Enclosure Management.
@ 2012-02-14 17:30 Robert Woodworth
2012-02-14 20:31 ` NeilBrown
2012-02-14 20:38 ` Joe Landman
0 siblings, 2 replies; 15+ messages in thread
From: Robert Woodworth @ 2012-02-14 17:30 UTC (permalink / raw)
To: linux-raid
Has anyone ever thought of integrating SES managed enclosures into the
kernel RAID system? I briefly looked through the archives and have
not found anything on the topic.
Some HW based RAID controllers do this flawlessly now, there is no
reason why the kernel RAID cannot also. (LSI MegaRAID)
1) When a drive is part if a managed enclosure, the RAID system should
address it by location instead of by enumerated device node. The SES
device in the enclosure can map the physical slot to a physical drive.
The RAID admin (mdamd) should be able to add/fail/identify devices
based on slot.
2) If the RAID system fails a drive, it should notify the SES
management and turn on the fail bit and the fail LED.
3) The RAID system should be able to turn on the 'identify' bit and
LEDs for an array or a single drive.
I'm currently doing firmware on a managed enclosure. Although my
vendor only supports LSI MegaRAID, there is no reason why my
enclosures cannot work in the same manner on a kernel RAID system.
Request for comments...
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 17:30 SES Enclosure Management Robert Woodworth
@ 2012-02-14 20:31 ` NeilBrown
2012-02-14 20:42 ` Joe Landman
2012-02-14 20:46 ` Robert Woodworth
2012-02-14 20:38 ` Joe Landman
1 sibling, 2 replies; 15+ messages in thread
From: NeilBrown @ 2012-02-14 20:31 UTC (permalink / raw)
To: Robert Woodworth; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 2451 bytes --]
On Tue, 14 Feb 2012 10:30:37 -0700 Robert Woodworth
<robertjwoodworth@gmail.com> wrote:
> Has anyone ever thought of integrating SES managed enclosures into the
> kernel RAID system? I briefly looked through the archives and have
> not found anything on the topic.
>
> Some HW based RAID controllers do this flawlessly now, there is no
> reason why the kernel RAID cannot also. (LSI MegaRAID)
>
> 1) When a drive is part if a managed enclosure, the RAID system should
> address it by location instead of by enumerated device node. The SES
> device in the enclosure can map the physical slot to a physical drive.
> The RAID admin (mdamd) should be able to add/fail/identify devices
> based on slot.
Does this just mean that the admin should using names in /dev/disk/by-path/
rather than /dev/sdXX to address devices? What can md or mdadm do to help?
>
> 2) If the RAID system fails a drive, it should notify the SES
> management and turn on the fail bit and the fail LED.
"mdadm --monitor" will run a script on drive failure. This could easily
notify the SES management.
So maybe all we need here is a script to plug in to mdadm... Would you like
to write one?
>
> 3) The RAID system should be able to turn on the 'identify' bit and
> LEDs for an array or a single drive.
Again, it sounds like you just need a script to ask mdadm which devices are
included in a given array, and then do whatever magic is needed to turn on
the light.
It is fairly easy to extra the device list from the output of
mdadm --detail --brief --verbose /dev/md/whatever
but it might be good to make it easier to extra from
mdadm --detail --export /dev/md/whatever
Would you like to write such a script?
>
> I'm currently doing firmware on a managed enclosure. Although my
> vendor only supports LSI MegaRAID, there is no reason why my
> enclosures cannot work in the same manner on a kernel RAID system.
>
>
>
> Request for comments...
It sounds to me like you just need a few scripts to provide some
enclosure-specific functionality. I would be happy to include them in the
mdadm distribution.
Or maybe there is something that I didn't understand??
Thanks,
NeilBrown
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 17:30 SES Enclosure Management Robert Woodworth
2012-02-14 20:31 ` NeilBrown
@ 2012-02-14 20:38 ` Joe Landman
1 sibling, 0 replies; 15+ messages in thread
From: Joe Landman @ 2012-02-14 20:38 UTC (permalink / raw)
To: Robert Woodworth; +Cc: linux-raid
On 02/14/2012 12:30 PM, Robert Woodworth wrote:
> Has anyone ever thought of integrating SES managed enclosures into the
> kernel RAID system? I briefly looked through the archives and have
> not found anything on the topic.
>
> Some HW based RAID controllers do this flawlessly now, there is no
> reason why the kernel RAID cannot also. (LSI MegaRAID)
>
> 1) When a drive is part if a managed enclosure, the RAID system should
> address it by location instead of by enumerated device node. The SES
> device in the enclosure can map the physical slot to a physical drive.
> The RAID admin (mdamd) should be able to add/fail/identify devices
> based on slot.
We've got something like this going now, but the SES interface is
usually through an HBA vendor specific cli tool. Is there somewhere
else we can/should look for SES bits?
>
> 2) If the RAID system fails a drive, it should notify the SES
> management and turn on the fail bit and the fail LED.
Agreed. This said, we've found some backplanes that simply do not work
correctly for SES or SGPIO modes.
>
> 3) The RAID system should be able to turn on the 'identify' bit and
> LEDs for an array or a single drive.
>
> I'm currently doing firmware on a managed enclosure. Although my
> vendor only supports LSI MegaRAID, there is no reason why my
> enclosures cannot work in the same manner on a kernel RAID system.
>
>
>
> Request for comments...
If there are libraries we can hook into to light this up correctly, let
me know. We have some of the elements of this in place via a number of
tools we've developed for our units. But we rely upon an HBA vendor
tool to communicate with the backplane. Would certainly like to talk
"directly" to the backplane. Any hints on how to do this?
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web : http://scalableinformatics.com
http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 20:31 ` NeilBrown
@ 2012-02-14 20:42 ` Joe Landman
2012-02-14 20:53 ` Robert Woodworth
2012-02-14 20:46 ` Robert Woodworth
1 sibling, 1 reply; 15+ messages in thread
From: Joe Landman @ 2012-02-14 20:42 UTC (permalink / raw)
To: NeilBrown; +Cc: Robert Woodworth, linux-raid
On 02/14/2012 03:31 PM, NeilBrown wrote:
> On Tue, 14 Feb 2012 10:30:37 -0700 Robert Woodworth
> <robertjwoodworth@gmail.com> wrote:
>
>> Has anyone ever thought of integrating SES managed enclosures into the
>> kernel RAID system? I briefly looked through the archives and have
>> not found anything on the topic.
>>
>> Some HW based RAID controllers do this flawlessly now, there is no
>> reason why the kernel RAID cannot also. (LSI MegaRAID)
>>
>> 1) When a drive is part if a managed enclosure, the RAID system should
>> address it by location instead of by enumerated device node. The SES
>> device in the enclosure can map the physical slot to a physical drive.
>> The RAID admin (mdamd) should be able to add/fail/identify devices
>> based on slot.
>
> Does this just mean that the admin should using names in /dev/disk/by-path/
> rather than /dev/sdXX to address devices? What can md or mdadm do to help?
>
Not sure on the SES (or SGPIO side), but one of the things we've been
doing has been to create a file with disk placement "coordinates", so as
to map serial number and device to physical location.
>>
>> 2) If the RAID system fails a drive, it should notify the SES
>> management and turn on the fail bit and the fail LED.
>
> "mdadm --monitor" will run a script on drive failure. This could easily
> notify the SES management.
Yes, we are using this now for notifications and logging.
>
> So maybe all we need here is a script to plug in to mdadm... Would you like
> to write one?
>
Just need a "standard" SES (or SGPIO) mechanism to hook into, and we
should be able to support this. Right now we have to work through HBA
scripts.
>>
>> 3) The RAID system should be able to turn on the 'identify' bit and
>> LEDs for an array or a single drive.
>
> Again, it sounds like you just need a script to ask mdadm which devices are
> included in a given array, and then do whatever magic is needed to turn on
> the light.
> It is fairly easy to extra the device list from the output of
> mdadm --detail --brief --verbose /dev/md/whatever
>
> but it might be good to make it easier to extra from
> mdadm --detail --export /dev/md/whatever
>
> Would you like to write such a script?
>
>>
>> I'm currently doing firmware on a managed enclosure. Although my
>> vendor only supports LSI MegaRAID, there is no reason why my
>> enclosures cannot work in the same manner on a kernel RAID system.
>>
>>
>>
>> Request for comments...
>
> It sounds to me like you just need a few scripts to provide some
> enclosure-specific functionality. I would be happy to include them in the
> mdadm distribution.
>
> Or maybe there is something that I didn't understand??
>
> Thanks,
> NeilBrown
>
>
>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web : http://scalableinformatics.com
http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 20:31 ` NeilBrown
2012-02-14 20:42 ` Joe Landman
@ 2012-02-14 20:46 ` Robert Woodworth
2012-02-14 21:16 ` NeilBrown
1 sibling, 1 reply; 15+ messages in thread
From: Robert Woodworth @ 2012-02-14 20:46 UTC (permalink / raw)
To: NeilBrown; +Cc: linux-raid
On 02/14/2012 01:31 PM, NeilBrown wrote:
> On Tue, 14 Feb 2012 10:30:37 -0700 Robert Woodworth
> <robertjwoodworth@gmail.com> wrote:
>
>> Has anyone ever thought of integrating SES managed enclosures into the
>> kernel RAID system? I briefly looked through the archives and have
>> not found anything on the topic.
>>
>> Some HW based RAID controllers do this flawlessly now, there is no
>> reason why the kernel RAID cannot also. (LSI MegaRAID)
>>
>> 1) When a drive is part if a managed enclosure, the RAID system should
>> address it by location instead of by enumerated device node. The SES
>> device in the enclosure can map the physical slot to a physical drive.
>> The RAID admin (mdamd) should be able to add/fail/identify devices
>> based on slot.
> Does this just mean that the admin should using names in /dev/disk/by-path/
> rather than /dev/sdXX to address devices? What can md or mdadm do to help?
I don't like the /by-path/ too well, it is not human readable. I would
propose that the SES module that is already in the base kernel be
expanded to provide a /dev/disk/by-location/<enclosureID>/<slot> ->
/dev/sdXY
If you have ever worked with AIX, all devices have physical locations in
the configurations held by the kernel.
>> 2) If the RAID system fails a drive, it should notify the SES
>> management and turn on the fail bit and the fail LED.
> "mdadm --monitor" will run a script on drive failure. This could easily
> notify the SES management.
>
> So maybe all we need here is a script to plug in to mdadm... Would you like
> to write one?
>
Yes, I thought of that too.
>> 3) The RAID system should be able to turn on the 'identify' bit and
>> LEDs for an array or a single drive.
> Again, it sounds like you just need a script to ask mdadm which devices are
> included in a given array, and then do whatever magic is needed to turn on
> the light.
> It is fairly easy to extra the device list from the output of
> mdadm --detail --brief --verbose /dev/md/whatever
>
> but it might be good to make it easier to extra from
> mdadm --detail --export /dev/md/whatever
>
> Would you like to write such a script?
Yes, I could take that on.
>> I'm currently doing firmware on a managed enclosure. Although my
>> vendor only supports LSI MegaRAID, there is no reason why my
>> enclosures cannot work in the same manner on a kernel RAID system.
>>
>>
>>
>> Request for comments...
> It sounds to me like you just need a few scripts to provide some
> enclosure-specific functionality. I would be happy to include them in the
> mdadm distribution.
>
> Or maybe there is something that I didn't understand??
>
> Thanks,
> NeilBrown
>
>
The one piece that could be included in the kernel level is the display
of /proc/mdstat to show the enclosureID and slot number
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 20:42 ` Joe Landman
@ 2012-02-14 20:53 ` Robert Woodworth
2012-02-14 20:58 ` Joe Landman
2012-02-16 19:00 ` Benjamin ESTRABAUD
0 siblings, 2 replies; 15+ messages in thread
From: Robert Woodworth @ 2012-02-14 20:53 UTC (permalink / raw)
To: Joe Landman; +Cc: NeilBrown, linux-raid
On 02/14/2012 01:42 PM, Joe Landman wrote:
> On 02/14/2012 03:31 PM, NeilBrown wrote:
>> On Tue, 14 Feb 2012 10:30:37 -0700 Robert Woodworth
>> <robertjwoodworth@gmail.com> wrote:
>>
>>> Has anyone ever thought of integrating SES managed enclosures into the
>>> kernel RAID system? I briefly looked through the archives and have
>>> not found anything on the topic.
>>>
>>> Some HW based RAID controllers do this flawlessly now, there is no
>>> reason why the kernel RAID cannot also. (LSI MegaRAID)
>>>
>>> 1) When a drive is part if a managed enclosure, the RAID system should
>>> address it by location instead of by enumerated device node. The SES
>>> device in the enclosure can map the physical slot to a physical drive.
>>> The RAID admin (mdamd) should be able to add/fail/identify devices
>>> based on slot.
>>
>> Does this just mean that the admin should using names in
>> /dev/disk/by-path/
>> rather than /dev/sdXX to address devices? What can md or mdadm do to
>> help?
>>
>
> Not sure on the SES (or SGPIO side), but one of the things we've been
> doing has been to create a file with disk placement "coordinates", so
> as to map serial number and device to physical location.
>
With real SES managed enclosures, you issue a SCSI command to read SES
Page1 and Page2 to get the details about the drives in any given slot.
This currently works fine in Linux with the sg_utils3 package. From the
command line, 'sg_ses -p 2 /dev/sgXX` where the device is the SES device.
Take a look at your systems, if you see a device at
/sys/class/enclosure/XXXX/ then you have a managed enclosure attached.
>>>
>>> 2) If the RAID system fails a drive, it should notify the SES
>>> management and turn on the fail bit and the fail LED.
>>
>> "mdadm --monitor" will run a script on drive failure. This could easily
>> notify the SES management.
>
> Yes, we are using this now for notifications and logging.
>
>>
>> So maybe all we need here is a script to plug in to mdadm... Would
>> you like
>> to write one?
>>
>
> Just need a "standard" SES (or SGPIO) mechanism to hook into, and we
> should be able to support this. Right now we have to work through HBA
> scripts.
A true managed enclosure has nothing to do with the HBA. A managed
enclosure provides a device on the SCSI bus and you exclusively
communicate with that device regardless of the HBA. Most HW RAIDs (LSI
MegaRAID) will hide the SES device exactly like they hide the physical
disks.
>
>>>
>>> 3) The RAID system should be able to turn on the 'identify' bit and
>>> LEDs for an array or a single drive.
>>
>> Again, it sounds like you just need a script to ask mdadm which
>> devices are
>> included in a given array, and then do whatever magic is needed to
>> turn on
>> the light.
>> It is fairly easy to extra the device list from the output of
>> mdadm --detail --brief --verbose /dev/md/whatever
>>
>> but it might be good to make it easier to extra from
>> mdadm --detail --export /dev/md/whatever
>>
>> Would you like to write such a script?
>>
>>>
>>> I'm currently doing firmware on a managed enclosure. Although my
>>> vendor only supports LSI MegaRAID, there is no reason why my
>>> enclosures cannot work in the same manner on a kernel RAID system.
>>>
>>>
>>>
>>> Request for comments...
>>
>> It sounds to me like you just need a few scripts to provide some
>> enclosure-specific functionality. I would be happy to include them
>> in the
>> mdadm distribution.
>>
>> Or maybe there is something that I didn't understand??
>>
>> Thanks,
>> NeilBrown
>>
>>
>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 20:53 ` Robert Woodworth
@ 2012-02-14 20:58 ` Joe Landman
2012-02-14 21:06 ` Jeff Johnson
2012-02-16 19:00 ` Benjamin ESTRABAUD
1 sibling, 1 reply; 15+ messages in thread
From: Joe Landman @ 2012-02-14 20:58 UTC (permalink / raw)
To: Robert Woodworth; +Cc: NeilBrown, linux-raid
On 02/14/2012 03:53 PM, Robert Woodworth wrote:
> On 02/14/2012 01:42 PM, Joe Landman wrote:
>> On 02/14/2012 03:31 PM, NeilBrown wrote:
>>> On Tue, 14 Feb 2012 10:30:37 -0700 Robert Woodworth
>>> <robertjwoodworth@gmail.com> wrote:
>>>
>>>> Has anyone ever thought of integrating SES managed enclosures into the
>>>> kernel RAID system? I briefly looked through the archives and have
>>>> not found anything on the topic.
>>>>
>>>> Some HW based RAID controllers do this flawlessly now, there is no
>>>> reason why the kernel RAID cannot also. (LSI MegaRAID)
>>>>
>>>> 1) When a drive is part if a managed enclosure, the RAID system should
>>>> address it by location instead of by enumerated device node. The SES
>>>> device in the enclosure can map the physical slot to a physical drive.
>>>> The RAID admin (mdamd) should be able to add/fail/identify devices
>>>> based on slot.
>>>
>>> Does this just mean that the admin should using names in
>>> /dev/disk/by-path/
>>> rather than /dev/sdXX to address devices? What can md or mdadm do to
>>> help?
>>>
>>
>> Not sure on the SES (or SGPIO side), but one of the things we've been
>> doing has been to create a file with disk placement "coordinates", so
>> as to map serial number and device to physical location.
>>
>
> With real SES managed enclosures, you issue a SCSI command to read SES
> Page1 and Page2 to get the details about the drives in any given slot.
> This currently works fine in Linux with the sg_utils3 package. From the
> command line, 'sg_ses -p 2 /dev/sgXX` where the device is the SES device.
>
> Take a look at your systems, if you see a device at
> /sys/class/enclosure/XXXX/ then you have a managed enclosure attached.
>
Got it. Thanks. Will look and see. Should be pretty straight forward
to do this.
>
>>>>
>>>> 2) If the RAID system fails a drive, it should notify the SES
>>>> management and turn on the fail bit and the fail LED.
>>>
>>> "mdadm --monitor" will run a script on drive failure. This could easily
>>> notify the SES management.
>>
>> Yes, we are using this now for notifications and logging.
>>
>>>
>>> So maybe all we need here is a script to plug in to mdadm... Would
>>> you like
>>> to write one?
>>>
>>
>> Just need a "standard" SES (or SGPIO) mechanism to hook into, and we
>> should be able to support this. Right now we have to work through HBA
>> scripts.
> A true managed enclosure has nothing to do with the HBA. A managed
> enclosure provides a device on the SCSI bus and you exclusively
> communicate with that device regardless of the HBA. Most HW RAIDs (LSI
> MegaRAID) will hide the SES device exactly like they hide the physical
> disks.
>
Ok. Let me look to see if we can do this. If so, we should be able to
help contribute some scripts.
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web : http://scalableinformatics.com
http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 20:58 ` Joe Landman
@ 2012-02-14 21:06 ` Jeff Johnson
2012-02-14 21:10 ` Robert Woodworth
0 siblings, 1 reply; 15+ messages in thread
From: Jeff Johnson @ 2012-02-14 21:06 UTC (permalink / raw)
To: linux-raid
On 2/14/12 12:58 PM, Joe Landman wrote:
> On 02/14/2012 03:53 PM, Robert Woodworth wrote:
>> On 02/14/2012 01:42 PM, Joe Landman wrote:
>>> On 02/14/2012 03:31 PM, NeilBrown wrote:
>>>> On Tue, 14 Feb 2012 10:30:37 -0700 Robert Woodworth
>>>> <robertjwoodworth@gmail.com> wrote:
>>>>
>>>>> Has anyone ever thought of integrating SES managed enclosures into
>>>>> the
>>>>> kernel RAID system? I briefly looked through the archives and have
>>>>> not found anything on the topic.
>>>>>
>>>>> Some HW based RAID controllers do this flawlessly now, there is no
>>>>> reason why the kernel RAID cannot also. (LSI MegaRAID)
>>>>>
>>>>> 1) When a drive is part if a managed enclosure, the RAID system
>>>>> should
>>>>> address it by location instead of by enumerated device node. The SES
>>>>> device in the enclosure can map the physical slot to a physical
>>>>> drive.
>>>>> The RAID admin (mdamd) should be able to add/fail/identify devices
>>>>> based on slot.
>>>>
>>>> Does this just mean that the admin should using names in
>>>> /dev/disk/by-path/
>>>> rather than /dev/sdXX to address devices? What can md or mdadm do to
>>>> help?
>>>>
>>>
>>> Not sure on the SES (or SGPIO side), but one of the things we've been
>>> doing has been to create a file with disk placement "coordinates", so
>>> as to map serial number and device to physical location.
>>>
>>
>> With real SES managed enclosures, you issue a SCSI command to read SES
>> Page1 and Page2 to get the details about the drives in any given slot.
>> This currently works fine in Linux with the sg_utils3 package. From the
>> command line, 'sg_ses -p 2 /dev/sgXX` where the device is the SES
>> device.
>>
>> Take a look at your systems, if you see a device at
>> /sys/class/enclosure/XXXX/ then you have a managed enclosure attached.
>>
>
> Got it. Thanks. Will look and see. Should be pretty straight
> forward to do this.
>
>>
>>>>>
>>>>> 2) If the RAID system fails a drive, it should notify the SES
>>>>> management and turn on the fail bit and the fail LED.
>>>>
>>>> "mdadm --monitor" will run a script on drive failure. This could
>>>> easily
>>>> notify the SES management.
>>>
>>> Yes, we are using this now for notifications and logging.
>>>
>>>>
>>>> So maybe all we need here is a script to plug in to mdadm... Would
>>>> you like
>>>> to write one?
>>>>
>>>
>>> Just need a "standard" SES (or SGPIO) mechanism to hook into, and we
>>> should be able to support this. Right now we have to work through HBA
>>> scripts.
>> A true managed enclosure has nothing to do with the HBA. A managed
>> enclosure provides a device on the SCSI bus and you exclusively
>> communicate with that device regardless of the HBA. Most HW RAIDs (LSI
>> MegaRAID) will hide the SES device exactly like they hide the physical
>> disks.
>>
>
> Ok. Let me look to see if we can do this. If so, we should be able
> to help contribute some scripts.
I've been doing this for a while. In SES the various elements (slot,
power supplies, fans, etc) are named. At least if your JBOD vendor has
their **** together. There is no need to decipher arrays of hex values
and modify bits to assert control. Once you know the names of your
elements you can address them by name for status and control (fault LED,
power on/off, temps, etc).
--Jeff
--
------------------------------
Jeff Johnson
Manager
Aeon Computing
jeff.johnson@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x101 f: 858-412-3845
m: 619-204-9061
4905 Morena Boulevard, Suite 1313 - San Diego, CA 92117
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 21:06 ` Jeff Johnson
@ 2012-02-14 21:10 ` Robert Woodworth
2012-02-14 21:24 ` Jeff Johnson
0 siblings, 1 reply; 15+ messages in thread
From: Robert Woodworth @ 2012-02-14 21:10 UTC (permalink / raw)
To: Jeff Johnson; +Cc: linux-raid
On 02/14/2012 02:06 PM, Jeff Johnson wrote:
> On 2/14/12 12:58 PM, Joe Landman wrote:
>> On 02/14/2012 03:53 PM, Robert Woodworth wrote:
>>> On 02/14/2012 01:42 PM, Joe Landman wrote:
>>>> On 02/14/2012 03:31 PM, NeilBrown wrote:
>>>>> On Tue, 14 Feb 2012 10:30:37 -0700 Robert Woodworth
>>>>> <robertjwoodworth@gmail.com> wrote:
>>>>>
>>>>>> Has anyone ever thought of integrating SES managed enclosures
>>>>>> into the
>>>>>> kernel RAID system? I briefly looked through the archives and have
>>>>>> not found anything on the topic.
>>>>>>
>>>>>> Some HW based RAID controllers do this flawlessly now, there is no
>>>>>> reason why the kernel RAID cannot also. (LSI MegaRAID)
>>>>>>
>>>>>> 1) When a drive is part if a managed enclosure, the RAID system
>>>>>> should
>>>>>> address it by location instead of by enumerated device node. The SES
>>>>>> device in the enclosure can map the physical slot to a physical
>>>>>> drive.
>>>>>> The RAID admin (mdamd) should be able to add/fail/identify devices
>>>>>> based on slot.
>>>>>
>>>>> Does this just mean that the admin should using names in
>>>>> /dev/disk/by-path/
>>>>> rather than /dev/sdXX to address devices? What can md or mdadm do to
>>>>> help?
>>>>>
>>>>
>>>> Not sure on the SES (or SGPIO side), but one of the things we've been
>>>> doing has been to create a file with disk placement "coordinates", so
>>>> as to map serial number and device to physical location.
>>>>
>>>
>>> With real SES managed enclosures, you issue a SCSI command to read SES
>>> Page1 and Page2 to get the details about the drives in any given slot.
>>> This currently works fine in Linux with the sg_utils3 package. From the
>>> command line, 'sg_ses -p 2 /dev/sgXX` where the device is the SES
>>> device.
>>>
>>> Take a look at your systems, if you see a device at
>>> /sys/class/enclosure/XXXX/ then you have a managed enclosure attached.
>>>
>>
>> Got it. Thanks. Will look and see. Should be pretty straight
>> forward to do this.
>>
>>>
>>>>>>
>>>>>> 2) If the RAID system fails a drive, it should notify the SES
>>>>>> management and turn on the fail bit and the fail LED.
>>>>>
>>>>> "mdadm --monitor" will run a script on drive failure. This could
>>>>> easily
>>>>> notify the SES management.
>>>>
>>>> Yes, we are using this now for notifications and logging.
>>>>
>>>>>
>>>>> So maybe all we need here is a script to plug in to mdadm... Would
>>>>> you like
>>>>> to write one?
>>>>>
>>>>
>>>> Just need a "standard" SES (or SGPIO) mechanism to hook into, and we
>>>> should be able to support this. Right now we have to work through HBA
>>>> scripts.
>>> A true managed enclosure has nothing to do with the HBA. A managed
>>> enclosure provides a device on the SCSI bus and you exclusively
>>> communicate with that device regardless of the HBA. Most HW RAIDs (LSI
>>> MegaRAID) will hide the SES device exactly like they hide the physical
>>> disks.
>>>
>>
>> Ok. Let me look to see if we can do this. If so, we should be able
>> to help contribute some scripts.
> I've been doing this for a while. In SES the various elements (slot,
> power supplies, fans, etc) are named. At least if your JBOD vendor
> has their **** together. There is no need to decipher arrays of hex
> values and modify bits to assert control. Once you know the names of
> your elements you can address them by name for status and control
> (fault LED, power on/off, temps, etc).
>
> --Jeff
Correct! Now I just want to connect the RAID system to the SES system
so that when a disk fails, the kernel module that failed the disk can
light up the LED.
I work for one of those vendors, it's my job to have our **** together.
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 20:46 ` Robert Woodworth
@ 2012-02-14 21:16 ` NeilBrown
0 siblings, 0 replies; 15+ messages in thread
From: NeilBrown @ 2012-02-14 21:16 UTC (permalink / raw)
To: Robert Woodworth; +Cc: linux-raid
[-- Attachment #1: Type: text/plain, Size: 1960 bytes --]
On Tue, 14 Feb 2012 13:46:32 -0700 Robert Woodworth
<robertjwoodworth@gmail.com> wrote:
> On 02/14/2012 01:31 PM, NeilBrown wrote:
> > On Tue, 14 Feb 2012 10:30:37 -0700 Robert Woodworth
> > <robertjwoodworth@gmail.com> wrote:
> >
> >> Has anyone ever thought of integrating SES managed enclosures into the
> >> kernel RAID system? I briefly looked through the archives and have
> >> not found anything on the topic.
> >>
> >> Some HW based RAID controllers do this flawlessly now, there is no
> >> reason why the kernel RAID cannot also. (LSI MegaRAID)
> >>
> >> 1) When a drive is part if a managed enclosure, the RAID system should
> >> address it by location instead of by enumerated device node. The SES
> >> device in the enclosure can map the physical slot to a physical drive.
> >> The RAID admin (mdamd) should be able to add/fail/identify devices
> >> based on slot.
> > Does this just mean that the admin should using names in /dev/disk/by-path/
> > rather than /dev/sdXX to address devices? What can md or mdadm do to help?
> I don't like the /by-path/ too well, it is not human readable. I would
> propose that the SES module that is already in the base kernel be
> expanded to provide a /dev/disk/by-location/<enclosureID>/<slot> ->
> /dev/sdXY
It isn't the kernel that provides these links, it is udev.
If there is some tool that can extract the enclosureID and slot values, then
one or 2 lines in a udev 'rules' file should be able to create the links.
> >
> The one piece that could be included in the kernel level is the display
> of /proc/mdstat to show the enclosureID and slot number
I'm finding /proc/mdstat increasingly distasteful. It is very convenient but
hard to extend usefully.
I' rather create something like
mdadm --status
which reported the status of all (or some) arrays with similar usability to
/proc/mdstat, but more room to extend it.
Thanks,
NeilBrown
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 21:10 ` Robert Woodworth
@ 2012-02-14 21:24 ` Jeff Johnson
2012-02-15 14:54 ` Joe Landman
0 siblings, 1 reply; 15+ messages in thread
From: Jeff Johnson @ 2012-02-14 21:24 UTC (permalink / raw)
To: linux-raid
On 2/14/12 1:10 PM, Robert Woodworth wrote:
> On 02/14/2012 02:06 PM, Jeff Johnson wrote:
>> On 2/14/12 12:58 PM, Joe Landman wrote:
>>> <..snip...>
>>>> enclosure provides a device on the SCSI bus and you exclusively
>>>> communicate with that device regardless of the HBA. Most HW RAIDs (LSI
>>>> MegaRAID) will hide the SES device exactly like they hide the physical
>>>> disks.
>>>>
>>> A true managed enclosure has nothing to do with the HBA. A managed
>>>
>>> Ok. Let me look to see if we can do this. If so, we should be able
>>> to help contribute some scripts.
>> I've been doing this for a while. In SES the various elements (slot,
>> power supplies, fans, etc) are named. At least if your JBOD vendor
>> has their **** together. There is no need to decipher arrays of hex
>> values and modify bits to assert control. Once you know the names of
>> your elements you can address them by name for status and control
>> (fault LED, power on/off, temps, etc).
>>
>> --Jeff
>
> Correct! Now I just want to connect the RAID system to the SES system
> so that when a disk fails, the kernel module that failed the disk can
> light up the LED.
>
> I work for one of those vendors, it's my job to have our **** together.
>
The trick is to map the disk element names to the block device names.
Different SAS HBAs and drivers can enumerate the devices differently.
Persistence settings can muck things up as well. Sometimes a failed
block device at /dev/sdf can appear as /dev/sdr when replaced. You could
use udev rules to create alternate block device names but so far, for
important data, I've seen no substitute for a pair of knowledgeable
human eyes analyzing a failure and confirming a failed drive by
corelating WWNs, etc.
--Jeff
--
------------------------------
Jeff Johnson
Manager
Aeon Computing
jeff.johnson@aeoncomputing.com
www.aeoncomputing.com
t: 858-412-3810 x101 f: 858-412-3845
m: 619-204-9061
4905 Morena Boulevard, Suite 1313 - San Diego, CA 92117
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 21:24 ` Jeff Johnson
@ 2012-02-15 14:54 ` Joe Landman
0 siblings, 0 replies; 15+ messages in thread
From: Joe Landman @ 2012-02-15 14:54 UTC (permalink / raw)
To: Jeff Johnson; +Cc: linux-raid
On 02/14/2012 04:24 PM, Jeff Johnson wrote:
>> I work for one of those vendors, it's my job to have our **** together.
>>
> The trick is to map the disk element names to the block device names.
> Different SAS HBAs and drivers can enumerate the devices differently.
> Persistence settings can muck things up as well. Sometimes a failed
> block device at /dev/sdf can appear as /dev/sdr when replaced. You could
> use udev rules to create alternate block device names but so far, for
> important data, I've seen no substitute for a pair of knowledgeable
> human eyes analyzing a failure and confirming a failed drive by
> corelating WWNs, etc.
The tools we've been working on have been trying to correlate this by
various methods though these vary by HBA and other issues. UDEV rules
can often produce some interesting results (and not in a good way, and
not just for disks).
--
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman@scalableinformatics.com
web : http://scalableinformatics.com
http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax : +1 866 888 3112
cell : +1 734 612 4615
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-14 20:53 ` Robert Woodworth
2012-02-14 20:58 ` Joe Landman
@ 2012-02-16 19:00 ` Benjamin ESTRABAUD
2012-02-16 19:12 ` Robert Woodworth
1 sibling, 1 reply; 15+ messages in thread
From: Benjamin ESTRABAUD @ 2012-02-16 19:00 UTC (permalink / raw)
To: Robert Woodworth; +Cc: Joe Landman, NeilBrown, linux-raid
On 14/02/12 20:53, Robert Woodworth wrote:
> On 02/14/2012 01:42 PM, Joe Landman wrote:
>> On 02/14/2012 03:31 PM, NeilBrown wrote:
>>> On Tue, 14 Feb 2012 10:30:37 -0700 Robert Woodworth
>>> <robertjwoodworth@gmail.com> wrote:
>>>
>>>> Has anyone ever thought of integrating SES managed enclosures into the
>>>> kernel RAID system? I briefly looked through the archives and have
>>>> not found anything on the topic.
>>>>
>>>> Some HW based RAID controllers do this flawlessly now, there is no
>>>> reason why the kernel RAID cannot also. (LSI MegaRAID)
>>>>
>>>> 1) When a drive is part if a managed enclosure, the RAID system should
>>>> address it by location instead of by enumerated device node. The SES
>>>> device in the enclosure can map the physical slot to a physical drive.
>>>> The RAID admin (mdamd) should be able to add/fail/identify devices
>>>> based on slot.
>>>
>>> Does this just mean that the admin should using names in
>>> /dev/disk/by-path/
>>> rather than /dev/sdXX to address devices? What can md or mdadm do
>>> to help?
>>>
>>
>> Not sure on the SES (or SGPIO side), but one of the things we've been
>> doing has been to create a file with disk placement "coordinates", so
>> as to map serial number and device to physical location.
>>
>
> With real SES managed enclosures, you issue a SCSI command to read SES
> Page1 and Page2 to get the details about the drives in any given
> slot. This currently works fine in Linux with the sg_utils3 package.
> From the command line, 'sg_ses -p 2 /dev/sgXX` where the device is
> the SES device.
>
> Take a look at your systems, if you see a device at
> /sys/class/enclosure/XXXX/ then you have a managed enclosure attached.
>
Hi,
True, but your definition of SES "slot 0" might be different from
someone else's. The end user will still need to map "slot 0" to a
physical slot on his system, instead of mapping a drive's serial number
to a physical slot. There is no standard for slot numbering and most
vendors use different schemes.
>
>>>>
>>>> 2) If the RAID system fails a drive, it should notify the SES
>>>> management and turn on the fail bit and the fail LED.
>>>
>>> "mdadm --monitor" will run a script on drive failure. This could
>>> easily
>>> notify the SES management.
>>
>> Yes, we are using this now for notifications and logging.
>>
>>>
>>> So maybe all we need here is a script to plug in to mdadm... Would
>>> you like
>>> to write one?
>>>
>>
>> Just need a "standard" SES (or SGPIO) mechanism to hook into, and we
>> should be able to support this. Right now we have to work through
>> HBA scripts.
> A true managed enclosure has nothing to do with the HBA. A managed
> enclosure provides a device on the SCSI bus and you exclusively
> communicate with that device regardless of the HBA. Most HW RAIDs
> (LSI MegaRAID) will hide the SES device exactly like they hide the
> physical disks.
>
>>
>>>>
>>>> 3) The RAID system should be able to turn on the 'identify' bit and
>>>> LEDs for an array or a single drive.
>>>
This is fine too, as you pointed out SES gives you information on disks,
but in order to map a block device sda to a device in SES, won't you
need to rely on its SAS address? or even the SAS address arbitrarily
given by the expander to a SATA device?
Your expander's SES device might display this address for both SAS and
SATA drives in SES page 0xa, but most SES devices won't, at least for
SATA drives, as the SES spec specifically requires SATA drive's address
to be set to 0x0.
So if drive "sd"a fails, you'll be able to tell which expander it is
attached using kernel infos, tell its SAS address as you see it, but you
won't be able to map it to a particular slot number in SES, and will not
be able to set its failed LED reliably.
>>> Again, it sounds like you just need a script to ask mdadm which
>>> devices are
>>> included in a given array, and then do whatever magic is needed to
>>> turn on
>>> the light.
>>> It is fairly easy to extra the device list from the output of
>>> mdadm --detail --brief --verbose /dev/md/whatever
>>>
>>> but it might be good to make it easier to extra from
>>> mdadm --detail --export /dev/md/whatever
>>>
>>> Would you like to write such a script?
>>>
>>>>
>>>> I'm currently doing firmware on a managed enclosure. Although my
>>>> vendor only supports LSI MegaRAID, there is no reason why my
>>>> enclosures cannot work in the same manner on a kernel RAID system.
>>>>
>>>>
>>>>
>>>> Request for comments...
>>>
>>> It sounds to me like you just need a few scripts to provide some
>>> enclosure-specific functionality. I would be happy to include them
>>> in the
>>> mdadm distribution.
>>>
>>> Or maybe there is something that I didn't understand??
>>>
>>> Thanks,
>>> NeilBrown
>>>
>>>
Regards,
Ben.
>>>
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe
>>>> linux-raid" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-16 19:00 ` Benjamin ESTRABAUD
@ 2012-02-16 19:12 ` Robert Woodworth
2012-02-21 18:55 ` Benjamin ESTRABAUD
0 siblings, 1 reply; 15+ messages in thread
From: Robert Woodworth @ 2012-02-16 19:12 UTC (permalink / raw)
To: Benjamin ESTRABAUD; +Cc: Joe Landman, NeilBrown, linux-raid
On 02/16/2012 12:00 PM, Benjamin ESTRABAUD wrote:
> On 14/02/12 20:53, Robert Woodworth wrote:
>> On 02/14/2012 01:42 PM, Joe Landman wrote:
>>> On 02/14/2012 03:31 PM, NeilBrown wrote:
>>>> On Tue, 14 Feb 2012 10:30:37 -0700 Robert Woodworth
>>>> <robertjwoodworth@gmail.com> wrote:
>>>>
>>>>> Has anyone ever thought of integrating SES managed enclosures into
>>>>> the
>>>>> kernel RAID system? I briefly looked through the archives and have
>>>>> not found anything on the topic.
>>>>>
>>>>> Some HW based RAID controllers do this flawlessly now, there is no
>>>>> reason why the kernel RAID cannot also. (LSI MegaRAID)
>>>>>
>>>>> 1) When a drive is part if a managed enclosure, the RAID system
>>>>> should
>>>>> address it by location instead of by enumerated device node. The SES
>>>>> device in the enclosure can map the physical slot to a physical
>>>>> drive.
>>>>> The RAID admin (mdamd) should be able to add/fail/identify devices
>>>>> based on slot.
>>>>
>>>> Does this just mean that the admin should using names in
>>>> /dev/disk/by-path/
>>>> rather than /dev/sdXX to address devices? What can md or mdadm do
>>>> to help?
>>>>
>>>
>>> Not sure on the SES (or SGPIO side), but one of the things we've
>>> been doing has been to create a file with disk placement
>>> "coordinates", so as to map serial number and device to physical
>>> location.
>>>
>>
>> With real SES managed enclosures, you issue a SCSI command to read
>> SES Page1 and Page2 to get the details about the drives in any given
>> slot. This currently works fine in Linux with the sg_utils3
>> package. From the command line, 'sg_ses -p 2 /dev/sgXX` where the
>> device is the SES device.
>>
>> Take a look at your systems, if you see a device at
>> /sys/class/enclosure/XXXX/ then you have a managed enclosure attached.
>>
>
> Hi,
>
> True, but your definition of SES "slot 0" might be different from
> someone else's. The end user will still need to map "slot 0" to a
> physical slot on his system, instead of mapping a drive's serial
> number to a physical slot. There is no standard for slot numbering and
> most vendors use different schemes.
On the boxes I write SES firmware for, we have a sticker label on the
slot 'slot1' and its my job to make sure the SES firmware always maps
the physical label 'slot1' with the drive in that slot to the data in
the SCSI SES page 10 (0x0A).
That is the whole point of the SES device.
>>
>>>>>
>>>>> 2) If the RAID system fails a drive, it should notify the SES
>>>>> management and turn on the fail bit and the fail LED.
>>>>
>>>> "mdadm --monitor" will run a script on drive failure. This could
>>>> easily
>>>> notify the SES management.
>>>
>>> Yes, we are using this now for notifications and logging.
>>>
>>>>
>>>> So maybe all we need here is a script to plug in to mdadm... Would
>>>> you like
>>>> to write one?
>>>>
>>>
>>> Just need a "standard" SES (or SGPIO) mechanism to hook into, and we
>>> should be able to support this. Right now we have to work through
>>> HBA scripts.
>> A true managed enclosure has nothing to do with the HBA. A managed
>> enclosure provides a device on the SCSI bus and you exclusively
>> communicate with that device regardless of the HBA. Most HW RAIDs
>> (LSI MegaRAID) will hide the SES device exactly like they hide the
>> physical disks.
>>
>>>
>>>>>
>>>>> 3) The RAID system should be able to turn on the 'identify' bit and
>>>>> LEDs for an array or a single drive.
>>>>
> This is fine too, as you pointed out SES gives you information on
> disks, but in order to map a block device sda to a device in SES,
> won't you need to rely on its SAS address? or even the SAS address
> arbitrarily given by the expander to a SATA device?
Look at SCSI SES Page 10 (0x0A), that is the correct mapping of disk to
slot on the host side.
>
> Your expander's SES device might display this address for both SAS and
> SATA drives in SES page 0xa, but most SES devices won't, at least for
> SATA drives, as the SES spec specifically requires SATA drive's
> address to be set to 0x0.
>
> So if drive "sd"a fails, you'll be able to tell which expander it is
> attached using kernel infos, tell its SAS address as you see it, but
> you won't be able to map it to a particular slot number in SES, and
> will not be able to set its failed LED reliably.
As stated above. If the firmware on the SES enclosure is written
correctly the SES page1 and page2 will correctly map the physical
devices *and LEDs* to the devices in the SCSI SES pages. This does work
on proper enclosures, yes you can rely on it (on my boxes anyway).
Maybe people have bad experiences with bad firmware, but if you get a
box that my expanders and SES firmware is in, they will work correctly.
>>>> Again, it sounds like you just need a script to ask mdadm which
>>>> devices are
>>>> included in a given array, and then do whatever magic is needed to
>>>> turn on
>>>> the light.
>>>> It is fairly easy to extra the device list from the output of
>>>> mdadm --detail --brief --verbose /dev/md/whatever
>>>>
>>>> but it might be good to make it easier to extra from
>>>> mdadm --detail --export /dev/md/whatever
>>>>
>>>> Would you like to write such a script?
>>>>
>>>>>
>>>>> I'm currently doing firmware on a managed enclosure. Although my
>>>>> vendor only supports LSI MegaRAID, there is no reason why my
>>>>> enclosures cannot work in the same manner on a kernel RAID system.
>>>>>
>>>>>
>>>>>
>>>>> Request for comments...
>>>>
>>>> It sounds to me like you just need a few scripts to provide some
>>>> enclosure-specific functionality. I would be happy to include them
>>>> in the
>>>> mdadm distribution.
>>>>
>>>> Or maybe there is something that I didn't understand??
>>>>
>>>> Thanks,
>>>> NeilBrown
>>>>
>>>>
> Regards,
> Ben.
>>>>
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>> linux-raid" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-raid" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
* Re: SES Enclosure Management.
2012-02-16 19:12 ` Robert Woodworth
@ 2012-02-21 18:55 ` Benjamin ESTRABAUD
0 siblings, 0 replies; 15+ messages in thread
From: Benjamin ESTRABAUD @ 2012-02-21 18:55 UTC (permalink / raw)
To: Robert Woodworth; +Cc: Joe Landman, NeilBrown, linux-raid
On 16/02/12 19:12, Robert Woodworth wrote:
> On 02/16/2012 12:00 PM, Benjamin ESTRABAUD wrote:
>> On 14/02/12 20:53, Robert Woodworth wrote:
>>> On 02/14/2012 01:42 PM, Joe Landman wrote:
>>>> On 02/14/2012 03:31 PM, NeilBrown wrote:
>>>>> On Tue, 14 Feb 2012 10:30:37 -0700 Robert Woodworth
>>>>> <robertjwoodworth@gmail.com> wrote:
>>>>>
>>>>>> Has anyone ever thought of integrating SES managed enclosures
>>>>>> into the
>>>>>> kernel RAID system? I briefly looked through the archives and have
>>>>>> not found anything on the topic.
>>>>>>
>>>>>> Some HW based RAID controllers do this flawlessly now, there is no
>>>>>> reason why the kernel RAID cannot also. (LSI MegaRAID)
>>>>>>
>>>>>> 1) When a drive is part if a managed enclosure, the RAID system
>>>>>> should
>>>>>> address it by location instead of by enumerated device node. The
>>>>>> SES
>>>>>> device in the enclosure can map the physical slot to a physical
>>>>>> drive.
>>>>>> The RAID admin (mdamd) should be able to add/fail/identify devices
>>>>>> based on slot.
>>>>>
>>>>> Does this just mean that the admin should using names in
>>>>> /dev/disk/by-path/
>>>>> rather than /dev/sdXX to address devices? What can md or mdadm do
>>>>> to help?
>>>>>
>>>>
>>>> Not sure on the SES (or SGPIO side), but one of the things we've
>>>> been doing has been to create a file with disk placement
>>>> "coordinates", so as to map serial number and device to physical
>>>> location.
>>>>
>>>
>>> With real SES managed enclosures, you issue a SCSI command to read
>>> SES Page1 and Page2 to get the details about the drives in any given
>>> slot. This currently works fine in Linux with the sg_utils3
>>> package. From the command line, 'sg_ses -p 2 /dev/sgXX` where the
>>> device is the SES device.
>>>
>>> Take a look at your systems, if you see a device at
>>> /sys/class/enclosure/XXXX/ then you have a managed enclosure attached.
>>>
>>
>> Hi,
>>
>> True, but your definition of SES "slot 0" might be different from
>> someone else's. The end user will still need to map "slot 0" to a
>> physical slot on his system, instead of mapping a drive's serial
>> number to a physical slot. There is no standard for slot numbering
>> and most vendors use different schemes.
> On the boxes I write SES firmware for, we have a sticker label on the
> slot 'slot1' and its my job to make sure the SES firmware always maps
> the physical label 'slot1' with the drive in that slot to the data in
> the SCSI SES page 10 (0x0A).
>
> That is the whole point of the SES device.
>
Very few vendor actually do this, although it makes sense I completely
agree.
>>>
>>>>>>
>>>>>> 2) If the RAID system fails a drive, it should notify the SES
>>>>>> management and turn on the fail bit and the fail LED.
>>>>>
>>>>> "mdadm --monitor" will run a script on drive failure. This could
>>>>> easily
>>>>> notify the SES management.
>>>>
>>>> Yes, we are using this now for notifications and logging.
>>>>
>>>>>
>>>>> So maybe all we need here is a script to plug in to mdadm... Would
>>>>> you like
>>>>> to write one?
>>>>>
>>>>
>>>> Just need a "standard" SES (or SGPIO) mechanism to hook into, and
>>>> we should be able to support this. Right now we have to work
>>>> through HBA scripts.
>>> A true managed enclosure has nothing to do with the HBA. A managed
>>> enclosure provides a device on the SCSI bus and you exclusively
>>> communicate with that device regardless of the HBA. Most HW RAIDs
>>> (LSI MegaRAID) will hide the SES device exactly like they hide the
>>> physical disks.
>>>
>>>>
>>>>>>
>>>>>> 3) The RAID system should be able to turn on the 'identify' bit and
>>>>>> LEDs for an array or a single drive.
>>>>>
>> This is fine too, as you pointed out SES gives you information on
>> disks, but in order to map a block device sda to a device in SES,
>> won't you need to rely on its SAS address? or even the SAS address
>> arbitrarily given by the expander to a SATA device?
> Look at SCSI SES Page 10 (0x0A), that is the correct mapping of disk
> to slot on the host side.
>
>>
>> Your expander's SES device might display this address for both SAS
>> and SATA drives in SES page 0xa, but most SES devices won't, at least
>> for SATA drives, as the SES spec specifically requires SATA drive's
>> address to be set to 0x0.
>>
>> So if drive "sd"a fails, you'll be able to tell which expander it is
>> attached using kernel infos, tell its SAS address as you see it, but
>> you won't be able to map it to a particular slot number in SES, and
>> will not be able to set its failed LED reliably.
> As stated above. If the firmware on the SES enclosure is written
> correctly the SES page1 and page2 will correctly map the physical
> devices *and LEDs* to the devices in the SCSI SES pages. This does
> work on proper enclosures, yes you can rely on it (on my boxes anyway).
>
> Maybe people have bad experiences with bad firmware, but if you get a
> box that my expanders and SES firmware is in, they will work correctly.
>
Well, the problem here is not that the SES page1 and page2 do not map to
a physical device in the right slot, but how to identify them.
The only info in common between the SES device and the kernel about a
disk is its SAS address. You can get a map of SAS address <> Slot in SES
page 0xa as you correctly mentioned, but when using STP (SATA drives),
that address will often be blank (0x0). This is not out of bad design
and merely following the SES specs (which, for this particular issue do
not make sense to me).
In this particular case, there will be no way to map the "sdX" block
device to a SES slot id. Therefore you will not know which slot to send
the command to when you send your SES control.
This is an inevitable issue, unfortunately, which is a great barrier to
integrate SES to the Linux kernel's block drivers in my opinion.
If SES had a common information to point out a disk with the kernel, it
could easily be done. Without it it's kind of hard, if purely using a
SES device and nothing more.
Regards,
Ben.
>
>
>>>>> Again, it sounds like you just need a script to ask mdadm which
>>>>> devices are
>>>>> included in a given array, and then do whatever magic is needed to
>>>>> turn on
>>>>> the light.
>>>>> It is fairly easy to extra the device list from the output of
>>>>> mdadm --detail --brief --verbose /dev/md/whatever
>>>>>
>>>>> but it might be good to make it easier to extra from
>>>>> mdadm --detail --export /dev/md/whatever
>>>>>
>>>>> Would you like to write such a script?
>>>>>
>>>>>>
>>>>>> I'm currently doing firmware on a managed enclosure. Although my
>>>>>> vendor only supports LSI MegaRAID, there is no reason why my
>>>>>> enclosures cannot work in the same manner on a kernel RAID system.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Request for comments...
>>>>>
>>>>> It sounds to me like you just need a few scripts to provide some
>>>>> enclosure-specific functionality. I would be happy to include
>>>>> them in the
>>>>> mdadm distribution.
>>>>>
>>>>> Or maybe there is something that I didn't understand??
>>>>>
>>>>> Thanks,
>>>>> NeilBrown
>>>>>
>>>>>
>> Regards,
>> Ben.
>>>>>
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>> linux-raid" in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>>
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe
>>> linux-raid" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at http://vger.kernel.org/majordomo-info.html
>>>
>>
>
>
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2012-02-21 18:55 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-02-14 17:30 SES Enclosure Management Robert Woodworth
2012-02-14 20:31 ` NeilBrown
2012-02-14 20:42 ` Joe Landman
2012-02-14 20:53 ` Robert Woodworth
2012-02-14 20:58 ` Joe Landman
2012-02-14 21:06 ` Jeff Johnson
2012-02-14 21:10 ` Robert Woodworth
2012-02-14 21:24 ` Jeff Johnson
2012-02-15 14:54 ` Joe Landman
2012-02-16 19:00 ` Benjamin ESTRABAUD
2012-02-16 19:12 ` Robert Woodworth
2012-02-21 18:55 ` Benjamin ESTRABAUD
2012-02-14 20:46 ` Robert Woodworth
2012-02-14 21:16 ` NeilBrown
2012-02-14 20:38 ` Joe Landman
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).