SATA hotplug notifications for BMC inventory updates

SATA hotplug notifications for BMC inventory updates

[Paul Fertser]

Hi,

An OCP Tioga Pass platform features up to 4 drive bays for SATA
storage accessible from the front and fully hot-pluggable. The
backplane is usually connected directly to Intel C620 PCH (aka
Lewisburg).

It would be nice to have information about the currently attached
devices (model and serial number) available in BMC inventory but what
are the ways to get this information?

Is it possible there's some additional module in the all-mighty ME
that can supply BMC with the current state of PCH SATA controller? Or
is there some other sensible mechanism other than talking to UEFI on
startup (which would mean BMC is showing stale and thus misleading
data till the next reboot)?

Even though the question is quite specific we'd appreciate any related
insights on the topic; there must be some side-channels to RAID
controllers and such for BMC to be obtaining essential information but
so far I wasn't able to find any documentation about this.

-- 
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:fercerpav@gmail.com

Re: SATA hotplug notifications for BMC inventory updates

[Johnathan Mantey]

[-- Attachment #1: Type: text/plain, Size: 1654 bytes --]

On 9/5/22 04:11, Paul Fertser wrote:
> Hi,
>
> An OCP Tioga Pass platform features up to 4 drive bays for SATA
> storage accessible from the front and fully hot-pluggable. The
> backplane is usually connected directly to Intel C620 PCH (aka
> Lewisburg).
>
> It would be nice to have information about the currently attached
> devices (model and serial number) available in BMC inventory but what
> are the ways to get this information?

Intel systems only get drive mdl/sn from NVMe devices that provide that 
information via MCTP over SMBus. SAS/SATA drives, to my current 
knowledge, don't supply this info in that mode.

The insertion/removal events are also managed from the BMC via I2C to a 
FPGA solution residing on the Hot Swap Backplane. The FPGA provides 
registers that allow the BMC to query the PRSTn/IFDETn pins.

>
> Is it possible there's some additional module in the all-mighty ME
> that can supply BMC with the current state of PCH SATA controller? Or
> is there some other sensible mechanism other than talking to UEFI on
> startup (which would mean BMC is showing stale and thus misleading
> data till the next reboot)?
>
> Even though the question is quite specific we'd appreciate any related
> insights on the topic; there must be some side-channels to RAID
> controllers and such for BMC to be obtaining essential information but
> so far I wasn't able to find any documentation about this.
RAID solutions are outside of scope for the Intel BMC.

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com

[-- Attachment #2: Type: text/html, Size: 2748 bytes --]

Re: SATA hotplug notifications for BMC inventory updates

[Johnathan Mantey]

[-- Attachment #1: Type: text/plain, Size: 3304 bytes --]

On 9/6/22 23:47, Konstantin Klubnichkin wrote:
> Hi Johnathan,
> What do you think about implementing this feature using SMI?
> Is it possible to raise SMI on drive plug/removal?

Paul, my experience with SMI is limited. Systems I've used that issued 
SMI only did so for PSUs. The details have gotten fuzzy over the years. 
I believe the SMI only asserted on removal of the power cord(s). The SMI 
event resulted in a high priority interrupt designed to tell the BMC to 
get state saved before complete power failure.

Even if you had the SMI connection from the HDD enclosure, I'm not 
certain how you'd acquire the information about which HDD was 
inserted/removed. That is an implementation detail of your HW.

A back of envelope thought I had is perhaps the host side could interact 
with the BMC via a mailbox. The issue being you'd have to have SW 
running during the BIOS time, and when the OS was in control an OS 
helper app pushing the data. I'm not sure how feasible that idea is.

> Thank you!
> 06.09.2022, 18:54, "Johnathan Mantey" <johnathanx.mantey@intel.com>:
>
>     On 9/5/22 04:11, Paul Fertser wrote:
>
>         Hi,
>
>         An OCP Tioga Pass platform features up to 4 drive bays for SATA
>         storage accessible from the front and fully hot-pluggable. The
>         backplane is usually connected directly to Intel C620 PCH (aka
>         Lewisburg).
>
>         It would be nice to have information about the currently attached
>         devices (model and serial number) available in BMC inventory but what
>         are the ways to get this information?
>
>     Intel systems only get drive mdl/sn from NVMe devices that provide
>     that information via MCTP over SMBus. SAS/SATA drives, to my
>     current knowledge, don't supply this info in that mode.
>
>     The insertion/removal events are also managed from the BMC via I2C
>     to a FPGA solution residing on the Hot Swap Backplane. The FPGA
>     provides registers that allow the BMC to query the PRSTn/IFDETn pins.
>
>         Is it possible there's some additional module in the all-mighty ME
>         that can supply BMC with the current state of PCH SATA controller? Or
>         is there some other sensible mechanism other than talking to UEFI on
>         startup (which would mean BMC is showing stale and thus misleading
>         data till the next reboot)?
>
>         Even though the question is quite specific we'd appreciate any related
>         insights on the topic; there must be some side-channels to RAID
>         controllers and such for BMC to be obtaining essential information but
>         so far I wasn't able to find any documentation about this.
>
>     RAID solutions are outside of scope for the Intel BMC.
>
>       
>
>     --
>     Johnathan Mantey
>     Senior Software Engineer
>     *azad technology partners*
>     Contributing to Technology Innovation since 1992
>     Phone: (503) 712-6764
>     Email: johnathanx.mantey@intel.com
>
> -- 
> Best regards,
> Konstantin Klubnichkin,
> lead firmware engineer,
> server hardware R&D group,
> Yandex Moscow office.
> tel: +7-903-510-33-33
-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com

[-- Attachment #2: Type: text/html, Size: 5348 bytes --]

Re: SATA hotplug notifications for BMC inventory updates

[Paul Fertser]

(skipped discussion about SMI, as I guess an SMI handler can't
meaningfully communicate with the BMC anyway since that would add an
unacceptable delay to the host operation; I can imagine SMI sending
data to ME and then BMC retrieving it via IPMB but, again, I can't
find any good reference about ME facilities...)

On Wed, Sep 07, 2022 at 07:04:48AM -0700, Johnathan Mantey wrote:
> A back of envelope thought I had is perhaps the host side could interact with
> the BMC via a mailbox. The issue being you'd have to have SW running during the
> BIOS time, and when the OS was in control an OS helper app pushing the data. I'm
> not sure how feasible that idea is.

Some kind of agent running on host can certainly do that. But
apparently no popular GNU/Linux distros package anything like that, so
it must be a rare need.

I asked experienced VMWare ESXi admins and they told me even SATA
drives show in BMC properly without any additional vendor "vibs"
installed on the host system so there must be some side channel
present on many production server systems. 

-- 
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:fercerpav@gmail.com

Re: SATA hotplug notifications for BMC inventory updates

[Paul Fertser]

Hello Johnathan,

Thank you for answering, please see inline.

On Tue, Sep 06, 2022 at 08:53:09AM -0700, Johnathan Mantey wrote:
> On 9/5/22 04:11, Paul Fertser wrote:
> 
>  Hi,
> 
>  An OCP Tioga Pass platform features up to 4 drive bays for SATA
>  storage accessible from the front and fully hot-pluggable. The
>  backplane is usually connected directly to Intel C620 PCH (aka
>  Lewisburg).
> 
>  It would be nice to have information about the currently attached
>  devices (model and serial number) available in BMC inventory but what
>  are the ways to get this information?
> 
> Intel systems only get drive mdl/sn from NVMe devices that provide that
> information via MCTP over SMBus. SAS/SATA drives, to my current knowledge, don't
> supply this info in that mode.

Indeed, NVMe provides this additional side-channel over SMBus
specifically for management, alas nothing like that is present in
SAS/SATA standards.

> The insertion/removal events are also managed from the BMC via I2C to a FPGA
> solution residing on the Hot Swap Backplane. The FPGA provides registers that
> allow the BMC to query the PRSTn/IFDETn pins.

I can also imagine I2C GPIO expanders doing the job, yes, this seems
to be reasonably straightforward.

>  Is it possible there's some additional module in the all-mighty ME
>  that can supply BMC with the current state of PCH SATA controller? Or
>  is there some other sensible mechanism other than talking to UEFI on
>  startup (which would mean BMC is showing stale and thus misleading
>  data till the next reboot)?
> 
>  Even though the question is quite specific we'd appreciate any related
>  insights on the topic; there must be some side-channels to RAID
>  controllers and such for BMC to be obtaining essential information but
>  so far I wasn't able to find any documentation about this.
> 
> RAID solutions are outside of scope for the Intel BMC.

Intel VROC too? Why? We'd be willing to contribute some development
effort into bringing RAID monitoring and management to OpenBMC,
wouldn't the community benefit from that? Many other vendors currently
offer this feature, why shouldn't OpenBMC? The problem seems to be
with the documentation availability; can you please tell where the
relevant hardware and protocols are described?

-- 
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:fercerpav@gmail.com

Re: SATA hotplug notifications for BMC inventory updates

[Johnathan Mantey]

[-- Attachment #1: Type: text/plain, Size: 895 bytes --]


On 9/7/22 07:16, Paul Fertser wrote:
> Hello Johnathan,
>
> Thank you for answering, please see inline.
>
<snip>
> RAID solutions are outside of scope for the Intel BMC.
> Intel VROC too? Why? We'd be willing to contribute some development
> effort into bringing RAID monitoring and management to OpenBMC,
> wouldn't the community benefit from that? Many other vendors currently
> offer this feature, why shouldn't OpenBMC? The problem seems to be
> with the documentation availability; can you please tell where the
> relevant hardware and protocols are described?
>
Paul, this question now crosses a demarcation line. As a contract worker 
I'm not comfortable discussing the topics you raise in that paragraph.
-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com

[-- Attachment #2: Type: text/html, Size: 1987 bytes --]

Re: SATA hotplug notifications for BMC inventory updates

[Paul Fertser]

Hello Johnathan,

On Wed, Sep 07, 2022 at 07:36:08AM -0700, Johnathan Mantey wrote:
>>> RAID solutions are outside of scope for the Intel BMC.
>>
>> Intel VROC too? Why? We'd be willing to contribute some development
>> effort into bringing RAID monitoring and management to OpenBMC,
>> wouldn't the community benefit from that? Many other vendors
>> currently offer this feature, why shouldn't OpenBMC? The problem
>> seems to be with the documentation availability; can you please
>> tell where the relevant hardware and protocols are described?
> 
> Paul, this question now crosses a demarcation line. As a contract
> worker I'm not comfortable discussing the topics you raise in that
> paragraph.

I'm not fluent in business slang so I take it you mean the feature
might be nice but Intel doesn't want the OpenBMC community to have
it. Well, too bad.

-- 
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:fercerpav@gmail.com

Re: SATA hotplug notifications for BMC inventory updates

[Johnathan Mantey]

[-- Attachment #1: Type: text/plain, Size: 1333 bytes --]


On 9/9/22 01:32, Paul Fertser wrote:
> Hello Johnathan,
>
> On Wed, Sep 07, 2022 at 07:36:08AM -0700, Johnathan Mantey wrote:
<snip>
>> Paul, this question now crosses a demarcation line. As a contract
>> worker I'm not comfortable discussing the topics you raise in that
>> paragraph.
> I'm not fluent in business slang so I take it you mean the feature
> might be nice but Intel doesn't want the OpenBMC community to have
> it. Well, too bad.

What I'm trying to convey is that as a contract worker I have access to 
some information I may not be authorized to divulge. This is a public 
forum, and I need to tread carefully so that I don't divulge information 
that impacts my employment.

The starting point for this conversation was HDD insertion/removal event 
detection.
Which then evolved to HDD MFR/Model Num access.
Which then evolved into RAID controllers.

It may turn out that Intel does want this feature as part of OBMC.
It may turn out that Intel is amenable to the community implementing a 
RAID/BMC feature.

The transition to RAID controller has initiated an investigation into 
what Intel may be willing to enable regarding OBMC.

-- 
Johnathan Mantey
Senior Software Engineer
*azad te**chnology partners*
Contributing to Technology Innovation since 1992
Phone: (503) 712-6764
Email: johnathanx.mantey@intel.com

[-- Attachment #2: Type: text/html, Size: 2592 bytes --]

Re: SATA hotplug notifications for BMC inventory updates

[Bills, Jason M]

On 9/9/2022 8:12 AM, Johnathan Mantey wrote:
> 
> On 9/9/22 01:32, Paul Fertser wrote:
>> Hello Johnathan,
>>
>> On Wed, Sep 07, 2022 at 07:36:08AM -0700, Johnathan Mantey wrote:
> <snip>
>>> Paul, this question now crosses a demarcation line. As a contract
>>> worker I'm not comfortable discussing the topics you raise in that
>>> paragraph.
>> I'm not fluent in business slang so I take it you mean the feature
>> might be nice but Intel doesn't want the OpenBMC community to have
>> it. Well, too bad.
> 
> What I'm trying to convey is that as a contract worker I have access to 
> some information I may not be authorized to divulge. This is a public 
> forum, and I need to tread carefully so that I don't divulge information 
> that impacts my employment.
> 
> The starting point for this conversation was HDD insertion/removal event 
> detection.
> Which then evolved to HDD MFR/Model Num access.
> Which then evolved into RAID controllers.
> 
> It may turn out that Intel does want this feature as part of OBMC.
> It may turn out that Intel is amenable to the community implementing a 
> RAID/BMC feature.
> 
> The transition to RAID controller has initiated an investigation into 
> what Intel may be willing to enable regarding OBMC.

I have reached out to some of our RAID team internally.  They have a 
software-based management solution that they use, today, but it only 
covers NVMe-MI based drives.

I also asked about documentation that may help with this, but since they 
only support NVMe-MI, they don't have much info about SATA support.

I'd als recommend contacting your Intel support representative to ask 
about relevant documentation.

> 
> -- 
> Johnathan Mantey
> Senior Software Engineer
> *azad te**chnology partners*
> Contributing to Technology Innovation since 1992
> Phone: (503) 712-6764
> Email: johnathanx.mantey@intel.com
> 

Re: SATA hotplug notifications for BMC inventory updates

[Paul Fertser]

Hello Jason,

Thank you very much for the reply, please see inline.

On Fri, Sep 09, 2022 at 03:19:45PM -0600, Bills, Jason M wrote:
> > The transition to RAID controller has initiated an investigation into
> > what Intel may be willing to enable regarding OBMC.
> 
> I have reached out to some of our RAID team internally.  They have a
> software-based management solution that they use, today, but it only covers
> NVMe-MI based drives.
> 
> I also asked about documentation that may help with this, but since they
> only support NVMe-MI, they don't have much info about SATA support.

I have seen the Intel VROC slides [0], and they mention some
facilities for integration with BMCs. I assumed they're generic for
all the hardware handled by VROC (including SATA as RSTe was
apparently made part of VROC) but now that I re-read it, I see "NVMe
RAID BMC OOB management via OS, UEFI OOB management coming with Intel
VROC 8.0" is apparently NVMe-specific.

So even if this is not useful for Tioga Pass it would still be nice to
support this feature from OpenBMC side for the next generation
servers, and we plan to work on that.

> I'd als recommend contacting your Intel support representative to ask about
> relevant documentation.

Due to the horrible actions of Russia we are no longer permitted to
talk to the Intel support representatives. So we have to count on open
documentation and communication, and we are willing to share
everything useful with the OpenBMC upstream.


[0] https://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/vroc-overview.pdf
-- 
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:fercerpav@gmail.com

Re: SATA hotplug notifications for BMC inventory updates

[Paul Fertser]

Hello Johnathan,

On Fri, Sep 09, 2022 at 07:12:21AM -0700, Johnathan Mantey wrote:
> What I'm trying to convey is that as a contract worker I have access to some
> information I may not be authorized to divulge. This is a public forum, and I
> need to tread carefully so that I don't divulge information that impacts my
> employment.

Thank you for the reply, it's now all pretty clear to me, much
appreciated.

> The starting point for this conversation was HDD insertion/removal event
> detection.
> Which then evolved to HDD MFR/Model Num access.

Manufacturer, model and serial number are all needed for proper
inventory. I mentioned hotplug events because cold plug is not
problematic at all: UEFI can enumerate all the devices and send
information via IPMB to BMC on startup.

> Which then evolved into RAID controllers.

That also follows naturally as when the storage is attached to a RAID
controller, or an HBA in non-RAID mode, or handled by Intel VROC the
BMC might need to communicate with them to show accurate
inventory. And if we're talking to a controller anyway it's nice to
implement RAID management as well. And I know that's working with
other BMCs, so the hardware connections must be there already, we just
need to write software to make use of them.

-- 
Be free, use free (http://www.gnu.org/philosophy/free-sw.html) software!
mailto:fercerpav@gmail.com