Design to isolate BMC service access

Design to isolate BMC service access

[Joseph Reynolds]

This is to gather information needed to write a design(1) for a BMC 
function to upload a "service access token" which gives access to BMC 
internals to a service agent.

BMCWeb uses the DMTF Redfish standard(2)(3), and I will pursue a Redfish 
spec change for this (or a reading that we should create a custom OEM 
solution which I would pursue at the OpenBMC community level).  I 
believe we can and should (at least partially) standardize some REST 
APIs in this area.

This topic was introduced and briefly discussed in the OpenBMC Security 
Working Group meeting(4) on 2023-03-29 where I agreed to write a design 
for this.  I intend to move forward with this email and a proper design.

If this interests you, please read and reply with changes needed to fit 
your scheme.  Study questions are at the bottom of the design sketch.

- Joseph

References:

 1. https://github.com/openbmc/docs/blob/master/designs/design-template.md
 2. https://redfish.dmtf.org/
 3. https://github.com/openbmc/bmcweb/blob/master/Redfish.md
 4. https://docs.google.com/document/d/1b7x9BaxsfcukQDqbvZsU2ehMq4xoJRQvLxxsDUWmAOI


------------------------------------------------------------------------

*Problem description*

The BMC has:

 1. External interfaces such as REST APIs.  These provide full
    operational control over the BMC.
 2. Internal interfaces within the BMC such as D-Bus, Systemd, and the
    sysfs file system.  These provide full access to the BMC's raw
    capabilities, are needed to provide the BMC's external function, and
    are needed to diagnose and fix problems.  Direct access to these
    internal interfaces is typically via SSH to the BMC's command shell.
 3. A default root user who has full administrator access to the BMC's
    external interfaces and access to the BMC's internal interfaces.


In some use cases, it is desirable to isolate the internal interfaces 
away from administrator users.  For example, when the BMC part of a 
system which has sensitive data, you want to isolate the BMC user from 
what little access the BMC has to that data.  When this use case is 
desired, a typical deployment involves three organizations:

 1. Development.  The development team creates the BMC's firmware, is
    responsible for the function of all internal and external
    interfaces, and may be needed to debug complex problems on
    operational systems.
 2. Operations.  The operations team installs the BMC, and the BMC
    administrator operates the BMC *only* via its intended external
    interface, and does not allow access to BMC internals.
 3. Service.  The service team diagnoses and debugs problems on
    operational BMCs, and sometimes needs to use the BMC's internal
    interfaces.


The trust relationships from an operational point of view:

 1. BMC administrators are trained only to operate the BMC's external
    interfaces.  When they find a problem, they call for authorized
    service agents.  To be extra careful, they may remove any sensitive
    workloads from the system before allowing service technicians to
    access the BMC.
 2. BMC service technicians are trained to use the BMC's internal
    interfaces.  They may not be fully trusted by the operations team
    who may carefully watch them or may operate the BMC's internal
    controls for them.
 3. BMC developers may be called to diagnose problems, and sometimes
    need access to the failing BMC's internal interfaces.


In addition, the development team may not trust the operations team to 
have access to BMC internals.  For example, operations' access to BMC 
internals can cause problems for which development may be blamed.  
Developers want to lock out everyone except (trained and trusted) 
authorized service agents from the BMC's internal interfaces.

To be clear, the problem is:

 1. The administrator users have access to the BMC's internal interfaces
    which they do not need, and that access can be used to harm the
    system.  Also developers may want to lock out access to the BMC's
    internal interfaces.
 2. If this access it taken away then this limits the capability to
    service the BMC.  We must retain the ability to access the BMC's
    internal interfaces so we can debug the BMC.
 3. Both the BMC administrator and service must agree before access to
    BMC internals is granted.



*The solution*

A way to solve this access problem has three pieces (all needed for a 
complete solution):

 1. Make BMC firmware so administrators are only allowed to access to
    the BMC's intended operational interfaces (and are not allowed to
    access BMC internal interfaces).  For example, move away from root
    account logins, and create a new "admin" account which can access
    only the BMC's external interfaces.
 2. Add function to the BMC so it can be accessed by a service user,
    where access is disabled by default, and where access can be to
    varying levels of BMC internal functions (for example, using
    "service" APIs to perform common functions, or root access to the
    BMC command shell for the deepest or most permissive level of access).
 3. Provide a way for a BMC admin user to request service access to
    their BMC, with 2 requirements:
     1. Only the BMC admin should be allowed to enable this access,
        meaning a service user should not be allowed to self-enable
        their own access (presuming they don't also have admin access).
     2. When service access is enabled, the admin user should not have
        service access.  For example, the service user should have to
        authenticate to the BMC using credentials not known to the admin.


Note: This design does not give the solution to create an "admin" 
account as in solution point 1 above.  That part of the solution is 
necessary, but it can be provided elsewhere.  This design addresses 
points 2 and 3.

A "service access token" is proposed.  Details are below but for now, a 
service access token:

  * Is a small file (kilobytes), a digitally-signed request to access a
    specific BMC function on a specific BMC for a limited time window. 
    This token may have additional information about its origin, etc.
  * Is created by an authorized service agent.  Only service agents can
    digitally sign the tokens so they can be verified by the BMC.
  * Is uploaded to the BMC by an admin user to perform a specific
    service function.
  * Has nothing that is secret to the BMC admin user.  If the token
    encodes a password, it is stored in the form of a secure hash.


Here is a sketch of the steps a BMC admin and their service agent would 
use to make a service call to gain access to BMC internals:

 1. The administrator gathers information about their BMC.  They may get
    the system model and serial number, or use the system to generate a
    token (via the system's root of trust) needed to make the service
    call.  The admin passes this data to the service agent along with
    their request for service.
 2. The service agent receives the data and (using their privileged
    position behind their organization’s firewall) creates a "service
    access token" needed to gain service access to the BMC.
 3. The service rep gives (via shared storage, or email, etc) the
    service access token to the BMC admin.
 4. The BMC admin uploads the service access token to the BMC. Doing so
    enables the service function indicated within the service access
    token.  Design question: Should the function be activated when it is
    uploaded, or via a separate activate function?
 5. If the service function is to allow root login to the BMC command
    shell, the service user can now login to the BMC, using a unique
    password associated with the service access token, and known only them.
 6. Other popular functions might be to recover the admin account,
    disable various security features, or perform a service dump.
    Example: Customers regularly call for service because they lost
    access to their admin account.  Recovery means, for example:
    recreate the admin account or set it to a usable state, and set its
    password to a known value, reset its password lockout, etc.
 7. The service agent then deletes or deactivates the service access
    token or allows it to expire.


To simplify the design and implementation, at most one service access 
token is allowed on the BMC at any given time.  Design question: Is this 
okay with the service planner?

Anti-replay protection for these access tokens is assumed.  For example, 
an access token used to get access to a BMC command shell could not be 
used twice: the second attempt to upload it should result in permission 
denied with reason: anti-replay protection. (In this example, login 
access is allowed multiple times until the ACF expires or is deleted.)


New BMC functions

This new "service access" function is optionally built into the firmware 
image, controlled by an image feature, and defaults to disabled (which 
means the REST APIs and underlying function is not present).  Enabling 
this feature includes the new REST APIs and its implementation on the BMC.

The new BMC REST API functions (used by an admin user) for this are:

 1. Generate request for service.  The BMC returns a small file to the
    admin (which contains the BMC model and serial number, or a signed
    request, etc).
 2. Upload (POST) the service access token.  The BMC admin uses this
    upload the service access token to the BMC.
 3. Inspect (GET) information about the service access token.
 4. Activate the service access token.  This causes the service access
    token to do its thing (take a service dump, activate  the "service"
    account login, etc.)
 5. Delete the service access token.


The backend function which runs on the BMC would be a new D-Bus service 
to handle the "service access token", possibly with 2 popular functions: 
(1) enable service account access to the BMC command shell, and (2) 
recover admin account access.  The implementor would have freedom to 
customize these functions and to provide their own custom functions.  
Each manufacturer would have a different set of purposes for the access 
tokens, and the use cases would not be shared.

This function could be used to begin to standardize various existing 
custom schemes such as IBM's Access Control File (ACF), Microsoft's 
Secure Unlock, and others.


*Questions*

 1. Is this design sketch clear?  What improvements are needed?
 2. Who in the OpenBMC community can use this?  IBM and Microsoft have
    discussed this.
 3. Is this the right set of BMC functions to support all the use
    cases?  What else is needed?
 4. Should the "service access token" be an X.509 certificate?  Or is
    that inappropriate?
 5. Is Redfish interested in putting this into the spec (or is this
    already spec'd)?  Versus OpenBMC OEM solution.
 6. Does threading this through BMCWeb, D-Bus, and a new Systemd D-Bus
    service seem right?
 7. Should uploading the service access token also activate it
    immediately, or should that be a separate step?  For example, a BMC
    admin might want to: (A) upload a token, (B) inspect the token
    (using the BMC function) to ensure it looks legitimate and perform
    the function they agreed to, and then finally (C) activate the
    token, for example, to disable secure boot.
 8. Does it make sense to have a common implementation for the functions
    as listed above (like to recover admin account access).

Re: Design to isolate BMC service access

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 3136 bytes --]


Joseph Reynolds <jrey@linux.ibm.com> wrote:
    > A "service access token" is proposed.  Details are below but for now, a
    > service access token:

    >  * Is a small file (kilobytes), a digitally-signed request to access a
    > specific BMC function on a specific BMC for a limited time window. 
    > This token may have additional information about its origin, etc.  * Is
    > created by an authorized service agent.  Only service agents can
    > digitally sign the tokens so they can be verified by the BMC.  * Is
    > uploaded to the BMC by an admin user to perform a specific service
    > function.  * Has nothing that is secret to the BMC admin user.  If the
    > token encodes a password, it is stored in the form of a secure hash.

So it's a bearer token (?) that is encrypted by the BMC to itself?
Or it's created by an external entity?

Very much exactly OAUTH2-like: JWT, CWT. In fact... I suggest not
re-inventing the wheel here.

Do you intend to sometimes bind the token to the specific user (service team,
I think).

    > call.  The admin passes this data to the service agent along with their
    > request for service.

There are now some IETF protocols (TIGRESS WG) that enable this secure transfer.

    > indicated within the service access token.  Design question: Should the
    > function be activated when it is uploaded, or via a separate activate
    > function?

When it is uploaded.
If you want it to take effect in the future, then the token should have a
notBefore entry.

    > 5. If the service function is to allow root login to the BMC
    > command shell, the service user can now login to the BMC, using a
    > unique password associated with the service access token, and known
    > only them.

No passwords.
If you are going to use SSH, then use SSH keys here.

    > 6. Other popular functions might be to recover the admin
    > account, disable various security features, or perform a service dump.
    > Example: Customers regularly call for service because they lost access
    > to their admin account.  Recovery means, for example: recreate the
    > admin account or set it to a usable state, and set its password to a
    > known value, reset its password lockout, etc.

That seems like a chicken and egg situation, but maybe I don't understand it.

    > 1. Is this design sketch clear?  What improvements are needed?

Relatively.

    > 4. Should the "service access
    > token" be an X.509 certificate?  Or is that inappropriate?

No.  It's just gonna confuse people.

    > should that be a separate step?  For example, a BMC admin might want
    > to: (A) upload a token, (B) inspect the token (using the BMC function)
    > to ensure it looks legitimate and perform the function they agreed to,
    > and then finally (C) activate the token, for example, to disable secure
    > boot.

Maybe that's a reasonable work flow.

    > 8. Does it make sense to have a common implementation for the
    > functions as listed above (like to recover admin account access).

I don't know.



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 511 bytes --]

Re: Design to isolate BMC service access

[Ed Tanous]

On Thu, Apr 6, 2023 at 12:10 PM Michael Richardson <mcr@sandelman.ca> wrote:
>
>
> Very much exactly OAUTH2-like: JWT, CWT. In fact... I suggest not
> re-inventing the wheel here.

+1.  My first thought here was "Doesn't Redfish already support
oauth?".  If this is different, let's call out exactly why it's
different.

Re: Design to isolate BMC service access

[Joseph Reynolds]

On 4/6/23 2:00 PM, Michael Richardson wrote:
> Joseph Reynolds <jrey@linux.ibm.com> wrote:
>      > A "service access token" is proposed.  Details are below but for now, a
>      > service access token:
>
>      >  * Is a small file (kilobytes), a digitally-signed request to access a
>      > specific BMC function on a specific BMC for a limited time window.
>      > This token may have additional information about its origin, etc.  * Is
>      > created by an authorized service agent.  Only service agents can
>      > digitally sign the tokens so they can be verified by the BMC.  * Is
>      > uploaded to the BMC by an admin user to perform a specific service
>      > function.  * Has nothing that is secret to the BMC admin user.  If the
>      > token encodes a password, it is stored in the form of a secure hash.
>
> So it's a bearer token (?) that is encrypted by the BMC to itself?
> Or it's created by an external entity?

Michael,  +cc: Ed Tanous
Thanks for your input!

This token is digitally signed by the service organization behind their 
firewall.  When it is uploaded to the BMC the signature is verified by 
the BMC.  I didn't describe the infrastructure needed for this, but it 
includes the following pieces:
1. The BMC functions described in this design sketch.
2. The service organization creates a key pair for the service access token:
2A. The private key is held by the service organization to digitally 
sign the token.
2B. The public keys gets built into the BMC firmware image to validate 
the token's signature.

The "steps" in the next paragraph of the original design sketch are 
intended to show when the token is signed and validated...
In step 2, the service agent creates the token and uses their privileged 
access (behind their organization's firewall) to the private key to sign 
the token.
In step 3, the BMC admin gets a copy of this signed token and uploads it 
to the BMC.
In step 4, the BMC validates the token's signature.

Note that step 1 describes the BMC optionally generating a "service 
request token" which might be signed by the BMC and validated by the 
service organization (the flow is the opposite direction from the 
"service access token").


> Very much exactly OAUTH2-like: JWT, CWT. In fact... I suggest not
> re-inventing the wheel here.

Thanks for that!  I think that would work.  The service organization 
would build an OAuth2 JSON Web Token (JWT) which is uploaded to the 
BMC.  I will study up and try to redo this design sketch in those terms.


>
> Do you intend to sometimes bind the token to the specific user (service team,
> I think).

Yes, the basic premise is: the token is (a) created by the service agent 
(person) to (b) do something specific on the BMC while using the BMC's 
service account.


>
>      > call.  The admin passes this data to the service agent along with their
>      > request for service.
>
> There are now some IETF protocols (TIGRESS WG) that enable this secure transfer.
>
>      > indicated within the service access token.  Design question: Should the
>      > function be activated when it is uploaded, or via a separate activate
>      > function?
>
> When it is uploaded.
> If you want it to take effect in the future, then the token should have a
> notBefore entry.
>
>      > 5. If the service function is to allow root login to the BMC
>      > command shell, the service user can now login to the BMC, using a
>      > unique password associated with the service access token, and known
>      > only them.
>
> No passwords.
> If you are going to use SSH, then use SSH keys here.

I agree that makes sense for SSH users, but our service organization has 
situations where they are not allowed to take any electronic materials 
into the customer site (and the customer will not carry it either), so 
our service techs sometimes still need an option for password access.  
Perhaps password access should not be the default option.


>
>      > 6. Other popular functions might be to recover the admin
>      > account, disable various security features, or perform a service dump.
>      > Example: Customers regularly call for service because they lost access
>      > to their admin account.  Recovery means, for example: recreate the
>      > admin account or set it to a usable state, and set its password to a
>      > known value, reset its password lockout, etc.
>
> That seems like a chicken and egg situation, but maybe I don't understand it.

You got it, "How can an admin give permission to a service agent to 
recover the admin account of their BMC, if the admin is locked out?" 
-->  One way to accomplish this is to have assert "physical presence" 
(such as invoking function 74 on the operator panel -- which can only be 
done by someone physically present at the system), and when this is done 
the BMC will omit the admin authorization required to upload the service 
token (but perform all other checks).  (Then the system owner has to 
keep authorized service agents away from their systems!)  I do not 
intend to try to standardize this approach.
Is there a better way to solve this problem?


>
>      > 1. Is this design sketch clear?  What improvements are needed?
>
> Relatively.
>
>      > 4. Should the "service access
>      > token" be an X.509 certificate?  Or is that inappropriate?
>
> No.  It's just gonna confuse people.
>
>      > should that be a separate step?  For example, a BMC admin might want
>      > to: (A) upload a token, (B) inspect the token (using the BMC function)
>      > to ensure it looks legitimate and perform the function they agreed to,
>      > and then finally (C) activate the token, for example, to disable secure
>      > boot.
>
> Maybe that's a reasonable work flow.
>
>      > 8. Does it make sense to have a common implementation for the
>      > functions as listed above (like to recover admin account access).
>
> I don't know.
>
>

Re: Design to isolate BMC service access

[Michael Richardson]

[-- Attachment #1: Type: text/plain, Size: 4099 bytes --]


Joseph Reynolds <jrey@linux.ibm.com> wrote:
    > On 4/6/23 2:00 PM, Michael Richardson wrote:
    >> Joseph Reynolds <jrey@linux.ibm.com> wrote: > A "service access token"
    >> is proposed.  Details are below but for now, a > service access token:
    >> 
    >> > * Is a small file (kilobytes), a digitally-signed request to access
    >> a > specific BMC function on a specific BMC for a limited time window.
    >> > This token may have additional information about its origin, etc.  *
    >> Is > created by an authorized service agent.  Only service agents can
    >> > digitally sign the tokens so they can be verified by the BMC.  * Is
    >> > uploaded to the BMC by an admin user to perform a specific service >
    >> function.  * Has nothing that is secret to the BMC admin user.  If the
    >> > token encodes a password, it is stored in the form of a secure hash.
    >> 
    >> So it's a bearer token (?) that is encrypted by the BMC to itself?  Or
    >> it's created by an external entity?

    > Michael,  +cc: Ed Tanous Thanks for your input!

    > This token is digitally signed by the service organization behind their
    > firewall.  When it is uploaded to the BMC the signature is verified by
    > the BMC.  I didn't describe the infrastructure needed for this, but it
    > includes the following pieces: 1. The BMC functions described in this
    > design sketch.  2. The service organization creates a key pair for the
    > service access token: 2A. The private key is held by the service
    > organization to digitally sign the token.  2B. The public keys gets
    > built into the BMC firmware image to validate the token's signature.

2B seems like a kicker to me.
I have been working on (too slowly, without enough resources), an RFC8995
client (pledge) for OpenBMC, which would allow for that token validation
public key/trust-anchor to be delivered to the BMC at boot time/installation.

RFC7030 (EST) seems like a very good way to do this, and even without RFC8995
to automate it, entering a URL into the BMC web interface and having it
transfer the trust-anchor seems like a good thing.  That also gets it access
to CRLs and to refreshing the trust-anchor.

I think that there should also be a self-signed option where the BMC creates
the token itself.  In that case, having it be a bearer token has pluses and
minuses.  linking it to a TLS Client certificate would be good in some use
cases.

I think that there are two deployment environments:
1) the facebook/etc. scale places where they just need automation and can pay
   for it.
   
2) the SMEs where they have a few dozen systems, installed by a variety of
   people with a variety of skills, probably at least one of them have left.
   They are chronically under-resourced, and being able to recover easily is
   important.

(The really SMALL places with four systems can probably just poke a button to reset
the BMC to factory defaults.  I had to do that to a 2010 era server on
Wednesday after the ice storm killed power, and the UPS went with it... Gosh
I wish that system was running something modern)

    >> Very much exactly OAUTH2-like: JWT, CWT. In fact... I suggest not
    >> re-inventing the wheel here.

    > Thanks for that!  I think that would work.  The service organization
    > would build an OAuth2 JSON Web Token (JWT) which is uploaded to the
    > BMC.  I will study up and try to redo this design sketch in those
    > terms.

Yes.  That way, you can leverage not just the crypto primitives, but also the
operational experience around that.

    >> Do you intend to sometimes bind the token to the specific user
    >> (service team, I think).

    > Yes, the basic premise is: the token is (a) created by the service
    > agent (person) to (b) do something specific on the BMC while using the
    > BMC's service account.

Good.


-- 
Michael Richardson <mcr+IETF@sandelman.ca>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide





[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 515 bytes --]