All of lore.kernel.org
 help / color / mirror / Atom feed
* Security Working Group - Wednesday July 22
@ 2020-07-20 13:57 Joseph Reynolds
  2020-07-23 14:11 ` Security Working Group - Wednesday July 22 - results Joseph Reynolds
  0 siblings, 1 reply; 8+ messages in thread
From: Joseph Reynolds @ 2020-07-20 13:57 UTC (permalink / raw)
  To: openbmc

This is a reminder of the OpenBMC Security Working Group meeting 
scheduled for this Wednesday July 22 at 10:00am PDT.

We'll discuss current development items, and anything else that comes up.

1. The OpenBMC interface overview is merged into the docs repository 
here: 
https://github.com/openbmc/docs/blob/master/architecture/interface-overview.md.  
Is there interest in building a threat model on top of this?

2. A gerrit review merged. It is a rework of BMCWeb authorization flow: 
https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/30994

and tweaks some security settings.  Is there interest in reviewing the 
code or changed settings?  (Please note: This changed was introduced 
Months ago and it went unnoticed in the security workgroup.  Better late 
than never.)

3.Gerrit review: Firmware minimum ship level (can help with host 
firmware anti-rollback protection) 
https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-bmc-code-mgmt/+/29914
Access, agenda, and notes are in the wiki:
https://github.com/openbmc/openbmc/wiki/Security-working-group

- Joseph

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Security Working Group - Wednesday July 22 - results
  2020-07-20 13:57 Security Working Group - Wednesday July 22 Joseph Reynolds
@ 2020-07-23 14:11 ` Joseph Reynolds
  2020-07-23 15:13   ` Ed Tanous
  0 siblings, 1 reply; 8+ messages in thread
From: Joseph Reynolds @ 2020-07-23 14:11 UTC (permalink / raw)
  To: openbmc



On 7/20/20 8:57 AM, Joseph Reynolds wrote:
> This is a reminder of the OpenBMC Security Working Group meeting 
> scheduled for this Wednesday July 22 at 10:00am PDT.
>
> We'll discuss current development items, and anything else that comes up.
>
> 1. The OpenBMC interface overview is merged into the docs repository 
> here: 
> https://github.com/openbmc/docs/blob/master/architecture/interface-overview.md. 
> Is there interest in building a threat model on top of this?
No discussion.

>
> 2. A gerrit review merged. It is a rework of BMCWeb authorization 
> flow: https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/30994
>
> and tweaks some security settings.  Is there interest in reviewing the 
> code or changed settings?  (Please note: This changed was introduced 
> Months ago and it went unnoticed in the security workgroup.  Better 
> late than never.)
No discussion
>
> 3.Gerrit review: Firmware minimum ship level (can help with host 
> firmware anti-rollback protection) 
> https://gerrit.openbmc-project.xyz/c/openbmc/phosphor-bmc-code-mgmt/+/29914
> Access, agenda, and notes are in the wiki:
> https://github.com/openbmc/openbmc/wiki/Security-working-group
No discussion

Topics added after the meeting notice was sent:

4 Question: If BMCWeb finds an unusable HTTPS site identity certificate, 
it DELETES it and self-generates one.  This has caused problems for 
certificates that are not valid until a future date. In general, what 
certificate management support should we have for BMCWeb?  What is needed?
ANSWER:
There were two discussion threads: The BMC’s notion of time of day 
(TOD), and how BMCWeb should handle certificates.

Does the BMC have a battery backed TOD clock?  Depends on BMC model.  
Can it validate if it has access to its NTP server (when configured)?  
Does the BMC know if its time was set correctly?

How does the BMC know if the BMC has the correct time?  Have a BMC flag 
that says, “Look like the BMC TOD clock is not working.”  Does the BMC 
know if we got a good time from an  NTP server?  Can we read the GPS 
signal?  What is the industry solution?
Should the BMC store its idea of what date it is?  So it can report if 
the time changes significantly.  Or will this lead to a bigger problem?  
Is it better/simpler to check for TOD = beginning-of-era-1/1/1970?  → 
start an email thread

BMCWeb configuration?  Configure option: delete cert and generate 
self-signed -vs- use defective certificate.  What is the purpose of 
deleting the unusable cert?  Should “out of date” not be part of the 
“unusable” definition? ⇒ Ideas: 1. If bmcweb finds a usable cert but is 
out of date, that cert can still be used.  2. Leave the defective 
certificate (do not delete it) and log an error.

The group consensus was that BMCWeb should treat its HTTPS site identity 
certificate like this:
1 certificate is perfectly good - Use the certificate
2 certificate is good but expired or not yet valid - Use the certificate 
and log a warning
3 certificate is missing or bad format or algorithm too old - Use 
another certificate or self-generate a certificate (and log that action)
There are no cases where BMCWeb should delete any certificate.

Next steps: discuss on email list, write patch.

5 Fuzzing.  We briefly discussed the existing test infrastructure 
https://github.com/openbmc/openbmc-test-automation/ and previous calls 
for fuzzing.
If someone wanted to perform fuzzing, where would they start, what tools 
should they use, etc.

6. Can we fill out the information in 
https://github.com/openbmc/openbmc/security ?


- Joseph

>
> - Joseph

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Security Working Group - Wednesday July 22 - results
  2020-07-23 14:11 ` Security Working Group - Wednesday July 22 - results Joseph Reynolds
@ 2020-07-23 15:13   ` Ed Tanous
  2020-07-23 16:04     ` Joseph Reynolds
                       ` (2 more replies)
  0 siblings, 3 replies; 8+ messages in thread
From: Ed Tanous @ 2020-07-23 15:13 UTC (permalink / raw)
  To: Joseph Reynolds; +Cc: openbmc

First off, if you are finding certificates are expiring in a
production environment, you need to rethink your certificate
replacement strategy.  The BMC cannot be the primary entity in charge
of certificate replacement and revocation, and the fact that this is
happening points to a greater organizational problem than the bmc can
cover.

One thing to note;  At one point, I had talked through how to
prototype ACME protocol replacement of certificates automatically, so,
given an ACME server on the network, the BMC could essentially
automatically provision itself and keep its certs up to date.  If
someone wanted to run with that, it might reduce some of the pain here
(and be extremely cool).

On Thu, Jul 23, 2020 at 7:24 AM Joseph Reynolds <jrey@linux.ibm.com> wrote:
>
>
> 4 Question: If BMCWeb finds an unusable HTTPS site identity certificate,
> it DELETES it and self-generates one.  This has caused problems for
> certificates that are not valid until a future date. In general, what
> certificate management support should we have for BMCWeb?  What is needed?

This is not how bmcweb was intended to work, and I had explicitly
tested that it doesn't have this behavior (prior to it being moved
over to certificate manager).  The only time it previously would
replace a certificate was if the cert file didn't exist, or was
completely unreadable.  I suspect the right thing here is to bisect
the commit that broke this behavior (I'll bet it's easy to find) and
get a bug filed with the author so they can either fix it or revert
the patch.

This one smells awfully suspicious: 082f28fd2588fd9fcd9452ad38234ce875319163.

> ANSWER:
> There were two discussion threads: The BMC’s notion of time of day
> (TOD), and how BMCWeb should handle certificates.
>
> Does the BMC have a battery backed TOD clock?  Depends on BMC model.
> Can it validate if it has access to its NTP server (when configured)?
> Does the BMC know if its time was set correctly?
>
> How does the BMC know if the BMC has the correct time?  Have a BMC flag
> that says, “Look like the BMC TOD clock is not working.”  Does the BMC
> know if we got a good time from an  NTP server?  Can we read the GPS
> signal?  What is the industry solution?
> Should the BMC store its idea of what date it is?  So it can report if
> the time changes significantly.  Or will this lead to a bigger problem?
> Is it better/simpler to check for TOD = beginning-of-era-1/1/1970?  →
> start an email thread

The above is all asking the wrong question: "Can we determine if the
certificate is valid?"  This is irrelevant, the question is: "Should
we ever be replacing a user provided certificate with one generated on
the BMC."  The answer previously has been no.  In almost all cases the
user provided certificate, even an expired one, will still be better
than one the BMC self signs.  Between having an invalid certificate
chain, and an invalid date, I'll take the invalid date every time.

It should be noted, most browsers (in my testing) seem to ignore the
HTTP date header entirely, so the BMC doesn't even need the correct
time to set up a proper encryption channel.

>
> BMCWeb configuration?  Configure option: delete cert and generate
> self-signed -vs- use defective certificate.  What is the purpose of
> deleting the unusable cert?

This question needs answered.  I suspect this was unintentional, and
someone simply used some bad openssl code to attempt to verify the
cert against the (possibly non existent) chain without realizing this
requirement.

>  Should “out of date” not be part of the
> “unusable” definition? ⇒ Ideas: 1. If bmcweb finds a usable cert but is
> out of date, that cert can still be used.  2. Leave the defective
> certificate (do not delete it) and log an error.

A lot of BMCs don't have a dedicated RTC, and rely on other systems
(like the PCH or NTP) to get the correct time.  bmcweb needs to come
up long before the PCH or NTP (both of which are also optional) so as
a general rule, using these for valid time is a non-starter.  I could
see logging an error _if_ you know time is valid, but I'm not sure how
a bmc could know that.

>
> The group consensus was that BMCWeb should treat its HTTPS site identity
> certificate like this:
> 1 certificate is perfectly good - Use the certificate
> 2 certificate is good but expired or not yet valid - Use the certificate
> and log a warning

Add "and we know time has been set appropriately".  Also, be careful
with not yet valid.  I don't know how openssl chains handle clock skew
between certificate generator and client.  If the BMC has a time
that's 1 minute fast, is the certificate "not yet valid"?

> 3 certificate is missing or bad format or algorithm too old - Use
> another certificate or self-generate a certificate (and log that action)

We shouldn't be replacing certificates unless it's completely missing
(ie, we're on a first boot or a factory reset) and wouldn't be
switching certificates on the fly.  If the algorithm is too old, the
user is free to replace it with their own, which is the right
procedure anyway.  We also have no idea if the user has added this
public certificate to their local root store, so replacing it out of
the blue might look like an attack.  Also, we have no idea if the
users' client systems support the new crypto format.  We could be
unintentionally DOSing them if, say, we turn on a new keytype or size
and their clients don't support that key.

> There are no cases where BMCWeb should delete any certificate.

I think I know what you mean, but to be clear, we should delete the
old certificates when they're replaced.  We should delete root trust
store certificates when requested, ect.  I don't think we can say "no
cases" here.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Security Working Group - Wednesday July 22 - results
  2020-07-23 15:13   ` Ed Tanous
@ 2020-07-23 16:04     ` Joseph Reynolds
  2020-07-23 19:05     ` Michael Richardson
  2020-08-04  9:36     ` Jayanth Othayoth
  2 siblings, 0 replies; 8+ messages in thread
From: Joseph Reynolds @ 2020-07-23 16:04 UTC (permalink / raw)
  To: Ed Tanous; +Cc: openbmc

On 7/23/20 10:13 AM, Ed Tanous wrote:
> First off, if you are finding certificates are expiring in a
> production environment, you need to rethink your certificate
> replacement strategy.  The BMC cannot be the primary entity in charge
> of certificate replacement and revocation, and the fact that this is
> happening points to a greater organizational problem than the bmc can
> cover.
>
> One thing to note;  At one point, I had talked through how to
> prototype ACME protocol replacement of certificates automatically, so,
> given an ACME server on the network, the BMC could essentially
> automatically provision itself and keep its certs up to date.  If
> someone wanted to run with that, it might reduce some of the pain here
> (and be extremely cool).
>
> On Thu, Jul 23, 2020 at 7:24 AM Joseph Reynolds <jrey@linux.ibm.com> wrote:
>>
>> 4 Question: If BMCWeb finds an unusable HTTPS site identity certificate,
>> it DELETES it and self-generates one.  This has caused problems for
>> certificates that are not valid until a future date. In general, what
>> certificate management support should we have for BMCWeb?  What is needed?
> This is not how bmcweb was intended to work, and I had explicitly
> tested that it doesn't have this behavior (prior to it being moved
> over to certificate manager).  The only time it previously would
> replace a certificate was if the cert file didn't exist, or was
> completely unreadable.  I suspect the right thing here is to bisect
> the commit that broke this behavior (I'll bet it's easy to find) and
> get a bug filed with the author so they can either fix it or revert
> the patch.
>
> This one smells awfully suspicious: 082f28fd2588fd9fcd9452ad38234ce875319163.
>
>> ANSWER:
>> There were two discussion threads: The BMC’s notion of time of day
>> (TOD), and how BMCWeb should handle certificates.
>>
>> Does the BMC have a battery backed TOD clock?  Depends on BMC model.
>> Can it validate if it has access to its NTP server (when configured)?
>> Does the BMC know if its time was set correctly?
>>
>> How does the BMC know if the BMC has the correct time?  Have a BMC flag
>> that says, “Look like the BMC TOD clock is not working.”  Does the BMC
>> know if we got a good time from an  NTP server?  Can we read the GPS
>> signal?  What is the industry solution?
>> Should the BMC store its idea of what date it is?  So it can report if
>> the time changes significantly.  Or will this lead to a bigger problem?
>> Is it better/simpler to check for TOD = beginning-of-era-1/1/1970?  →
>> start an email thread
> The above is all asking the wrong question: "Can we determine if the
> certificate is valid?"  This is irrelevant, the question is: "Should
> we ever be replacing a user provided certificate with one generated on
> the BMC."  The answer previously has been no.  In almost all cases the
> user provided certificate, even an expired one, will still be better
> than one the BMC self signs.  Between having an invalid certificate
> chain, and an invalid date, I'll take the invalid date every time.
>
> It should be noted, most browsers (in my testing) seem to ignore the
> HTTP date header entirely, so the BMC doesn't even need the correct
> time to set up a proper encryption channel.

We weren't sure what question to ask.  Thanks for the discussion.

I sent a followup email with some better-phrased questions, but our 
email crossed.

>
>> BMCWeb configuration?  Configure option: delete cert and generate
>> self-signed -vs- use defective certificate.  What is the purpose of
>> deleting the unusable cert?
> This question needs answered.  I suspect this was unintentional, and
> someone simply used some bad openssl code to attempt to verify the
> cert against the (possibly non existent) chain without realizing this
> requirement.
>
>>   Should “out of date” not be part of the
>> “unusable” definition? ⇒ Ideas: 1. If bmcweb finds a usable cert but is
>> out of date, that cert can still be used.  2. Leave the defective
>> certificate (do not delete it) and log an error.
> A lot of BMCs don't have a dedicated RTC, and rely on other systems
> (like the PCH or NTP) to get the correct time.  bmcweb needs to come
> up long before the PCH or NTP (both of which are also optional) so as
> a general rule, using these for valid time is a non-starter.  I could
> see logging an error _if_ you know time is valid, but I'm not sure how
> a bmc could know that.
>
>> The group consensus was that BMCWeb should treat its HTTPS site identity
>> certificate like this:
>> 1 certificate is perfectly good - Use the certificate
>> 2 certificate is good but expired or not yet valid - Use the certificate
>> and log a warning
> Add "and we know time has been set appropriately".  Also, be careful
> with not yet valid.  I don't know how openssl chains handle clock skew
> between certificate generator and client.  If the BMC has a time
> that's 1 minute fast, is the certificate "not yet valid"?
>
>> 3 certificate is missing or bad format or algorithm too old - Use
>> another certificate or self-generate a certificate (and log that action)
> We shouldn't be replacing certificates unless it's completely missing
> (ie, we're on a first boot or a factory reset) and wouldn't be
> switching certificates on the fly.  If the algorithm is too old, the
> user is free to replace it with their own, which is the right
> procedure anyway.  We also have no idea if the user has added this
> public certificate to their local root store, so replacing it out of
> the blue might look like an attack.  Also, we have no idea if the
> users' client systems support the new crypto format.  We could be
> unintentionally DOSing them if, say, we turn on a new keytype or size
> and their clients don't support that key.
>
>> There are no cases where BMCWeb should delete any certificate.
> I think I know what you mean, but to be clear, we should delete the
> old certificates when they're replaced.  We should delete root trust
> store certificates when requested, ect.  I don't think we can say "no
> cases" here.

Correct.  I meant BMCWeb should not delete an unusable certifivate when 
it generates a new one.  Thanks for clarifying that.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Security Working Group - Wednesday July 22 - results
  2020-07-23 15:13   ` Ed Tanous
  2020-07-23 16:04     ` Joseph Reynolds
@ 2020-07-23 19:05     ` Michael Richardson
  2020-07-23 20:09       ` Ed Tanous
  2020-08-04  9:36     ` Jayanth Othayoth
  2 siblings, 1 reply; 8+ messages in thread
From: Michael Richardson @ 2020-07-23 19:05 UTC (permalink / raw)
  To: Ed Tanous, Joseph Reynolds, openbmc

[-- Attachment #1: Type: text/plain, Size: 2589 bytes --]


Ed Tanous <ed@tanous.net> wrote:
    > One thing to note;  At one point, I had talked through how to
    > prototype ACME protocol replacement of certificates automatically, so,
    > given an ACME server on the network, the BMC could essentially
    > automatically provision itself and keep its certs up to date.  If
    > someone wanted to run with that, it might reduce some of the pain here
    > (and be extremely cool).

I have running code, but to use ACME, requires some initial trust
relationship.  The manufacturer can do that if they want.

One can also use draft-ietf-anima-bootstrapping-keyinfra + EST (RFC7030).
These two are not mutually exclusive.

I hope to clear my plate enough before the end of the year to demonstrate
this on OpenBMC.

    > The above is all asking the wrong question: "Can we determine if the
    > certificate is valid?"  This is irrelevant, the question is: "Should
    > we ever be replacing a user provided certificate with one generated on
    > the BMC."  The answer previously has been no.  In almost all cases the
    > user provided certificate, even an expired one, will still be better
    > than one the BMC self signs.  Between having an invalid certificate
    > chain, and an invalid date, I'll take the invalid date every time.

I agree.

    > It should be noted, most browsers (in my testing) seem to ignore the
    > HTTP date header entirely, so the BMC doesn't even need the correct
    > time to set up a proper encryption channel.

That's very surprising and counter to my experience.
The more likely case is that the OpenBMC has the wrong date.

    >> Should “out of date” not be part of the
    >> “unusable” definition? ⇒ Ideas: 1. If bmcweb finds a usable cert but is
    >> out of date, that cert can still be used.  2. Leave the defective
    >> certificate (do not delete it) and log an error.

    > A lot of BMCs don't have a dedicated RTC, and rely on other systems
    > (like the PCH or NTP) to get the correct time.  bmcweb needs to come
    > up long before the PCH or NTP (both of which are also optional) so as
    > a general rule, using these for valid time is a non-starter.  I could
    > see logging an error _if_ you know time is valid, but I'm not sure how
    > a bmc could know that.

Agreed.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Security Working Group - Wednesday July 22 - results
  2020-07-23 19:05     ` Michael Richardson
@ 2020-07-23 20:09       ` Ed Tanous
  2020-07-23 21:34         ` Michael Richardson
  0 siblings, 1 reply; 8+ messages in thread
From: Ed Tanous @ 2020-07-23 20:09 UTC (permalink / raw)
  To: Michael Richardson; +Cc: Joseph Reynolds, openbmc

On Thu, Jul 23, 2020 at 12:05 PM Michael Richardson <mcr@sandelman.ca> wrote:
>
>
> Ed Tanous <ed@tanous.net> wrote:
>     > One thing to note;  At one point, I had talked through how to
>     > prototype ACME protocol replacement of certificates automatically, so,
>     > given an ACME server on the network, the BMC could essentially
>     > automatically provision itself and keep its certs up to date.  If
>     > someone wanted to run with that, it might reduce some of the pain here
>     > (and be extremely cool).
>
> I have running code, but to use ACME, requires some initial trust
> relationship.  The manufacturer can do that if they want.

Lots of (mostly private) meta layers have this set up already for
internal use and add the relevant CA cert to the build.  Also, I think
(I could be wrong) the ca-certificates package is included in most
builds already so we can handle trust with foreign servers (for things
like HTTP event push).  Presumably ACME uses the same trust
relationship, or does it have a specific mechanism that's unique?

>
> One can also use draft-ietf-anima-bootstrapping-keyinfra + EST (RFC7030).

... has been added to my nightly reading list.

> These two are not mutually exclusive.
>
> I hope to clear my plate enough before the end of the year to demonstrate
> this on OpenBMC.

Awesome.  Looking forward to it.

>     > It should be noted, most browsers (in my testing) seem to ignore the
>     > HTTP date header entirely, so the BMC doesn't even need the correct
>     > time to set up a proper encryption channel.
>
> That's very surprising and counter to my experience.
> The more likely case is that the OpenBMC has the wrong date.
>

IIIIInteresting.  Clearly I need to do more testing.  Just to be
clear, I'm talking about the HTTP response date:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Date

Not the validity dates in the TLS certificate.  There were a couple
versions of bmcweb where the Date field was broken as well as systems
with a reset CMOS where the date is incorrectly set to epoch.  In both
cases, no browsers threw any kind of warning that I recall, we just
happened to notice it on the debug output.

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Security Working Group - Wednesday July 22 - results
  2020-07-23 20:09       ` Ed Tanous
@ 2020-07-23 21:34         ` Michael Richardson
  0 siblings, 0 replies; 8+ messages in thread
From: Michael Richardson @ 2020-07-23 21:34 UTC (permalink / raw)
  To: Ed Tanous, openbmc

[-- Attachment #1: Type: text/plain, Size: 3696 bytes --]


Ed Tanous <ed@tanous.net> wrote:
    >> Ed Tanous <ed@tanous.net> wrote:
    >> > One thing to note;  At one point, I had talked through how to
    >> > prototype ACME protocol replacement of certificates automatically, so,
    >> > given an ACME server on the network, the BMC could essentially
    >> > automatically provision itself and keep its certs up to date.  If
    >> > someone wanted to run with that, it might reduce some of the pain here
    >> > (and be extremely cool).
    >>
    >> I have running code, but to use ACME, requires some initial trust
    >> relationship.  The manufacturer can do that if they want.

    > Lots of (mostly private) meta layers have this set up already for
    > internal use and add the relevant CA cert to the build.  Also, I think
    > (I could be wrong) the ca-certificates package is included in most
    > builds already so we can handle trust with foreign servers (for things
    > like HTTP event push).  Presumably ACME uses the same trust
    > relationship, or does it have a specific mechanism that's unique?

yes, the ca-certificate package provides CABForum listed keys, but that won't
include my local private-CA, unless I put a custom build in.

ACME requires that the machine that wants a certificate has to prove it's
name somehow.  The tools are https-01 or dns-01 challenge.

https-01 trust requires that the ACME server be able to reach the server (the
BMC) on port-443 to see the challenge.  And do so by DNS name!
The public LetsEncrypt systems need this to be a public name, and there must
be public port-443 connectivity.
But, if you run your own ACME servers, then you can do something different.
(but, you'd have to configure OpenBMC to talk to your servers, so you'd need
a way to tell it do that, so you'd already need admin access...)

If this is done by dns-01 challenge, then the ACME server needs to be able to
do a Dynamic DNS Update.

    >> One can also use draft-ietf-anima-bootstrapping-keyinfra + EST (RFC7030).

    > ... has been added to my nightly reading list.

waiting on a MISREF to become RFC.
https://www.sandelman.ca/SSW/ietf/brski-links/ contains a few videos that
might lighten your load.

    >> > It should be noted, most browsers (in my testing) seem to ignore the
    >> > HTTP date header entirely, so the BMC doesn't even need the correct
    >> > time to set up a proper encryption channel.
    >>
    >> That's very surprising and counter to my experience.
    >> The more likely case is that the OpenBMC has the wrong date.
    >>

    > IIIIInteresting.  Clearly I need to do more testing.  Just to be
    > clear, I'm talking about the HTTP response date:
    > https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Date

    > Not the validity dates in the TLS certificate.  There were a couple
    > versions of bmcweb where the Date field was broken as well as systems
    > with a reset CMOS where the date is incorrectly set to epoch.  In both
    > cases, no browsers threw any kind of warning that I recall, we just
    > happened to notice it on the debug output.

So, the BMC has the wrong date, but the certificate was still valid.
(Browser time >= notBefore, browser time <= notAfter)
I don't expect the browser to care about the date the server thinks it is,
only if the certificate has become invalid.

The proposed code to kill the certificate if it was invalid would have
rendered the certificate unuseable in this context.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     mcr@sandelman.ca  http://www.sandelman.ca/        |   ruby on rails    [



[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: Security Working Group - Wednesday July 22 - results
  2020-07-23 15:13   ` Ed Tanous
  2020-07-23 16:04     ` Joseph Reynolds
  2020-07-23 19:05     ` Michael Richardson
@ 2020-08-04  9:36     ` Jayanth Othayoth
  2 siblings, 0 replies; 8+ messages in thread
From: Jayanth Othayoth @ 2020-08-04  9:36 UTC (permalink / raw)
  To: Ed Tanous; +Cc: Joseph Reynolds, openbmc

[-- Attachment #1: Type: text/plain, Size: 6498 bytes --]

On Thu, Jul 23, 2020 at 8:46 PM Ed Tanous <ed@tanous.net> wrote:

> First off, if you are finding certificates are expiring in a
> production environment, you need to rethink your certificate
> replacement strategy.  The BMC cannot be the primary entity in charge
> of certificate replacement and revocation, and the fact that this is
> happening points to a greater organizational problem than the bmc can
> cover.
>
> One thing to note;  At one point, I had talked through how to
> prototype ACME protocol replacement of certificates automatically, so,
> given an ACME server on the network, the BMC could essentially
> automatically provision itself and keep its certs up to date.  If
> someone wanted to run with that, it might reduce some of the pain here
> (and be extremely cool).
>
> On Thu, Jul 23, 2020 at 7:24 AM Joseph Reynolds <jrey@linux.ibm.com>
> wrote:
> >
> >
> > 4 Question: If BMCWeb finds an unusable HTTPS site identity certificate,
> > it DELETES it and self-generates one.  This has caused problems for
> > certificates that are not valid until a future date. In general, what
> > certificate management support should we have for BMCWeb?  What is
> needed?
>
> This is not how bmcweb was intended to work, and I had explicitly
> tested that it doesn't have this behavior (prior to it being moved
> over to certificate manager).  The only time it previously would
> replace a certificate was if the cert file didn't exist, or was
> completely unreadable.  I suspect the right thing here is to bisect
> the commit that broke this behavior (I'll bet it's easy to find) and
> get a bug filed with the author so they can either fix it or revert
> the patch.
>
> This one smells awfully suspicious:
> 082f28fd2588fd9fcd9452ad38234ce875319163.
>

  Ed,
        https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/22938
 introduced to fix issue https://github.com/openbmc/bmcweb/issues/91.
     We should also add check to ignore “X509_V_ERR_CERT_NOT_YET_VALID” to
allow user to upload “NOT_YET_VALID” certificate similar to certificate
manager verification function.


>
> > ANSWER:
> > There were two discussion threads: The BMC’s notion of time of day
> > (TOD), and how BMCWeb should handle certificates.
> >
> > Does the BMC have a battery backed TOD clock?  Depends on BMC model.
> > Can it validate if it has access to its NTP server (when configured)?
> > Does the BMC know if its time was set correctly?
> >
> > How does the BMC know if the BMC has the correct time?  Have a BMC flag
> > that says, “Look like the BMC TOD clock is not working.”  Does the BMC
> > know if we got a good time from an  NTP server?  Can we read the GPS
> > signal?  What is the industry solution?
> > Should the BMC store its idea of what date it is?  So it can report if
> > the time changes significantly.  Or will this lead to a bigger problem?
> > Is it better/simpler to check for TOD = beginning-of-era-1/1/1970?  →
> > start an email thread
>
> The above is all asking the wrong question: "Can we determine if the
> certificate is valid?"  This is irrelevant, the question is: "Should
> we ever be replacing a user provided certificate with one generated on
> the BMC."  The answer previously has been no.  In almost all cases the
> user provided certificate, even an expired one, will still be better
> than one the BMC self signs.  Between having an invalid certificate
> chain, and an invalid date, I'll take the invalid date every time.
>
> It should be noted, most browsers (in my testing) seem to ignore the
> HTTP date header entirely, so the BMC doesn't even need the correct
> time to set up a proper encryption channel.
>
> >
> > BMCWeb configuration?  Configure option: delete cert and generate
> > self-signed -vs- use defective certificate.  What is the purpose of
> > deleting the unusable cert?
>
> This question needs answered.  I suspect this was unintentional, and
> someone simply used some bad openssl code to attempt to verify the
> cert against the (possibly non existent) chain without realizing this
> requirement.
>
> >  Should “out of date” not be part of the
> > “unusable” definition? ⇒ Ideas: 1. If bmcweb finds a usable cert but is
> > out of date, that cert can still be used.  2. Leave the defective
> > certificate (do not delete it) and log an error.
>
> A lot of BMCs don't have a dedicated RTC, and rely on other systems
> (like the PCH or NTP) to get the correct time.  bmcweb needs to come
> up long before the PCH or NTP (both of which are also optional) so as
> a general rule, using these for valid time is a non-starter.  I could
> see logging an error _if_ you know time is valid, but I'm not sure how
> a bmc could know that.
>
> >
> > The group consensus was that BMCWeb should treat its HTTPS site identity
> > certificate like this:
> > 1 certificate is perfectly good - Use the certificate
> > 2 certificate is good but expired or not yet valid - Use the certificate
> > and log a warning
>
> Add "and we know time has been set appropriately".  Also, be careful
> with not yet valid.  I don't know how openssl chains handle clock skew
> between certificate generator and client.  If the BMC has a time
> that's 1 minute fast, is the certificate "not yet valid"?
>
> > 3 certificate is missing or bad format or algorithm too old - Use
> > another certificate or self-generate a certificate (and log that action)
>
> We shouldn't be replacing certificates unless it's completely missing
> (ie, we're on a first boot or a factory reset) and wouldn't be
> switching certificates on the fly.  If the algorithm is too old, the
> user is free to replace it with their own, which is the right
> procedure anyway.  We also have no idea if the user has added this
> public certificate to their local root store, so replacing it out of
> the blue might look like an attack.  Also, we have no idea if the
> users' client systems support the new crypto format.  We could be
> unintentionally DOSing them if, say, we turn on a new keytype or size
> and their clients don't support that key.
>
> > There are no cases where BMCWeb should delete any certificate.
>
> I think I know what you mean, but to be clear, we should delete the
> old certificates when they're replaced.  We should delete root trust
> store certificates when requested, ect.  I don't think we can say "no
> cases" here.
>

[-- Attachment #2: Type: text/html, Size: 7701 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2020-08-04  9:36 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-07-20 13:57 Security Working Group - Wednesday July 22 Joseph Reynolds
2020-07-23 14:11 ` Security Working Group - Wednesday July 22 - results Joseph Reynolds
2020-07-23 15:13   ` Ed Tanous
2020-07-23 16:04     ` Joseph Reynolds
2020-07-23 19:05     ` Michael Richardson
2020-07-23 20:09       ` Ed Tanous
2020-07-23 21:34         ` Michael Richardson
2020-08-04  9:36     ` Jayanth Othayoth

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.