* Re: Security Working Group - Wednesday July 22 - results
2020-07-23 15:13 ` Ed Tanous
@ 2020-07-23 16:04 ` Joseph Reynolds
2020-07-23 19:05 ` Michael Richardson
2020-08-04 9:36 ` Jayanth Othayoth
2 siblings, 0 replies; 8+ messages in thread
From: Joseph Reynolds @ 2020-07-23 16:04 UTC (permalink / raw)
To: Ed Tanous; +Cc: openbmc
On 7/23/20 10:13 AM, Ed Tanous wrote:
> First off, if you are finding certificates are expiring in a
> production environment, you need to rethink your certificate
> replacement strategy. The BMC cannot be the primary entity in charge
> of certificate replacement and revocation, and the fact that this is
> happening points to a greater organizational problem than the bmc can
> cover.
>
> One thing to note; At one point, I had talked through how to
> prototype ACME protocol replacement of certificates automatically, so,
> given an ACME server on the network, the BMC could essentially
> automatically provision itself and keep its certs up to date. If
> someone wanted to run with that, it might reduce some of the pain here
> (and be extremely cool).
>
> On Thu, Jul 23, 2020 at 7:24 AM Joseph Reynolds <jrey@linux.ibm.com> wrote:
>>
>> 4 Question: If BMCWeb finds an unusable HTTPS site identity certificate,
>> it DELETES it and self-generates one. This has caused problems for
>> certificates that are not valid until a future date. In general, what
>> certificate management support should we have for BMCWeb? What is needed?
> This is not how bmcweb was intended to work, and I had explicitly
> tested that it doesn't have this behavior (prior to it being moved
> over to certificate manager). The only time it previously would
> replace a certificate was if the cert file didn't exist, or was
> completely unreadable. I suspect the right thing here is to bisect
> the commit that broke this behavior (I'll bet it's easy to find) and
> get a bug filed with the author so they can either fix it or revert
> the patch.
>
> This one smells awfully suspicious: 082f28fd2588fd9fcd9452ad38234ce875319163.
>
>> ANSWER:
>> There were two discussion threads: The BMC’s notion of time of day
>> (TOD), and how BMCWeb should handle certificates.
>>
>> Does the BMC have a battery backed TOD clock? Depends on BMC model.
>> Can it validate if it has access to its NTP server (when configured)?
>> Does the BMC know if its time was set correctly?
>>
>> How does the BMC know if the BMC has the correct time? Have a BMC flag
>> that says, “Look like the BMC TOD clock is not working.” Does the BMC
>> know if we got a good time from an NTP server? Can we read the GPS
>> signal? What is the industry solution?
>> Should the BMC store its idea of what date it is? So it can report if
>> the time changes significantly. Or will this lead to a bigger problem?
>> Is it better/simpler to check for TOD = beginning-of-era-1/1/1970? →
>> start an email thread
> The above is all asking the wrong question: "Can we determine if the
> certificate is valid?" This is irrelevant, the question is: "Should
> we ever be replacing a user provided certificate with one generated on
> the BMC." The answer previously has been no. In almost all cases the
> user provided certificate, even an expired one, will still be better
> than one the BMC self signs. Between having an invalid certificate
> chain, and an invalid date, I'll take the invalid date every time.
>
> It should be noted, most browsers (in my testing) seem to ignore the
> HTTP date header entirely, so the BMC doesn't even need the correct
> time to set up a proper encryption channel.
We weren't sure what question to ask. Thanks for the discussion.
I sent a followup email with some better-phrased questions, but our
email crossed.
>
>> BMCWeb configuration? Configure option: delete cert and generate
>> self-signed -vs- use defective certificate. What is the purpose of
>> deleting the unusable cert?
> This question needs answered. I suspect this was unintentional, and
> someone simply used some bad openssl code to attempt to verify the
> cert against the (possibly non existent) chain without realizing this
> requirement.
>
>> Should “out of date” not be part of the
>> “unusable” definition? ⇒ Ideas: 1. If bmcweb finds a usable cert but is
>> out of date, that cert can still be used. 2. Leave the defective
>> certificate (do not delete it) and log an error.
> A lot of BMCs don't have a dedicated RTC, and rely on other systems
> (like the PCH or NTP) to get the correct time. bmcweb needs to come
> up long before the PCH or NTP (both of which are also optional) so as
> a general rule, using these for valid time is a non-starter. I could
> see logging an error _if_ you know time is valid, but I'm not sure how
> a bmc could know that.
>
>> The group consensus was that BMCWeb should treat its HTTPS site identity
>> certificate like this:
>> 1 certificate is perfectly good - Use the certificate
>> 2 certificate is good but expired or not yet valid - Use the certificate
>> and log a warning
> Add "and we know time has been set appropriately". Also, be careful
> with not yet valid. I don't know how openssl chains handle clock skew
> between certificate generator and client. If the BMC has a time
> that's 1 minute fast, is the certificate "not yet valid"?
>
>> 3 certificate is missing or bad format or algorithm too old - Use
>> another certificate or self-generate a certificate (and log that action)
> We shouldn't be replacing certificates unless it's completely missing
> (ie, we're on a first boot or a factory reset) and wouldn't be
> switching certificates on the fly. If the algorithm is too old, the
> user is free to replace it with their own, which is the right
> procedure anyway. We also have no idea if the user has added this
> public certificate to their local root store, so replacing it out of
> the blue might look like an attack. Also, we have no idea if the
> users' client systems support the new crypto format. We could be
> unintentionally DOSing them if, say, we turn on a new keytype or size
> and their clients don't support that key.
>
>> There are no cases where BMCWeb should delete any certificate.
> I think I know what you mean, but to be clear, we should delete the
> old certificates when they're replaced. We should delete root trust
> store certificates when requested, ect. I don't think we can say "no
> cases" here.
Correct. I meant BMCWeb should not delete an unusable certifivate when
it generates a new one. Thanks for clarifying that.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Security Working Group - Wednesday July 22 - results
2020-07-23 15:13 ` Ed Tanous
2020-07-23 16:04 ` Joseph Reynolds
@ 2020-07-23 19:05 ` Michael Richardson
2020-07-23 20:09 ` Ed Tanous
2020-08-04 9:36 ` Jayanth Othayoth
2 siblings, 1 reply; 8+ messages in thread
From: Michael Richardson @ 2020-07-23 19:05 UTC (permalink / raw)
To: Ed Tanous, Joseph Reynolds, openbmc
[-- Attachment #1: Type: text/plain, Size: 2589 bytes --]
Ed Tanous <ed@tanous.net> wrote:
> One thing to note; At one point, I had talked through how to
> prototype ACME protocol replacement of certificates automatically, so,
> given an ACME server on the network, the BMC could essentially
> automatically provision itself and keep its certs up to date. If
> someone wanted to run with that, it might reduce some of the pain here
> (and be extremely cool).
I have running code, but to use ACME, requires some initial trust
relationship. The manufacturer can do that if they want.
One can also use draft-ietf-anima-bootstrapping-keyinfra + EST (RFC7030).
These two are not mutually exclusive.
I hope to clear my plate enough before the end of the year to demonstrate
this on OpenBMC.
> The above is all asking the wrong question: "Can we determine if the
> certificate is valid?" This is irrelevant, the question is: "Should
> we ever be replacing a user provided certificate with one generated on
> the BMC." The answer previously has been no. In almost all cases the
> user provided certificate, even an expired one, will still be better
> than one the BMC self signs. Between having an invalid certificate
> chain, and an invalid date, I'll take the invalid date every time.
I agree.
> It should be noted, most browsers (in my testing) seem to ignore the
> HTTP date header entirely, so the BMC doesn't even need the correct
> time to set up a proper encryption channel.
That's very surprising and counter to my experience.
The more likely case is that the OpenBMC has the wrong date.
>> Should “out of date” not be part of the
>> “unusable” definition? ⇒ Ideas: 1. If bmcweb finds a usable cert but is
>> out of date, that cert can still be used. 2. Leave the defective
>> certificate (do not delete it) and log an error.
> A lot of BMCs don't have a dedicated RTC, and rely on other systems
> (like the PCH or NTP) to get the correct time. bmcweb needs to come
> up long before the PCH or NTP (both of which are also optional) so as
> a general rule, using these for valid time is a non-starter. I could
> see logging an error _if_ you know time is valid, but I'm not sure how
> a bmc could know that.
Agreed.
--
] Never tell me the odds! | ipv6 mesh networks [
] Michael Richardson, Sandelman Software Works | IoT architect [
] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: Security Working Group - Wednesday July 22 - results
2020-07-23 19:05 ` Michael Richardson
@ 2020-07-23 20:09 ` Ed Tanous
2020-07-23 21:34 ` Michael Richardson
0 siblings, 1 reply; 8+ messages in thread
From: Ed Tanous @ 2020-07-23 20:09 UTC (permalink / raw)
To: Michael Richardson; +Cc: Joseph Reynolds, openbmc
On Thu, Jul 23, 2020 at 12:05 PM Michael Richardson <mcr@sandelman.ca> wrote:
>
>
> Ed Tanous <ed@tanous.net> wrote:
> > One thing to note; At one point, I had talked through how to
> > prototype ACME protocol replacement of certificates automatically, so,
> > given an ACME server on the network, the BMC could essentially
> > automatically provision itself and keep its certs up to date. If
> > someone wanted to run with that, it might reduce some of the pain here
> > (and be extremely cool).
>
> I have running code, but to use ACME, requires some initial trust
> relationship. The manufacturer can do that if they want.
Lots of (mostly private) meta layers have this set up already for
internal use and add the relevant CA cert to the build. Also, I think
(I could be wrong) the ca-certificates package is included in most
builds already so we can handle trust with foreign servers (for things
like HTTP event push). Presumably ACME uses the same trust
relationship, or does it have a specific mechanism that's unique?
>
> One can also use draft-ietf-anima-bootstrapping-keyinfra + EST (RFC7030).
... has been added to my nightly reading list.
> These two are not mutually exclusive.
>
> I hope to clear my plate enough before the end of the year to demonstrate
> this on OpenBMC.
Awesome. Looking forward to it.
> > It should be noted, most browsers (in my testing) seem to ignore the
> > HTTP date header entirely, so the BMC doesn't even need the correct
> > time to set up a proper encryption channel.
>
> That's very surprising and counter to my experience.
> The more likely case is that the OpenBMC has the wrong date.
>
IIIIInteresting. Clearly I need to do more testing. Just to be
clear, I'm talking about the HTTP response date:
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Date
Not the validity dates in the TLS certificate. There were a couple
versions of bmcweb where the Date field was broken as well as systems
with a reset CMOS where the date is incorrectly set to epoch. In both
cases, no browsers threw any kind of warning that I recall, we just
happened to notice it on the debug output.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Security Working Group - Wednesday July 22 - results
2020-07-23 20:09 ` Ed Tanous
@ 2020-07-23 21:34 ` Michael Richardson
0 siblings, 0 replies; 8+ messages in thread
From: Michael Richardson @ 2020-07-23 21:34 UTC (permalink / raw)
To: Ed Tanous, openbmc
[-- Attachment #1: Type: text/plain, Size: 3696 bytes --]
Ed Tanous <ed@tanous.net> wrote:
>> Ed Tanous <ed@tanous.net> wrote:
>> > One thing to note; At one point, I had talked through how to
>> > prototype ACME protocol replacement of certificates automatically, so,
>> > given an ACME server on the network, the BMC could essentially
>> > automatically provision itself and keep its certs up to date. If
>> > someone wanted to run with that, it might reduce some of the pain here
>> > (and be extremely cool).
>>
>> I have running code, but to use ACME, requires some initial trust
>> relationship. The manufacturer can do that if they want.
> Lots of (mostly private) meta layers have this set up already for
> internal use and add the relevant CA cert to the build. Also, I think
> (I could be wrong) the ca-certificates package is included in most
> builds already so we can handle trust with foreign servers (for things
> like HTTP event push). Presumably ACME uses the same trust
> relationship, or does it have a specific mechanism that's unique?
yes, the ca-certificate package provides CABForum listed keys, but that won't
include my local private-CA, unless I put a custom build in.
ACME requires that the machine that wants a certificate has to prove it's
name somehow. The tools are https-01 or dns-01 challenge.
https-01 trust requires that the ACME server be able to reach the server (the
BMC) on port-443 to see the challenge. And do so by DNS name!
The public LetsEncrypt systems need this to be a public name, and there must
be public port-443 connectivity.
But, if you run your own ACME servers, then you can do something different.
(but, you'd have to configure OpenBMC to talk to your servers, so you'd need
a way to tell it do that, so you'd already need admin access...)
If this is done by dns-01 challenge, then the ACME server needs to be able to
do a Dynamic DNS Update.
>> One can also use draft-ietf-anima-bootstrapping-keyinfra + EST (RFC7030).
> ... has been added to my nightly reading list.
waiting on a MISREF to become RFC.
https://www.sandelman.ca/SSW/ietf/brski-links/ contains a few videos that
might lighten your load.
>> > It should be noted, most browsers (in my testing) seem to ignore the
>> > HTTP date header entirely, so the BMC doesn't even need the correct
>> > time to set up a proper encryption channel.
>>
>> That's very surprising and counter to my experience.
>> The more likely case is that the OpenBMC has the wrong date.
>>
> IIIIInteresting. Clearly I need to do more testing. Just to be
> clear, I'm talking about the HTTP response date:
> https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Date
> Not the validity dates in the TLS certificate. There were a couple
> versions of bmcweb where the Date field was broken as well as systems
> with a reset CMOS where the date is incorrectly set to epoch. In both
> cases, no browsers threw any kind of warning that I recall, we just
> happened to notice it on the debug output.
So, the BMC has the wrong date, but the certificate was still valid.
(Browser time >= notBefore, browser time <= notAfter)
I don't expect the browser to care about the date the server thinks it is,
only if the certificate has become invalid.
The proposed code to kill the certificate if it was invalid would have
rendered the certificate unuseable in this context.
--
] Never tell me the odds! | ipv6 mesh networks [
] Michael Richardson, Sandelman Software Works | IoT architect [
] mcr@sandelman.ca http://www.sandelman.ca/ | ruby on rails [
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 487 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Security Working Group - Wednesday July 22 - results
2020-07-23 15:13 ` Ed Tanous
2020-07-23 16:04 ` Joseph Reynolds
2020-07-23 19:05 ` Michael Richardson
@ 2020-08-04 9:36 ` Jayanth Othayoth
2 siblings, 0 replies; 8+ messages in thread
From: Jayanth Othayoth @ 2020-08-04 9:36 UTC (permalink / raw)
To: Ed Tanous; +Cc: Joseph Reynolds, openbmc
[-- Attachment #1: Type: text/plain, Size: 6498 bytes --]
On Thu, Jul 23, 2020 at 8:46 PM Ed Tanous <ed@tanous.net> wrote:
> First off, if you are finding certificates are expiring in a
> production environment, you need to rethink your certificate
> replacement strategy. The BMC cannot be the primary entity in charge
> of certificate replacement and revocation, and the fact that this is
> happening points to a greater organizational problem than the bmc can
> cover.
>
> One thing to note; At one point, I had talked through how to
> prototype ACME protocol replacement of certificates automatically, so,
> given an ACME server on the network, the BMC could essentially
> automatically provision itself and keep its certs up to date. If
> someone wanted to run with that, it might reduce some of the pain here
> (and be extremely cool).
>
> On Thu, Jul 23, 2020 at 7:24 AM Joseph Reynolds <jrey@linux.ibm.com>
> wrote:
> >
> >
> > 4 Question: If BMCWeb finds an unusable HTTPS site identity certificate,
> > it DELETES it and self-generates one. This has caused problems for
> > certificates that are not valid until a future date. In general, what
> > certificate management support should we have for BMCWeb? What is
> needed?
>
> This is not how bmcweb was intended to work, and I had explicitly
> tested that it doesn't have this behavior (prior to it being moved
> over to certificate manager). The only time it previously would
> replace a certificate was if the cert file didn't exist, or was
> completely unreadable. I suspect the right thing here is to bisect
> the commit that broke this behavior (I'll bet it's easy to find) and
> get a bug filed with the author so they can either fix it or revert
> the patch.
>
> This one smells awfully suspicious:
> 082f28fd2588fd9fcd9452ad38234ce875319163.
>
Ed,
https://gerrit.openbmc-project.xyz/c/openbmc/bmcweb/+/22938
introduced to fix issue https://github.com/openbmc/bmcweb/issues/91.
We should also add check to ignore “X509_V_ERR_CERT_NOT_YET_VALID” to
allow user to upload “NOT_YET_VALID” certificate similar to certificate
manager verification function.
>
> > ANSWER:
> > There were two discussion threads: The BMC’s notion of time of day
> > (TOD), and how BMCWeb should handle certificates.
> >
> > Does the BMC have a battery backed TOD clock? Depends on BMC model.
> > Can it validate if it has access to its NTP server (when configured)?
> > Does the BMC know if its time was set correctly?
> >
> > How does the BMC know if the BMC has the correct time? Have a BMC flag
> > that says, “Look like the BMC TOD clock is not working.” Does the BMC
> > know if we got a good time from an NTP server? Can we read the GPS
> > signal? What is the industry solution?
> > Should the BMC store its idea of what date it is? So it can report if
> > the time changes significantly. Or will this lead to a bigger problem?
> > Is it better/simpler to check for TOD = beginning-of-era-1/1/1970? →
> > start an email thread
>
> The above is all asking the wrong question: "Can we determine if the
> certificate is valid?" This is irrelevant, the question is: "Should
> we ever be replacing a user provided certificate with one generated on
> the BMC." The answer previously has been no. In almost all cases the
> user provided certificate, even an expired one, will still be better
> than one the BMC self signs. Between having an invalid certificate
> chain, and an invalid date, I'll take the invalid date every time.
>
> It should be noted, most browsers (in my testing) seem to ignore the
> HTTP date header entirely, so the BMC doesn't even need the correct
> time to set up a proper encryption channel.
>
> >
> > BMCWeb configuration? Configure option: delete cert and generate
> > self-signed -vs- use defective certificate. What is the purpose of
> > deleting the unusable cert?
>
> This question needs answered. I suspect this was unintentional, and
> someone simply used some bad openssl code to attempt to verify the
> cert against the (possibly non existent) chain without realizing this
> requirement.
>
> > Should “out of date” not be part of the
> > “unusable” definition? ⇒ Ideas: 1. If bmcweb finds a usable cert but is
> > out of date, that cert can still be used. 2. Leave the defective
> > certificate (do not delete it) and log an error.
>
> A lot of BMCs don't have a dedicated RTC, and rely on other systems
> (like the PCH or NTP) to get the correct time. bmcweb needs to come
> up long before the PCH or NTP (both of which are also optional) so as
> a general rule, using these for valid time is a non-starter. I could
> see logging an error _if_ you know time is valid, but I'm not sure how
> a bmc could know that.
>
> >
> > The group consensus was that BMCWeb should treat its HTTPS site identity
> > certificate like this:
> > 1 certificate is perfectly good - Use the certificate
> > 2 certificate is good but expired or not yet valid - Use the certificate
> > and log a warning
>
> Add "and we know time has been set appropriately". Also, be careful
> with not yet valid. I don't know how openssl chains handle clock skew
> between certificate generator and client. If the BMC has a time
> that's 1 minute fast, is the certificate "not yet valid"?
>
> > 3 certificate is missing or bad format or algorithm too old - Use
> > another certificate or self-generate a certificate (and log that action)
>
> We shouldn't be replacing certificates unless it's completely missing
> (ie, we're on a first boot or a factory reset) and wouldn't be
> switching certificates on the fly. If the algorithm is too old, the
> user is free to replace it with their own, which is the right
> procedure anyway. We also have no idea if the user has added this
> public certificate to their local root store, so replacing it out of
> the blue might look like an attack. Also, we have no idea if the
> users' client systems support the new crypto format. We could be
> unintentionally DOSing them if, say, we turn on a new keytype or size
> and their clients don't support that key.
>
> > There are no cases where BMCWeb should delete any certificate.
>
> I think I know what you mean, but to be clear, we should delete the
> old certificates when they're replaced. We should delete root trust
> store certificates when requested, ect. I don't think we can say "no
> cases" here.
>
[-- Attachment #2: Type: text/html, Size: 7701 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread