Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

[Stephan Mueller]

Am Mittwoch, 7. August 2024, 14:50:32 MESZ schrieb Jeff Barnes:

Hi Jeff,

> Hello,
> 
> We are currently migrating to kernel 6.6.14 and encountering intermittent
> EHEALTH errors that cause a kernel panic in initrd (FIPS mode). The error
> occurs in the following section of the code:
> 
> crypto/jitterentropy.c
> 722                 /* Validate health test result */
> 723                 if (jent_health_failure(&ec))
> 724                         return JENT_EHEALTH;
> 
> This is called from jent_mod_init():
> 
> 337         ret = jent_entropy_init(desc);
> 338         shash_desc_zero(desc);
> 339         crypto_free_shash(tfm);
> 340         if (ret) {
> 341                 /* Handle permanent health test error */
> 342                 if (fips_enabled)
> 343                         panic("jitterentropy: Initialization failed with
> host not compliant with requirements: %d\n", ret);
> 
> We are experiencing up to a 90% failure rate.
> 
> In my troubleshooting efforts, I followed the call to jent_condition_data()
> and attempted to increase the SHA3_HASH_LOOP to give the CPU more work,
> hoping to collect more entropy:

The proper way to handle it is the following: set 
CONFIG_CRYPTO_JITTERENTROPY_OSR to a higer value as it is - like 3 (the 
default is 1). The higher you set it the slower the collection will get as 
more samples are collected.
> 
> 356
> -#define SHA3_HASH_LOOP (1<<3)
> +#define SHA3_HASH_LOOP (1<<4)
> 
> This adjustment reduced the failure rate to 40-50%, but the issue persists.
> It is intermittent. It is also intermittent without the change. Sometimes I
> get a 90% failure rate on 10 reboots, sometimes 0%.
> 
> Given the difficulty in reproducing the kernel panic consistently, is there
> a more effective workaround or solution for this problem?
> 
> Your assistance is greatly appreciated.
> 
> Best regards,
> Jeff Barnes


Ciao
Stephan

Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

[Herbert Xu]

On Wed, Aug 07, 2024 at 03:21:04PM +0200, Stephan Mueller wrote:
>
> The proper way to handle it is the following: set 
> CONFIG_CRYPTO_JITTERENTROPY_OSR to a higer value as it is - like 3 (the 
> default is 1). The higher you set it the slower the collection will get as 
> more samples are collected.

Stephan, it's not acceptable to rely on adjusting a Kconfig knob
to obtain a working system.  This needs to work out of the box.

Thanks,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

[Stephan Mueller]

Am Donnerstag, 8. August 2024, 07:17:01 MESZ schrieb Herbert Xu:

Hi Herbert,

> On Wed, Aug 07, 2024 at 03:21:04PM +0200, Stephan Mueller wrote:
> > The proper way to handle it is the following: set
> > CONFIG_CRYPTO_JITTERENTROPY_OSR to a higer value as it is - like 3 (the
> > default is 1). The higher you set it the slower the collection will get as
> > more samples are collected.
> 
> Stephan, it's not acceptable to rely on adjusting a Kconfig knob
> to obtain a working system.  This needs to work out of the box.

The user-space version uses an OSR of 3. Using this value, I have not heard of 
any problems. I will prepare a patch.
> 
> Thanks,


Ciao
Stephan

Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

[Herbert Xu]

On Thu, Aug 08, 2024 at 07:56:39AM +0200, Stephan Mueller wrote:
>
> The user-space version uses an OSR of 3. Using this value, I have not heard of 
> any problems. I will prepare a patch.

Thanks Stephan.  Jeff, could you please let us know if the value of 3
fixes your problems?

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

[Stephan Mueller]

Am Donnerstag, 8. August 2024, 08:10:15 MESZ schrieb Herbert Xu:

Hi Herbert,

> On Thu, Aug 08, 2024 at 07:56:39AM +0200, Stephan Mueller wrote:
> > The user-space version uses an OSR of 3. Using this value, I have not
> > heard of any problems. I will prepare a patch.
> 
> Thanks Stephan.  Jeff, could you please let us know if the value of 3
> fixes your problems?

However, the heart of the problem is the following: This failure mode is 
probabilistic in nature. A number of folks trying to push rules that the 
failure does not need to be handled with a panic.

A changed OSR only changes the probability, but that probability is always 
strictly higher than zero.
> 
> Cheers,


Ciao
Stephan

Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

[Herbert Xu]

On Thu, Aug 08, 2024 at 08:13:56AM +0200, Stephan Mueller wrote:
>
> However, the heart of the problem is the following: This failure mode is 
> probabilistic in nature. A number of folks trying to push rules that the 
> failure does not need to be handled with a panic.
> 
> A changed OSR only changes the probability, but that probability is always 
> strictly higher than zero.

That's fine.  There are many places in the kernel that will fail
with a probably that is non-zero.  It is considered to be acceptable
as long as the value is negligible (e.g., equal or less than the
probablility of cosmic rays hitting DRAM).

But if it happens reproducibly it clearly is not acceptable.

Cheers,
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: [EXTERNAL] Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

[Jeff Barnes]

By setting the configuration option
+CONFIG_CRYPTO_JITTERENTROPY_OSR=3,

I ran the following ad hoc test.

50 consecutive boots with fips=1 on the command line with the same image.

No jitterentropy health check failure. Booted successfully.

Thanks for the help. I plan to try a value of 2 to see if that will fix it too.

Jeff

________________________________________
From: Herbert Xu <herbert@gondor.apana.org.au>
Sent: Thursday, August 8, 2024 2:20 AM
To: Stephan Mueller
Cc: linux-crypto@vger.kernel.org; Jeff Barnes; Vladis Dronov; marcelo.cerri@canonical.com; Tyler Hicks; Shyam Saini
Subject: [EXTERNAL] Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

[You don't often get email from herbert@gondor.apana.org.au. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]

On Thu, Aug 08, 2024 at 08:13:56AM +0200, Stephan Mueller wrote:
>
> However, the heart of the problem is the following: This failure mode is
> probabilistic in nature. A number of folks trying to push rules that the
> failure does not need to be handled with a panic.
>
> A changed OSR only changes the probability, but that probability is always
> strictly higher than zero.

That's fine.  There are many places in the kernel that will fail
with a probably that is non-zero.  It is considered to be acceptable
as long as the value is negligible (e.g., equal or less than the
probablility of cosmic rays hitting DRAM).

But if it happens reproducibly it clearly is not acceptable.

Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

[PATCH] crypto: JENT - set default OSR to 3

[Stephan Mueller]

The user space Jitter RNG library uses the oversampling rate of 3 which
implies that each time stamp is credited with 1/3 bit of entropy. To
obtain 256 bits of entropy, 768 time stamps need to be sampled. The
increase in OSR is applied based on a report where the Jitter RNG is
used on a system exhibiting a challenging environment to collect
entropy.

This OSR default value is now applied to the Linux kernel version of
the Jitter RNG as well.

The increase in the OSR from 1 to 3 also implies that the Jitter RNG is
now slower by default.

Reported-by: Jeff Barnes <jeffbarnes@microsoft.com>
Signed-off-by: Stephan Mueller <smueller@chronox.com>
---
 crypto/Kconfig | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/crypto/Kconfig b/crypto/Kconfig
index 72e2decb8c6a..a779cab668c2 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -1305,7 +1305,7 @@ config CRYPTO_JITTERENTROPY_MEMORY_BLOCKSIZE
 config CRYPTO_JITTERENTROPY_OSR
 	int "CPU Jitter RNG Oversampling Rate"
 	range 1 15
-	default 1
+	default 3
 	help
 	  The Jitter RNG allows the specification of an oversampling rate (OSR).
 	  The Jitter RNG operation requires a fixed amount of timing
-- 
2.46.0

Re: [PATCH] crypto: JENT - set default OSR to 3

[Herbert Xu]

On Mon, Aug 12, 2024 at 08:25:42AM +0200, Stephan Mueller wrote:
> The user space Jitter RNG library uses the oversampling rate of 3 which
> implies that each time stamp is credited with 1/3 bit of entropy. To
> obtain 256 bits of entropy, 768 time stamps need to be sampled. The
> increase in OSR is applied based on a report where the Jitter RNG is
> used on a system exhibiting a challenging environment to collect
> entropy.
> 
> This OSR default value is now applied to the Linux kernel version of
> the Jitter RNG as well.
> 
> The increase in the OSR from 1 to 3 also implies that the Jitter RNG is
> now slower by default.
> 
> Reported-by: Jeff Barnes <jeffbarnes@microsoft.com>
> Signed-off-by: Stephan Mueller <smueller@chronox.com>
> ---
>  crypto/Kconfig | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)

Patch applied.  Thanks.
-- 
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt

Re: Intermittent EHEALTH Failure in FIPS Mode - jitterentropy jent_entropy_init() in Kernel 6.6.14

[Jeff Barnes]

Resending in plain text

Hello,

We are currently migrating to kernel 6.6.14 and encountering intermittent EHEALTH errors that cause a kernel panic in initrd (FIPS mode). The error occurs in the following section of the code:

crypto/jitterentropy.c
722                 /* Validate health test result */
723                 if (jent_health_failure(&ec))
724                         return JENT_EHEALTH;

This is called from jent_mod_init():

337         ret = jent_entropy_init(desc);
338         shash_desc_zero(desc);
339         crypto_free_shash(tfm);
340         if (ret) {
341                 /* Handle permanent health test error */
342                 if (fips_enabled)
343                         panic("jitterentropy: Initialization failed with host not compliant with requirements: %d\n", ret);

We are experiencing up to a 90% failure rate.

In my troubleshooting efforts, I followed the call to jent_condition_data() and attempted to increase the SHA3_HASH_LOOP to give the CPU more work, hoping to collect more entropy:

356
-#define SHA3_HASH_LOOP (1<<3)
+#define SHA3_HASH_LOOP (1<<4)

This adjustment reduced the failure rate to 40-50%, but the issue persists. It is intermittent. It is also intermittent without the change. Sometimes I get a 90% failure rate on 10 reboots, sometimes 0%.

Given the difficulty in reproducing the kernel panic consistently, is there a more effective workaround or solution for this problem?

Your assistance is greatly appreciated.

Best regards,
Jeff Barnes