* [PATCH] x86/microcode/intel: Panic on partial microcode update
@ 2026-06-30 19:13 Chang S. Bae
2026-06-30 21:38 ` Dave Hansen
2026-07-08 21:18 ` [PATCH v2] x86/microcode/intel: Taint kernel on partial update Chang S. Bae
0 siblings, 2 replies; 21+ messages in thread
From: Chang S. Bae @ 2026-06-30 19:13 UTC (permalink / raw)
To: linux-kernel; +Cc: x86, tglx, mingo, bp, dave.hansen, hpa, chang.seok.bae
The MSR IA32_MCU_STATUS, if available, may report a partial update status
on a microcode update. This indicates that only a subset of microcode
components was updated successfully while other parts of updates failed,
which in turn leaves the system in an undefined and (potentially)
unreliable state.
Panic when such a fatal condition is reported, since the system can no
longer be trusted to make forward progress safely.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
---
This patch was once considered bundled with another feature enabling
(uniform). But it can stand on its own as a reliability improvement.
The status is possible on newer CPUs. While validation is expected to
prevent these error conditions in normal deployments, handling them
explicitly protects systems against an otherwise undefined state.
Considered refactoring the enumeration code was obvious enough to fold
into a signle patch here. Otherwise, next revision may separate it out.
---
arch/x86/include/asm/msr-index.h | 4 +++
arch/x86/kernel/cpu/microcode/intel.c | 35 +++++++++++++++++++++++++--
2 files changed, 37 insertions(+), 2 deletions(-)
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 18c4be75e927..7273d340470d 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -977,6 +977,10 @@
#define MSR_IA32_MCU_ENUMERATION 0x0000007b
#define MCU_STAGING BIT(4)
+#define MSR_IA32_MCU_STATUS 0x0000007c
+#define MCU_PARTIAL_UPDATE BIT(0)
+#define AUTH_FAIL_ON_MCU_COMPONENT BIT(1)
+
#define MSR_IA32_UCODE_REV 0x0000008b
/* Intel SGX Launch Enclave Public Key Hash MSRs */
diff --git a/arch/x86/kernel/cpu/microcode/intel.c b/arch/x86/kernel/cpu/microcode/intel.c
index f4a444e6114d..0b474a7c6986 100644
--- a/arch/x86/kernel/cpu/microcode/intel.c
+++ b/arch/x86/kernel/cpu/microcode/intel.c
@@ -76,6 +76,9 @@ static struct microcode_intel *ucode_patch_late __read_mostly;
/* last level cache size per core */
static unsigned int llc_size_per_core __ro_after_init;
+/* CPU capability for update status and staging support */
+static bool cpu_has_mcu __ro_after_init;
+
/* microcode format is extended from prescott processors */
struct extended_signature {
unsigned int sig;
@@ -702,6 +705,16 @@ static enum ucode_state __apply_microcode(struct ucode_cpu_info *uci,
/* write microcode via MSR 0x79 */
native_wrmsrq(MSR_IA32_UCODE_WRITE, (unsigned long)mc->bits);
+ /* Check if the update put the system in an unreliable state */
+ if (cpu_has_mcu) {
+ u64 status = native_rdmsrq(MSR_IA32_MCU_STATUS);
+
+ if (status & (MCU_PARTIAL_UPDATE | AUTH_FAIL_ON_MCU_COMPONENT)) {
+ pr_emerg("Partial update: MSR_IA32_MCU_STATUS=0x%llx\n", status);
+ nmi_panic(NULL, "Microcode load: fatal status from partial update");
+ }
+ }
+
rev = intel_get_microcode_revision();
if (rev != mc->hdr.rev)
return UCODE_ERROR;
@@ -779,11 +792,30 @@ static int __init save_builtin_microcode(void)
}
early_initcall(save_builtin_microcode);
+#define CPUID_EDX_ARCH_CAP BIT(29)
+
+static __init bool mcu_capable(void)
+{
+ if (native_cpuid_eax(0) < 7)
+ return false;
+
+ if (!(native_cpuid_edx(7) & CPUID_EDX_ARCH_CAP))
+ return false;
+
+ if (!(native_rdmsrq(MSR_IA32_ARCH_CAPABILITIES) & ARCH_CAP_MCU_ENUM))
+ return false;
+
+ return true;
+}
+
/* Load microcode on BSP from initrd or builtin blobs */
void __init load_ucode_intel_bsp(struct early_load_data *ed)
{
struct ucode_cpu_info uci;
+ /* Indicate early enough to cover the early-loading path */
+ cpu_has_mcu = mcu_capable();
+
uci.mc = get_microcode_blob(&uci, false);
ed->old_rev = uci.cpu_sig.rev;
@@ -1023,8 +1055,7 @@ static __init bool staging_available(void)
{
u64 val;
- val = x86_read_arch_cap_msr();
- if (!(val & ARCH_CAP_MCU_ENUM))
+ if (!cpu_has_mcu)
return false;
rdmsrq(MSR_IA32_MCU_ENUMERATION, val);
--
2.51.0
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-06-30 19:13 [PATCH] x86/microcode/intel: Panic on partial microcode update Chang S. Bae
@ 2026-06-30 21:38 ` Dave Hansen
2026-06-30 21:47 ` Dave Hansen
2026-07-08 21:18 ` [PATCH v2] x86/microcode/intel: Taint kernel on partial update Chang S. Bae
1 sibling, 1 reply; 21+ messages in thread
From: Dave Hansen @ 2026-06-30 21:38 UTC (permalink / raw)
To: Chang S. Bae, linux-kernel; +Cc: x86, tglx, mingo, bp, dave.hansen, hpa
On 6/30/26 12:13, Chang S. Bae wrote:
> The MSR IA32_MCU_STATUS, if available, may report a partial update status
> on a microcode update. This indicates that only a subset of microcode
> components was updated successfully while other parts of updates failed,
> which in turn leaves the system in an undefined and (potentially)
> unreliable state.
A modern individual microcode update update contains firmware for many
pieces of silicon inside the CPU. Sometimes, a single update operation
successfully updates some components and not others leaving a
partially-applied update.
This leaves the system in an undefined and unreliable state.
> The status is possible on newer CPUs. While validation is expected to
> prevent these error conditions in normal deployments, handling them
> explicitly protects systems against an otherwise undefined state.
I think that this is saying that CPUs try hard not to throw up their
hands and give up mid-update. But, the world is hard and things don't
always go to plan.
> +#define MSR_IA32_MCU_STATUS 0x0000007c
> +#define MCU_PARTIAL_UPDATE BIT(0)
> +#define AUTH_FAIL_ON_MCU_COMPONENT BIT(1)
Let's just make a :
#define MCU_STATUS_FAILURE_MASK (MCU_PARTIAL_UPDATE \
AUTH_FAIL_ON_MCU_COMPONENT)
> diff --git a/arch/x86/kernel/cpu/microcode/intel.c b/arch/x86/kernel/cpu/microcode/intel.c
> index f4a444e6114d..0b474a7c6986 100644
> --- a/arch/x86/kernel/cpu/microcode/intel.c
> +++ b/arch/x86/kernel/cpu/microcode/intel.c
> @@ -76,6 +76,9 @@ static struct microcode_intel *ucode_patch_late __read_mostly;
> /* last level cache size per core */
> static unsigned int llc_size_per_core __ro_after_init;
>
> +/* CPU capability for update status and staging support */
> +static bool cpu_has_mcu __ro_after_init;
These are *SUCH* slow, rare paths that I'm not even sure we need to
cache this.
> /* microcode format is extended from prescott processors */
> struct extended_signature {
> unsigned int sig;
> @@ -702,6 +705,16 @@ static enum ucode_state __apply_microcode(struct ucode_cpu_info *uci,
> /* write microcode via MSR 0x79 */
> native_wrmsrq(MSR_IA32_UCODE_WRITE, (unsigned long)mc->bits);
>
> + /* Check if the update put the system in an unreliable state */
> + if (cpu_has_mcu) {
> + u64 status = native_rdmsrq(MSR_IA32_MCU_STATUS);
> +
> + if (status & (MCU_PARTIAL_UPDATE | AUTH_FAIL_ON_MCU_COMPONENT)) {
> + pr_emerg("Partial update: MSR_IA32_MCU_STATUS=0x%llx\n", status);
> + nmi_panic(NULL, "Microcode load: fatal status from partial update");
> + }
> + }
Are both messages needed? Is this because nmi_panic() doesn't do
printk() formatting?
Also, let's just say:
nmi_panic(NULL, "Fatal microcode update failure");
> rev = intel_get_microcode_revision();
> if (rev != mc->hdr.rev)
> return UCODE_ERROR;
> @@ -779,11 +792,30 @@ static int __init save_builtin_microcode(void)
> }
> early_initcall(save_builtin_microcode);
>
> +#define CPUID_EDX_ARCH_CAP BIT(29)
> +
> +static __init bool mcu_capable(void)
> +{
> + if (native_cpuid_eax(0) < 7)
> + return false;
> +
> + if (!(native_cpuid_edx(7) & CPUID_EDX_ARCH_CAP))
> + return false;
> +
> + if (!(native_rdmsrq(MSR_IA32_ARCH_CAPABILITIES) & ARCH_CAP_MCU_ENUM))
> + return false;
> +
> + return true;
> +}
Comments, please.
"mcu_capable" sounds like "Is this CPU capable of microcode updates",
which doesn't seem quite right or logically sensible.
Second, this needs to say why it's using raw CPUID functions and not
X86_FEATURE* bits or cpuid().
> /* Load microcode on BSP from initrd or builtin blobs */
> void __init load_ucode_intel_bsp(struct early_load_data *ed)
> {
> struct ucode_cpu_info uci;
>
> + /* Indicate early enough to cover the early-loading path */
> + cpu_has_mcu = mcu_capable();
> +
> uci.mc = get_microcode_blob(&uci, false);
> ed->old_rev = uci.cpu_sig.rev;
>
> @@ -1023,8 +1055,7 @@ static __init bool staging_available(void)
> {
> u64 val;
>
> - val = x86_read_arch_cap_msr();
> - if (!(val & ARCH_CAP_MCU_ENUM))
> + if (!cpu_has_mcu)
> return false;
>
> rdmsrq(MSR_IA32_MCU_ENUMERATION, val);
Yeah, I'm not sure we need to cache mcu_capable(). Just call it twice.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-06-30 21:38 ` Dave Hansen
@ 2026-06-30 21:47 ` Dave Hansen
2026-06-30 23:21 ` Borislav Petkov
` (2 more replies)
0 siblings, 3 replies; 21+ messages in thread
From: Dave Hansen @ 2026-06-30 21:47 UTC (permalink / raw)
To: Chang S. Bae, linux-kernel; +Cc: x86, tglx, mingo, bp, dave.hansen, hpa
Thinking about this a bit more: I don't think we should panic. It's
perfectly fine to spew a nice scary warning. But we really should
continue unless we really *know* that something has gone so horribly
wrong that it's dangerous to continue.
Think about things like folks who are ssh'd in. Say they have a bad,
partial ucode update. With this approach, they get a dead ssh session.
If you WARN() and keep going, they at least have a chance of seeing the
spew and getting it off the system.
I mean, the ucode update guys themselves could definitely have reset the
system if it needed to. They also know when it is dangerous to keep the
CPU running. They obviously don't think that this partial update thing
is *THAT* dangerous or they wouldn't have even let the CPU keep
churning. Right?
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-06-30 21:47 ` Dave Hansen
@ 2026-06-30 23:21 ` Borislav Petkov
2026-07-01 19:02 ` Chang S. Bae
2026-07-18 22:54 ` Maciej W. Rozycki
2 siblings, 0 replies; 21+ messages in thread
From: Borislav Petkov @ 2026-06-30 23:21 UTC (permalink / raw)
To: Dave Hansen
Cc: Chang S. Bae, linux-kernel, x86, tglx, mingo, dave.hansen, hpa
On Tue, Jun 30, 2026 at 02:47:46PM -0700, Dave Hansen wrote:
> Thinking about this a bit more: I don't think we should panic. It's
> perfectly fine to spew a nice scary warning. But we really should
> continue unless we really *know* that something has gone so horribly
> wrong that it's dangerous to continue.
>
> Think about things like folks who are ssh'd in. Say they have a bad,
> partial ucode update. With this approach, they get a dead ssh session.
>
> If you WARN() and keep going, they at least have a chance of seeing the
> spew and getting it off the system.
>
> I mean, the ucode update guys themselves could definitely have reset the
> system if it needed to. They also know when it is dangerous to keep the
> CPU running. They obviously don't think that this partial update thing
> is *THAT* dangerous or they wouldn't have even let the CPU keep
> churning. Right?
You want to taint because panic_on_warn will kill the system.
But lemme look at v2 first in detail, once it appears...
Thx.
--
Regards/Gruss,
Boris.
https://people.kernel.org/tglx/notes-about-netiquette
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-06-30 21:47 ` Dave Hansen
2026-06-30 23:21 ` Borislav Petkov
@ 2026-07-01 19:02 ` Chang S. Bae
2026-07-18 22:54 ` Maciej W. Rozycki
2 siblings, 0 replies; 21+ messages in thread
From: Chang S. Bae @ 2026-07-01 19:02 UTC (permalink / raw)
To: Dave Hansen, linux-kernel; +Cc: x86, tglx, mingo, bp, dave.hansen, hpa
On 6/30/2026 2:47 PM, Dave Hansen wrote:
>
> I mean, the ucode update guys themselves could definitely have reset the
> system if it needed to. They also know when it is dangerous to keep the
> CPU running. They obviously don't think that this partial update thing
> is *THAT* dangerous or they wouldn't have even let the CPU keep
> churning. Right?
A revision of the spec doc adds some description of the severity, which
looks worth referencing here for the review / decision.
Spec - https://cdrdv2.intel.com/v1/dl/getContent/782715
There are two bits in MSR_IA32_MCU_STATUS:
* Bit 0 indicates a partial update occured
* Bit 1 indicates authentication failed _after_ the update was
committed and the revision ID was updated.
Section 2.7 summarizes the implications:
+----------------+----------------+--------------------------+
| partial update | authentication | note / recommendation |
+----------------+----------------+--------------------------+
| no | fail | impossible |
|----------------+----------------+--------------------------|
| yes | success | log the error |
|----------------+----------------+--------------------------|
| yes | fail | raise a fatal error |
+----------------+----------------+--------------------------+
It could be an option to handle those cases separately. But yes, a sane
ucode can escalate an unrecoverable condition via #MC. I'll revise V2 to
simply taint the kernel (TAINT_WARN) on any of those.
Thanks,
Chang
^ permalink raw reply [flat|nested] 21+ messages in thread
* [PATCH v2] x86/microcode/intel: Taint kernel on partial update
2026-06-30 19:13 [PATCH] x86/microcode/intel: Panic on partial microcode update Chang S. Bae
2026-06-30 21:38 ` Dave Hansen
@ 2026-07-08 21:18 ` Chang S. Bae
2026-07-27 11:27 ` Nikolay Borisov
1 sibling, 1 reply; 21+ messages in thread
From: Chang S. Bae @ 2026-07-08 21:18 UTC (permalink / raw)
To: linux-kernel; +Cc: x86, tglx, mingo, bp, dave.hansen, hpa, chang.seok.bae
A modern individual microcode update contains firmware for many pieces of
silicon inside the CPU. Sometimes, a single update operation successfully
updates some components and not others leaving a partially-applied
update.
This may leave the system in an undefined and unreliable state. Fatal
failures are expected to be handled by the CPU itself, by raising #MC for
example. If the CPU instead reports a partial update, warn and taint the
kernel at least.
A partial update may still update the revision. Continue checking the
revision rather than an immediate error return so the update result
remains consistent with the revision changes.
Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
---
V1 -> V2: Do not raise panic, inline cpuid check and massage the changelog (Dave)
---
arch/x86/include/asm/msr-index.h | 4 +++
arch/x86/kernel/cpu/microcode/intel.c | 37 +++++++++++++++++++++++++++
2 files changed, 41 insertions(+)
diff --git a/arch/x86/include/asm/msr-index.h b/arch/x86/include/asm/msr-index.h
index 18c4be75e927..7273d340470d 100644
--- a/arch/x86/include/asm/msr-index.h
+++ b/arch/x86/include/asm/msr-index.h
@@ -977,6 +977,10 @@
#define MSR_IA32_MCU_ENUMERATION 0x0000007b
#define MCU_STAGING BIT(4)
+#define MSR_IA32_MCU_STATUS 0x0000007c
+#define MCU_PARTIAL_UPDATE BIT(0)
+#define AUTH_FAIL_ON_MCU_COMPONENT BIT(1)
+
#define MSR_IA32_UCODE_REV 0x0000008b
/* Intel SGX Launch Enclave Public Key Hash MSRs */
diff --git a/arch/x86/kernel/cpu/microcode/intel.c b/arch/x86/kernel/cpu/microcode/intel.c
index f4a444e6114d..913b0b32e39b 100644
--- a/arch/x86/kernel/cpu/microcode/intel.c
+++ b/arch/x86/kernel/cpu/microcode/intel.c
@@ -69,6 +69,9 @@ static const char ucode_path[] = "kernel/x86/microcode/GenuineIntel.bin";
#define MBOX_XACTION_TIMEOUT_MS (10 * MSEC_PER_SEC)
+#define CPUID_EDX_ARCH_CAP BIT(29)
+#define MCU_STATUS_FAILURE_MASK (MCU_PARTIAL_UPDATE | AUTH_FAIL_ON_MCU_COMPONENT)
+
/* Current microcode patch used in early patching on the APs. */
static struct microcode_intel *ucode_patch_va __read_mostly;
static struct microcode_intel *ucode_patch_late __read_mostly;
@@ -679,6 +682,24 @@ static void stage_microcode(void)
pr_info("Staging of patch revision 0x%x succeeded.\n", ucode_patch_late->hdr.rev);
}
+/*
+ * __apply_microcode() is the only caller which may be invoked in the early
+ * loading path. Use raw CPUID/RDMSR functions.
+ */
+static bool update_status_available(void)
+{
+ if (native_cpuid_eax(0) < 7)
+ return false;
+
+ if (!(native_cpuid_edx(7) & CPUID_EDX_ARCH_CAP))
+ return false;
+
+ if (!(native_rdmsrq(MSR_IA32_ARCH_CAPABILITIES) & ARCH_CAP_MCU_ENUM))
+ return false;
+
+ return true;
+}
+
static enum ucode_state __apply_microcode(struct ucode_cpu_info *uci,
struct microcode_intel *mc,
u32 *cur_rev)
@@ -702,6 +723,22 @@ static enum ucode_state __apply_microcode(struct ucode_cpu_info *uci,
/* write microcode via MSR 0x79 */
native_wrmsrq(MSR_IA32_UCODE_WRITE, (unsigned long)mc->bits);
+ /*
+ * Warn and taint the kernel on a partial update. Fatal conditions are
+ * expected to be handled by the CPU itself.
+ *
+ * Then, continue checking the revision since a partial update may still
+ * advance the microcode revision.
+ */
+ if (update_status_available()) {
+ u64 status = native_rdmsrq(MSR_IA32_MCU_STATUS);
+
+ if (status & MCU_STATUS_FAILURE_MASK) {
+ pr_warn_once("update incomplete (MSR_IA32_MCU_STATUS=0x%llx).\n", status);
+ add_taint(TAINT_CPU_OUT_OF_SPEC, LOCKDEP_STILL_OK);
+ }
+ }
+
rev = intel_get_microcode_revision();
if (rev != mc->hdr.rev)
return UCODE_ERROR;
--
2.51.0
^ permalink raw reply related [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-06-30 21:47 ` Dave Hansen
2026-06-30 23:21 ` Borislav Petkov
2026-07-01 19:02 ` Chang S. Bae
@ 2026-07-18 22:54 ` Maciej W. Rozycki
2026-07-19 15:52 ` Dave Hansen
2 siblings, 1 reply; 21+ messages in thread
From: Maciej W. Rozycki @ 2026-07-18 22:54 UTC (permalink / raw)
To: Dave Hansen
Cc: Chang S. Bae, linux-kernel, x86, tglx, Ingo Molnar, bp,
dave.hansen, hpa
On Tue, 30 Jun 2026, Dave Hansen wrote:
> Thinking about this a bit more: I don't think we should panic. It's
> perfectly fine to spew a nice scary warning. But we really should
> continue unless we really *know* that something has gone so horribly
> wrong that it's dangerous to continue.
Is there no risk for silent data corruption such as in storage after a
broken ucode update?
> Think about things like folks who are ssh'd in. Say they have a bad,
> partial ucode update. With this approach, they get a dead ssh session.
>
> If you WARN() and keep going, they at least have a chance of seeing the
> spew and getting it off the system.
A panic() message may not make it there, but KERN_EMERG messages ought to
make it to all the terminal sessions via syslogd(8) where logging has been
correctly configured for a system (so up to the local policy to get it
right). A way to force it through could be needed here though.
Then there's the system console the kernel messages are usually produced
to (again, subject to the local policy) one could watch; one might argue
even that such a dangerous operation on a live system would best be done
via the console terminal (even if it means IPMI nowadays).
> I mean, the ucode update guys themselves could definitely have reset the
> system if it needed to. They also know when it is dangerous to keep the
> CPU running. They obviously don't think that this partial update thing
> is *THAT* dangerous or they wouldn't have even let the CPU keep
> churning. Right?
Umm, what the reasonable alternative would be for the RTL designer, a CPU
shutdown? Then you'd have no good way to figure out what really happened.
Maciej
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-18 22:54 ` Maciej W. Rozycki
@ 2026-07-19 15:52 ` Dave Hansen
2026-07-23 22:42 ` Maciej W. Rozycki
2026-07-27 20:36 ` Chang S. Bae
0 siblings, 2 replies; 21+ messages in thread
From: Dave Hansen @ 2026-07-19 15:52 UTC (permalink / raw)
To: Maciej W. Rozycki
Cc: Chang S. Bae, linux-kernel, x86, tglx, Ingo Molnar, bp,
dave.hansen, hpa
On 7/18/26 15:54, Maciej W. Rozycki wrote:
> On Tue, 30 Jun 2026, Dave Hansen wrote:
>> Thinking about this a bit more: I don't think we should panic. It's
>> perfectly fine to spew a nice scary warning. But we really should
>> continue unless we really *know* that something has gone so horribly
>> wrong that it's dangerous to continue.
>
> Is there no risk for silent data corruption such as in storage after a
> broken ucode update?
Is there no risk for silent data corruption such as in storage after a
kernel panic?
Is there no risk for silent data corruption such as in storage after a
use after free or a stray write? Or any other bog standard kernel bug?
We don't aim for "zero risk", IMNHO. We can't.
>> I mean, the ucode update guys themselves could definitely have reset the
>> system if it needed to. They also know when it is dangerous to keep the
>> CPU running. They obviously don't think that this partial update thing
>> is *THAT* dangerous or they wouldn't have even let the CPU keep
>> churning. Right?
>
> Umm, what the reasonable alternative would be for the RTL designer, a CPU
> shutdown? Then you'd have no good way to figure out what really happened.
There are actually a huge number of things that can happen to a CPU that
result in a reset or other conditions that are really hard to debug.
Thankfully, most of them get debugged out of the system before end users
ever get them. But folks who work with early silicon frequently get to
experience the joy of having no good way to figure out what really happened.
Those "early" folks have lots of other tools at their disposal. But,
believe me, the RTL designers and ucode writers are not shy about
killing the system if data corruption is at all likely.
Chang, have you seen anything at all in the specs or from our Intel
colleagues that makes you think that it's _dangerous_ to keep running
rather than killing the system as soon as possible?
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-19 15:52 ` Dave Hansen
@ 2026-07-23 22:42 ` Maciej W. Rozycki
2026-07-27 12:35 ` Nikolay Borisov
2026-07-27 20:36 ` Chang S. Bae
1 sibling, 1 reply; 21+ messages in thread
From: Maciej W. Rozycki @ 2026-07-23 22:42 UTC (permalink / raw)
To: Dave Hansen
Cc: Chang S. Bae, linux-kernel, x86, tglx, Ingo Molnar, bp,
dave.hansen, hpa
On Sun, 19 Jul 2026, Dave Hansen wrote:
> >> Thinking about this a bit more: I don't think we should panic. It's
> >> perfectly fine to spew a nice scary warning. But we really should
> >> continue unless we really *know* that something has gone so horribly
> >> wrong that it's dangerous to continue.
> >
> > Is there no risk for silent data corruption such as in storage after a
> > broken ucode update?
> Is there no risk for silent data corruption such as in storage after a
> kernel panic?
>
> Is there no risk for silent data corruption such as in storage after a
> use after free or a stray write? Or any other bog standard kernel bug?
>
> We don't aim for "zero risk", IMNHO. We can't.
We aim not to set a policy either, but at this point we have become aware
the CPU is in a bad state, so I think at the very least there needs to be
a way for the site manager to configure handling according to their needs,
and I think it should be killing the system by default.
> >> I mean, the ucode update guys themselves could definitely have reset the
> >> system if it needed to. They also know when it is dangerous to keep the
> >> CPU running. They obviously don't think that this partial update thing
> >> is *THAT* dangerous or they wouldn't have even let the CPU keep
> >> churning. Right?
> >
> > Umm, what the reasonable alternative would be for the RTL designer, a CPU
> > shutdown? Then you'd have no good way to figure out what really happened.
>
> There are actually a huge number of things that can happen to a CPU that
> result in a reset or other conditions that are really hard to debug.
> Thankfully, most of them get debugged out of the system before end users
> ever get them. But folks who work with early silicon frequently get to
> experience the joy of having no good way to figure out what really happened.
>
> Those "early" folks have lots of other tools at their disposal. But,
> believe me, the RTL designers and ucode writers are not shy about
> killing the system if data corruption is at all likely.
That is true, however I think there is essential difference between CPU
bringup and faults triggered in the field. And you do actually have extra
tools for bringup, such as hardware emulation, bond-outs, JTAG debug, etc.
> Chang, have you seen anything at all in the specs or from our Intel
> colleagues that makes you think that it's _dangerous_ to keep running
> rather than killing the system as soon as possible?
From the implications already quoted:
On Wed, 1 Jul 2026, Chang S. Bae wrote:
> Section 2.7 summarizes the implications:
>
> +----------------+----------------+--------------------------+
> | partial update | authentication | note / recommendation |
> +----------------+----------------+--------------------------+
> | no | fail | impossible |
> |----------------+----------------+--------------------------|
> | yes | success | log the error |
> |----------------+----------------+--------------------------|
> | yes | fail | raise a fatal error |
> +----------------+----------------+--------------------------+
I infer the action you recommend to be taken is suitable for the second
case, and in the third case the system is supposed to stop operating -- a
fatal error in my interpretation means termination and for hardware it
implies killing the system once the error has been reported.
This would match my recommendation, and with a BMC in the picture I think
it's even easier, because it's external to the main CPU(s) and therefore
all that is required is sending the error message across right before
termination. The message can then be processed by the BMC accordingly as
there's no need for the BMC to become disrupted in any way with a main
CPU's fault.
Maciej
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH v2] x86/microcode/intel: Taint kernel on partial update
2026-07-08 21:18 ` [PATCH v2] x86/microcode/intel: Taint kernel on partial update Chang S. Bae
@ 2026-07-27 11:27 ` Nikolay Borisov
0 siblings, 0 replies; 21+ messages in thread
From: Nikolay Borisov @ 2026-07-27 11:27 UTC (permalink / raw)
To: Chang S. Bae, linux-kernel; +Cc: x86, tglx, mingo, bp, dave.hansen, hpa
On 7/9/26 00:18, Chang S. Bae wrote:
> A modern individual microcode update contains firmware for many pieces of
> silicon inside the CPU. Sometimes, a single update operation successfully
> updates some components and not others leaving a partially-applied
> update.
>
> This may leave the system in an undefined and unreliable state. Fatal
> failures are expected to be handled by the CPU itself, by raising #MC for
> example. If the CPU instead reports a partial update, warn and taint the
> kernel at least.
>
> A partial update may still update the revision. Continue checking the
> revision rather than an immediate error return so the update result
> remains consistent with the revision changes.
>
> Signed-off-by: Chang S. Bae <chang.seok.bae@intel.com>
> ---
> V1 -> V2: Do not raise panic, inline cpuid check and massage the changelog (Dave)
Reviewed-by: Nikolay Borisov <nik.borisov@suse.com>
<rant>
What a long-winded way to check whether MCU_ENUM is present... I
understand it's outside of the purview of software but couldn't this
been put into some customs CPUID leaf ... sigh
</rant>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-23 22:42 ` Maciej W. Rozycki
@ 2026-07-27 12:35 ` Nikolay Borisov
2026-07-28 21:53 ` Maciej W. Rozycki
0 siblings, 1 reply; 21+ messages in thread
From: Nikolay Borisov @ 2026-07-27 12:35 UTC (permalink / raw)
To: Maciej W. Rozycki, Dave Hansen
Cc: Chang S. Bae, linux-kernel, x86, tglx, Ingo Molnar, bp,
dave.hansen, hpa
On 7/24/26 01:42, Maciej W. Rozycki wrote:
> We aim not to set a policy either, but at this point we have become aware
> the CPU is in a bad state, so I think at the very least there needs to be
> a way for the site manager to configure handling according to their needs,
> and I think it should be killing the system by default.
For people who care that much about this, they likely are running huge
fleets and already have daemons/agents(sic) which are monitoring dmesg
and are taking actions accordingly. So that use case is not lost.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-19 15:52 ` Dave Hansen
2026-07-23 22:42 ` Maciej W. Rozycki
@ 2026-07-27 20:36 ` Chang S. Bae
2026-07-27 21:59 ` Sohil Mehta
1 sibling, 1 reply; 21+ messages in thread
From: Chang S. Bae @ 2026-07-27 20:36 UTC (permalink / raw)
To: Dave Hansen, Maciej W. Rozycki
Cc: linux-kernel, x86, tglx, Ingo Molnar, bp, dave.hansen, hpa
On 7/19/2026 8:52 AM, Dave Hansen wrote:
>
> Chang, have you seen anything at all in the specs or from our Intel
> colleagues that makes you think that it's _dangerous_ to keep running
> rather than killing the system as soon as possible?
I double-checked with the microcode guy, and here are what I heard so far:
* The microcode implementation makes best effort to avoid partial
updates in the first place
* If an update leaves any critical state, e.g. data corruption and
serious security breach, the processor will terminate execution
rather than relying on the OS to take action.
* That said, no implementation can perfectly cover every possible case.
At the same time, panicking on every partial update could be also an
overkill. It will depend on the specific failure condition in detail.
Then, they've defined those bits seemingly intended to summary the
condition at abstraction.
I think the first two points have played in this thread already. There
seems to be some caution, specifically around the authentication-failure
as per spec. Perhaps it might be reassuring if handling that case
conservatively (unless too much complicated).
In summary, I think options are:
1. Panic on any partial update is the most conservative and secure
option, but it may unnecessarily disrupt systems and users
2. Warn admins that the system has entered a taint state, assuming the
processor handles truly unrecoverable cases itself.
3. Different reactions - for example, panic on an authentication
failure but warn/taint otherwise.
Thanks,
Chang
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-27 20:36 ` Chang S. Bae
@ 2026-07-27 21:59 ` Sohil Mehta
2026-07-28 2:58 ` Chang S. Bae
0 siblings, 1 reply; 21+ messages in thread
From: Sohil Mehta @ 2026-07-27 21:59 UTC (permalink / raw)
To: Chang S. Bae, Dave Hansen, Maciej W. Rozycki
Cc: linux-kernel, x86, tglx, Ingo Molnar, bp, dave.hansen, hpa
On 7/27/2026 1:36 PM, Chang S. Bae wrote:
> I think the first two points have played in this thread already. There
> seems to be some caution, specifically around the authentication-failure
> as per spec.
What does an authentication failure mean in this context?
Does that mean unauthenticated microcode components somehow got loaded?
or that
Someone attempted to load bogus microcode which was rejected by the
processor?
If it was rejected, then there is no need to panic, right? Just warning
about the attempt should be enough.
>
> In summary, I think options are:
>
> 1. Panic on any partial update is the most conservative and secure
> option, but it may unnecessarily disrupt systems and users
My understanding of a panic is the OS saying "something catastrophic has
happened and I don't know how to make forward progress". So, it's better
to crash at this point than spew out incoherent information.
> 2. Warn admins that the system has entered a taint state, assuming the
> processor handles truly unrecoverable cases itself.
If the processor chooses not to terminate itself after a partial update
maybe the kernel shouldn't either and give the admin the opportunity to
reconcile at their convenience. I think a strong warning makes the most
sense here.
In addition to a taint, should there also be a WARN()? That gives any
paranoid admin the option to automatically convert WARN => PANIC via
panic_on_warn. They can set panic_on_warn at runtime just before the
microcode update and then clear it once the update is done.
There seems to be a kernel command line option called panic_on_taint
which can be configured to panic only for TAINT_CPU_OUT_OF_SPEC. But
that may not be very useful because there isn't any runtime sysctl option.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-27 21:59 ` Sohil Mehta
@ 2026-07-28 2:58 ` Chang S. Bae
2026-07-28 14:12 ` Dave Hansen
0 siblings, 1 reply; 21+ messages in thread
From: Chang S. Bae @ 2026-07-28 2:58 UTC (permalink / raw)
To: Sohil Mehta, Dave Hansen, Maciej W. Rozycki
Cc: linux-kernel, x86, tglx, Ingo Molnar, bp, dave.hansen, hpa
On 7/27/2026 2:59 PM, Sohil Mehta wrote:
>
> What does an authentication failure mean in this context?
The spec has this:
When set to 1, it indicates that an authentication failure occurred on
some portion of the MCU after another portion had already been
committed and the Revision ID had already been updated on the most
recent write to IA32_BIOS_UPDT_TRIG.
It would be helpful if including some examples, but those appear to fall
under microcode implementation details as I heard.
> Does that mean unauthenticated microcode components somehow got loaded?
>
> or that
>
> Someone attempted to load bogus microcode which was rejected by the
> processor?
First, the SDM describes the legacy behavior in Vol.3A 12.11.7 "Update
Signature and Verification". In the legacy flow, an invalid or
unauthenticated patch is rejected _before_ the revision ID is updated.
So the outcome is effectively binary: either the update succeeds, or it
is rejected.
The partial-update status introduce a third outcome. Some portion of the
update has already been committed and (maybe, therefore) the revision ID
has been updated, but another portion failed. The result is not the
simple "accepted" or "rejected" state. This thus could be seen as a
bogus state from this perspective, TBH.
> If it was rejected, then there is no need to panic, right? Just warning
> about the attempt should be enough.
In this new state, the processor may either continue operating or end up
running with a combination of sub-components that has never been tested.
Given that, I think one question to ask in the end is whether we trust
the microcode's own error handling (completely), or not.
Thanks,
Chang
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-28 2:58 ` Chang S. Bae
@ 2026-07-28 14:12 ` Dave Hansen
2026-07-28 18:56 ` David Laight
0 siblings, 1 reply; 21+ messages in thread
From: Dave Hansen @ 2026-07-28 14:12 UTC (permalink / raw)
To: Chang S. Bae, Sohil Mehta, Maciej W. Rozycki
Cc: linux-kernel, x86, tglx, Ingo Molnar, bp, dave.hansen, hpa
On 7/27/26 19:58, Chang S. Bae wrote:
> In this new state, the processor may either continue operating or end up
> running with a combination of sub-components that has never been tested.
> Given that, I think one question to ask in the end is whether we trust
> the microcode's own error handling (completely), or not.
We *have* to trust it completely.
Think of it like this: CPUs are fundamentally analog, physical things.
They malfunction. They break. They include human-induced design errors.
But we don't have all kinds of code in the kernel to make sure that
gates in the ALU don't go bonkers and do 1+1=3.
It's not because kernel developers don't think that CPUs can break. It's
because if we went trying to detect things like that ^, the kernel
wouldn't be able to do anything else useful.
It's the same for the microcode update. The kernel first trusts that the
CPU does what the spec says. If that turns out not to be enough in the
real world, the kernel adds some paranoia later. But it's done
defensively against real, actual screwups. Not preemptively against
every imaginable screwup.
WARN(), please. Not panic().
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-28 14:12 ` Dave Hansen
@ 2026-07-28 18:56 ` David Laight
2026-07-28 20:48 ` Dave Hansen
0 siblings, 1 reply; 21+ messages in thread
From: David Laight @ 2026-07-28 18:56 UTC (permalink / raw)
To: Dave Hansen
Cc: Chang S. Bae, Sohil Mehta, Maciej W. Rozycki, linux-kernel, x86,
tglx, Ingo Molnar, bp, dave.hansen, hpa
On Tue, 28 Jul 2026 07:12:13 -0700
Dave Hansen <dave.hansen@intel.com> wrote:
> On 7/27/26 19:58, Chang S. Bae wrote:
> > In this new state, the processor may either continue operating or end up
> > running with a combination of sub-components that has never been tested.
> > Given that, I think one question to ask in the end is whether we trust
> > the microcode's own error handling (completely), or not.
>
> We *have* to trust it completely.
>
> Think of it like this: CPUs are fundamentally analog, physical things.
> They malfunction. They break. They include human-induced design errors.
> But we don't have all kinds of code in the kernel to make sure that
> gates in the ALU don't go bonkers and do 1+1=3.
>
> It's not because kernel developers don't think that CPUs can break. It's
> because if we went trying to detect things like that ^, the kernel
> wouldn't be able to do anything else useful.
>
> It's the same for the microcode update. The kernel first trusts that the
> CPU does what the spec says. If that turns out not to be enough in the
> real world, the kernel adds some paranoia later. But it's done
> defensively against real, actual screwups. Not preemptively against
> every imaginable screwup.
>
> WARN(), please. Not panic().
>
PANIC_ON_WARN ....
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-28 18:56 ` David Laight
@ 2026-07-28 20:48 ` Dave Hansen
2026-07-28 21:47 ` Maciej W. Rozycki
2026-07-28 22:09 ` David Laight
0 siblings, 2 replies; 21+ messages in thread
From: Dave Hansen @ 2026-07-28 20:48 UTC (permalink / raw)
To: David Laight
Cc: Chang S. Bae, Sohil Mehta, Maciej W. Rozycki, linux-kernel, x86,
tglx, Ingo Molnar, bp, dave.hansen, hpa
On 7/28/26 11:56, David Laight wrote:
>> WARN(), please. Not panic().
>>
> PANIC_ON_WARN ....
Right. That's kinda my point.
If you are of the tin-foil-hat-wearing persuasion, you are paranoid and
can't get enough panic()s in your life. You have panic_on_warn=1 to
magically transform all warnings into panic()s and warm your heart.
There's no warn_on_panic=1. The trusting don't have an analog.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-28 20:48 ` Dave Hansen
@ 2026-07-28 21:47 ` Maciej W. Rozycki
2026-07-28 21:53 ` Dave Hansen
2026-07-28 22:09 ` David Laight
1 sibling, 1 reply; 21+ messages in thread
From: Maciej W. Rozycki @ 2026-07-28 21:47 UTC (permalink / raw)
To: Dave Hansen
Cc: David Laight, Chang S. Bae, Sohil Mehta, linux-kernel, x86, tglx,
Ingo Molnar, bp, dave.hansen, hpa
On Tue, 28 Jul 2026, Dave Hansen wrote:
> >> WARN(), please. Not panic().
> >>
> > PANIC_ON_WARN ....
>
> Right. That's kinda my point.
>
> If you are of the tin-foil-hat-wearing persuasion, you are paranoid and
> can't get enough panic()s in your life. You have panic_on_warn=1 to
> magically transform all warnings into panic()s and warm your heart.
This OTOH seems excessive, there are warnings of varying severity, e.g.
you don't usually want to kill your system on an odd error with a low
importance peripheral.
Thinking more as to not enforcing any policy while letting people apply
one according to their site rules, how about communicating the condition
via the severity level, such as say KERN_ALERT or maybe KERN_EMERG for
failed authentication and plain ordinary KERN_ERR for a partial update
that has succeeded authenticating? Then the receiver of the messages can
decide what to do about them, such as the BMC power-cycling the system.
Have we come up with any guidelines beyond common sense as to how to
apply these severity levels to any conditions encountered being reported?
Maciej
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-28 21:47 ` Maciej W. Rozycki
@ 2026-07-28 21:53 ` Dave Hansen
0 siblings, 0 replies; 21+ messages in thread
From: Dave Hansen @ 2026-07-28 21:53 UTC (permalink / raw)
To: Maciej W. Rozycki
Cc: David Laight, Chang S. Bae, Sohil Mehta, linux-kernel, x86, tglx,
Ingo Molnar, bp, dave.hansen, hpa
On 7/28/26 14:47, Maciej W. Rozycki wrote:
> Have we come up with any guidelines beyond common sense as to how to
> apply these severity levels to any conditions encountered being reported?
There are quite a few guidelines in here:
Documentation/process/coding-style.rst
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-27 12:35 ` Nikolay Borisov
@ 2026-07-28 21:53 ` Maciej W. Rozycki
0 siblings, 0 replies; 21+ messages in thread
From: Maciej W. Rozycki @ 2026-07-28 21:53 UTC (permalink / raw)
To: Nikolay Borisov
Cc: Dave Hansen, Chang S. Bae, linux-kernel, x86, tglx, Ingo Molnar,
bp, dave.hansen, hpa
On Mon, 27 Jul 2026, Nikolay Borisov wrote:
> > We aim not to set a policy either, but at this point we have become aware
> > the CPU is in a bad state, so I think at the very least there needs to be
> > a way for the site manager to configure handling according to their needs,
> > and I think it should be killing the system by default.
>
> For people who care that much about this, they likely are running huge fleets
> and already have daemons/agents(sic) which are monitoring dmesg and are taking
> actions accordingly. So that use case is not lost.
But do they keep up to date databases of possible messages to match so as
to act accordingly? I think there is a need to communicate the severity
of the situation somehow; cf.
<https://lore.kernel.org/r/alpine.DEB.2.21.2607282230170.14485@angie.orcam.me.uk/>.
Maciej
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [PATCH] x86/microcode/intel: Panic on partial microcode update
2026-07-28 20:48 ` Dave Hansen
2026-07-28 21:47 ` Maciej W. Rozycki
@ 2026-07-28 22:09 ` David Laight
1 sibling, 0 replies; 21+ messages in thread
From: David Laight @ 2026-07-28 22:09 UTC (permalink / raw)
To: Dave Hansen
Cc: Chang S. Bae, Sohil Mehta, Maciej W. Rozycki, linux-kernel, x86,
tglx, Ingo Molnar, bp, dave.hansen, hpa
On Tue, 28 Jul 2026 13:48:20 -0700
Dave Hansen <dave.hansen@intel.com> wrote:
> On 7/28/26 11:56, David Laight wrote:
> >> WARN(), please. Not panic().
> >>
> > PANIC_ON_WARN ....
>
> Right. That's kinda my point.
>
> If you are of the tin-foil-hat-wearing persuasion, you are paranoid and
> can't get enough panic()s in your life. You have panic_on_warn=1 to
> magically transform all warnings into panic()s and warm your heart.
>
> There's no warn_on_panic=1. The trusting don't have an analog.
And the panics are much harder to debug.
It isn't as though the code waits long enough after generating all the
'console' output to let syslogd get it to filestore.
We we trying to find why a customer system kept rebooting.
Eventually realised PANIC_ON_OOPS was set, turned it off and the silly bug
was obvious.
David
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2026-07-28 22:09 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-06-30 19:13 [PATCH] x86/microcode/intel: Panic on partial microcode update Chang S. Bae
2026-06-30 21:38 ` Dave Hansen
2026-06-30 21:47 ` Dave Hansen
2026-06-30 23:21 ` Borislav Petkov
2026-07-01 19:02 ` Chang S. Bae
2026-07-18 22:54 ` Maciej W. Rozycki
2026-07-19 15:52 ` Dave Hansen
2026-07-23 22:42 ` Maciej W. Rozycki
2026-07-27 12:35 ` Nikolay Borisov
2026-07-28 21:53 ` Maciej W. Rozycki
2026-07-27 20:36 ` Chang S. Bae
2026-07-27 21:59 ` Sohil Mehta
2026-07-28 2:58 ` Chang S. Bae
2026-07-28 14:12 ` Dave Hansen
2026-07-28 18:56 ` David Laight
2026-07-28 20:48 ` Dave Hansen
2026-07-28 21:47 ` Maciej W. Rozycki
2026-07-28 21:53 ` Dave Hansen
2026-07-28 22:09 ` David Laight
2026-07-08 21:18 ` [PATCH v2] x86/microcode/intel: Taint kernel on partial update Chang S. Bae
2026-07-27 11:27 ` Nikolay Borisov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox