Re: x86: missing FRED #PF event data?

[Dmytro Maluka <dmaluka@google.com>]

On Tue, Aug 11, 2026 at 04:05:36PM +0900, Sergey Senozhatsky wrote:
> On (26/08/10 22:37), Xin Li wrote:
> > > On August 10, 2026 6:47:13 PM PDT, Sergey Senozhatsky <senozhatsky@chromium.org> wrote:
> > >> On (26/08/10 08:40), H. Peter Anvin wrote:
> > >>> On August 10, 2026 1:58:18 AM PDT, Sergey Senozhatsky <senozhatsky@chromium.org> wrote:
> > >>>> On (26/08/10 16:38), Sergey Senozhatsky wrote:
> > >>>>> [..]
> > >>>>>> All the crashes are reported as NULL ptr derefs, however, I believe this
> > >>>>>> is not exactly the case.  In all crashes CR2 is 0x1000 aligned (we always
> > >>>>>> crash accessing first byte of a page).  It seems that csum_partial() calls
> > >>>>>> load_unaligned_zeropad() and we hit what load_unaligned_zeropad() comment
> > >>>>>> describes as very unlikely) case: "word being a page-crosser and the
> > >>>>>> next page not being mapped").  So instead of reading 4 remaining bytes
> > >>>>>> of the page and zeroes for trailing 4 bytes, we panic().  It appears that
> > >>>>>> FRED #PF is set to 0 while CR2 points to a correct page address.  I added
> > >>>>>> a simple printk to exc_page_fault:
> > >>>>>> 
> > >>>>>>        address = cpu_feature_enabled(X86_FEATURE_FRED) ? fred_event_data(regs) : read_cr2();
> > >>>>>>        /* Fall back to CR2 if FRED event data was empty */
> > >>>>>>        if (unlikely(!address)) {
> > >>>>>>                address = read_cr2();
> > >>>>>>                pr_err(":: fixed up address to %lx [[fred: %lx cr2: %lx]]\n", address, fred_event_data(regs), read_cr2());
> > >>>>>>        }
> > >>>>>> 
> > >>>>>> and got the following while running my tests (and well, we don't crash
> > >>>>>> anymore):
> > >>>>>> 
> > >>>>>> [  254.040223] :: fixed up address to ffff9c4d64af4000  [[fred: 0 cr2: ffff9c4d64af4000]]
> > >>>>>> ...
> > >>>>>> [ 1821.904563] :: fixed up address to ffff9c4e9dd0a000  [[fred: 0 cr2: ffff9c4e9dd0a000]]
> > >>>>>> 
> > >>>>>> Does any of this make sense to you?
> > >>>>> 
> > >>>>> I think the explanation is some pKVM shenanigans.  Sorry for the noise.
> > >>>> 
> > >>>> No, I think we are back at square one.  I thought that maybe pKVM
> > >>>> was disabling FRED and that was causing issues.  But I actually see
> > >>>> that both cpu_feature_enabled(X86_FEATURE_FRED) and (cr4 & X86_CR4_FRED)
> > >>>> claim FRED is enabled, yet fred #PF data is 0 while CR2 holds the correct
> > >>>> address.
> > >>> 
> > >>> What is pKVM? Paravirtualized KVM?
> > >> 
> > >> Protected KVM.
> > >> 
> > >>> In that case, it is most likely pKVM not filling in the relevant fields
> > >>> in the FRED stack frame, which would be a very serious bug.
> > >>> 
> > >>> I cannot think of any other way that that could possibly happen otherwise;
> > >>> on bare metal those fields are set by hardware and Linux only consumes them.
> > >> 
> > >> I agree.  I'll look at it from the pKVM side.  I was not aware of pKVM
> > >> when I started this discussion, I found out about it later.
> > > 
> > > If that code calls the FRED entry from KVM routine, that routine doesn't have support for setting event_data in upstream. This would be fixed if necessary.
> > 
> > Per Sean, it’s “host” running in a VM, so it’s kind of like a filter
> > hypervisor you ever mentioned; part of the “host" running in non-root mode.
> > 
> > So where is this page fault from?  If it’s from non-root mode, does this
> > page fault cause a VM exit?  If yes and pKVM forwards it to FRED entry, I
> > would guess it is exactly the case.
> 
> Added Chuanxiao and Dmytro, folks please correct me.
> 
> What I see: the page fault is happening in the non-root mode (native
> MMU?).  I don't see a VM exit - I tried injecting FRED #PF data but
> exc_page_fault() still reads 0x00 FRED #PF data.  What I also see is
> that... it seems to be a hybrid configurations.  From what I can tell,
> guests have FRED enabled in CR4, while hypervisor has FRED disabled in
> CR4.  So maybe this mix of FRED modes is what pushes empty FRED #PF frame?
> 
> Sorry if I babbled complete nonsense.  I'll happily hand it over to
> Chuanxiao and Dmytro at this point.

We already figured the problem out offline, let me describe it here for
posterity.

There is actually a VM exit. What is happening is: with our out-of-tree
pKVM-x86 patches, load_unaligned_zeropad() legitimately crosses a page
that is protected from the host by pKVM (i.e. unmapped in the host VM's
EPT page tables) yet still mapped in the host's own stage-1 page tables
(e.g. as a part of the kernel direct map). So this doesn't trigger a
native #PF within the host, it triggers an EPT violation, and then pKVM
synthesizes a #PF and injects it into the host VM to let
load_unaligned_zeropad() work seamlessly.

And basically hpa's guess is spot on: the problem is that for injecting
this #PF, pKVM is reusing KVM's vmx_inject_exception(), which doesn't
support the case when FRED is enabled in the guest and thus doesn't set
the event_data (not until Xin's patches [1] are merged). I've quickly
patched that up in [2].

As for the FRED setup in hypervisor vs host VM (which is rather
orthogonal to the above problem): indeed, FRED is disabled in the pKVM
hypervisor [3] while in the host VM it is kept enabled if it was
enabled before deprivileging [4], and the host VM "owns" FRED, i.e. all
FRED-related MSRs are passed-through to the host VM, and
VM_EXIT_SAVE_IA32_FRED and VM_EXIT_LOAD_IA32_FRED are *not* enabled for
the host VM. Please anyone let me know if such a setup is problematic in
any way.

[1] https://lore.kernel.org/kvm/20251026201911.505204-1-xin@zytor.com/
[2] https://android-review.googlesource.com/4225278
[3] https://android.googlesource.com/kernel/common/+/d13d0c68ee9106a26a20cbab4a653f0d8dd4691f/arch/x86/kvm/vmx/pkvm_init.c#909
[4] https://android.googlesource.com/kernel/common/+/d13d0c68ee9106a26a20cbab4a653f0d8dd4691f/arch/x86/kvm/vmx/pkvm_init.c#794