Xen-unstable Linux-6.1.0-rc5 BUG: unable to handle page fault for address: ffff8880083374d0

Xen-unstable Linux-6.1.0-rc5 BUG: unable to handle page fault for address: ffff8880083374d0

[Sander Eikelenboom]

Hi Yu / Juergen,

This night I got a dom0 kernel crash on my new Ryzen box running Xen-unstable and a Linux-6.1.0-rc5 kernel.
I did enable the new and shiny MGLRU, could this be related ?

--
Sander


Nov 19 06:30:11 serveerstertje kernel: [68959.647371] BUG: unable to handle page fault for address: ffff8880083374d0
Nov 19 06:30:11 serveerstertje kernel: [68959.663555] #PF: supervisor write access in kernel mode
Nov 19 06:30:11 serveerstertje kernel: [68959.677542] #PF: error_code(0x0003) - permissions violation
Nov 19 06:30:11 serveerstertje kernel: [68959.691181] PGD 3026067 P4D 3026067 PUD 3027067 PMD 7fee5067 PTE 8010000008337065
Nov 19 06:30:11 serveerstertje kernel: [68959.705084] Oops: 0003 [#1] PREEMPT SMP NOPTI
Nov 19 06:30:11 serveerstertje kernel: [68959.718710] CPU: 7 PID: 158 Comm: kswapd0 Not tainted 6.1.0-rc5-20221118-doflr-mac80211debug+ #1
Nov 19 06:30:11 serveerstertje kernel: [68959.732457] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./B450 Pro4 R2.0, BIOS P5.60 10/20/2022
Nov 19 06:30:11 serveerstertje kernel: [68959.746391] RIP: e030:pmdp_test_and_clear_young+0x25/0x40
Nov 19 06:30:11 serveerstertje kernel: [68959.760294] Code: 00 00 00 66 90 48 b9 ff ff 1f 00 00 00 f0 ff 48 8b 02 48 be ff 0f 00 00 00 00 f0 ff a8 80 48 0f 44 ce 48 21 c8 83 e0 20 74 0c <f0> 48 0f ba 32 05 0f 92 c0 0f b6 c0 c3 cc cc c
Nov 19 06:30:11 serveerstertje kernel: [68959.787908] RSP: e02b:ffffc9000161f940 EFLAGS: 00010202
Nov 19 06:30:11 serveerstertje kernel: [68959.801637] RAX: 0000000000000020 RBX: 0000000000000000 RCX: fff0000000000fff
Nov 19 06:30:11 serveerstertje kernel: [68959.815243] RDX: ffff8880083374d0 RSI: fff0000000000fff RDI: ffff888010f41000
Nov 19 06:30:11 serveerstertje kernel: [68959.828683] RBP: ffffc9000161fa70 R08: 000ffffffffff000 R09: 00005654134b5000
Nov 19 06:30:11 serveerstertje kernel: [68959.842026] R10: 000000000000689e R11: 0000000000000000 R12: ffff8880083374d0
Nov 19 06:30:11 serveerstertje kernel: [68959.855214] R13: ffff88807fc1a000 R14: ffff8880083374d0 R15: 0000000000000000
Nov 19 06:30:11 serveerstertje kernel: [68959.868118] FS:  0000000000000000(0000) GS:ffff8880801c0000(0000) knlGS:0000000000000000
Nov 19 06:30:11 serveerstertje kernel: [68959.880689] CS:  e030 DS: 0000 ES: 0000 CR0: 0000000080050033
Nov 19 06:30:11 serveerstertje kernel: [68959.893457] CR2: ffff8880083374d0 CR3: 000000000f33c000 CR4: 0000000000050660
Nov 19 06:30:11 serveerstertje kernel: [68959.906377] Call Trace:
Nov 19 06:30:11 serveerstertje kernel: [68959.919219]  <TASK>
Nov 19 06:30:11 serveerstertje kernel: [68959.931844]  walk_pmd_range_locked.isra.87+0x2e9/0x4e0
Nov 19 06:30:11 serveerstertje kernel: [68959.944840]  walk_pud_range+0x69c/0x980
Nov 19 06:30:11 serveerstertje kernel: [68959.957562]  walk_pgd_range+0xe9/0x810
Nov 19 06:30:11 serveerstertje kernel: [68959.970161]  ? mt_find+0x1f8/0x3c0
Nov 19 06:30:11 serveerstertje kernel: [68959.982808]  __walk_page_range+0x17b/0x180
Nov 19 06:30:11 serveerstertje kernel: [68959.995440]  walk_page_range+0x106/0x170
Nov 19 06:30:11 serveerstertje kernel: [68960.008014]  try_to_inc_max_seq+0x40a/0x9e0
Nov 19 06:30:11 serveerstertje kernel: [68960.020262]  lru_gen_age_node+0x1d3/0x280
Nov 19 06:30:11 serveerstertje kernel: [68960.032222]  ? shrink_node+0x294/0x710
Nov 19 06:30:11 serveerstertje kernel: [68960.044129]  balance_pgdat+0x1c3/0x650
Nov 19 06:30:11 serveerstertje kernel: [68960.055995]  ? prepare_to_wait_event+0x110/0x110
Nov 19 06:30:11 serveerstertje kernel: [68960.068022]  kswapd+0x1f0/0x3a0
Nov 19 06:30:11 serveerstertje kernel: [68960.079997]  ? prepare_to_

Re: Xen-unstable Linux-6.1.0-rc5 BUG: unable to handle page fault for address: ffff8880083374d0

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 1841 bytes --]

On 19.11.22 09:28, Sander Eikelenboom wrote:
> Hi Yu / Juergen,
> 
> This night I got a dom0 kernel crash on my new Ryzen box running Xen-unstable 
> and a Linux-6.1.0-rc5 kernel.
> I did enable the new and shiny MGLRU, could this be related ?

It might be related, but I think it could happen independently from it.

> Nov 19 06:30:11 serveerstertje kernel: [68959.647371] BUG: unable to handle page 
> fault for address: ffff8880083374d0
> Nov 19 06:30:11 serveerstertje kernel: [68959.663555] #PF: supervisor write 
> access in kernel mode
> Nov 19 06:30:11 serveerstertje kernel: [68959.677542] #PF: error_code(0x0003) - 
> permissions violation
> Nov 19 06:30:11 serveerstertje kernel: [68959.691181] PGD 3026067 P4D 3026067 
> PUD 3027067 PMD 7fee5067 PTE 8010000008337065
> Nov 19 06:30:11 serveerstertje kernel: [68959.705084] Oops: 0003 [#1] PREEMPT 
> SMP NOPTI
> Nov 19 06:30:11 serveerstertje kernel: [68959.718710] CPU: 7 PID: 158 Comm: 
> kswapd0 Not tainted 6.1.0-rc5-20221118-doflr-mac80211debug+ #1
> Nov 19 06:30:11 serveerstertje kernel: [68959.732457] Hardware name: To Be 
> Filled By O.E.M. To Be Filled By O.E.M./B450 Pro4 R2.0, BIOS P5.60 10/20/2022
> Nov 19 06:30:11 serveerstertje kernel: [68959.746391] RIP: 
> e030:pmdp_test_and_clear_young+0x25/0x40

The kernel tired to reset the "accessed" bit in the pmd entry.

It does so only since commit eed9a328aa1ae. Before that
pmdp_test_and_clear_young() could be called only for huge pages, which are
disabled in Xen PV guests.

pmdp_test_and_clear_young() does a test_and_clear_bit() of the pmd entry, which
is failing since the hypervisor is emulating pte entry modifications only (pmd
and pud entries can be set via hypercalls only).

Could you please test the attached patch whether it fixes the issue for you?


Juergen


[-- Attachment #1.1.2: 0001-x86-mm-fix-pmdp_test_and_clear_young-for-Xen-PV-gues.patch --]
[-- Type: text/x-patch, Size: 1999 bytes --]

From e89ea813cc09ca7c31af81a87b4856cd3eba3ab9 Mon Sep 17 00:00:00 2001
From: Juergen Gross <jgross@suse.com>
Date: Mon, 21 Nov 2022 07:41:14 +0100
Subject: [PATCH] x86/mm: fix pmdp_test_and_clear_young() for Xen PV guests

When running as a Xen PV guests commit eed9a328aa1a ("mm: x86: add
CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG") can cause a protection violation
in pmdp_test_and_clear_young():

 BUG: unable to handle page fault for address: ffff8880083374d0
 #PF: supervisor write access in kernel mode
 #PF: error_code(0x0003) - permissions violation
 PGD 3026067 P4D 3026067 PUD 3027067 PMD 7fee5067 PTE 8010000008337065
 Oops: 0003 [#1] PREEMPT SMP NOPTI
 CPU: 7 PID: 158 Comm: kswapd0 Not tainted 6.1.0-rc5-20221118-doflr+ #1
 RIP: e030:pmdp_test_and_clear_young+0x25/0x40

This happens because the Xen hypervisor can't emulate direct writes to
page table entries other than PTEs.

In order to fix that do the PMD access bit resetting only when not
running as a Xen PV guest. Note that PUD entries are no issue, as those
won't be written directly by the kernel when running as a Xen PV guest
due to transparent huge pages being disabled in that case.

Fixes: eed9a328aa1a ("mm: x86: add CONFIG_ARCH_HAS_NONLEAF_PMD_YOUNG")
Reported-by: Sander Eikelenboom <linux@eikelenboom.it>
Signed-off-by: Juergen Gross <jgross@suse.com>
---
 arch/x86/mm/pgtable.c | 11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)

diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c
index 8525f2876fb4..076a99e77e28 100644
--- a/arch/x86/mm/pgtable.c
+++ b/arch/x86/mm/pgtable.c
@@ -556,9 +556,14 @@ int pmdp_test_and_clear_young(struct vm_area_struct *vma,
 {
 	int ret = 0;
 
-	if (pmd_young(*pmdp))
-		ret = test_and_clear_bit(_PAGE_BIT_ACCESSED,
-					 (unsigned long *)pmdp);
+	if (pmd_young(*pmdp)) {
+		if (cpu_feature_enabled(X86_FEATURE_XENPV)) {
+			ret = 1;
+		} else {
+			ret = test_and_clear_bit(_PAGE_BIT_ACCESSED,
+						 (unsigned long *)pmdp);
+		}
+	}
 
 	return ret;
 }
-- 
2.35.3


[-- Attachment #1.1.3: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]

Re: Xen-unstable Linux-6.1.0-rc5 BUG: unable to handle page fault for address: ffff8880083374d0

[Yu Zhao]

On Mon, Nov 21, 2022 at 12:10 AM Juergen Gross <jgross@suse.com> wrote:
>
> On 19.11.22 09:28, Sander Eikelenboom wrote:
> > Hi Yu / Juergen,

Hi Sander / Juergen,

Thanks for the report and the analysis.

> > This night I got a dom0 kernel crash on my new Ryzen box running Xen-unstable
> > and a Linux-6.1.0-rc5 kernel.
> > I did enable the new and shiny MGLRU, could this be related ?
>
> It might be related, but I think it could happen independently from it.

Yes, I think it's related.

> > Nov 19 06:30:11 serveerstertje kernel: [68959.647371] BUG: unable to handle page
> > fault for address: ffff8880083374d0
> > Nov 19 06:30:11 serveerstertje kernel: [68959.663555] #PF: supervisor write
> > access in kernel mode
> > Nov 19 06:30:11 serveerstertje kernel: [68959.677542] #PF: error_code(0x0003) -
> > permissions violation
> > Nov 19 06:30:11 serveerstertje kernel: [68959.691181] PGD 3026067 P4D 3026067
> > PUD 3027067 PMD 7fee5067 PTE 8010000008337065
> > Nov 19 06:30:11 serveerstertje kernel: [68959.705084] Oops: 0003 [#1] PREEMPT
> > SMP NOPTI
> > Nov 19 06:30:11 serveerstertje kernel: [68959.718710] CPU: 7 PID: 158 Comm:
> > kswapd0 Not tainted 6.1.0-rc5-20221118-doflr-mac80211debug+ #1
> > Nov 19 06:30:11 serveerstertje kernel: [68959.732457] Hardware name: To Be
> > Filled By O.E.M. To Be Filled By O.E.M./B450 Pro4 R2.0, BIOS P5.60 10/20/2022
> > Nov 19 06:30:11 serveerstertje kernel: [68959.746391] RIP:
> > e030:pmdp_test_and_clear_young+0x25/0x40
>
> The kernel tired to reset the "accessed" bit in the pmd entry.

Correct.

> It does so only since commit eed9a328aa1ae. Before that
> pmdp_test_and_clear_young() could be called only for huge pages, which are
> disabled in Xen PV guests.

Correct. After that commit, we also can clear the accessed bit in
non-leaf PMD entries (pointing to PTE tables).

> pmdp_test_and_clear_young() does a test_and_clear_bit() of the pmd entry, which
> is failing since the hypervisor is emulating pte entry modifications only (pmd
> and pud entries can be set via hypercalls only).
>
> Could you please test the attached patch whether it fixes the issue for you?

There is a runtime kill switch for ARCH_HAS_NONLEAF_PMD_YOUNG, since I
wasn't able to verify this capability on all x86 varieties. The following
should do it:

  # cat /sys/kernel/mm/lru_gen/enabled
  0x0007
  # echo 3 >/sys/kernel/mm/lru_gen/enabled

Details are in Documentation/admin-guide/mm/multigen_lru.rst.

Alternatively, we could make ARCH_HAS_NONLEAF_PMD_YOUNG a runtime
check similar to arch_has_hw_pte_young() on arm64.

Re: Xen-unstable Linux-6.1.0-rc5 BUG: unable to handle page fault for address: ffff8880083374d0

[Juergen Gross]

[-- Attachment #1.1.1: Type: text/plain, Size: 2831 bytes --]

On 21.11.22 09:18, Yu Zhao wrote:
> On Mon, Nov 21, 2022 at 12:10 AM Juergen Gross <jgross@suse.com> wrote:
>>
>> On 19.11.22 09:28, Sander Eikelenboom wrote:
>>> Hi Yu / Juergen,
> 
> Hi Sander / Juergen,
> 
> Thanks for the report and the analysis.
> 
>>> This night I got a dom0 kernel crash on my new Ryzen box running Xen-unstable
>>> and a Linux-6.1.0-rc5 kernel.
>>> I did enable the new and shiny MGLRU, could this be related ?
>>
>> It might be related, but I think it could happen independently from it.
> 
> Yes, I think it's related.
> 
>>> Nov 19 06:30:11 serveerstertje kernel: [68959.647371] BUG: unable to handle page
>>> fault for address: ffff8880083374d0
>>> Nov 19 06:30:11 serveerstertje kernel: [68959.663555] #PF: supervisor write
>>> access in kernel mode
>>> Nov 19 06:30:11 serveerstertje kernel: [68959.677542] #PF: error_code(0x0003) -
>>> permissions violation
>>> Nov 19 06:30:11 serveerstertje kernel: [68959.691181] PGD 3026067 P4D 3026067
>>> PUD 3027067 PMD 7fee5067 PTE 8010000008337065
>>> Nov 19 06:30:11 serveerstertje kernel: [68959.705084] Oops: 0003 [#1] PREEMPT
>>> SMP NOPTI
>>> Nov 19 06:30:11 serveerstertje kernel: [68959.718710] CPU: 7 PID: 158 Comm:
>>> kswapd0 Not tainted 6.1.0-rc5-20221118-doflr-mac80211debug+ #1
>>> Nov 19 06:30:11 serveerstertje kernel: [68959.732457] Hardware name: To Be
>>> Filled By O.E.M. To Be Filled By O.E.M./B450 Pro4 R2.0, BIOS P5.60 10/20/2022
>>> Nov 19 06:30:11 serveerstertje kernel: [68959.746391] RIP:
>>> e030:pmdp_test_and_clear_young+0x25/0x40
>>
>> The kernel tired to reset the "accessed" bit in the pmd entry.
> 
> Correct.
> 
>> It does so only since commit eed9a328aa1ae. Before that
>> pmdp_test_and_clear_young() could be called only for huge pages, which are
>> disabled in Xen PV guests.
> 
> Correct. After that commit, we also can clear the accessed bit in
> non-leaf PMD entries (pointing to PTE tables).
> 
>> pmdp_test_and_clear_young() does a test_and_clear_bit() of the pmd entry, which
>> is failing since the hypervisor is emulating pte entry modifications only (pmd
>> and pud entries can be set via hypercalls only).
>>
>> Could you please test the attached patch whether it fixes the issue for you?
> 
> There is a runtime kill switch for ARCH_HAS_NONLEAF_PMD_YOUNG, since I
> wasn't able to verify this capability on all x86 varieties. The following
> should do it:
> 
>    # cat /sys/kernel/mm/lru_gen/enabled
>    0x0007
>    # echo 3 >/sys/kernel/mm/lru_gen/enabled
> 
> Details are in Documentation/admin-guide/mm/multigen_lru.rst.
> 
> Alternatively, we could make ARCH_HAS_NONLEAF_PMD_YOUNG a runtime
> check similar to arch_has_hw_pte_young() on arm64.

I like this idea.

The patch should be rather trivial. Let me have a try ...


Juergen

[-- Attachment #1.1.2: OpenPGP public key --]
[-- Type: application/pgp-keys, Size: 3149 bytes --]

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 495 bytes --]