From: Gavin Shan <gshan@redhat.com>
To: Will Deacon <will@kernel.org>, kvmarm@lists.cs.columbia.edu
Cc: Suzuki Poulose <suzuki.poulose@arm.com>,
Marc Zyngier <maz@kernel.org>,
Quentin Perret <qperret@google.com>,
James Morse <james.morse@arm.com>,
Catalin Marinas <catalin.marinas@arm.com>,
kernel-team@android.com, linux-arm-kernel@lists.infradead.org
Subject: Re: [PATCH v3 00/21] KVM: arm64: Rewrite page-table code and fault handling
Date: Thu, 3 Sep 2020 17:34:02 +1000 [thread overview]
Message-ID: <f7e20f9c-e14e-5314-f78f-1d73743da84c@redhat.com> (raw)
In-Reply-To: <20200825093953.26493-1-will@kernel.org>
Hi Will,
On 8/25/20 7:39 PM, Will Deacon wrote:
> Hello folks,
>
> This is version three of the KVM page-table rework that I previously posted
> here:
>
> v1: https://lore.kernel.org/r/20200730153406.25136-1-will@kernel.org
> v2: https://lore.kernel.org/r/20200818132818.16065-1-will@kernel.org
>
> Changes since v2 include:
>
> * Rebased onto -rc2, which includes the conflicting OOM blocking fixes
> * Dropped the patch trying to "fix" the memcache in kvm_phys_addr_ioremap()
>
It's really nice work, making the code unified/simplified greatly.
However, it seems it doesn't work well with HugeTLBfs. Please refer
to the following test result and see if you have quick idea, or I
can debug it a bit :)
Note: I think the failing cases (FAIL[1] and FAIL[2]) would be
caused by same issue.
Machine Host Guest Result
===============================================================
ThunderX2 VA_BITS: 42 PAGE_SIZE: 4KB Passed
PAGE_SIZE: 64KB 64KB passed
THP: disabled
HugeTLB: disabled
---------------------------------------------------------------
ThunderX2 VA_BITS: 42 PAGE_SIZE: 4KB Passed
PAGE_SIZE: 64KB 64KB passed
THP: enabled
HugeTLB: disabled
----------------------------------------------------------------
ThunderX2 VA_BITS: 42 PAGE_SIZE: 4KB Fail[1]
PAGE_SIZE: 64KB 64KB Fail[1]
THP: disabled
HugeTLB: enabled
---------------------------------------------------------------
ThunderX2 VA_BITS: 39 PAGE_SIZE: 4KB Passed
PAGE_SIZE: 4KB 64KB Passed
THP: disabled
HugeTLB: disabled
---------------------------------------------------------------
ThunderX2 VA_BITS: 39 PAGE_SIZE: 4KB Passed
PAGE_SIZE: 4KB 64KB Passed
THP: enabled
HugeTLB: disabled
--------------------------------------------------------------
ThunderX2 VA_BITS: 39 PAGE_SIZE: 4KB Fail[2]
PAGE_SIZE: 4KB 64KB Fail[2]
THP: disabled
HugeTLB: enabled
NOTE: The commands used to start VM are same for FAIL[1] and
FAIL[2] and the host kernel log are similar. So I don't provide
the kernel log for FAIL[2]. I guess they're caused by same
issue.
Fail[1]
===============================================================
start_vm_aarch64_hugetlbfs() {
echo 16 > /sys/kernel/mm/hugepages/hugepages-524288kB/nr_hugepages
/home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
--enable-kvm -machine virt,gic-version=host \
-cpu host -smp 8,sockets=8,cores=1,threads=1 \
-m 4G -mem-prealloc -mem-path /dev/hugepages \
-monitor none -serial mon:stdio -nographic -s \
-bios /home/gavin/sandbox/qemu.main/pc-bios/edk2-aarch64-code.fd \
-kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
-initrd /home/gavin/sandbox/images/rootfs.cpio.xz \
-append "earlycon=pl011,mmio,0x9000000" \
-device virtio-net-pci,netdev=unet,mac=52:54:00:f1:26:a6 \
-netdev user,id=unet,hostfwd=tcp::50959-:22 \
-drive file=/home/gavin/sandbox/images/vm.img,if=none,format=raw,id=nvme0 \
-device nvme,drive=nvme0,serial=foo \
-drive file=/home/gavin/sandbox/images/vm1.img,if=none,format=raw,id=nvme1 \
-device nvme,drive=nvme1,serial=foo1
}
[ 160.889802] Unable to handle kernel paging request at virtual address 003fffff7fc00034
[ 160.897712] Mem abort info:
[ 160.900507] ESR = 0x96000004
[ 160.903550] EC = 0x25: DABT (current EL), IL = 32 bits
[ 160.908848] SET = 0, FnV = 0
[ 160.911896] EA = 0, S1PTW = 0
[ 160.915024] Data abort info:
[ 160.917891] ISV = 0, ISS = 0x00000004
[ 160.921722] CM = 0, WnR = 0
[ 160.924678] [003fffff7fc00034] address between user and kernel address ranges
[ 160.931808] Internal error: Oops: 96000004 [#1] SMP
[ 160.936676] Modules linked in: xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun bridge stp llc rfkill ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib vfat fat ib_umad rpcrdma sunrpc rdma_ucm ib_iser rdma_cm iw_cm ib_cm libiscsi scsi_transport_iscsi ipmi_ssif qedr ib_uverbs crct10dif_ce i2c_smbus ghash_ce sha2_ce sha256_arm64 ib_core sha1_ce ipmi_devintf ipmi_msghandler thunderx2_pmu ip_tables xfs libcrc32c sg ast i2c_algo_bit drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper qede ttm mpt3sas qed drm raid_class e1000e scsi_transport_sas crc8 gpio_xlp i2c_xlp9xx dm_mirror dm_region_hash dm_log dm_mod
[ 161.007565] CPU: 222 PID: 4559 Comm: qemu-system-aar Not tainted 5.9.0-rc3-gavin+ #4
[ 161.015293] Hardware name: Default string MT91-FS1/MT91-FS1, BIOS 28m 12/14/2019
[ 161.022676] pstate: 60400009 (nZCv daif +PAN -UAO BTYPE=--)
[ 161.028250] pc : __free_pages+0x24/0x60
[ 161.032074] lr : free_pages.part.102+0x2c/0x38
[ 161.036504] sp : fffffe0031b2f8a0
[ 161.039805] x29: fffffe0031b2f8a0 x28: 0000000040000000
[ 161.045104] x27: fffffe0031b2f9c8 x26: 0000000000000007
[ 161.050402] x25: 0000000000000003 x24: fffffe0010f16000
[ 161.055700] x23: 0000000020000000 x22: 0000000040000000
[ 161.060998] x21: 0000000000000002 x20: 0000000060000000
[ 161.066296] x19: fffffc0f1b050010 x18: 0000000000000000
[ 161.071595] x17: 0000000000000000 x16: 0000000000000000
[ 161.076893] x15: 0000000000000000 x14: 0000000000000000
[ 161.082191] x13: 0000000000000000 x12: 0000000000000001
[ 161.087489] x11: 0000000000000003 x10: 0000000000000002
[ 161.092787] x9 : fffffe001035fca4 x8 : 0000000000000007
[ 161.098085] x7 : 00000000fffffff3 x6 : fffffe0010126370
[ 161.103383] x5 : fffffe0031b2f9e8 x4 : 0000040080000000
[ 161.108681] x3 : 003fffff7fc00034 x2 : 00000000ffffffff
[ 161.113979] x1 : 0000000000000000 x0 : 003fffff7fc00000
[ 161.119277] Call trace:
[ 161.121713] __free_pages+0x24/0x60
[ 161.125189] free_pages.part.102+0x2c/0x38
[ 161.129272] free_pages+0x1c/0x28
[ 161.132586] stage2_map_walker+0xbc/0x218
[ 161.136584] __kvm_pgtable_walk+0xec/0x1c8
[ 161.140667] _kvm_pgtable_walk+0xa4/0xe0
[ 161.144578] kvm_pgtable_stage2_map+0xa4/0x118
[ 161.149022] kvm_handle_guest_abort+0x48c/0xa08
[ 161.153543] handle_exit+0x134/0x198
[ 161.157107] kvm_arch_vcpu_ioctl_run+0x4f0/0x880
[ 161.161721] kvm_vcpu_ioctl+0x3a8/0x808
[ 161.165546] __arm64_sys_ioctl+0x1dc/0xcf8
[ 161.169642] do_el0_svc+0xf4/0x1b8
[ 161.173039] el0_sync_handler+0xf8/0x124
[ 161.176949] el0_sync+0x140/0x180
[ 161.180254] Code: d503201f 9100d003 52800022 4b0203e2 (b8e20064)
[ 161.186408] ---[ end trace d0b1b117875f8fcd ]---
[ 161.191012] Kernel panic - not syncing: Fatal exception
[ 161.196247] SMP: stopping secondary CPUs
[ 161.200206] Kernel Offset: 0xc0000 from 0xfffffe0010000000
[ 161.205677] PHYS_OFFSET: 0x80000000
[ 161.209154] CPU features: 0x0046002,22800c38
[ 161.213410] Memory Limit: none
[ 161.216474] ---[ end Kernel panic - not syncing: Fatal exception ]---
FAIL[2]
================================================================
start_vm_aarch64_hugetlbfs() {
echo 4096 > /sys/kernel/mm/hugepages/hugepages-2048kB/nr_hugepages
/home/gavin/sandbox/qemu.main/build/qemu-system-aarch64 \
--enable-kvm -machine virt,gic-version=host \
-cpu host -smp 8,sockets=8,cores=1,threads=1 \
-m 4G -mem-prealloc -mem-path /dev/hugepages \
-monitor none -serial mon:stdio -nographic -s \
-bios /home/gavin/sandbox/qemu.main/pc-bios/edk2-aarch64-code.fd \
-kernel /home/gavin/sandbox/linux.guest/arch/arm64/boot/Image \
-initrd /home/gavin/sandbox/images/rootfs.cpio.xz \
-append "earlycon=pl011,mmio,0x9000000" \
-device virtio-net-pci,netdev=unet,mac=52:54:00:f1:26:a6 \
-netdev user,id=unet,hostfwd=tcp::50959-:22 \
-drive file=/home/gavin/sandbox/images/vm.img,if=none,format=raw,id=nvme0 \
-device nvme,drive=nvme0,serial=foo \
-drive file=/home/gavin/sandbox/images/vm1.img,if=none,format=raw,id=nvme1 \
-device nvme,drive=nvme1,serial=foo1
}
[ 666.278391] Unable to handle kernel paging request at virtual address 03fffffefde00034
[ 666.286304] Mem abort info:
[ 666.289086] ESR = 0x96000004
[ 666.292142] EC = 0x25: DABT (current EL), IL = 32 bits
[ 666.297440] SET = 0, FnV = 0
[ 666.300481] EA = 0, S1PTW = 0
[ 666.303616] Data abort info:
[ 666.306484] ISV = 0, ISS = 0x00000004
[ 666.310306] CM = 0, WnR = 0
[ 666.313269] [03fffffefde00034] address between user and kernel address ranges
[ 666.320393] Internal error: Oops: 96000004 [#1] SMP
[ 666.325259] Modules linked in: xt_CHECKSUM xt_MASQUERADE xt_conntrack ipt_REJECT nf_reject_ipv4 nft_compat nft_counter nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables nfnetlink tun bridge stp llc rfkill ib_isert iscsi_target_mod ib_srpt target_core_mod ib_srp scsi_transport_srp ib_ipoib ib_umad vfat fat rpcrdma sunrpc rdma_ucm ib_iser rdma_cm iw_cm ib_cm libiscsi scsi_transport_iscsi qedr ib_uverbs ipmi_ssif ib_core crct10dif_ce i2c_smbus ghash_ce sha2_ce sha256_arm64 sha1_ce ipmi_devintf ipmi_msghandler thunderx2_pmu ip_tables xfs libcrc32c sg ast i2c_algo_bit drm_vram_helper drm_kms_helper syscopyarea sysfillrect sysimgblt fb_sys_fops drm_ttm_helper qede ttm qed mpt3sas drm e1000e raid_class crc8 scsi_transport_sas gpio_xlp i2c_xlp9xx dm_mirror dm_region_hash dm_log dm_mod
[ 666.396150] CPU: 168 PID: 42112 Comm: qemu-system-aar Not tainted 5.9.0-rc3-gavin+ #5
[ 666.403965] Hardware name: Default string MT91-FS1/MT91-FS1, BIOS 28m 12/14/2019
[ 666.411348] pstate: 60400009 (nZCv daif +PAN -UAO BTYPE=--)
[ 666.416922] pc : __free_pages+0x24/0x60
[ 666.420746] lr : free_pages.part.102+0x2c/0x38
[ 666.425176] sp : ffffffc024a23840
[ 666.428477] x29: ffffffc024a23840 x28: 0000000040000000
[ 666.433776] x27: ffffffc024a239c8 x26: 0000000000000007
[ 666.439074] x25: 0000000000000003 x24: ffffffc010ec0000
[ 666.444373] x23: 0000000000200000 x22: 0000000040200000
[ 666.449671] x21: 0000000040000000 x20: ffffff8f34576000
[ 666.454969] x19: 0000000000000002 x18: 0000000000000000
[ 666.460267] x17: 0000000000000000 x16: 0000000000000000
[ 666.465565] x15: 0000000000000000 x14: 0000000000000000
[ 666.470863] x13: 0000000000000000 x12: 0000000000000001
[ 666.476161] x11: 0000000000000003 x10: 0000000000000002
[ 666.481459] x9 : ffffffc0103522f4 x8 : 0000000000000007
[ 666.486757] x7 : ffffffc0249960f8 x6 : ffffffc0101162f8
[ 666.492055] x5 : ffffffc024a239e8 x4 : 0000000000000004
[ 666.497353] x3 : 03fffffefde00034 x2 : 00000000ffffffff
[ 666.502651] x1 : 0000000000000000 x0 : 03fffffefde00000
[ 666.507950] Call trace:
[ 666.510385] __free_pages+0x24/0x60
[ 666.513861] free_pages.part.102+0x2c/0x38
[ 666.517945] free_pages+0x1c/0x28
[ 666.521260] stage2_map_walker+0xb0/0x208
[ 666.525257] __kvm_pgtable_walk+0xe0/0x1b8
[ 666.529340] __kvm_pgtable_walk+0xb8/0x1b8
[ 666.533424] _kvm_pgtable_walk+0xa4/0xe0
[ 666.537334] kvm_pgtable_stage2_map+0xa0/0x118
[ 666.541779] kvm_handle_guest_abort+0x48c/0xa38
[ 666.546300] handle_exit+0x134/0x198
[ 666.549864] kvm_arch_vcpu_ioctl_run+0x4f0/0x880
[ 666.554479] kvm_vcpu_ioctl+0x3a8/0x808
[ 666.558304] __arm64_sys_ioctl+0x1dc/0xcf8
[ 666.562402] do_el0_svc+0xf4/0x1b8
[ 666.565799] el0_sync_handler+0xf8/0x124
[ 666.569709] el0_sync+0x140/0x180
[ 666.573014] Code: d503201f 9100d003 52800022 4b0203e2 (b8e20064)
[ 666.579197] ---[ end trace 52b60e2f408396b6 ]---
[ 666.583801] Kernel panic - not syncing: Fatal exception
[ 666.589035] SMP: stopping secondary CPUs
[ 666.592996] Kernel Offset: 0xb0000 from 0xffffffc010000000
[ 666.598467] PHYS_OFFSET: 0x80000000
[ 666.601944] CPU features: 0x0046002,22800c38
[ 666.606200] Memory Limit: none
[ 666.609264] ---[ end Kernel panic - not syncing: Fatal exception ]---
Thanks,
Gavin
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2020-09-03 7:35 UTC|newest]
Thread overview: 86+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-08-25 9:39 [PATCH v3 00/21] KVM: arm64: Rewrite page-table code and fault handling Will Deacon
2020-08-25 9:39 ` [PATCH v3 01/21] KVM: arm64: Remove kvm_mmu_free_memory_caches() Will Deacon
2020-08-25 9:39 ` [PATCH v3 02/21] KVM: arm64: Add stand-alone page-table walker infrastructure Will Deacon
2020-08-27 16:27 ` Alexandru Elisei
2020-08-28 15:43 ` Alexandru Elisei
2020-09-02 10:36 ` Will Deacon
2020-08-28 15:51 ` Alexandru Elisei
2020-09-02 10:49 ` Will Deacon
2020-09-02 6:31 ` Gavin Shan
2020-09-02 11:02 ` Will Deacon
2020-09-03 1:11 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 03/21] KVM: arm64: Add support for creating kernel-agnostic stage-1 page tables Will Deacon
2020-08-28 15:35 ` Alexandru Elisei
2020-09-02 10:06 ` Will Deacon
2020-08-25 9:39 ` [PATCH v3 04/21] KVM: arm64: Use generic allocator for hyp stage-1 page-tables Will Deacon
2020-08-28 16:32 ` Alexandru Elisei
2020-09-02 11:35 ` Will Deacon
2020-09-02 14:48 ` Alexandru Elisei
2020-08-25 9:39 ` [PATCH v3 05/21] KVM: arm64: Add support for creating kernel-agnostic stage-2 page tables Will Deacon
2020-09-02 6:40 ` Gavin Shan
2020-09-02 11:30 ` Will Deacon
2020-08-25 9:39 ` [PATCH v3 06/21] KVM: arm64: Add support for stage-2 map()/unmap() in generic page-table Will Deacon
2020-09-01 16:24 ` Alexandru Elisei
2020-09-02 11:46 ` Will Deacon
2020-09-03 2:57 ` Gavin Shan
2020-09-03 5:27 ` Gavin Shan
2020-09-03 11:18 ` Gavin Shan
2020-09-03 12:30 ` Will Deacon
2020-09-03 16:15 ` Will Deacon
2020-09-04 0:47 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 07/21] KVM: arm64: Convert kvm_phys_addr_ioremap() to generic page-table API Will Deacon
2020-09-01 17:08 ` Alexandru Elisei
2020-09-02 11:48 ` Will Deacon
2020-09-03 3:57 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 08/21] KVM: arm64: Convert kvm_set_spte_hva() " Will Deacon
2020-09-02 15:37 ` Alexandru Elisei
2020-09-03 16:37 ` Will Deacon
2020-09-03 4:13 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 09/21] KVM: arm64: Convert unmap_stage2_range() " Will Deacon
2020-09-02 16:23 ` Alexandru Elisei
2020-09-02 18:44 ` Alexandru Elisei
2020-09-03 17:57 ` Will Deacon
2020-09-08 13:07 ` Alexandru Elisei
2020-09-09 10:57 ` Alexandru Elisei
2020-09-03 4:19 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 10/21] KVM: arm64: Add support for stage-2 page-aging in generic page-table Will Deacon
2020-09-03 4:33 ` Gavin Shan
2020-09-03 16:48 ` Will Deacon
2020-09-04 1:01 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 11/21] KVM: arm64: Convert page-aging and access faults to generic page-table API Will Deacon
2020-09-03 4:37 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 12/21] KVM: arm64: Add support for stage-2 write-protect in generic page-table Will Deacon
2020-09-03 4:47 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 13/21] KVM: arm64: Convert write-protect operation to generic page-table API Will Deacon
2020-09-03 4:48 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 14/21] KVM: arm64: Add support for stage-2 cache flushing in generic page-table Will Deacon
2020-09-03 4:51 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 15/21] KVM: arm64: Convert memslot cache-flushing code to generic page-table API Will Deacon
2020-09-03 4:52 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 16/21] KVM: arm64: Add support for relaxing stage-2 perms in generic page-table code Will Deacon
2020-09-03 4:55 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 17/21] KVM: arm64: Convert user_mem_abort() to generic page-table API Will Deacon
2020-09-03 6:05 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 18/21] KVM: arm64: Check the pgt instead of the pgd when modifying page-table Will Deacon
2020-09-03 5:00 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 19/21] KVM: arm64: Remove unused page-table code Will Deacon
2020-09-03 6:02 ` Gavin Shan
2020-08-25 9:39 ` [PATCH v3 20/21] KVM: arm64: Remove unused 'pgd' field from 'struct kvm_s2_mmu' Will Deacon
2020-09-03 5:07 ` Gavin Shan
2020-09-03 16:50 ` Will Deacon
2020-09-04 0:59 ` Gavin Shan
2020-09-04 10:02 ` Marc Zyngier
2020-08-25 9:39 ` [PATCH v3 21/21] KVM: arm64: Don't constrain maximum IPA size based on host configuration Will Deacon
2020-09-03 5:09 ` Gavin Shan
2020-08-27 16:26 ` [PATCH v3 00/21] KVM: arm64: Rewrite page-table code and fault handling Alexandru Elisei
2020-09-01 16:15 ` Will Deacon
2020-09-03 7:34 ` Gavin Shan [this message]
2020-09-03 11:13 ` Gavin Shan
2020-09-03 11:48 ` Gavin Shan
2020-09-03 12:16 ` Will Deacon
2020-09-04 0:51 ` Gavin Shan
2020-09-04 10:07 ` Marc Zyngier
2020-09-05 3:56 ` Gavin Shan
2020-09-05 9:33 ` Marc Zyngier
2020-09-07 9:27 ` Will Deacon
2020-09-03 18:52 ` Will Deacon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=f7e20f9c-e14e-5314-f78f-1d73743da84c@redhat.com \
--to=gshan@redhat.com \
--cc=catalin.marinas@arm.com \
--cc=james.morse@arm.com \
--cc=kernel-team@android.com \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=maz@kernel.org \
--cc=qperret@google.com \
--cc=suzuki.poulose@arm.com \
--cc=will@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).