Re: KVM crash on Jetson TK1

[Jan Kiszka <jan.kiszka@web.de>]

[-- Attachment #1: Type: text/plain, Size: 7635 bytes --]

On 2015-02-07 21:26, Marc Zyngier wrote:
> On Sat, 7 Feb 2015 20:09:14 +0000
> Jan Kiszka <jan.kiszka@web.de> wrote:
> 
> Hi Jan,
> 
>> Hi,
>>
>> using Ian Campbell's PSCI patches for U-Boot (ported to master, see
>> [1]), I managed to enable KVM on the Jetson TK1 board. Unfortunately,
>> I'm getting a crash on the host when starting QEMU:
>>
>> [   59.164674] kernel BUG at ../arch/arm/include/asm/kvm_mmu.h:189!
>> [   59.173139] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
>> [   59.181442] Modules linked in:
>> [   59.186960] CPU: 0 PID: 820 Comm: qemu-system-arm Not tainted
>> 3.19.0-rc7-00050-gdf75905-dirty #6 [   59.198353] Hardware name:
>> NVIDIA Tegra SoC (Flattened Device Tree) [   59.207249] task:
>> ed33b340 ti: ed15e000 task.ti: ed15e000 [   59.215288] PC is at
>> kvm_handle_guest_abort+0x2c0/0x320 [   59.223161] LR is at
>> kvm_handle_guest_abort+0x2c0/0x320 [   59.230996] pc :
>> [<c00175c4>]    lr : [<c00175c4>]    psr: 600f0013 [   59.230996]
>> sp : ed15fe70  ip : 00000000  fp : 00000000 [   59.247782] r10:
>> 00000000  r9 : 00000000  r8 : 000aafa6 [   59.255678] r7 : ed117000
>> r6 : 0000016f  r5 : 00080000  r4 : ed2e5630 [   59.264912] r3 :
>> 00000010  r2 : 0008027c  r1 : eed564c0  r0 : 00000000 [   59.274146]
>> Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
>> [   59.284044] Control: 30c5387d  Table: ad9d0ac0  DAC: fffffffd
>> [   59.292558] Process qemu-system-arm (pid: 820, stack limit =
>> 0xed15e238) [   59.302114] Stack: (0xed15fe70 to 0xed160000)
>> [   59.309340] fe60:                                     00000000
>> ed15fe87 00000000 73800000 [   59.320462] fe80: 0000000f 01010000
>> 00000504 ed2e5630 c0917450 00000007 00010000 00000001 [   59.331630]
>> fea0: edac4000 c0960164 c0917450 c0014710 00000000 00010000 ed91a580
>> 7ffbfeff [   59.342833] fec0: fffffffe fffffffb 00000001 00000000
>> 0000ae80 ed2e5630 ed114040 00000000 [   59.354049] fee0: 00000000
>> ed114040 ed2e5630 00000000 ed15e000 00000000 737fed4c c001063c
>> [   59.365272] ff00: 737fea4c c0042688 00000240 00000000 00000000
>> 00000000 0000000a 00000000 [   59.376493] ff20: edc0de78 00000000
>> ed114040 0000000b 00000000 c0109678 00000001 b749ba00 [   59.387737]
>> ff40: 000000f0 00000000 00000001 00000081 b7032dc0 c009627c 00000002
>> 00000000 [   59.399006] ff60: ffffffff 00000000 00000000 737feb4c
>> ed114040 ed114041 0000000b ed114040 [   59.410271] ff80: 0000ae80
>> 00000000 ed15e000 c01098a0 b6a62eb0 00004254 b749ba00 00000036
>> [   59.421558] ffa0: c001e2e4 c001e160 b6a62eb0 00004254 0000000b
>> 0000ae80 00000000 cc628500 [   59.432884] ffc0: b6a62eb0 00004254
>> b749ba00 00000036 b76baba8 bebc9da8 00000000 737fed4c [   59.444244]
>> ffe0: b6fc33f8 737feb74 b6b3afcc b57f3abc 600f0010 0000000b 00000000
>> 00000000 [   59.455661] [<c00175c4>] (kvm_handle_guest_abort) from
>> [<c0014710>] (kvm_arch_vcpu_ioctl_run+0x160/0x400) [   59.468552]
>> [<c0014710>] (kvm_arch_vcpu_ioctl_run) from [<c001063c>]
>> (kvm_vcpu_ioctl+0x2e4/0x6ec) [   59.480803] [<c001063c>]
>> (kvm_vcpu_ioctl) from [<c0109678>] (do_vfs_ioctl+0x40c/0x600)
>> [   59.492142] [<c0109678>] (do_vfs_ioctl) from [<c01098a0>]
>> (SyS_ioctl+0x34/0x5c) [   59.502909] [<c01098a0>] (SyS_ioctl) from
>> [<c001e160>] (ret_fast_syscall+0x0/0x34) [   59.513944] Code:
>> 0a000002 e1a00008 e1a01009 ebffe2c7 (e7f001f2) 
>>
>> Host kernel is from Linus tree. I can try a different one if there are
>> pending fixes somewhere.
>>
>> Any ideas? Anything I should look at in particular?
> 
> That crash doesn't make much sense. If you look at the BUG_ON you're
> hitting, we check that the size to flush is a multiple of PAGE_SIZE.
> 
> The only two call sites do pass  either PMD_SIZE or PAGE_SIZE, so it is
> hard to imagine how this can trigger.
> 
> What compiler are you using? Can you force the compiler not to inline

Linaro 4.9-2014.09

> this function and see if that makes a difference? I'd be very
> interested in seeing the assembly code generated for this particular
> function.

Here we go:

[   41.973321] kernel BUG at ../arch/arm/include/asm/kvm_mmu.h:189!
[   41.981747] Internal error: Oops - BUG: 0 [#1] PREEMPT SMP ARM
[   41.990011] Modules linked in:
[   41.995498] CPU: 2 PID: 807 Comm: qemu-system-arm Not tainted 3.19.0-rc7-00221-gfd7a168-dirty #8
[   42.006850] Hardware name: NVIDIA Tegra SoC (Flattened Device Tree)
[   42.015707] task: ec354d80 ti: ed31a000 task.ti: ed31a000
[   42.023728] PC is at __coherent_cache_guest_page.constprop.37+0x0/0x4
[   42.032808] LR is at kvm_handle_guest_abort+0x2e0/0x340
[   42.040678] pc : [<c0016bc0>]    lr : [<c00175e8>]    psr: 600f0013
[   42.040678] sp : ed31be68  ip : 00000000  fp : 73800000
[   42.057517] r10: 00000000  r9 : 00000000  r8 : 000f7ede
[   42.065465] r7 : 00000170  r6 : eda3e800  r5 : 00000000  r4 : edb4b630
[   42.074730] r3 : 00000000  r2 : 000f7ede  r1 : ef6f4bc0  r0 : edb4b630
[   42.083989] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment user
[   42.093932] Control: 30c5387d  Table: ad1ae700  DAC: 55555555
[   42.102470] Process qemu-system-arm (pid: 807, stack limit = 0xed31a238)
[   42.112069] Stack: (0xed31be68 to 0xed31c000)
[   42.119326] be60:                   00000000 ed31be87 c0013fa0 00000000 ec390000 ed00b008
[   42.130477] be80: 0000000f 01010000 00000504 edb4b630 c0915450 00000007 00010000 00000001
[   42.141659] bea0: ec390000 c095e164 c0915450 c0014710 00000000 00010000 edb8ec80 7ffbfeff
[   42.152875] bec0: fffffffe fffffffb 00000001 00000000 0000ae80 edb4b630 edaf6b80 00000000
[   42.164115] bee0: 00000000 edaf6b80 edb4b630 00000000 ed31a000 00000000 737fed4c c001063c
[   42.175356] bf00: 737fea4c c0042688 00000240 00000000 00000000 00000000 0000000a 00000000
[   42.186607] bf20: edc0de78 00000000 edaf6b80 0000000b 00000000 c0109724 00000001 b74e1a80
[   42.197866] bf40: 000000f0 00000000 00000001 00000081 b7078dc0 c0096310 00000002 00000000
[   42.209184] bf60: ffffffff 00000000 00000000 737feb4c edaf6b80 edaf6b81 0000000b edaf6b80
[   42.220531] bf80: 0000ae80 00000000 ed31a000 c010994c b6aa8eb0 00004254 b74e1a80 00000036
[   42.231880] bfa0: c001e324 c001e1a0 b6aa8eb0 00004254 0000000b 0000ae80 00000000 622b3f00
[   42.243254] bfc0: b6aa8eb0 00004254 b74e1a80 00000036 b7700c48 bedaad98 00000000 737fed4c
[   42.254662] bfe0: b70093f8 737feb74 b6b80fcc b5839abc 600f0010 0000000b 00000000 00000000
[   42.266123] [<c0016bc0>] (__coherent_cache_guest_page.constprop.37) from [<c00175e8>] (kvm_handle_guest_abort+0x2e0/0x340)
[   42.280587] [<c00175e8>] (kvm_handle_guest_abort) from [<c0014710>] (kvm_arch_vcpu_ioctl_run+0x160/0x400)
[   42.293631] [<c0014710>] (kvm_arch_vcpu_ioctl_run) from [<c001063c>] (kvm_vcpu_ioctl+0x2e4/0x6ec)
[   42.306056] [<c001063c>] (kvm_vcpu_ioctl) from [<c0109724>] (do_vfs_ioctl+0x40c/0x600)
[   42.317561] [<c0109724>] (do_vfs_ioctl) from [<c010994c>] (SyS_ioctl+0x34/0x5c)
[   42.328482] [<c010994c>] (SyS_ioctl) from [<c001e1a0>] (ret_fast_syscall+0x0/0x34)
[   42.339686] Code: e3e0000b e8bd8010 e3a00000 e8bd8010 (e7f001f2) 

And that functions looks, well, simple:

000084c0 <__coherent_cache_guest_page.constprop.37>:
static inline bool vcpu_has_cache_enabled(struct kvm_vcpu *vcpu)
{
	return (vcpu->arch.cp15[c1_SCTLR] & 0b101) == 0b101;
}

static noinline void __coherent_cache_guest_page(struct kvm_vcpu *vcpu, pfn_t pfn,
    84c0:	e7f001f2 	.word	0xe7f001f2

000084c4 <free_boot_hyp_pgd>:
...

Seems the size parameter is constant and fulfills the BUG_ON condition.

Jan


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 181 bytes --]