Re: [PATCH V2 1/2] kvm tools: Add initial SPAPR PPC64 architecture support

[Matt Evans <matt@ozlabs.org>]

On 15/12/11 12:37, Alexander Graf wrote:
> 
> On 15.12.2011, at 02:27, Matt Evans wrote:
> 
>> Heya Alex,
>>
>> On 13/12/11 19:23, Alexander Graf wrote:
>>>
>>> On 13.12.2011, at 08:00, Matt Evans <matt@ozlabs.org> wrote:
>>>
>>>> This patch adds a new arch directory, powerpc, basic file structure, register
>>>> setup and where necessary stubs out arch-specific functions (e.g. interrupts,
>>>> runloop exits) that later patches will provide.  The target is an
>>>> SPAPR-compliant PPC64 machine (i.e. pSeries); there is no support for PPC32 or
>>>> 'bare metal' PPC64 guests as yet.  Subsequent patches implement the hcalls and
>>>> RTAS required to boot SPAPR pSeries kernels.
>>>>
>>>> Memory is mapped from hugetlbfs (as that is currently required by upstream PPC64
>>>> HV-mode KVM).  The mapping of a VRMA region is yet to be implemented; this is
>>>> only necessary on processors that don't support VRMA, e.g. <= P6.  Work is
>>>> therefore needed to get this going on pre-P7 CPUs.
>>>>
>>>> Processor state is set up as a guest kernel would expect (both primary and
>>>> secondaries), and SMP is fully supported.
>>>>
>>>> Finally, support is added for simply loading flat binary kernels (plus initrd).
>>>> (bzImages are not used on PPC, and this series does not add zImage support or an
>>>> ELF loader.)  The intention is to later support loading firmware such as SLOF.
>>>>
>>>> Signed-off-by: Matt Evans <matt@ozlabs.org>
>>>> ---
>>>> tools/kvm/Makefile                           |   10 +
>>>> tools/kvm/kvm.c                              |    3 +
>>>> tools/kvm/powerpc/include/kvm/barrier.h      |    6 +
>>>> tools/kvm/powerpc/include/kvm/kvm-arch.h     |   72 ++++++++
>>>> tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h |   66 ++++++++
>>>> tools/kvm/powerpc/ioport.c                   |   18 ++
>>>> tools/kvm/powerpc/irq.c                      |   40 +++++
>>>> tools/kvm/powerpc/kvm-cpu.c                  |  233 ++++++++++++++++++++++++++
>>>> tools/kvm/powerpc/kvm.c                      |  187 +++++++++++++++++++++
>>>> 9 files changed, 635 insertions(+), 0 deletions(-)
>>>> create mode 100644 tools/kvm/powerpc/include/kvm/barrier.h
>>>> create mode 100644 tools/kvm/powerpc/include/kvm/kvm-arch.h
>>>> create mode 100644 tools/kvm/powerpc/include/kvm/kvm-cpu-arch.h
>>>> create mode 100644 tools/kvm/powerpc/ioport.c
>>>> create mode 100644 tools/kvm/powerpc/irq.c
>>>> create mode 100644 tools/kvm/powerpc/kvm-cpu.c
>>>> create mode 100644 tools/kvm/powerpc/kvm.c
>>>>
>>>> diff --git a/tools/kvm/Makefile b/tools/kvm/Makefile
>>>> index 2bf70c9..3f1e84a 100644
>>>> --- a/tools/kvm/Makefile
>>>> +++ b/tools/kvm/Makefile
>>>> @@ -124,6 +124,16 @@ ifeq ($(ARCH),x86)
>>>>   OTHEROBJS    += x86/bios/bios-rom.o
>>>>   ARCH_INCLUDE := x86/include
>>>> endif
>>>> +# POWER/ppc:  Actually only support ppc64 currently.
>>>
>>> Why? I usually run ppc32 user land. Doesn't that expose 'ppc' here?
>>
>> Not quite sure what you mean here; do you mean 32bit distro?  (Will still get 'ppc64' from a 64-bit kernel.)
> 
> Eh. Yes. Sorry, my bad.
> 
>> There is clearly some work required here to determine what to build for when we
>> eventually support PPC32 guests/hosts though I'm not sure how that will look
>> yet.  This is designed to break if you build on a 32bit kernel, as if it DID
>> build, it wouldn't run anyway.  (It's building -m64 too...
> 
> Yeah, running -M pseries on PPC32 hosts doesn't make sense really.
> 
>>
>>>> +ifeq ($(uname_M), ppc64)
>>>> +    DEFINES += -DCONFIG_PPC
>>>> +    OBJS    += powerpc/ioport.o
>>>> +    OBJS    += powerpc/irq.o
>>>> +    OBJS    += powerpc/kvm.o
>>>> +    OBJS    += powerpc/kvm-cpu.o
>>>> +    ARCH_INCLUDE := powerpc/include
>>>> +    CFLAGS += -m64
>>
>> ...here.)
>>
>>>> +endif
>>>>
>>>> ###
>>>>
>>>> diff --git a/tools/kvm/kvm.c b/tools/kvm/kvm.c
>>>> index 35ca2c5..3fb46f6 100644
>>>> --- a/tools/kvm/kvm.c
>>>> +++ b/tools/kvm/kvm.c
>>>> @@ -49,6 +49,9 @@ const char *kvm_exit_reasons[] = {
>>>>   DEFINE_KVM_EXIT_REASON(KVM_EXIT_DCR),
>>>>   DEFINE_KVM_EXIT_REASON(KVM_EXIT_NMI),
>>>>   DEFINE_KVM_EXIT_REASON(KVM_EXIT_INTERNAL_ERROR),
>>>> +#ifdef CONFIG_PPC64
>>>> +    DEFINE_KVM_EXIT_REASON(KVM_EXIT_PAPR_HCALL),
>>>> +#endif
>>>> };
>>>>
>>>> extern struct kvm *kvm;
>>>> diff --git a/tools/kvm/powerpc/include/kvm/barrier.h b/tools/kvm/powerpc/include/kvm/barrier.h
>>>> new file mode 100644
>>>> index 0000000..bc7d179
>>>> --- /dev/null
>>>> +++ b/tools/kvm/powerpc/include/kvm/barrier.h
>>>> @@ -0,0 +1,6 @@
>>>> +#ifndef _KVM_BARRIER_H_
>>>> +#define _KVM_BARRIER_H_
>>>> +
>>>> +#include <asm/system.h>
>>>> +
>>>> +#endif /* _KVM_BARRIER_H_ */
>>>> diff --git a/tools/kvm/powerpc/include/kvm/kvm-arch.h b/tools/kvm/powerpc/include/kvm/kvm-arch.h
>>>> new file mode 100644
>>>> index 0000000..da61774
>>>> --- /dev/null
>>>> +++ b/tools/kvm/powerpc/include/kvm/kvm-arch.h
>>>> @@ -0,0 +1,72 @@
>>>> +/*
>>>> + * PPC64 architecture-specific definitions
>>>> + *
>>>> + * Copyright 2011 Matt Evans <matt@ozlabs.org>, IBM Corporation.
>>>> + *
>>>> + * This program is free software; you can redistribute it and/or modify it
>>>> + * under the terms of the GNU General Public License version 2 as published
>>>> + * by the Free Software Foundation.
>>>> + */
>>>> +
>>>> +#ifndef KVM__KVM_ARCH_H
>>>> +#define KVM__KVM_ARCH_H
>>>> +
>>>> +#include <stdbool.h>
>>>> +#include <linux/types.h>
>>>> +#include <time.h>
>>>> +
>>>> +#define KVM_NR_CPUS            (255)
>>>
>>> Why?
>>
>> Good question; that's arbitrary & cut-paste I missed. :-)
>>
>> I'll make this 1024, to match the max sensible NR_CPUS in the PPC64 kernel
>> (which in turn limits KVM_MAX_VCPUS).
> 
> I thought Sasha converted this to a queryable interface?

Yeah, nah; that's (urgh) something different.  The actual number of VCPUs
created is limited by the KVM_CAP_NR_VCPUS/KVM_CAP_MAX_VCPUS stuff.  This
#define is actually only used to size a static array, so on second thoughts I
think I'll just malloc() it instead, i.e. remove this #define.

>>> [snip]
>>>
>>> Have you tried running this on PR KVM? That should also need HIOR synchronization.
>>
>> I have, but only briefly and I admit I built it from a random tree I had lying
>> around, certainly stale, and immediately hit some "can't emulate MMIO" errors on
>> some stdu instructions.  I will give it another go with your tree and see if I
>> can get it working with kvmtool, it would be very cool for that to work.
> 
> Yup, it would also make your work executable to people outside of IBM :)

That's totally not lost on me ;-)  (and :( )


Cheers,


Matt