[Qemu-devel] plans for QEMU support for KVM on ARM

[Qemu-devel] plans for QEMU support for KVM on ARM

[Peter Maydell]

This email is just a quick summary of what we (Linaro) are
planning in the way of QEMU work to support KVM on ARM Cortex-A15.
The idea is to let people know what's coming up, find out if we've
forgotten anything, and avoid people duplicating work unnecessarily.
Most of this is based on a useful session at the recent 'ARM server
mini-summit' in Orlando (UDS/Linaro Connect) at the beginning of
this month.

The work we're currently proposing to do falls into three parts:

 * refactor QEMU's cp15 register handling

At the moment QEMU handles cp15 accesses by calling out to a single
helper function which is an enormous set of nested switch statements
to handle the different coprocessor registers. Access permissions are
checked separately at translate time. This design makes specifying
board-dependent or cpu-dependent registers somewhat painful; it's also
easy for the access permission checks to be out of sync. There is no
support for banked cp15 registers either (needed for trustzone and
virtualisation). We need a better design which lets a board or core
register handler routines for cp15 registers. This will make the code
cleaner and more maintainable as a base for new features.

This isn't strictly a requirement for KVM, but we're going to want
KVM to be able to hand off cp15 accesses to QEMU, and I don't think
that's going to be maintainable or reliable without this refactoring.

(https://blueprints.launchpad.net/qemu-linaro/+spec/cp15-rework)

 * A15 system model

Basically a QEMU model of a Versatile-Express with a Cortex-A15
minus the virtualization and LPAE extensions. This needs the
A15 private peripherals (just the GIC in the right place in
the memory map, really; generic timer not required) and the
new memory map version of the vexpress board model, plus some
new cp15 registers. (Bill Carson has already done some patches
in this area but they need a little rework and may have minor
missing pieces.)

https://blueprints.launchpad.net/qemu-linaro/+spec/initial-a15-system-model

 * miscellaneous integration work

We're aiming for a reasonable working prototype of A15 guest on
an A15 Fast Model host here; we need to fix at least some of
the bugs which currently mean upstream QEMU doesn't work on ARM hosts,
sort out which kernel and qemu trees we are developing from, and
get things running in our validation lab's continuous integration
setup.

https://blueprints.launchpad.net/qemu-linaro/+spec/qemu-kvm-getting-started

Also on the radar is a fourth piece of work:

 * QEMU virtio-mmio support

This is adding support for the 'mmio' virtio transport, which will
allow virtio support in a versatile-express model. We're going to
need this at some point but the current thought is that we want
to do the above listed more important bits of work first...
(The exception would probably be if it turned out that this was
sufficiently useful for making early KVM development easier)

https://blueprints.launchpad.net/qemu-linaro/+spec/add-amba-virtio-support

So, questions:
(1) did we forget something important?
(2) is anybody else already planning to do any of this (or would
like to start)? if so we should coordinate...
(3) is there anything that the kernel folk need/want earlier
rather than later?

thanks
-- PMM

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Alexander Graf]

On 22.11.2011, at 15:44, Peter Maydell <peter.maydell@linaro.org> wrote:

> This email is just a quick summary of what we (Linaro) are
> planning in the way of QEMU work to support KVM on ARM Cortex-A15.
> The idea is to let people know what's coming up, find out if we've
> forgotten anything, and avoid people duplicating work unnecessarily.
> Most of this is based on a useful session at the recent 'ARM server
> mini-summit' in Orlando (UDS/Linaro Connect) at the beginning of
> this month.
> 
> The work we're currently proposing to do falls into three parts:
> 
> * refactor QEMU's cp15 register handling
> 
> At the moment QEMU handles cp15 accesses by calling out to a single
> helper function which is an enormous set of nested switch statements
> to handle the different coprocessor registers. Access permissions are
> checked separately at translate time. This design makes specifying
> board-dependent or cpu-dependent registers somewhat painful; it's also
> easy for the access permission checks to be out of sync. There is no
> support for banked cp15 registers either (needed for trustzone and
> virtualisation). We need a better design which lets a board or core
> register handler routines for cp15 registers. This will make the code
> cleaner and more maintainable as a base for new features.
> 
> This isn't strictly a requirement for KVM, but we're going to want
> KVM to be able to hand off cp15 accesses to QEMU, and I don't think
> that's going to be maintainable or reliable without this refactoring.
> 
> (https://blueprints.launchpad.net/qemu-linaro/+spec/cp15-rework)
> 
> * A15 system model
> 
> Basically a QEMU model of a Versatile-Express with a Cortex-A15
> minus the virtualization and LPAE extensions. This needs the
> A15 private peripherals (just the GIC in the right place in
> the memory map, really; generic timer not required) and the
> new memory map version of the vexpress board model, plus some
> new cp15 registers. (Bill Carson has already done some patches
> in this area but they need a little rework and may have minor
> missing pieces.)
> 
> https://blueprints.launchpad.net/qemu-linaro/+spec/initial-a15-system-model
> 
> * miscellaneous integration work
> 
> We're aiming for a reasonable working prototype of A15 guest on
> an A15 Fast Model host here; we need to fix at least some of
> the bugs which currently mean upstream QEMU doesn't work on ARM hosts,

I thought there was upstream tcg support for arm now? What specifically doesn't work?

Alex

> sort out which kernel and qemu trees we are developing from, and
> get things running in our validation lab's continuous integration
> setup.
> 
> https://blueprints.launchpad.net/qemu-linaro/+spec/qemu-kvm-getting-started
> 
> Also on the radar is a fourth piece of work:
> 
> * QEMU virtio-mmio support
> 
> This is adding support for the 'mmio' virtio transport, which will
> allow virtio support in a versatile-express model. We're going to
> need this at some point but the current thought is that we want
> to do the above listed more important bits of work first...
> (The exception would probably be if it turned out that this was
> sufficiently useful for making early KVM development easier)
> 
> https://blueprints.launchpad.net/qemu-linaro/+spec/add-amba-virtio-support
> 
> So, questions:
> (1) did we forget something important?
> (2) is anybody else already planning to do any of this (or would
> like to start)? if so we should coordinate...
> (3) is there anything that the kernel folk need/want earlier
> rather than later?
> 
> thanks
> -- PMM
> _______________________________________________
> Android-virt mailing list
> Android-virt@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/android-virt

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Peter Maydell]

On 22 November 2011 15:05, Alexander Graf <agraf@suse.de> wrote:
> On 22.11.2011, at 15:44, Peter Maydell <peter.maydell@linaro.org> wrote:
>> We're aiming for a reasonable working prototype of A15 guest on
>> an A15 Fast Model host here; we need to fix at least some of
>> the bugs which currently mean upstream QEMU doesn't work on ARM hosts,
>
> I thought there was upstream tcg support for arm now? What specifically
> doesn't work?

 * coroutine implementation doesn't cope if libc provides a makecontext()
   that returns ENOSYS (so we abort on startup)
 * the MAP_FIXED static codegen buffer is put at an address which overlaps
   the glibc heap so we crash about 50% of the time
 * if you try to start a standard i386 debian image which works fine on
   x86 host, the guest reboots just before displaying the blue-background
   grub image -- clearly something wrong here, not tracked down what yet
 * if the C code was compiled -mthumb, our choice of TCG_AREG0 clashes
   with the frame pointer (trivial fix but could do with a quick benchmark
   run to see whether making AREG0 r6 or r10 (ie thumb lowreg or high)
   has any effect on performance)

Those are just the ones I know about... The coroutine one is the only
one that KVM specifically will require a fix for, but being able to
cross-compare "run with KVM" vs "run with TCG" will be handy for
debugging I suspect.

ARM host support is documented as "known not to work" in the 1.0
release notes (see earlier mail to qemu-devel).

-- PMM

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Paolo Bonzini]

On 11/22/2011 04:19 PM, Peter Maydell wrote:
>   * coroutine implementation doesn't cope if libc provides a makecontext()
>     that returns ENOSYS (so we abort on startup)

I have a patch that replaces makecontext() with sigaltstack(), i.e. 
going from nice code that does not work to hideous code that works.

I had not submitted it yet because it does not help for OpenBSD so we 
would need anyway coroutine-gthread.c, and because I would like to test 
it more thoroughly on non-Linux hosts.  If anyone can help with the 
testing, it is at git://github.com/bonzini/qemu.git branch coroutine-fix.

>  * the MAP_FIXED static codegen buffer is put at an address which overlaps
>    the glibc heap so we crash about 50% of the time

For 1.1 we can just use USE_STATIC_CODE_GEN_BUFFER.  It should not have 
major problems on Linux, since it is already tested for linux-user. 
That leaves out Windows of course.

Paolo

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Peter Maydell]

On 23 November 2011 08:45, Paolo Bonzini <pbonzini@redhat.com> wrote:
> On 11/22/2011 04:19 PM, Peter Maydell wrote:
>>  * the MAP_FIXED static codegen buffer is put at an address which overlaps
>>   the glibc heap so we crash about 50% of the time
>
> For 1.1 we can just use USE_STATIC_CODE_GEN_BUFFER.  It should not have
> major problems on Linux, since it is already tested for linux-user. That
> leaves out Windows of course.

This raises the question of why we don't use the static buffer everywhere.
Presumably there was a disadvantage that caused us to move to the dynamic
mmap for system mode...

(I don't actually think it should be very hard to make the ARM code
cope with the idea that jumps into host code might be long range.)

-- PMM

Re: [Qemu-devel] plans for QEMU support for KVM on ARM

[Jubi Taneja]

[-- Attachment #1: Type: text/plain, Size: 3758 bytes --]

Thanks for sharing this information. When is QEMU support for ARM Cortex
A15 (with LPAE) expected?

Jubi

On Tue, Nov 22, 2011 at 8:14 PM, Peter Maydell <peter.maydell@linaro.org>wrote:

> This email is just a quick summary of what we (Linaro) are
> planning in the way of QEMU work to support KVM on ARM Cortex-A15.
> The idea is to let people know what's coming up, find out if we've
> forgotten anything, and avoid people duplicating work unnecessarily.
> Most of this is based on a useful session at the recent 'ARM server
> mini-summit' in Orlando (UDS/Linaro Connect) at the beginning of
> this month.
>
> The work we're currently proposing to do falls into three parts:
>
>  * refactor QEMU's cp15 register handling
>
> At the moment QEMU handles cp15 accesses by calling out to a single
> helper function which is an enormous set of nested switch statements
> to handle the different coprocessor registers. Access permissions are
> checked separately at translate time. This design makes specifying
> board-dependent or cpu-dependent registers somewhat painful; it's also
> easy for the access permission checks to be out of sync. There is no
> support for banked cp15 registers either (needed for trustzone and
> virtualisation). We need a better design which lets a board or core
> register handler routines for cp15 registers. This will make the code
> cleaner and more maintainable as a base for new features.
>
> This isn't strictly a requirement for KVM, but we're going to want
> KVM to be able to hand off cp15 accesses to QEMU, and I don't think
> that's going to be maintainable or reliable without this refactoring.
>
> (https://blueprints.launchpad.net/qemu-linaro/+spec/cp15-rework)
>
>  * A15 system model
>
> Basically a QEMU model of a Versatile-Express with a Cortex-A15
> minus the virtualization and LPAE extensions. This needs the
> A15 private peripherals (just the GIC in the right place in
> the memory map, really; generic timer not required) and the
> new memory map version of the vexpress board model, plus some
> new cp15 registers. (Bill Carson has already done some patches
> in this area but they need a little rework and may have minor
> missing pieces.)
>
> https://blueprints.launchpad.net/qemu-linaro/+spec/initial-a15-system-model
>
>  * miscellaneous integration work
>
> We're aiming for a reasonable working prototype of A15 guest on
> an A15 Fast Model host here; we need to fix at least some of
> the bugs which currently mean upstream QEMU doesn't work on ARM hosts,
> sort out which kernel and qemu trees we are developing from, and
> get things running in our validation lab's continuous integration
> setup.
>
> https://blueprints.launchpad.net/qemu-linaro/+spec/qemu-kvm-getting-started
>
> Also on the radar is a fourth piece of work:
>
>  * QEMU virtio-mmio support
>
> This is adding support for the 'mmio' virtio transport, which will
> allow virtio support in a versatile-express model. We're going to
> need this at some point but the current thought is that we want
> to do the above listed more important bits of work first...
> (The exception would probably be if it turned out that this was
> sufficiently useful for making early KVM development easier)
>
> https://blueprints.launchpad.net/qemu-linaro/+spec/add-amba-virtio-support
>
> So, questions:
> (1) did we forget something important?
> (2) is anybody else already planning to do any of this (or would
> like to start)? if so we should coordinate...
> (3) is there anything that the kernel folk need/want earlier
> rather than later?
>
> thanks
> -- PMM
>
> _______________________________________________
> linaro-toolchain mailing list
> linaro-toolchain@lists.linaro.org
> http://lists.linaro.org/mailman/listinfo/linaro-toolchain
>

[-- Attachment #2: Type: text/html, Size: 4907 bytes --]

Re: [Qemu-devel] plans for QEMU support for KVM on ARM

[Peter Maydell]

On 23 November 2011 07:50, Jubi Taneja <jubitaneja@gmail.com> wrote:
> Thanks for sharing this information. When is QEMU support for ARM Cortex A15
> (with LPAE) expected?

Depends what you mean. LPAE is on the list of bits we currently
hope we won't have to implement for KVM's benefit, but which we
might have to. (Basically if we start with a guest kernel not
compiled for LPAE then there's no need.) So adding LPAE on top
of the "A15-without-virtualization" QEMU system model isn't in
this planned work. If we need it it'll probably be done some time
in the first quarter of next year.

A full QEMU system emulation including the virtualization extensions
(ie what you would need to run a hypervisor inside QEMU) is not
currently planned.

-- PMM

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Christoffer Dall]

>
> This isn't strictly a requirement for KVM, but we're going to want
> KVM to be able to hand off cp15 accesses to QEMU, and I don't think
> that's going to be maintainable or reliable without this refactoring.

Why do we need KVM to hand off cp15 accesses to QEMU? As of now almost
all of this is supported in-kernel. Do the CP15 behavior vary that
much according to the board config? I would think that other
co-processor accesses would actually be more important if they're used
as interfaces for DSPs etc. But in that case, yes, we need a way to
handle them at least.

>
> [snip]
>
> Also on the radar is a fourth piece of work:
>
>  * QEMU virtio-mmio support
>
> This is adding support for the 'mmio' virtio transport, which will
> allow virtio support in a versatile-express model. We're going to
> need this at some point but the current thought is that we want
> to do the above listed more important bits of work first...
> (The exception would probably be if it turned out that this was
> sufficiently useful for making early KVM development easier)

I don't really see why it would be. Is this not merely a question of
performance?

>
> So, questions:
> (1) did we forget something important?
> (2) is anybody else already planning to do any of this (or would
> like to start)? if so we should coordinate...
> (3) is there anything that the kernel folk need/want earlier
> rather than later?
>
Thanks for sending this out.

I would like some clarity (if possible) of which branch the KVM
support for ARM should be based on. Is it the Linaro version of QEMU
and basically just keep rebasing the changes off there until someone
accepts them or?

What would be helpful from the point of view of KVM is to have basic
ARM host support and the A15 system model upstream.

Currently there are a number of changes to the configure script to
make things work and we link against a prebuilt zlib library that we
keep distributing to people who wish to build QEMU for KVM/ARM. I am
not an expert on QEMU and would definitely like some help on doing
this the right way. What's the recommended procedure? Should I send
out the hack-ish patch we have now, and if so, what should be the base
of such a patch.

Best,
Christoffer

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Peter Maydell]

On 24 November 2011 20:11, Christoffer Dall <cdall@cs.columbia.edu> wrote:
>> This isn't strictly a requirement for KVM, but we're going to want
>> KVM to be able to hand off cp15 accesses to QEMU, and I don't think
>> that's going to be maintainable or reliable without this refactoring.
>
> Why do we need KVM to hand off cp15 accesses to QEMU? As of now almost
> all of this is supported in-kernel. Do the CP15 behavior vary that
> much according to the board config? I would think that other
> co-processor accesses would actually be more important if they're used
> as interfaces for DSPs etc. But in that case, yes, we need a way to
> handle them at least.

I think it was mostly the generic timer that inclined me that way:
it's a device that just happens to be hung off the cp15 interface
rather than being memory mapped.

My assumption was that initial default would be "hand everything
off to qemu" with the kernel then providing implementations of
things like timers for performance reasons later.

Also, there are lots of cp15 registers, and they vary between
implementations, so (a) it would be useful to share the emulation
implementation between qemu and kvm somehow and (b) are you really
going to implement cp15 emulation for half a dozen CPU implementations
in the kernel?

If it doesn't make sense to hand off cp15 accesses that's fine,
though -- I want to do this refactoring for the A15 system mode
implementation in qemu anyway, because I really don't think we
should try to shoehorn in yet another cpu implementation into the
current set of switch statements...

It looks (from a brief glance at the code) like ppc kvm does
handoff-to-qemu with the DCRs, incidentally.

>> Also on the radar is a fourth piece of work:
>>
>>  * QEMU virtio-mmio support
>>
>> This is adding support for the 'mmio' virtio transport, which will
>> allow virtio support in a versatile-express model. We're going to
>> need this at some point but the current thought is that we want
>> to do the above listed more important bits of work first...
>> (The exception would probably be if it turned out that this was
>> sufficiently useful for making early KVM development easier)
>
> I don't really see why it would be. Is this not merely a question of
> performance?

Yes; Marc Z was complaining at me earlier this week about SD
card emulated performance being particularly dire, though :-)
So this is kind of down here as a "if anybody wants this sooner
than some-time-late-next-Q1 now is a good time to scream" marker...

> I would like some clarity (if possible) of which branch the KVM
> support for ARM should be based on. Is it the Linaro version of QEMU
> and basically just keep rebasing the changes off there until someone
> accepts them or?
>
> What would be helpful from the point of view of KVM is to have basic
> ARM host support and the A15 system model upstream.

My plan here was that the bits like A15 system model which are
clearly upstreamable I would push directly upstream (and just keep
in qemu-linaro for the typically week-or-three things take to
go through upstream patch review). For the KVM support patch itself,
my thought was simply to carry that in qemu-linaro as an "unsupported
work in progress" patch until we think it's ready to commit upstream.
[I'm suggesting qemu-linaro here mostly because it's convenient to
me: it's a tree I already maintain and track upstream trunk with.
If that would be awkward for other people we can do something else.]

Pretty high up my todo list was rebasing your kvm patch on to
master / qemu-linaro (the two are more or less the same for this
purpose).

> Currently there are a number of changes to the configure script to
> make things work and we link against a prebuilt zlib library that we
> keep distributing to people who wish to build QEMU for KVM/ARM.

I have some instructions for Ubuntu oneiric hosts that work by
using the stock oneiric ARM zlib (installed via dpkg-cross):
see the section at the bottom of
https://wiki.linaro.org/PeterMaydell/A15OnFastModels
(that's "how to cross build upstream qemu master", not your qemu
with the kvm patch.)

You'll find that when we update to QEMU 1.0 you'll also need a
cross version of the glib/gthread libraries, which may make you
more reluctant to stick to the "hand out prebuilt cross library"
approach.

-- PMM

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Christoffer Dall]

On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 24 November 2011 20:11, Christoffer Dall <cdall@cs.columbia.edu> wrote:
>>> This isn't strictly a requirement for KVM, but we're going to want
>>> KVM to be able to hand off cp15 accesses to QEMU, and I don't think
>>> that's going to be maintainable or reliable without this refactoring.
>>
>> Why do we need KVM to hand off cp15 accesses to QEMU? As of now almost
>> all of this is supported in-kernel. Do the CP15 behavior vary that
>> much according to the board config? I would think that other
>> co-processor accesses would actually be more important if they're used
>> as interfaces for DSPs etc. But in that case, yes, we need a way to
>> handle them at least.
>
> I think it was mostly the generic timer that inclined me that way:
> it's a device that just happens to be hung off the cp15 interface
> rather than being memory mapped.
>
> My assumption was that initial default would be "hand everything
> off to qemu" with the kernel then providing implementations of
> things like timers for performance reasons later.

hmm yeah, that could have been an approach. What happened was that
before a15, I had to take care of stuff like cache flushes inside the
kernel, so it wasn't really an option anyway.

>
> Also, there are lots of cp15 registers, and they vary between
> implementations, so (a) it would be useful to share the emulation
> implementation between qemu and kvm somehow and (b) are you really
> going to implement cp15 emulation for half a dozen CPU implementations
> in the kernel?

regarding (a), yes most certainly and (b), not if I can avoid it :)

>
> If it doesn't make sense to hand off cp15 accesses that's fine,
> though -- I want to do this refactoring for the A15 system mode
> implementation in qemu anyway, because I really don't think we
> should try to shoehorn in yet another cpu implementation into the
> current set of switch statements...
>
> It looks (from a brief glance at the code) like ppc kvm does
> handoff-to-qemu with the DCRs, incidentally.
>

for the record, I wasn't questioning that the refactoring was a good
idea, I'm sure it is.

>>> Also on the radar is a fourth piece of work:
>>>
>>>  * QEMU virtio-mmio support
>>>
>>> This is adding support for the 'mmio' virtio transport, which will
>>> allow virtio support in a versatile-express model. We're going to
>>> need this at some point but the current thought is that we want
>>> to do the above listed more important bits of work first...
>>> (The exception would probably be if it turned out that this was
>>> sufficiently useful for making early KVM development easier)
>>
>> I don't really see why it would be. Is this not merely a question of
>> performance?
>
> Yes; Marc Z was complaining at me earlier this week about SD
> card emulated performance being particularly dire, though :-)
> So this is kind of down here as a "if anybody wants this sooner
> than some-time-late-next-Q1 now is a good time to scream" marker...
>

ok. I didn't do much performance tuning yet overall, but I do see
virtio and generic timers and being pretty high on the list once we
get there, so yes, let's start thinking about it.

>> I would like some clarity (if possible) of which branch the KVM
>> support for ARM should be based on. Is it the Linaro version of QEMU
>> and basically just keep rebasing the changes off there until someone
>> accepts them or?
>>
>> What would be helpful from the point of view of KVM is to have basic
>> ARM host support and the A15 system model upstream.
>
> My plan here was that the bits like A15 system model which are
> clearly upstreamable I would push directly upstream (and just keep
> in qemu-linaro for the typically week-or-three things take to
> go through upstream patch review). For the KVM support patch itself,
> my thought was simply to carry that in qemu-linaro as an "unsupported
> work in progress" patch until we think it's ready to commit upstream.
> [I'm suggesting qemu-linaro here mostly because it's convenient to
> me: it's a tree I already maintain and track upstream trunk with.
> If that would be awkward for other people we can do something else.]
>
> Pretty high up my todo list was rebasing your kvm patch on to
> master / qemu-linaro (the two are more or less the same for this
> purpose).
>

if you could take charge on that it would be awesome from my point of
view. I will send you a (slightly) cleaned up patch that you can use
or throw away.

>> Currently there are a number of changes to the configure script to
>> make things work and we link against a prebuilt zlib library that we
>> keep distributing to people who wish to build QEMU for KVM/ARM.
>
> I have some instructions for Ubuntu oneiric hosts that work by
> using the stock oneiric ARM zlib (installed via dpkg-cross):
> see the section at the bottom of
> https://wiki.linaro.org/PeterMaydell/A15OnFastModels
> (that's "how to cross build upstream qemu master", not your qemu
> with the kvm patch.)
>
> You'll find that when we update to QEMU 1.0 you'll also need a
> cross version of the glib/gthread libraries, which may make you
> more reluctant to stick to the "hand out prebuilt cross library"
> approach.

I'm already reluctant, so I'm happy to hear there will be some
"official" instructions available. I actually think it's important for
the adoption of KVM that things are somewhat possible to build without
too much jumping through hoops. A remaining item is to test this setup
with the KVM stuff, but it should be a minor thing as long as the
kernel headers for KVM/ARM can be integrated with the build process
somehow. (Or is there an alternative?).

-Christoffer

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Peter Maydell]

On 24 November 2011 22:02, Christoffer Dall <cdall@cs.columbia.edu> wrote:
> On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
>> Pretty high up my todo list was rebasing your kvm patch on to
>> master / qemu-linaro (the two are more or less the same for this
>> purpose).
>
> if you could take charge on that it would be awesome from my point of
> view. I will send you a (slightly) cleaned up patch that you can use
> or throw away.

That would be good, thanks. I'll try to get that rebasing started
tomorrow and done early next week.

> I'm already reluctant, so I'm happy to hear there will be some
> "official" instructions available. I actually think it's important for
> the adoption of KVM that things are somewhat possible to build without
> too much jumping through hoops.

Well, once we've got real hardware it'll be more straightforward
because building QEMU on the hardware won't be quite so slow...
Most of this is just because crosscompiling is and remains painful.
(Alas, you can't compile QEMU in a QEMU arm-linux-user chroot,
because gcc segfaults. I blame our mmap emulation layer.)

> A remaining item is to test this setup
> with the KVM stuff, but it should be a minor thing as long as the
> kernel headers for KVM/ARM can be integrated with the build process
> somehow. (Or is there an alternative?).

Ah, kernel headers, good point. QEMU now carries the KVM kernel
headers in its own git tree (this has changed since the QEMU version
you based your original kvm patches on, I think). So we'll have
to carry the ARM header changes in qemu-linaro too (I think).
I guess that should be OK, presumably we won't be revising the
kernel-userspace ABI at a rate of knots. (when the headers get
upstream in the kernel upstream qemu can update to get them from
there and we can drop the temporary copies in qemu-linaro).
[that's kind of off-the-top-of-my-head, yell if it sounds lunatic.]

-- PMM

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Alexander Graf]

On 25.11.2011, at 00:06, Peter Maydell wrote:

> On 24 November 2011 22:02, Christoffer Dall <cdall@cs.columbia.edu> wrote:
>> On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
>>> Pretty high up my todo list was rebasing your kvm patch on to
>>> master / qemu-linaro (the two are more or less the same for this
>>> purpose).
>> 
>> if you could take charge on that it would be awesome from my point of
>> view. I will send you a (slightly) cleaned up patch that you can use
>> or throw away.
> 
> That would be good, thanks. I'll try to get that rebasing started
> tomorrow and done early next week.
> 
>> I'm already reluctant, so I'm happy to hear there will be some
>> "official" instructions available. I actually think it's important for
>> the adoption of KVM that things are somewhat possible to build without
>> too much jumping through hoops.
> 
> Well, once we've got real hardware it'll be more straightforward
> because building QEMU on the hardware won't be quite so slow...
> Most of this is just because crosscompiling is and remains painful.
> (Alas, you can't compile QEMU in a QEMU arm-linux-user chroot,
> because gcc segfaults. I blame our mmap emulation layer.)

Just throw in MAP_32BIT in all mmaps and it should work like a charm :).

> 
>> A remaining item is to test this setup
>> with the KVM stuff, but it should be a minor thing as long as the
>> kernel headers for KVM/ARM can be integrated with the build process
>> somehow. (Or is there an alternative?).
> 
> Ah, kernel headers, good point. QEMU now carries the KVM kernel
> headers in its own git tree (this has changed since the QEMU version
> you based your original kvm patches on, I think). So we'll have
> to carry the ARM header changes in qemu-linaro too (I think).
> I guess that should be OK, presumably we won't be revising the
> kernel-userspace ABI at a rate of knots. (when the headers get
> upstream in the kernel upstream qemu can update to get them from
> there and we can drop the temporary copies in qemu-linaro).
> [that's kind of off-the-top-of-my-head, yell if it sounds lunatic.]

I fully agree. Treat the linaro tree as the temporary straighten-out-the-ABI-tree and then move to upstream for further development :)


Alex

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Peter Maydell]

On 24 November 2011 23:10, Alexander Graf <agraf@suse.de> wrote:
> On 25.11.2011, at 00:06, Peter Maydell wrote:
>> Well, once we've got real hardware it'll be more straightforward
>> because building QEMU on the hardware won't be quite so slow...
>> Most of this is just because crosscompiling is and remains painful.
>> (Alas, you can't compile QEMU in a QEMU arm-linux-user chroot,
>> because gcc segfaults. I blame our mmap emulation layer.)
>
> Just throw in MAP_32BIT in all mmaps and it should work like a charm :)

Want to submit a patch that does that if the host is x86-64?
I think I'd rather pragmatically do that and fix the issues
in the actually common case rather than hold out for a larger
scale reimplementation of the mmap layer which we aren't in
practice going to do...

-- PMM

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Alexander Graf]

On 25.11.2011, at 00:11, Peter Maydell wrote:

> On 24 November 2011 23:10, Alexander Graf <agraf@suse.de> wrote:
>> On 25.11.2011, at 00:06, Peter Maydell wrote:
>>> Well, once we've got real hardware it'll be more straightforward
>>> because building QEMU on the hardware won't be quite so slow...
>>> Most of this is just because crosscompiling is and remains painful.
>>> (Alas, you can't compile QEMU in a QEMU arm-linux-user chroot,
>>> because gcc segfaults. I blame our mmap emulation layer.)
>> 
>> Just throw in MAP_32BIT in all mmaps and it should work like a charm :)
> 
> Want to submit a patch that does that if the host is x86-64?
> I think I'd rather pragmatically do that and fix the issues
> in the actually common case rather than hold out for a larger
> scale reimplementation of the mmap layer which we aren't in
> practice going to do...

Happily, yes :).


Alex

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Christoffer Dall]

[snip]

>> A remaining item is to test this setup
>> with the KVM stuff, but it should be a minor thing as long as the
>> kernel headers for KVM/ARM can be integrated with the build process
>> somehow. (Or is there an alternative?).
>
> Ah, kernel headers, good point. QEMU now carries the KVM kernel
> headers in its own git tree (this has changed since the QEMU version
> you based your original kvm patches on, I think). So we'll have
> to carry the ARM header changes in qemu-linaro too (I think).
> I guess that should be OK, presumably we won't be revising the
> kernel-userspace ABI at a rate of knots. (when the headers get
> upstream in the kernel upstream qemu can update to get them from
> there and we can drop the temporary copies in qemu-linaro).
> [that's kind of off-the-top-of-my-head, yell if it sounds lunatic.]
>

sounds good to me.

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Peter Maydell]

On 24 November 2011 23:06, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 24 November 2011 22:02, Christoffer Dall <cdall@cs.columbia.edu> wrote:
>> On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
>>> Pretty high up my todo list was rebasing your kvm patch on to
>>> master / qemu-linaro (the two are more or less the same for this
>>> purpose).
>>
>> if you could take charge on that it would be awesome from my point of
>> view. I will send you a (slightly) cleaned up patch that you can use
>> or throw away.
>
> That would be good, thanks. I'll try to get that rebasing started
> tomorrow and done early next week.

I've now done this rebasing, fixed a few other minor issues and
put the resulting patches into qemu-linaro as "experimental work
in progress":

http://git.linaro.org/gitweb?p=qemu/qemu-linaro.git;a=summary

Note that I've defaulted KVM to off for ARM hosts (for the benefit
of other users of qemu-linaro who don't care about KVM yet), so run
configure with "--enable-kvm".

A note about the branches here: the 'rebasing' branch is the patch
stack, and (as the name suggests) it rebases on qemu upstream
fairly frequently. 'master' is a never-rebases branch which always
has identical contents to the head of 'rebasing' (generated by
stg publish). Pick whichever branch suits your use case best.

Instructions on how to cross-compile it are here:
https://wiki.linaro.org/PeterMaydell/A15OnFastModels
(if you're moving from Christoffer's qemu tree these are worth
checking, because QEMU now requires glib/gthread so a cross
compile setup that could compile Christoffer's tree won't
necessarily build this one.)

-- PMM

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Christoffer Dall]

On Tue, Nov 29, 2011 at 1:16 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
> On 24 November 2011 23:06, Peter Maydell <peter.maydell@linaro.org> wrote:
>> On 24 November 2011 22:02, Christoffer Dall <cdall@cs.columbia.edu> wrote:
>>> On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
>>>> Pretty high up my todo list was rebasing your kvm patch on to
>>>> master / qemu-linaro (the two are more or less the same for this
>>>> purpose).
>>>
>>> if you could take charge on that it would be awesome from my point of
>>> view. I will send you a (slightly) cleaned up patch that you can use
>>> or throw away.
>>
>> That would be good, thanks. I'll try to get that rebasing started
>> tomorrow and done early next week.
>
> I've now done this rebasing, fixed a few other minor issues and
> put the resulting patches into qemu-linaro as "experimental work
> in progress":
>
> http://git.linaro.org/gitweb?p=qemu/qemu-linaro.git;a=summary
>
> Note that I've defaulted KVM to off for ARM hosts (for the benefit
> of other users of qemu-linaro who don't care about KVM yet), so run
> configure with "--enable-kvm".
>
> A note about the branches here: the 'rebasing' branch is the patch
> stack, and (as the name suggests) it rebases on qemu upstream
> fairly frequently. 'master' is a never-rebases branch which always
> has identical contents to the head of 'rebasing' (generated by
> stg publish). Pick whichever branch suits your use case best.
>
> Instructions on how to cross-compile it are here:
> https://wiki.linaro.org/PeterMaydell/A15OnFastModels
> (if you're moving from Christoffer's qemu tree these are worth
> checking, because QEMU now requires glib/gthread so a cross
> compile setup that could compile Christoffer's tree won't
> necessarily build this one.)


this is great, thanks!

looking forward to trying it out later this week.

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Alexander Graf]

[-- Attachment #1: Type: text/plain, Size: 3245 bytes --]


On 24.11.2011, at 23:02, Christoffer Dall wrote:

> On Thu, Nov 24, 2011 at 4:27 PM, Peter Maydell <peter.maydell@linaro.org> wrote:
>> On 24 November 2011 20:11, Christoffer Dall <cdall@cs.columbia.edu> wrote:
> 

[...]

>>> I would like some clarity (if possible) of which branch the KVM
>>> support for ARM should be based on. Is it the Linaro version of QEMU
>>> and basically just keep rebasing the changes off there until someone
>>> accepts them or?
>>> 
>>> What would be helpful from the point of view of KVM is to have basic
>>> ARM host support and the A15 system model upstream.
>> 
>> My plan here was that the bits like A15 system model which are
>> clearly upstreamable I would push directly upstream (and just keep
>> in qemu-linaro for the typically week-or-three things take to
>> go through upstream patch review). For the KVM support patch itself,
>> my thought was simply to carry that in qemu-linaro as an "unsupported
>> work in progress" patch until we think it's ready to commit upstream.
>> [I'm suggesting qemu-linaro here mostly because it's convenient to
>> me: it's a tree I already maintain and track upstream trunk with.
>> If that would be awkward for other people we can do something else.]
>> 
>> Pretty high up my todo list was rebasing your kvm patch on to
>> master / qemu-linaro (the two are more or less the same for this
>> purpose).
>> 
> 
> if you could take charge on that it would be awesome from my point of
> view. I will send you a (slightly) cleaned up patch that you can use
> or throw away.

Please make sure that once there is upstream code in the kernel to actually implement A15 KVM, there should also be the patches ready for QEMU upstream inclusion, as that's the point where the ABI is stable :).

> 
>>> Currently there are a number of changes to the configure script to
>>> make things work and we link against a prebuilt zlib library that we
>>> keep distributing to people who wish to build QEMU for KVM/ARM.
>> 
>> I have some instructions for Ubuntu oneiric hosts that work by
>> using the stock oneiric ARM zlib (installed via dpkg-cross):
>> see the section at the bottom of
>> https://wiki.linaro.org/PeterMaydell/A15OnFastModels
>> (that's "how to cross build upstream qemu master", not your qemu
>> with the kvm patch.)
>> 
>> You'll find that when we update to QEMU 1.0 you'll also need a
>> cross version of the glib/gthread libraries, which may make you
>> more reluctant to stick to the "hand out prebuilt cross library"
>> approach.
> 
> I'm already reluctant, so I'm happy to hear there will be some
> "official" instructions available. I actually think it's important for
> the adoption of KVM that things are somewhat possible to build without
> too much jumping through hoops. A remaining item is to test this setup
> with the KVM stuff, but it should be a minor thing as long as the
> kernel headers for KVM/ARM can be integrated with the build process
> somehow. (Or is there an alternative?).

The kernel headers now all reside inside the QEMU tree, so you just sync them up to Linux's headers using a script and then have them around. That way building against KVM should be trivial :).


Alex


[-- Attachment #2: Type: text/html, Size: 5640 bytes --]

Re: [Qemu-devel] [Android-virt] plans for QEMU support for KVM on ARM

[Scott Wood]

On 11/24/2011 03:27 PM, Peter Maydell wrote:
> If it doesn't make sense to hand off cp15 accesses that's fine,
> though -- I want to do this refactoring for the A15 system mode
> implementation in qemu anyway, because I really don't think we
> should try to shoehorn in yet another cpu implementation into the
> current set of switch statements...
> 
> It looks (from a brief glance at the code) like ppc kvm does
> handoff-to-qemu with the DCRs, incidentally.

Not for SPRs, though, which may be more analogous to ARM's cp15 (and are
where timers are implemented).

DCRs are more like the the PPC 4xx family's version of x86 I/O ports.

-Scott