[Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Erlon Cruz]

Hi all,

We are planning to implement DLPAR capacity on QEMU pSeries. As we
lack of experience in the internals of the arch we would like you guys
to give us some design directions
and confirm if we going in the right direction. Our first idea is:

    1 - to patch 'spapr.c' so it can dynamically insert/remove basic
items into the device tree.
    2 - create a host side device that will be used with a guest side
driver to perform guest side operations and communicate changes from
host to the guest (like DynamicRM does in PowerVM LPARs). We are not
planning to use powerpc-tools and want to make resource management
transparent (i.e. no need to run daemons or userspace programs in the
guest, only this kernel driver).
    3 - create bindings to support adding/removal  ibmvscsi devices
    4 - create bindings to support adding/removal  ibmveth devices
    5 - create bindings to support adding/removal PCI devices
    6 - create bindings to support adding/removal of memory
        - Do we need to do this the way PowerVM does? We have tested
virtio ballooning and it can works with a few endiannes corrections.
    7 - create bindings to support adding/removal  CPUs
        - is SMP supported already? I tried to run SMP in a x86 host
and the guest stuck when SMP is enabled
        - would be possible to work on this without a P7 baremetal
machine? We have a P7 8205-E6B, is that possible to kick PHYP out?

Any ideia on how much effort (time/people) the hole thing would take?
Any consideration about this is much appreciated :)

Kind regards,
Erlon

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Alexander Graf]

On 09/12/2012 04:54 PM, Erlon Cruz wrote:
> Hi all,
>
> We are planning to implement DLPAR capacity on QEMU pSeries. As we

What is DLPAR? Hotplug support?

> lack of experience in the internals of the arch we would like you guys
> to give us some design directions
> and confirm if we going in the right direction. Our first idea is:
>
>      1 - to patch 'spapr.c' so it can dynamically insert/remove basic
> items into the device tree.

What exactly would you like to patch into it? We already do have support 
for dynamic dt creation with the spapr target.

>      2 - create a host side device that will be used with a guest side
> driver to perform guest side operations and communicate changes from
> host to the guest (like DynamicRM does in PowerVM LPARs). We are not

Why not just use hypercalls?

> planning to use powerpc-tools and want to make resource management
> transparent (i.e. no need to run daemons or userspace programs in the
> guest, only this kernel driver).
>      3 - create bindings to support adding/removal  ibmvscsi devices
>      4 - create bindings to support adding/removal  ibmveth devices
>      5 - create bindings to support adding/removal PCI devices
>      6 - create bindings to support adding/removal of memory

This is going to be the hardest part. I don't think QEMU supports memory 
hotplug yet.

>          - Do we need to do this the way PowerVM does? We have tested
> virtio ballooning and it can works with a few endiannes corrections.

I don't know how PowerVM works. But if normal ballooning is all you 
need, you should certainly just enable virtio-balloon.

>      7 - create bindings to support adding/removal  CPUs
>          - is SMP supported already? I tried to run SMP in a x86 host
> and the guest stuck when SMP is enabled

SMP should work just fine, yes. Where exactly does it get stuck?

>          - would be possible to work on this without a P7 baremetal
> machine?

At least for device hotplug, it should be perfectly possible to use an 
old G5 with PR KVM. I haven't gotten around to patch all the pieces of 
the puzzle to make -M pseries work with PR KVM when it's running on top 
of pHyp yet, so that won't work.

> We have a P7 8205-E6B, is that possible to kick PHYP out?

Ben?

> Any ideia on how much effort (time/people) the hole thing would take?
> Any consideration about this is much appreciated :)

Phew. It's hard to tell. Depends heavily on how good your people are :).


Alex

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Erlon Cruz]

On Wed, Sep 12, 2012 at 12:53 PM, Alexander Graf <agraf@suse.de> wrote:
> On 09/12/2012 04:54 PM, Erlon Cruz wrote:
>>
>> Hi all,
>>
>> We are planning to implement DLPAR capacity on QEMU pSeries. As we
>
>
> What is DLPAR? Hotplug support?

Yes, basically the way PowerVM uses to dynamically add memory, cpu,
and I/O slots to logical partitions (LPARs)

>
>> lack of experience in the internals of the arch we would like you guys
>> to give us some design directions
>> and confirm if we going in the right direction. Our first idea is:
>>
>>      1 - to patch 'spapr.c' so it can dynamically insert/remove basic
>> items into the device tree.
>
>
> What exactly would you like to patch into it? We already do have support for
> dynamic dt creation with the spapr target.
>

Actually we were not sure if the machine could do that. So we can add
things to the tree after booting it?

>>      2 - create a host side device that will be used with a guest side
>> driver to perform guest side operations and communicate changes from
>> host to the guest (like DynamicRM does in PowerVM LPARs). We are not
>
>
> Why not just use hypercalls?

The hypercalls are initiated from the guest side right? We also need a
way to the host send things to guest. Using hypercall also would
require a guest side KM.

>> planning to use powerpc-tools and want to make resource management
>> transparent (i.e. no need to run daemons or userspace programs in the
>> guest, only this kernel driver).
>>      3 - create bindings to support adding/removal  ibmvscsi devices
>>      4 - create bindings to support adding/removal  ibmveth devices
>>      5 - create bindings to support adding/removal PCI devices
>>      6 - create bindings to support adding/removal of memory
>
>
> This is going to be the hardest part. I don't think QEMU supports memory
> hotplug yet.

AFAIC ballonning is what QEMU provides so far which is fine to x86.

>
>>          - Do we need to do this the way PowerVM does? We have tested
>> virtio ballooning and it can works with a few endiannes corrections.
>
>
> I don't know how PowerVM works. But if normal ballooning is all you need,
> you should certainly just enable virtio-balloon.

PowerVM works with Logical Memory Blocks (LMB). The hypervisor
hotplugs memory blocks to guest's memory. Not only a 'borrowing' from
the guests, right Ben?

>>      7 - create bindings to support adding/removal  CPUs
>>          - is SMP supported already? I tried to run SMP in a x86 host
>> and the guest stuck when SMP is enabled
>
>
> SMP should work just fine, yes. Where exactly does it get stuck?

I think that is right after the guest kernel enables SMP

[    7.478259] Faulting instruction address: 0xc00000000053bbec
[    7.479521] Oops: Kernel access of bad area, sig: 11 [#1]
[    7.479694] SMP NR_CPUS=1024 NUMA pSeries

http://pastebin.com/VMtRyaTE

>
>
>>          - would be possible to work on this without a P7 baremetal
>> machine?
>
>
> At least for device hotplug, it should be perfectly possible to use an old
> G5 with PR KVM. I haven't gotten around to patch all the pieces of the
> puzzle to make -M pseries work with PR KVM when it's running on top of pHyp
> yet, so that won't work.
>
>
>> We have a P7 8205-E6B, is that possible to kick PHYP out?
>
>
> Ben?
>
>
>> Any ideia on how much effort (time/people) the hole thing would take?
>> Any consideration about this is much appreciated :)
>
>
> Phew. It's hard to tell. Depends heavily on how good your people are :).

Well, considering someone like you :p, so we would just need to
multiply it by 5 :P

>
>
> Alex
>
>

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Alexander Graf]

On 12.09.2012, at 22:56, Erlon Cruz <erlon.cruz@br.flextronics.com> wrote:

> On Wed, Sep 12, 2012 at 12:53 PM, Alexander Graf <agraf@suse.de> wrote:
>> On 09/12/2012 04:54 PM, Erlon Cruz wrote:
>>> 
>>> Hi all,
>>> 
>>> We are planning to implement DLPAR capacity on QEMU pSeries. As we
>> 
>> 
>> What is DLPAR? Hotplug support?
> 
> Yes, basically the way PowerVM uses to dynamically add memory, cpu,
> and I/O slots to logical partitions (LPARs)
> 
>> 
>>> lack of experience in the internals of the arch we would like you guys
>>> to give us some design directions
>>> and confirm if we going in the right direction. Our first idea is:
>>> 
>>>     1 - to patch 'spapr.c' so it can dynamically insert/remove basic
>>> items into the device tree.
>> 
>> 
>> What exactly would you like to patch into it? We already do have support for
>> dynamic dt creation with the spapr target.
>> 
> 
> Actually we were not sure if the machine could do that. So we can add
> things to the tree after booting it?

Linux gets a dt blob on boot, so that one is fixed. IIRC the user space tools for pHyp mess with the device tree through special hooks from the Linux side.

> 
>>>     2 - create a host side device that will be used with a guest side
>>> driver to perform guest side operations and communicate changes from
>>> host to the guest (like DynamicRM does in PowerVM LPARs). We are not
>> 
>> 
>> Why not just use hypercalls?
> 
> The hypercalls are initiated from the guest side right? We also need a
> way to the host send things to guest. Using hypercall also would
> require a guest side KM.

Ah, you need some interrupt mechanism. I see.

> 
>>> planning to use powerpc-tools and want to make resource management
>>> transparent (i.e. no need to run daemons or userspace programs in the
>>> guest, only this kernel driver).
>>>     3 - create bindings to support adding/removal  ibmvscsi devices
>>>     4 - create bindings to support adding/removal  ibmveth devices
>>>     5 - create bindings to support adding/removal PCI devices
>>>     6 - create bindings to support adding/removal of memory
>> 
>> 
>> This is going to be the hardest part. I don't think QEMU supports memory
>> hotplug yet.
> 
> AFAIC ballonning is what QEMU provides so far which is fine to x86.

Yes. But balooning can only shrink memory footprint of a VM, not increase its memory size.

> 
>> 
>>>         - Do we need to do this the way PowerVM does? We have tested
>>> virtio ballooning and it can works with a few endiannes corrections.
>> 
>> 
>> I don't know how PowerVM works. But if normal ballooning is all you need,
>> you should certainly just enable virtio-balloon.
> 
> PowerVM works with Logical Memory Blocks (LMB). The hypervisor
> hotplugs memory blocks to guest's memory. Not only a 'borrowing' from
> the guests, right Ben?
> 
>>>     7 - create bindings to support adding/removal  CPUs
>>>         - is SMP supported already? I tried to run SMP in a x86 host
>>> and the guest stuck when SMP is enabled
>> 
>> 
>> SMP should work just fine, yes. Where exactly does it get stuck?
> 
> I think that is right after the guest kernel enables SMP
> 
> [    7.478259] Faulting instruction address: 0xc00000000053bbec
> [    7.479521] Oops: Kernel access of bad area, sig: 11 [#1]
> [    7.479694] SMP NR_CPUS=1024 NUMA pSeries

Mind to attach gdb to the vm and break at that address? Then print a bt on all threads (vcpus) :).

> 
> http://pastebin.com/VMtRyaTE
> 
>> 
>> 
>>>         - would be possible to work on this without a P7 baremetal
>>> machine?
>> 
>> 
>> At least for device hotplug, it should be perfectly possible to use an old
>> G5 with PR KVM. I haven't gotten around to patch all the pieces of the
>> puzzle to make -M pseries work with PR KVM when it's running on top of pHyp
>> yet, so that won't work.
>> 
>> 
>>> We have a P7 8205-E6B, is that possible to kick PHYP out?
>> 
>> 
>> Ben?
>> 
>> 
>>> Any ideia on how much effort (time/people) the hole thing would take?
>>> Any consideration about this is much appreciated :)
>> 
>> 
>> Phew. It's hard to tell. Depends heavily on how good your people are :).
> 
> Well, considering someone like you :p, so we would just need to
> multiply it by 5 :P

It's still hard to guess. Full-time work probably about a month for a working prototype. But you might run into additional issues. And then there's the upstream review process which will take at least as long until thebthing is clean. And then quite some time to get it bug free ;).


Alex

> 
>> 
>> 
>> Alex
>> 
>> 

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Benjamin Herrenschmidt]

On Wed, 2012-09-12 at 17:53 +0200, Alexander Graf wrote:
> On 09/12/2012 04:54 PM, Erlon Cruz wrote:
> > Hi all,
> >
> > We are planning to implement DLPAR capacity on QEMU pSeries. As we
> 
> What is DLPAR? Hotplug support?

Yes.

> > lack of experience in the internals of the arch we would like you guys
> > to give us some design directions
> > and confirm if we going in the right direction. Our first idea is:
> >
> >      1 - to patch 'spapr.c' so it can dynamically insert/remove basic
> > items into the device tree.
> 
> What exactly would you like to patch into it? We already do have support 
> for dynamic dt creation with the spapr target.

No we don't. We don't have the necessary bits and pieces to pass the DT
updates down to the guest. PAPR defines a mechanism using RTAS calls
which we need to implement, but there are some issues remaining:

 - We don't have a way to "initiate" a DLPAR operation. This is
currently done by proprietary tools that communicate with the HMC. We
want to invent some kind of hotplug "interrupt" (using existing RTAS
event facilities). All it needs to do is indicate the DT path (ie.
connector) where something is to be plugged to or unplugged, which can
then trigger the relevant configure-connector calls to retrieve the DT
bits.

 - We have a problem with PCI. Currently, the content of the PCI
bus(ses) is discovered by SLOF running inside the guest. Not by qemu.
It's SLOF that assigns the BARs and create the device-tree nodes for the
various PCI devices. However, with hotplug, the guest expects to get
fully populated DT nodes for hotplugged PCI devices and fully assigned
BARS. Under pHyp that works because under the hood, RTAS contains an OFW
implementation which does all the assignment before passing the stuff to
the OS, but under qemu, RTAS is actually in qemu. This means we'll
probably have to move back the PCI device node creation and resource
assignment to qemu (like it was in the very early versions of the spapr
support).

> >      2 - create a host side device that will be used with a guest side
> > driver to perform guest side operations and communicate changes from
> > host to the guest (like DynamicRM does in PowerVM LPARs). We are not
>
> Why not just use hypercalls?

Actually there are existing RTAS calls to use for the actual passing of
the device-tree bits, the problem is purely how to "initiate" an
operation to trigger the guest code that will then perform the
appropriate calls.

qemu-ga is an option. But I was thinking more along the lines of adding
some new RTAS events, maybe EPOW style, a bit like ACPI does.

> > planning to use powerpc-tools and want to make resource management
> > transparent (i.e. no need to run daemons or userspace programs in the
> > guest, only this kernel driver).
> >      3 - create bindings to support adding/removal  ibmvscsi devices
> >      4 - create bindings to support adding/removal  ibmveth devices
> >      5 - create bindings to support adding/removal PCI devices
> >      6 - create bindings to support adding/removal of memory

There's already large parts of the necessary bits using RTAS in the
kernel (in recent kernels that is, older stuff really needed it all done
in userspace). The trigger mostly is missing.

> This is going to be the hardest part. I don't think QEMU supports memory 
> hotplug yet.

Missing from the above list is also CPU hotplug.

> >          - Do we need to do this the way PowerVM does? We have tested
> > virtio ballooning and it can works with a few endiannes corrections.
> 
> I don't know how PowerVM works. But if normal ballooning is all you 
> need, you should certainly just enable virtio-balloon.

Does virtio-balloon needs endian fixes ? We though it was just working !
Feel free to submit patches :)

> >      7 - create bindings to support adding/removal  CPUs
> >          - is SMP supported already? I tried to run SMP in a x86 host
> > and the guest stuck when SMP is enabled
> 
> SMP should work just fine, yes. Where exactly does it get stuck?

Right,it works fine as far as I can tell.

> >          - would be possible to work on this without a P7 baremetal
> > machine?
> 
> At least for device hotplug, it should be perfectly possible to use an 
> old G5 with PR KVM. I haven't gotten around to patch all the pieces of 
> the puzzle to make -M pseries work with PR KVM when it's running on top 
> of pHyp yet, so that won't work.
> 
> > We have a P7 8205-E6B, is that possible to kick PHYP out?
> 
> Ben?

Probably not. You need a 7R2.

> > Any ideia on how much effort (time/people) the hole thing would take?
> > Any consideration about this is much appreciated :)
> 
> Phew. It's hard to tell. Depends heavily on how good your people are :).
> 

Cheers,
Ben.

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Erlon Cruz]

> On Wed, Sep 12, 2012 at 6:48 PM, Benjamin Herrenschmidt <benh@kernel.crashing.org> wrote:
> On Wed, 2012-09-12 at 17:53 +0200, Alexander Graf wrote:
>> On 09/12/2012 04:54 PM, Erlon Cruz wrote:
>> > Hi all,
>> >
>> > We are planning to implement DLPAR capacity on QEMU pSeries. As we
>>
>> What is DLPAR? Hotplug support?
>
> Yes.
>
>> > lack of experience in the internals of the arch we would like you guys
>> > to give us some design directions
>> > and confirm if we going in the right direction. Our first idea is:
>> >
>> >      1 - to patch 'spapr.c' so it can dynamically insert/remove basic
>> > items into the device tree.
>>
>> What exactly would you like to patch into it? We already do have support
>> for dynamic dt creation with the spapr target.
>
> No we don't. We don't have the necessary bits and pieces to pass the DT
> updates down to the guest. PAPR defines a mechanism using RTAS calls
> which we need to implement, but there are some issues remaining:

Do we need any patching on SLOF to make this possible?

>  - We don't have a way to "initiate" a DLPAR operation. This is
> currently done by proprietary tools that communicate with the HMC. We
> want to invent some kind of hotplug "interrupt" (using existing RTAS
> event facilities). All it needs to do is indicate the DT path (ie.
> connector) where something is to be plugged to or unplugged, which can
> then trigger the relevant configure-connector calls to retrieve the DT
> bits.

I think that a device/guest driver will work for this purpose. It will
get interrupted when something on DT changes and trigger the kernel
routines that in the actual implementation are called by those
proprietary tool.

>  - We have a problem with PCI. Currently, the content of the PCI
> bus(ses) is discovered by SLOF running inside the guest. Not by qemu.
> It's SLOF that assigns the BARs and create the device-tree nodes for the
> various PCI devices. However, with hotplug, the guest expects to get
> fully populated DT nodes for hotplugged PCI devices and fully assigned
> BARS. Under pHyp that works because under the hood, RTAS contains an OFW
> implementation which does all the assignment before passing the stuff to
> the OS, but under qemu, RTAS is actually in qemu. This means we'll
> probably have to move back the PCI device node creation and resource
> assignment to qemu (like it was in the very early versions of the spapr
> support).

AFIK in the first versions of spapr there was no PCI support, right?
So I'm guessing that you refer to the first implementation of PCI.
Would we have to remove the PCI discovering functions from SLOF? I
have no idea on how to code into SLOF.

>> >      2 - create a host side device that will be used with a guest side
>> > driver to perform guest side operations and communicate changes from
>> > host to the guest (like DynamicRM does in PowerVM LPARs). We are not
>>
>> Why not just use hypercalls?
>
> Actually there are existing RTAS calls to use for the actual passing of
> the device-tree bits, the problem is purely how to "initiate" an
> operation to trigger the guest code that will then perform the
> appropriate calls.
> qemu-ga is an option. But I was thinking more along the lines of adding
> some new RTAS events, maybe EPOW style, a bit like ACPI does.
>
>> > planning to use powerpc-tools and want to make resource management
>> > transparent (i.e. no need to run daemons or userspace programs in the
>> > guest, only this kernel driver).
>> >      3 - create bindings to support adding/removal  ibmvscsi devices
>> >      4 - create bindings to support adding/removal  ibmveth devices
>> >      5 - create bindings to support adding/removal PCI devices
>> >      6 - create bindings to support adding/removal of memory
>
> There's already large parts of the necessary bits using RTAS in the
> kernel (in recent kernels that is, older stuff really needed it all done
> in userspace). The trigger mostly is missing.
>> This is going to be the hardest part. I don't think QEMU supports memory
>> hotplug yet.
>
> Missing from the above list is also CPU hotplug.
>
>> >          - Do we need to do this the way PowerVM does? We have tested
>> > virtio ballooning and it can works with a few endiannes corrections.
>>
>> I don't know how PowerVM works. But if normal ballooning is all you
>> need, you should certainly just enable virtio-balloon.
>
> Does virtio-balloon needs endian fixes ? We though it was just working !
> Feel free to submit patches :)
>
>> >      7 - create bindings to support adding/removal  CPUs
>> >          - is SMP supported already? I tried to run SMP in a x86 host
>> > and the guest stuck when SMP is enabled
>>
>> SMP should work just fine, yes. Where exactly does it get stuck?
>
> Right,it works fine as far as I can tell.
>
>> >          - would be possible to work on this without a P7 baremetal
>> > machine?
>>
>> At least for device hotplug, it should be perfectly possible to use an
>> old G5 with PR KVM. I haven't gotten around to patch all the pieces of
>> the puzzle to make -M pseries work with PR KVM when it's running on top
>> of pHyp yet, so that won't work.
>>
>> > We have a P7 8205-E6B, is that possible to kick PHYP out?
>>
>> Ben?
>
> Probably not. You need a 7R2.
>
>> > Any ideia on how much effort (time/people) the hole thing would take?
>> > Any consideration about this is much appreciated :)
>>
>> Phew. It's hard to tell. Depends heavily on how good your people are :).
>>
>
> Cheers,
> Ben.
>
>
>

Cheers,
Erlon

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Benjamin Herrenschmidt]

On Thu, 2012-09-13 at 12:15 -0300, Erlon Cruz wrote:

> >> > lack of experience in the internals of the arch we would like you guys
> >> > to give us some design directions
> >> > and confirm if we going in the right direction. Our first idea is:
> >> >
> >> >      1 - to patch 'spapr.c' so it can dynamically insert/remove basic
> >> > items into the device tree.
> >>
> >> What exactly would you like to patch into it? We already do have support
> >> for dynamic dt creation with the spapr target.
> >
> > No we don't. We don't have the necessary bits and pieces to pass the DT
> > updates down to the guest. PAPR defines a mechanism using RTAS calls
> > which we need to implement, but there are some issues remaining:
> 
> Do we need any patching on SLOF to make this possible?

For adding the RTAS calls, no. Under qemu, RTAS is currently entirely
provided by qemu itself (the RTAS blob that is carried around through
SLOF etc... is just a little 5 instructions wrapper that calls a hidden
hcall).

> >  - We don't have a way to "initiate" a DLPAR operation. This is
> > currently done by proprietary tools that communicate with the HMC. We
> > want to invent some kind of hotplug "interrupt" (using existing RTAS
> > event facilities). All it needs to do is indicate the DT path (ie.
> > connector) where something is to be plugged to or unplugged, which can
> > then trigger the relevant configure-connector calls to retrieve the DT
> > bits.
> 
> I think that a device/guest driver will work for this purpose. It will
> get interrupted when something on DT changes and trigger the kernel
> routines that in the actual implementation are called by those
> proprietary tool.

Right. My idea was to remain generally consistent with PAPR and use some
kind of existing RTAS even interrupt facility and extend it. I should
try to poke some of the IBM folks in charge of PAPR to see if they are
interested in actually architecting such a mechanism.

> >  - We have a problem with PCI. Currently, the content of the PCI
> > bus(ses) is discovered by SLOF running inside the guest. Not by qemu.
> > It's SLOF that assigns the BARs and create the device-tree nodes for the
> > various PCI devices. However, with hotplug, the guest expects to get
> > fully populated DT nodes for hotplugged PCI devices and fully assigned
> > BARS. Under pHyp that works because under the hood, RTAS contains an OFW
> > implementation which does all the assignment before passing the stuff to
> > the OS, but under qemu, RTAS is actually in qemu. This means we'll
> > probably have to move back the PCI device node creation and resource
> > assignment to qemu (like it was in the very early versions of the spapr
> > support).
> 
> AFIK in the first versions of spapr there was no PCI support, right?

Well, we added PCI pretty quickly.

> So I'm guessing that you refer to the first implementation of PCI.
> Would we have to remove the PCI discovering functions from SLOF? I
> have no idea on how to code into SLOF.

We would have to change them at least, as we still want SLOF to do the
driver matching part at least. I can help with SLOF and we have some
folks assigned to it as well in IBM, so don't worry too much about that
part. I think the first step is to get a proof of concept using PAPR VIO
first, which doesn't have that problem. Then we can look at the PCI
issues.

Cheers,
Ben.

> >> >      2 - create a host side device that will be used with a guest side
> >> > driver to perform guest side operations and communicate changes from
> >> > host to the guest (like DynamicRM does in PowerVM LPARs). We are not
> >>
> >> Why not just use hypercalls?
> >
> > Actually there are existing RTAS calls to use for the actual passing of
> > the device-tree bits, the problem is purely how to "initiate" an
> > operation to trigger the guest code that will then perform the
> > appropriate calls.
> > qemu-ga is an option. But I was thinking more along the lines of adding
> > some new RTAS events, maybe EPOW style, a bit like ACPI does.
> >
> >> > planning to use powerpc-tools and want to make resource management
> >> > transparent (i.e. no need to run daemons or userspace programs in the
> >> > guest, only this kernel driver).
> >> >      3 - create bindings to support adding/removal  ibmvscsi devices
> >> >      4 - create bindings to support adding/removal  ibmveth devices
> >> >      5 - create bindings to support adding/removal PCI devices
> >> >      6 - create bindings to support adding/removal of memory
> >
> > There's already large parts of the necessary bits using RTAS in the
> > kernel (in recent kernels that is, older stuff really needed it all done
> > in userspace). The trigger mostly is missing.
> >> This is going to be the hardest part. I don't think QEMU supports memory
> >> hotplug yet.
> >
> > Missing from the above list is also CPU hotplug.
> >
> >> >          - Do we need to do this the way PowerVM does? We have tested
> >> > virtio ballooning and it can works with a few endiannes corrections.
> >>
> >> I don't know how PowerVM works. But if normal ballooning is all you
> >> need, you should certainly just enable virtio-balloon.
> >
> > Does virtio-balloon needs endian fixes ? We though it was just working !
> > Feel free to submit patches :)
> >
> >> >      7 - create bindings to support adding/removal  CPUs
> >> >          - is SMP supported already? I tried to run SMP in a x86 host
> >> > and the guest stuck when SMP is enabled
> >>
> >> SMP should work just fine, yes. Where exactly does it get stuck?
> >
> > Right,it works fine as far as I can tell.
> >
> >> >          - would be possible to work on this without a P7 baremetal
> >> > machine?
> >>
> >> At least for device hotplug, it should be perfectly possible to use an
> >> old G5 with PR KVM. I haven't gotten around to patch all the pieces of
> >> the puzzle to make -M pseries work with PR KVM when it's running on top
> >> of pHyp yet, so that won't work.
> >>
> >> > We have a P7 8205-E6B, is that possible to kick PHYP out?
> >>
> >> Ben?
> >
> > Probably not. You need a 7R2.
> >
> >> > Any ideia on how much effort (time/people) the hole thing would take?
> >> > Any consideration about this is much appreciated :)
> >>
> >> Phew. It's hard to tell. Depends heavily on how good your people are :).
> >>
> >
> > Cheers,
> > Ben.
> >
> >
> >
> 
> Cheers,
> Erlon

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Erlon Cruz]

[-- Attachment #1: Type: text/plain, Size: 8105 bytes --]

Hi all,

Just retaking the discussion of some days ago. After some discussions here
and considering the suggestions in this thread, we got 3 main ideas for the
DLPAR on QEMU/KVM.

1 - introduce a new device/driver that will be used to communicate changes
in the DT to the guest
   + make thinks cleaner as there is no need to take any guest action to
have anything added
   + no userspace daemon needed
   - we would have to add a new driver. Not shure whether this is a bad
thing

2 - create a kernel task to listen to host changes
   + make thinks cleaner as there is no need to take any guest action to
have anything added
   + no userspace daemon needed
    - that might require some changes on the kernel core

3 - use a model as close as provide by PHYP: the idea here is to be able
use the same proprietary daemons used in the guest. As this daemon calls an
open tool(drmgr) to trigger DR events in in the kernel, it will also be
possible to use qemu-ga as an alternative  to the daemon.
    + this make thinks easier as we don't have to make any changes on guest
kernel
    + supported PowerVM tools will be able to work on this model
    -  this model (DynamicRM) will be is probably deprecated in the nearly
future

So far we are going with the third option. Any regards about that let us
know.

Kind Regards,
Erlon


On Thu, Sep 13, 2012 at 6:45 PM, Benjamin Herrenschmidt <
benh@kernel.crashing.org> wrote:

> On Thu, 2012-09-13 at 12:15 -0300, Erlon Cruz wrote:
>
> > >> > lack of experience in the internals of the arch we would like you
> guys
> > >> > to give us some design directions
> > >> > and confirm if we going in the right direction. Our first idea is:
> > >> >
> > >> >      1 - to patch 'spapr.c' so it can dynamically insert/remove
> basic
> > >> > items into the device tree.
> > >>
> > >> What exactly would you like to patch into it? We already do have
> support
> > >> for dynamic dt creation with the spapr target.
> > >
> > > No we don't. We don't have the necessary bits and pieces to pass the DT
> > > updates down to the guest. PAPR defines a mechanism using RTAS calls
> > > which we need to implement, but there are some issues remaining:
> >
> > Do we need any patching on SLOF to make this possible?
>
> For adding the RTAS calls, no. Under qemu, RTAS is currently entirely
> provided by qemu itself (the RTAS blob that is carried around through
> SLOF etc... is just a little 5 instructions wrapper that calls a hidden
> hcall).
>
> > >  - We don't have a way to "initiate" a DLPAR operation. This is
> > > currently done by proprietary tools that communicate with the HMC. We
> > > want to invent some kind of hotplug "interrupt" (using existing RTAS
> > > event facilities). All it needs to do is indicate the DT path (ie.
> > > connector) where something is to be plugged to or unplugged, which can
> > > then trigger the relevant configure-connector calls to retrieve the DT
> > > bits.
> >
> > I think that a device/guest driver will work for this purpose. It will
> > get interrupted when something on DT changes and trigger the kernel
> > routines that in the actual implementation are called by those
> > proprietary tool.
>
> Right. My idea was to remain generally consistent with PAPR and use some
> kind of existing RTAS even interrupt facility and extend it. I should
> try to poke some of the IBM folks in charge of PAPR to see if they are
> interested in actually architecting such a mechanism.
>
> > >  - We have a problem with PCI. Currently, the content of the PCI
> > > bus(ses) is discovered by SLOF running inside the guest. Not by qemu.
> > > It's SLOF that assigns the BARs and create the device-tree nodes for
> the
> > > various PCI devices. However, with hotplug, the guest expects to get
> > > fully populated DT nodes for hotplugged PCI devices and fully assigned
> > > BARS. Under pHyp that works because under the hood, RTAS contains an
> OFW
> > > implementation which does all the assignment before passing the stuff
> to
> > > the OS, but under qemu, RTAS is actually in qemu. This means we'll
> > > probably have to move back the PCI device node creation and resource
> > > assignment to qemu (like it was in the very early versions of the spapr
> > > support).
> >
> > AFIK in the first versions of spapr there was no PCI support, right?
>
> Well, we added PCI pretty quickly.
>
> > So I'm guessing that you refer to the first implementation of PCI.
> > Would we have to remove the PCI discovering functions from SLOF? I
> > have no idea on how to code into SLOF.
>
> We would have to change them at least, as we still want SLOF to do the
> driver matching part at least. I can help with SLOF and we have some
> folks assigned to it as well in IBM, so don't worry too much about that
> part. I think the first step is to get a proof of concept using PAPR VIO
> first, which doesn't have that problem. Then we can look at the PCI
> issues.
>
> Cheers,
> Ben.
>
> > >> >      2 - create a host side device that will be used with a guest
> side
> > >> > driver to perform guest side operations and communicate changes from
> > >> > host to the guest (like DynamicRM does in PowerVM LPARs). We are not
> > >>
> > >> Why not just use hypercalls?
> > >
> > > Actually there are existing RTAS calls to use for the actual passing of
> > > the device-tree bits, the problem is purely how to "initiate" an
> > > operation to trigger the guest code that will then perform the
> > > appropriate calls.
> > > qemu-ga is an option. But I was thinking more along the lines of adding
> > > some new RTAS events, maybe EPOW style, a bit like ACPI does.
> > >
> > >> > planning to use powerpc-tools and want to make resource management
> > >> > transparent (i.e. no need to run daemons or userspace programs in
> the
> > >> > guest, only this kernel driver).
> > >> >      3 - create bindings to support adding/removal  ibmvscsi devices
> > >> >      4 - create bindings to support adding/removal  ibmveth devices
> > >> >      5 - create bindings to support adding/removal PCI devices
> > >> >      6 - create bindings to support adding/removal of memory
> > >
> > > There's already large parts of the necessary bits using RTAS in the
> > > kernel (in recent kernels that is, older stuff really needed it all
> done
> > > in userspace). The trigger mostly is missing.
> > >> This is going to be the hardest part. I don't think QEMU supports
> memory
> > >> hotplug yet.
> > >
> > > Missing from the above list is also CPU hotplug.
> > >
> > >> >          - Do we need to do this the way PowerVM does? We have
> tested
> > >> > virtio ballooning and it can works with a few endiannes corrections.
> > >>
> > >> I don't know how PowerVM works. But if normal ballooning is all you
> > >> need, you should certainly just enable virtio-balloon.
> > >
> > > Does virtio-balloon needs endian fixes ? We though it was just working
> !
> > > Feel free to submit patches :)
> > >
> > >> >      7 - create bindings to support adding/removal  CPUs
> > >> >          - is SMP supported already? I tried to run SMP in a x86
> host
> > >> > and the guest stuck when SMP is enabled
> > >>
> > >> SMP should work just fine, yes. Where exactly does it get stuck?
> > >
> > > Right,it works fine as far as I can tell.
> > >
> > >> >          - would be possible to work on this without a P7 baremetal
> > >> > machine?
> > >>
> > >> At least for device hotplug, it should be perfectly possible to use an
> > >> old G5 with PR KVM. I haven't gotten around to patch all the pieces of
> > >> the puzzle to make -M pseries work with PR KVM when it's running on
> top
> > >> of pHyp yet, so that won't work.
> > >>
> > >> > We have a P7 8205-E6B, is that possible to kick PHYP out?
> > >>
> > >> Ben?
> > >
> > > Probably not. You need a 7R2.
> > >
> > >> > Any ideia on how much effort (time/people) the hole thing would
> take?
> > >> > Any consideration about this is much appreciated :)
> > >>
> > >> Phew. It's hard to tell. Depends heavily on how good your people are
> :).
> > >>
> > >
> > > Cheers,
> > > Ben.
> > >
> > >
> > >
> >
> > Cheers,
> > Erlon
>
>
>
>

[-- Attachment #2: Type: text/html, Size: 10143 bytes --]

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Anthony Liguori]

Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:

> On Wed, 2012-09-12 at 17:53 +0200, Alexander Graf wrote:
>> On 09/12/2012 04:54 PM, Erlon Cruz wrote:
>> > Hi all,
>> >
>> > We are planning to implement DLPAR capacity on QEMU pSeries. As we
>> 
>> What is DLPAR? Hotplug support?
>
> Yes.
>
>> > lack of experience in the internals of the arch we would like you guys
>> > to give us some design directions
>> > and confirm if we going in the right direction. Our first idea is:
>> >
>> >      1 - to patch 'spapr.c' so it can dynamically insert/remove basic
>> > items into the device tree.
>> 
>> What exactly would you like to patch into it? We already do have support 
>> for dynamic dt creation with the spapr target.
>
> No we don't. We don't have the necessary bits and pieces to pass the DT
> updates down to the guest. PAPR defines a mechanism using RTAS calls
> which we need to implement, but there are some issues remaining:
>
>  - We don't have a way to "initiate" a DLPAR operation. This is
> currently done by proprietary tools that communicate with the HMC. We
> want to invent some kind of hotplug "interrupt" (using existing RTAS
> event facilities). All it needs to do is indicate the DT path (ie.
> connector) where something is to be plugged to or unplugged, which can
> then trigger the relevant configure-connector calls to retrieve the DT
> bits.
>
>  - We have a problem with PCI. Currently, the content of the PCI
> bus(ses) is discovered by SLOF running inside the guest. Not by qemu.
> It's SLOF that assigns the BARs and create the device-tree nodes for the
> various PCI devices. However, with hotplug, the guest expects to get
> fully populated DT nodes for hotplugged PCI devices and fully assigned
> BARS. Under pHyp that works because under the hood, RTAS contains an OFW
> implementation which does all the assignment before passing the stuff to
> the OS, but under qemu, RTAS is actually in qemu. This means we'll
> probably have to move back the PCI device node creation and resource
> assignment to qemu (like it was in the very early versions of the spapr
> support).

I know you and I have discussed this in the past, but not on the list
and now I don't recall the reasoning.

Why not add a proper RTAS and do this work there?

Regards,

Anthony Liguori

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Erlon Cruz]

On 10/05/2012 11:42 AM, Anthony Liguori wrote:
> Benjamin Herrenschmidt <benh@kernel.crashing.org> writes:
>
>> On Wed, 2012-09-12 at 17:53 +0200, Alexander Graf wrote:
>>> On 09/12/2012 04:54 PM, Erlon Cruz wrote:
>>>> Hi all,
>>>>
>>>> We are planning to implement DLPAR capacity on QEMU pSeries. As we
>>> What is DLPAR? Hotplug support?
>> Yes.
>>
>>>> lack of experience in the internals of the arch we would like you guys
>>>> to give us some design directions
>>>> and confirm if we going in the right direction. Our first idea is:
>>>>
>>>>       1 - to patch 'spapr.c' so it can dynamically insert/remove basic
>>>> items into the device tree.
>>> What exactly would you like to patch into it? We already do have support
>>> for dynamic dt creation with the spapr target.
>> No we don't. We don't have the necessary bits and pieces to pass the DT
>> updates down to the guest. PAPR defines a mechanism using RTAS calls
>> which we need to implement, but there are some issues remaining:
>>
>>   - We don't have a way to "initiate" a DLPAR operation. This is
>> currently done by proprietary tools that communicate with the HMC. We
>> want to invent some kind of hotplug "interrupt" (using existing RTAS
>> event facilities). All it needs to do is indicate the DT path (ie.
>> connector) where something is to be plugged to or unplugged, which can
>> then trigger the relevant configure-connector calls to retrieve the DT
>> bits.
>>
>>   - We have a problem with PCI. Currently, the content of the PCI
>> bus(ses) is discovered by SLOF running inside the guest. Not by qemu.
>> It's SLOF that assigns the BARs and create the device-tree nodes for the
>> various PCI devices. However, with hotplug, the guest expects to get
>> fully populated DT nodes for hotplugged PCI devices and fully assigned
>> BARS. Under pHyp that works because under the hood, RTAS contains an OFW
>> implementation which does all the assignment before passing the stuff to
>> the OS, but under qemu, RTAS is actually in qemu. This means we'll
>> probably have to move back the PCI device node creation and resource
>> assignment to qemu (like it was in the very early versions of the spapr
>> support).
> I know you and I have discussed this in the past, but not on the list
> and now I don't recall the reasoning.
>
> Why not add a proper RTAS and do this work there?

The RTAS call is how the guest will communicate with the host. It 
doesn't have a way to get interrupted and notified about any changes. So 
to start the changing of something, the guest need some kind of trigger

> Regards,
>
> Anthony Liguori
> .
>

Legal Disclaimer:
The information contained in this message may be privileged and confidential. It is intended to be read only by the individual or entity to whom it is addressed or by their designee. If the reader of this message is not the intended recipient, you are on notice that any distribution of this message, in any form, is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete or destroy any copy of this message!

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Benjamin Herrenschmidt]

On Fri, 2012-10-05 at 08:26 -0700, Erlon Cruz wrote:
> >
> > Why not add a proper RTAS and do this work there?
> 
> The RTAS call is how the guest will communicate with the host. It 
> doesn't have a way to get interrupted and notified about any changes.
> So 
> to start the changing of something, the guest need some kind of
> trigger

I believe the question was more about where we put the resource
allocation.

The interrupt problem remains in any case yes, but inventing a new RTAT
event interrupt (or hijacking EPOW) isn't a big deal. 

Cheers,
Ben.

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Benjamin Herrenschmidt]

On Fri, 2012-10-05 at 09:42 -0500, Anthony Liguori wrote:
> >  - We have a problem with PCI. Currently, the content of the PCI
> > bus(ses) is discovered by SLOF running inside the guest. Not by qemu.
> > It's SLOF that assigns the BARs and create the device-tree nodes for the
> > various PCI devices. However, with hotplug, the guest expects to get
> > fully populated DT nodes for hotplugged PCI devices and fully assigned
> > BARS. Under pHyp that works because under the hood, RTAS contains an OFW
> > implementation which does all the assignment before passing the stuff to
> > the OS, but under qemu, RTAS is actually in qemu. This means we'll
> > probably have to move back the PCI device node creation and resource
> > assignment to qemu (like it was in the very early versions of the spapr
> > support).
> 
> I know you and I have discussed this in the past, but not on the list
> and now I don't recall the reasoning.
> 
> Why not add a proper RTAS and do this work there?

Because it's a PITA ? We could ... but that means we'd have to write
some kind of FW blob that is relocatable and capable of "live"
relocating itself (the versions that would run within SLOF need to
transfer control to one at a different address that runs within the OS
context since the OS decides where RTAS lives).

It would have to know the device-tree to handle DLPAR (thus get "synced"
by OF on any DT change until OF is killed by the OF).

Generally writing yet another blob means yet another piece of SW to
write from scratch, maintain, package, deal with dependencies, debug,
etc...

It is an option though, just not a nice one. And a fairly time consuming
one too.

Cheers,
Ben.

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[David Gibson]

On Sat, Oct 06, 2012 at 06:49:13AM +1000, Benjamin Herrenschmidt wrote:
> On Fri, 2012-10-05 at 09:42 -0500, Anthony Liguori wrote:
> > >  - We have a problem with PCI. Currently, the content of the PCI
> > > bus(ses) is discovered by SLOF running inside the guest. Not by qemu.
> > > It's SLOF that assigns the BARs and create the device-tree nodes for the
> > > various PCI devices. However, with hotplug, the guest expects to get
> > > fully populated DT nodes for hotplugged PCI devices and fully assigned
> > > BARS. Under pHyp that works because under the hood, RTAS contains an OFW
> > > implementation which does all the assignment before passing the stuff to
> > > the OS, but under qemu, RTAS is actually in qemu. This means we'll
> > > probably have to move back the PCI device node creation and resource
> > > assignment to qemu (like it was in the very early versions of the spapr
> > > support).
> > 
> > I know you and I have discussed this in the past, but not on the list
> > and now I don't recall the reasoning.
> > 
> > Why not add a proper RTAS and do this work there?
> 
> Because it's a PITA ? We could ... but that means we'd have to write
> some kind of FW blob that is relocatable and capable of "live"
> relocating itself (the versions that would run within SLOF need to
> transfer control to one at a different address that runs within the OS
> context since the OS decides where RTAS lives).
> 
> It would have to know the device-tree to handle DLPAR (thus get "synced"
> by OF on any DT change until OF is killed by the OF).
> 
> Generally writing yet another blob means yet another piece of SW to
> write from scratch, maintain, package, deal with dependencies, debug,
> etc...
> 
> It is an option though, just not a nice one. And a fairly time consuming
> one too.

I also don't see how it helps in this situation.  The hotplug will
still be initiated by qemu, so we'd have to work out how qemu
communicates the info to RTAS, and how RTAS communicates it to the
kernel, doubling the work.

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

Re: [Qemu-devel] Enablig DLPAR capacity on QEMU pSeries

[Benjamin Herrenschmidt]

On Sun, 2012-10-07 at 00:54 +1000, David Gibson wrote:
> I also don't see how it helps in this situation.  The hotplug will
> still be initiated by qemu, so we'd have to work out how qemu
> communicates the info to RTAS, and how RTAS communicates it to the
> kernel, doubling the work.

This is not about initiating the hotplug, this is about the resource
management. Those are two separate problems.

Cheers,
Ben.