Right location in sysfs for dlpar file

Right location in sysfs for dlpar file

[Benjamin Herrenschmidt]

Hi Greg,

So Nathan is working on a patch series to cleanup and improve our
"DLPAR" infrastructure which is basically our hotplug mechanism when
running under the PowerVM (aka pHyp) and KVM hypervisors.

I'll let Nathan give you a bit more details/background and answer
subsequent question you might have as this is really his area of
expertise.

To cut a long story short, we need a sysfs file that allows our
userspace tools to notify the kernel of hotplug events coming from
the management console (which talks to userspace daemons using a
proprietary protocol) to "initiate" the hotplug operations, which in
turn get dispatched internally in the kernel to the right subsystem
(memory, cpu, pci, ...) based on the resource type.

On IRC, Greg suggested /sys/firmware and /sys/hypervisor which both
look like a reasonable option to me, probably better than dlpar...

Cheers,
Ben.

Re: Right location in sysfs for dlpar file

[Nathan Fontenot]

On 11/26/2014 09:12 PM, Benjamin Herrenschmidt wrote:
> Hi Greg,
> 
> So Nathan is working on a patch series to cleanup and improve our
> "DLPAR" infrastructure which is basically our hotplug mechanism when
> running under the PowerVM (aka pHyp) and KVM hypervisors.

The cleanup to the dlpar infrastructure will move the entire operation
of hotplugging a device to the kernel instead of doing it partially in
userspace and partially in the kernel as is currently done.

> 
> I'll let Nathan give you a bit more details/background and answer
> subsequent question you might have as this is really his area of
> expertise.
> 
> To cut a long story short, we need a sysfs file that allows our
> userspace tools to notify the kernel of hotplug events coming from
> the management console (which talks to userspace daemons using a
> proprietary protocol) to "initiate" the hotplug operations, which in
> turn get dispatched internally in the kernel to the right subsystem
> (memory, cpu, pci, ...) based on the resource type.
> 
> On IRC, Greg suggested /sys/firmware and /sys/hypervisor which both
> look like a reasonable option to me, probably better than dlpar...

For PowerVM systems we need this sysfs file to deliver what is
essentially a binary blob (specifically a rtas error log) to the
kernel. The current patch set is creating /sys/kernel/dlpar. As Ben
mentioned we would like your input on what would be the proper place
to create this file.

-Nathan

Re: Right location in sysfs for dlpar file

[Greg KH]

On Mon, Dec 01, 2014 at 09:41:03AM -0600, Nathan Fontenot wrote:
> On 11/26/2014 09:12 PM, Benjamin Herrenschmidt wrote:
> > Hi Greg,
> > 
> > So Nathan is working on a patch series to cleanup and improve our
> > "DLPAR" infrastructure which is basically our hotplug mechanism when
> > running under the PowerVM (aka pHyp) and KVM hypervisors.
> 
> The cleanup to the dlpar infrastructure will move the entire operation
> of hotplugging a device to the kernel instead of doing it partially in
> userspace and partially in the kernel as is currently done.
> 
> > 
> > I'll let Nathan give you a bit more details/background and answer
> > subsequent question you might have as this is really his area of
> > expertise.
> > 
> > To cut a long story short, we need a sysfs file that allows our
> > userspace tools to notify the kernel of hotplug events coming from
> > the management console (which talks to userspace daemons using a
> > proprietary protocol) to "initiate" the hotplug operations, which in
> > turn get dispatched internally in the kernel to the right subsystem
> > (memory, cpu, pci, ...) based on the resource type.
> > 
> > On IRC, Greg suggested /sys/firmware and /sys/hypervisor which both
> > look like a reasonable option to me, probably better than dlpar...
> 
> For PowerVM systems we need this sysfs file to deliver what is
> essentially a binary blob (specifically a rtas error log) to the
> kernel. The current patch set is creating /sys/kernel/dlpar. As Ben
> mentioned we would like your input on what would be the proper place
> to create this file.

And what is the kernel supposed to do with such a binary blob?  Parse
it?  Or pass it to something else?

Anyway, let's see the patches before I guess anything else, that will
determine how things work out best.

thanks,

greg k-h

Re: Right location in sysfs for dlpar file

[Nathan Fontenot]

On 12/01/2014 10:26 PM, Greg KH wrote:
> On Mon, Dec 01, 2014 at 09:41:03AM -0600, Nathan Fontenot wrote:
>> On 11/26/2014 09:12 PM, Benjamin Herrenschmidt wrote:
>>> Hi Greg,
>>>
>>> So Nathan is working on a patch series to cleanup and improve our
>>> "DLPAR" infrastructure which is basically our hotplug mechanism when
>>> running under the PowerVM (aka pHyp) and KVM hypervisors.
>>
>> The cleanup to the dlpar infrastructure will move the entire operation
>> of hotplugging a device to the kernel instead of doing it partially in
>> userspace and partially in the kernel as is currently done.
>>
>>>
>>> I'll let Nathan give you a bit more details/background and answer
>>> subsequent question you might have as this is really his area of
>>> expertise.
>>>
>>> To cut a long story short, we need a sysfs file that allows our
>>> userspace tools to notify the kernel of hotplug events coming from
>>> the management console (which talks to userspace daemons using a
>>> proprietary protocol) to "initiate" the hotplug operations, which in
>>> turn get dispatched internally in the kernel to the right subsystem
>>> (memory, cpu, pci, ...) based on the resource type.
>>>
>>> On IRC, Greg suggested /sys/firmware and /sys/hypervisor which both
>>> look like a reasonable option to me, probably better than dlpar...
>>
>> For PowerVM systems we need this sysfs file to deliver what is
>> essentially a binary blob (specifically a rtas error log) to the
>> kernel. The current patch set is creating /sys/kernel/dlpar. As Ben
>> mentioned we would like your input on what would be the proper place
>> to create this file.
> 
> And what is the kernel supposed to do with such a binary blob?  Parse
> it?  Or pass it to something else?

The kernel will parse it and perform the requested hotplug operation.

> 
> Anyway, let's see the patches before I guess anything else, that will
> determine how things work out best.

I'll cc you on the next version of the patch set.

-Nathan

Re: Right location in sysfs for dlpar file

[Greg KH]

On Wed, Dec 03, 2014 at 09:07:27PM -0600, Nathan Fontenot wrote:
> On 12/01/2014 10:26 PM, Greg KH wrote:
> > On Mon, Dec 01, 2014 at 09:41:03AM -0600, Nathan Fontenot wrote:
> >> On 11/26/2014 09:12 PM, Benjamin Herrenschmidt wrote:
> >>> Hi Greg,
> >>>
> >>> So Nathan is working on a patch series to cleanup and improve our
> >>> "DLPAR" infrastructure which is basically our hotplug mechanism when
> >>> running under the PowerVM (aka pHyp) and KVM hypervisors.
> >>
> >> The cleanup to the dlpar infrastructure will move the entire operation
> >> of hotplugging a device to the kernel instead of doing it partially in
> >> userspace and partially in the kernel as is currently done.
> >>
> >>>
> >>> I'll let Nathan give you a bit more details/background and answer
> >>> subsequent question you might have as this is really his area of
> >>> expertise.
> >>>
> >>> To cut a long story short, we need a sysfs file that allows our
> >>> userspace tools to notify the kernel of hotplug events coming from
> >>> the management console (which talks to userspace daemons using a
> >>> proprietary protocol) to "initiate" the hotplug operations, which in
> >>> turn get dispatched internally in the kernel to the right subsystem
> >>> (memory, cpu, pci, ...) based on the resource type.
> >>>
> >>> On IRC, Greg suggested /sys/firmware and /sys/hypervisor which both
> >>> look like a reasonable option to me, probably better than dlpar...
> >>
> >> For PowerVM systems we need this sysfs file to deliver what is
> >> essentially a binary blob (specifically a rtas error log) to the
> >> kernel. The current patch set is creating /sys/kernel/dlpar. As Ben
> >> mentioned we would like your input on what would be the proper place
> >> to create this file.
> > 
> > And what is the kernel supposed to do with such a binary blob?  Parse
> > it?  Or pass it to something else?
> 
> The kernel will parse it and perform the requested hotplug operation.

Oh I was hoping you would not say that :(

Seriously?  Parsing binary blobs from userspace?  Don't do that, you
know better.

greg k-h

Re: Right location in sysfs for dlpar file

[Nathan Fontenot]

On 12/03/2014 10:31 PM, Greg KH wrote:
> On Wed, Dec 03, 2014 at 09:07:27PM -0600, Nathan Fontenot wrote:
>> On 12/01/2014 10:26 PM, Greg KH wrote:
>>> On Mon, Dec 01, 2014 at 09:41:03AM -0600, Nathan Fontenot wrote:
>>>> On 11/26/2014 09:12 PM, Benjamin Herrenschmidt wrote:
>>>>> Hi Greg,
>>>>>
>>>>> So Nathan is working on a patch series to cleanup and improve our
>>>>> "DLPAR" infrastructure which is basically our hotplug mechanism when
>>>>> running under the PowerVM (aka pHyp) and KVM hypervisors.
>>>>
>>>> The cleanup to the dlpar infrastructure will move the entire operation
>>>> of hotplugging a device to the kernel instead of doing it partially in
>>>> userspace and partially in the kernel as is currently done.
>>>>
>>>>>
>>>>> I'll let Nathan give you a bit more details/background and answer
>>>>> subsequent question you might have as this is really his area of
>>>>> expertise.
>>>>>
>>>>> To cut a long story short, we need a sysfs file that allows our
>>>>> userspace tools to notify the kernel of hotplug events coming from
>>>>> the management console (which talks to userspace daemons using a
>>>>> proprietary protocol) to "initiate" the hotplug operations, which in
>>>>> turn get dispatched internally in the kernel to the right subsystem
>>>>> (memory, cpu, pci, ...) based on the resource type.
>>>>>
>>>>> On IRC, Greg suggested /sys/firmware and /sys/hypervisor which both
>>>>> look like a reasonable option to me, probably better than dlpar...
>>>>
>>>> For PowerVM systems we need this sysfs file to deliver what is
>>>> essentially a binary blob (specifically a rtas error log) to the
>>>> kernel. The current patch set is creating /sys/kernel/dlpar. As Ben
>>>> mentioned we would like your input on what would be the proper place
>>>> to create this file.
>>>
>>> And what is the kernel supposed to do with such a binary blob?  Parse
>>> it?  Or pass it to something else?
>>
>> The kernel will parse it and perform the requested hotplug operation.
> 
> Oh I was hoping you would not say that :(

Heh! hint taken. I won't cc you.

> 
> Seriously?  Parsing binary blobs from userspace?  Don't do that, you
> know better.

Yes, not ideal. One thing to note here is that the code to parse these
binary blobs already exists in the kernel for powerpc. The binary blob
we need to pass in is the same format as rtas error logs that we can be
handed by firmware on powerpc systems.

-Nathan

Re: Right location in sysfs for dlpar file

[Benjamin Herrenschmidt]

On Mon, 2014-12-08 at 09:21 -0600, Nathan Fontenot wrote:
> > Oh I was hoping you would not say that :(
> 
> Heh! hint taken. I won't cc you.
> 
> > 
> > Seriously?  Parsing binary blobs from userspace?  Don't do that, you
> > know better.
> 
> Yes, not ideal. One thing to note here is that the code to parse these
> binary blobs already exists in the kernel for powerpc. The binary blob
> we need to pass in is the same format as rtas error logs that we can be
> handed by firmware on powerpc systems.

We should clarify that it's a small blob, it's an error log entry as defined
by our firmware interfaces. We already obtain these from the FW and parse them
(it's really just mapping the right structure based on the type in the header)
under various circumstances.

What happens is on KVM, we used that existing path to signal the kernel of an
hotplug event. On PowerVM (aka pHyp), we want to use the same kernel code path
by passing an equivalent event to the kernel.

Nathan, if Greg still strongly objects, we could have the sysfs interface be
something like a simple action + resource ID string, we don't *need* it to
be encapsulated in an error log, we would just pipe into the same infrastructure
in the kernel but at a slightly lower level...

Ben.

Re: Right location in sysfs for dlpar file

[Michael Ellerman]

On Mon, 2014-12-01 at 09:41 -0600, Nathan Fontenot wrote:
> On 11/26/2014 09:12 PM, Benjamin Herrenschmidt wrote:
> > Hi Greg,
> > 
> > So Nathan is working on a patch series to cleanup and improve our
> > "DLPAR" infrastructure which is basically our hotplug mechanism when
> > running under the PowerVM (aka pHyp) and KVM hypervisors.
> 
> The cleanup to the dlpar infrastructure will move the entire operation
> of hotplugging a device to the kernel instead of doing it partially in
> userspace and partially in the kernel as is currently done.
> 
...
> 
> For PowerVM systems we need this sysfs file to deliver what is
> essentially a binary blob (specifically a rtas error log) to the
> kernel.

Those two statements don't really agree with each other. ie. "move the entire
operation .. to the kernel", but then we need a sysfs file so userspace can
deliver us a blob?

I think what you mean is that all the actual logic will move into the kernel,
and the only thing userspace will do (on PowerVM) is write the blog to kick off
the process.

On PowerKVM the entire process will be handled in the kernel (after some
additional patches to hook up the rtas event to the hotplug).


As ugly as it is, we already have /proc/rtas, which includes a bunch of files,
including error_log, which is where you can *read* the RTAS error logs from.

So maybe we just extend that, either a new file, or just by making error_log
writable?

It'd be nice to drop all that rtas gunk and move to something cleaner in /sys,
but I don't think we can realistically do that any time soon anyway?

cheers

Re: Right location in sysfs for dlpar file

[Nathan Fontenot]

On 12/01/2014 11:09 PM, Michael Ellerman wrote:
> On Mon, 2014-12-01 at 09:41 -0600, Nathan Fontenot wrote:
>> On 11/26/2014 09:12 PM, Benjamin Herrenschmidt wrote:
>>> Hi Greg,
>>>
>>> So Nathan is working on a patch series to cleanup and improve our
>>> "DLPAR" infrastructure which is basically our hotplug mechanism when
>>> running under the PowerVM (aka pHyp) and KVM hypervisors.
>>
>> The cleanup to the dlpar infrastructure will move the entire operation
>> of hotplugging a device to the kernel instead of doing it partially in
>> userspace and partially in the kernel as is currently done.
>>
> ...
>>
>> For PowerVM systems we need this sysfs file to deliver what is
>> essentially a binary blob (specifically a rtas error log) to the
>> kernel.
> 
> Those two statements don't really agree with each other. ie. "move the entire
> operation .. to the kernel", but then we need a sysfs file so userspace can
> deliver us a blob?
> 
> I think what you mean is that all the actual logic will move into the kernel,
> and the only thing userspace will do (on PowerVM) is write the blog to kick off
> the process.
> 
> On PowerKVM the entire process will be handled in the kernel (after some
> additional patches to hook up the rtas event to the hotplug).
> 

This is correct. I should have explained it better.

> 
> As ugly as it is, we already have /proc/rtas, which includes a bunch of files,
> including error_log, which is where you can *read* the RTAS error logs from.
>

I have no problem making this a writeable file. I think keeping all of the rtas
error log communication between the kernel userspace in one place would be a good
idea.
 
> So maybe we just extend that, either a new file, or just by making error_log
> writable?
> 
> It'd be nice to drop all that rtas gunk and move to something cleaner in /sys,
> but I don't think we can realistically do that any time soon anyway?
> 

Perhaps doing this at a later time would be worthwhile.

-Nathan