Re: [Qemu-devel] Adding an IPMI BMC device to KVM

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Avi Kivity]

(copied qemu-devel)

On 05/04/2012 10:10 PM, Corey Minyard wrote:
> I've been working on adding an IPMI BMC as a virtual device under
> KVM.  I'm
> doing this for two primary reasons, one to have a better test
> environment than
> what I have now for testing IPMI issues, and second to be able to better
> simulate a legacy environment for customers porting legacy software.
>
> For those that don't know, IPMI is a system management interface. 
> Generally
> systems with IPMI have a small microcontroller, called a BMC, that is
> always on
> when the board is powered.  The BMC is capable of controlling power
> and reset
> on the board, and it is hooked to sensors on the board (voltage, current,
> temperature, the presence of things like DIMMS and power supplies,
> intrusion
> detection, and a host of other things).  The main processor on a
> system can
> communicate with the BMC over a device.  Often these systems also have
> a LAN
> interface that lets you control the system remotely even when it's off.
>
> In addition, IPMI provides access to FRU (Field Replaceable Unit) data
> that
> describes the components of the system that can be replaced.  It also
> has data
> records that describe the sensor, so it is possible to directly
> interpret the sensor
> data and know what the sensor is measuring without outside data.
>
> I've been struggling a bit with how to implement this.  There is a lot of
> configuration information, and you need ways to simulate the sensors. 
> This type
> of interface is a little sensitive, since it has direct access to the
> reset and power
> control of a system.
>
> I was at first considering having the BMC be an external program that KVM
> connected to over a chardev, with possibly a simulated LAN interface. 
> This has
> the advantage that the BMC can run even when KVM is down.  It could even
> start up KVM for a "power up", though I'm not sure how valuable that
> would be.
> Plus it could be used for other virtual machines.  However, that means
> there is
> an interface to KVM over a chardev that could do nasty things, and
> even be a
> possible intrusion point.  It also means there is a separate program
> to maintain.
>
> You could also include the BMC inside of KVM and run it as a separate
> thread.
> That way there doesn't have to be an insecure interface.  But the BMC
> will need
> a lot of configuration data and this will add a bunch of code to KVM
> that's only
> tangentially related to it.  And you would still need a way to
> simulate setting
> sensors and such for testing things.
>
> Either way, is this interesting for including into KVM?  

Not kvm, but certainly it would make a good addition to qemu, which kvm
then uses.

> Does anyone have any
> opinions on the possible ways to implement this?

My preference would be the second alternative.  The issue you raise is a
good one.  There are two ways we can approach it:

- have the management system intercept IPMI requests, start up a qemu
instance (if it's down), and let it handle the event.
- change the whole system to keep a running qemu even when the guest is
down.  This is a much larger change; it involves reducing the memory
footprint to almost nothing when the guest is down (deallocating memory
and threads) so it doesn't impact guest density, but it allows for other
minor features such as wake-on-LAN and RTC alarm wakeups.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Anthony Liguori]

On 05/06/2012 08:11 AM, Avi Kivity wrote:
> (copied qemu-devel)
>
> On 05/04/2012 10:10 PM, Corey Minyard wrote:
>>
>> Either way, is this interesting for including into KVM?
>
> Not kvm, but certainly it would make a good addition to qemu, which kvm
> then uses.
>
>> Does anyone have any
>> opinions on the possible ways to implement this?
>
> My preference would be the second alternative.  The issue you raise is a
> good one.  There are two ways we can approach it:
>
> - have the management system intercept IPMI requests, start up a qemu
> instance (if it's down), and let it handle the event.
> - change the whole system to keep a running qemu even when the guest is
> down.  This is a much larger change; it involves reducing the memory
> footprint to almost nothing when the guest is down (deallocating memory
> and threads) so it doesn't impact guest density, but it allows for other
> minor features such as wake-on-LAN and RTC alarm wakeups.

libvirt is essentially the BMC for a virtual guest.  I would suggest looking at 
implementing an IPMI interface to libvirt and exposing it to the guest through a 
USB RNDIS device.

Regards,

Anthony Liguori

>

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Avi Kivity]

On 05/06/2012 05:35 PM, Anthony Liguori wrote:
> On 05/06/2012 08:11 AM, Avi Kivity wrote:
>> (copied qemu-devel)
>>
>> On 05/04/2012 10:10 PM, Corey Minyard wrote:
>>>
>>> Either way, is this interesting for including into KVM?
>>
>> Not kvm, but certainly it would make a good addition to qemu, which kvm
>> then uses.
>>
>>> Does anyone have any
>>> opinions on the possible ways to implement this?
>>
>> My preference would be the second alternative.  The issue you raise is a
>> good one.  There are two ways we can approach it:
>>
>> - have the management system intercept IPMI requests, start up a qemu
>> instance (if it's down), and let it handle the event.
>> - change the whole system to keep a running qemu even when the guest is
>> down.  This is a much larger change; it involves reducing the memory
>> footprint to almost nothing when the guest is down (deallocating memory
>> and threads) so it doesn't impact guest density, but it allows for other
>> minor features such as wake-on-LAN and RTC alarm wakeups.
>
> libvirt is essentially the BMC for a virtual guest.  I would suggest
> looking at implementing an IPMI interface to libvirt and exposing it
> to the guest through a USB RNDIS device.
>

That's the first option.  One unanswered question is what to do when the
guest is down?  Someone should listen for IPMI events, but we can't make
it libvirt unconditionally, since many instances of libvirt are active
at any one time.

Note the IPMI external interface needs to be migrated, like any other.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Anthony Liguori]

On 05/06/2012 09:39 AM, Avi Kivity wrote:
> On 05/06/2012 05:35 PM, Anthony Liguori wrote:
>> On 05/06/2012 08:11 AM, Avi Kivity wrote:
>> libvirt is essentially the BMC for a virtual guest.  I would suggest
>> looking at implementing an IPMI interface to libvirt and exposing it
>> to the guest through a USB RNDIS device.
>>
>
> That's the first option.  One unanswered question is what to do when the
> guest is down?  Someone should listen for IPMI events, but we can't make
> it libvirt unconditionally, since many instances of libvirt are active
> at any one time.
>
> Note the IPMI external interface needs to be migrated, like any other.

For all intents and purposes, the BMC/RSA is a separate physical machine.  If 
you really wanted to model it, you would launch two instances of QEMU.  The BMC 
instance would have a virtual NIC and would share a USB bus with the slave QEMU 
instance (probably via USBoIP).  The USB bus is how the BMC exposes IPMI to the 
guest (via a USB rndis adapter), remote media, etc.  I believe some BMC's also 
expose IPMI over i2c but that's pretty low bandwidth.

At any rate, you would have some sort of virtual hardware device that 
essentially spoke QMP to the slave instance.  You could just do virtio-serial 
and call it a day actually.

It really boils down to what you are trying to do.  If you want to just get some 
piece of software working that expects to do IPMI, the easiest thing to do is 
run IPMI in the host and use a USB rndis interface to interact with it.

I don't think there's a tremendous amount of value in QEMU making itself look 
like an IBM IMM or whatever HP/Dell's equivalent is.  As I said, these stacks 
are hugely complicated and there are better ways of doing out of band management 
(like talk to libvirt directly).

So what's really the use case here?  Would an IPMI -> libvirt bridge get you 
what you need?  I really think that's the best path forward.

Regards,

Anthony Liguori

>

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Avi Kivity]

On 05/07/2012 05:30 PM, Anthony Liguori wrote:
> On 05/06/2012 09:39 AM, Avi Kivity wrote:
>> On 05/06/2012 05:35 PM, Anthony Liguori wrote:
>>> On 05/06/2012 08:11 AM, Avi Kivity wrote:
>>> libvirt is essentially the BMC for a virtual guest.  I would suggest
>>> looking at implementing an IPMI interface to libvirt and exposing it
>>> to the guest through a USB RNDIS device.
>>>
>>
>> That's the first option.  One unanswered question is what to do when the
>> guest is down?  Someone should listen for IPMI events, but we can't make
>> it libvirt unconditionally, since many instances of libvirt are active
>> at any one time.
>>
>> Note the IPMI external interface needs to be migrated, like any other.
>
> For all intents and purposes, the BMC/RSA is a separate physical
> machine.  

That's true for any other card on a machine.

> If you really wanted to model it, you would launch two instances of
> QEMU.  The BMC instance would have a virtual NIC and would share a USB
> bus with the slave QEMU instance (probably via USBoIP).  The USB bus
> is how the BMC exposes IPMI to the guest (via a USB rndis adapter),
> remote media, etc.  I believe some BMC's also expose IPMI over i2c but
> that's pretty low bandwidth.

That is one way to do it.  Figure out the interactions between two
different parts in a machine, define an interface for them to
communicate, and split them into two processes.  We don't usually do
that; I believe your motivation is that the two have different power
domains (but then so do NICs with wake-on-LAN support).

> At any rate, you would have some sort of virtual hardware device that
> essentially spoke QMP to the slave instance.  You could just do
> virtio-serial and call it a day actually.

Sorry I lost you.  Which is the master and which is the slave?

> It really boils down to what you are trying to do.  If you want to
> just get some piece of software working that expects to do IPMI, the
> easiest thing to do is run IPMI in the host and use a USB rndis
> interface to interact with it.

That would be most strange.  A remote client connecting to the IPMI
interface would control the power level of the host, not the guest.

> I don't think there's a tremendous amount of value in QEMU making
> itself look like an IBM IMM or whatever HP/Dell's equivalent is.  As I
> said, these stacks are hugely complicated and there are better ways of
> doing out of band management (like talk to libvirt directly).

I have to agree here.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Anthony Liguori]

On 05/07/2012 09:44 AM, Avi Kivity wrote:
> On 05/07/2012 05:30 PM, Anthony Liguori wrote:
>> On 05/06/2012 09:39 AM, Avi Kivity wrote:
>>> On 05/06/2012 05:35 PM, Anthony Liguori wrote:
>>>> On 05/06/2012 08:11 AM, Avi Kivity wrote:
>>>> libvirt is essentially the BMC for a virtual guest.  I would suggest
>>>> looking at implementing an IPMI interface to libvirt and exposing it
>>>> to the guest through a USB RNDIS device.
>>>>
>>>
>>> That's the first option.  One unanswered question is what to do when the
>>> guest is down?  Someone should listen for IPMI events, but we can't make
>>> it libvirt unconditionally, since many instances of libvirt are active
>>> at any one time.
>>>
>>> Note the IPMI external interface needs to be migrated, like any other.
>>
>> For all intents and purposes, the BMC/RSA is a separate physical
>> machine.
>
> That's true for any other card on a machine.

It has a separate power source for all intents and purposes.  If you think of it 
in QOM terms, what connects the nodes together ultimately is the "Vcc" pin that 
travels across all devices.  The RTC is a little special because it has a 
battery backed CMOS/clock but it's also handled specially.

The BMC does not share Vcc.  It's no different than a separate physical box.  It 
just shares a couple buses.

>> If you really wanted to model it, you would launch two instances of
>> QEMU.  The BMC instance would have a virtual NIC and would share a USB
>> bus with the slave QEMU instance (probably via USBoIP).  The USB bus
>> is how the BMC exposes IPMI to the guest (via a USB rndis adapter),
>> remote media, etc.  I believe some BMC's also expose IPMI over i2c but
>> that's pretty low bandwidth.
>
> That is one way to do it.  Figure out the interactions between two
> different parts in a machine, define an interface for them to
> communicate, and split them into two processes.  We don't usually do
> that; I believe your motivation is that the two have different power
> domains (but then so do NICs with wake-on-LAN support).

The power still comes from the PCI bus.

Think of something like a blade center.  Each individual blade does not have 
it's own BMC.  There's a single common BMC that provides an IPMI interface for 
all blades.  Yet each blade still sees an IPMI interface via a USB rndis device.

You can rip out the memory, PCI devices, etc. from a box while the Power is in 
and the BMC will be unaffected.

>
>> At any rate, you would have some sort of virtual hardware device that
>> essentially spoke QMP to the slave instance.  You could just do
>> virtio-serial and call it a day actually.
>
> Sorry I lost you.  Which is the master and which is the slave?

The BMC is the master, system being controlled is the slave.

>
>> It really boils down to what you are trying to do.  If you want to
>> just get some piece of software working that expects to do IPMI, the
>> easiest thing to do is run IPMI in the host and use a USB rndis
>> interface to interact with it.
>
> That would be most strange.  A remote client connecting to the IPMI
> interface would control the power level of the host, not the guest.

IPMI with a custom backend is what I mean.  That's what I mean by an IPMI -> 
libvirt bridge.  You could build a libvirt client that exposes an IPMI interface 
and when you issue IPMI commands, it translate it to libvirt operations.

This can run as a normal process on the host and then network it to the guest 
via an emulated USB rndis device.  Existing software on the guest shouldn't be 
able to tell the difference as long as it doesn't try to use I2C to talk to the BMC.

>
>> I don't think there's a tremendous amount of value in QEMU making
>> itself look like an IBM IMM or whatever HP/Dell's equivalent is.  As I
>> said, these stacks are hugely complicated and there are better ways of
>> doing out of band management (like talk to libvirt directly).
>
> I have to agree here.
>

Regards,

Anthony Liguori

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Avi Kivity]

On 05/07/2012 05:55 PM, Anthony Liguori wrote:
>>> For all intents and purposes, the BMC/RSA is a separate physical
>>> machine.
>>
>> That's true for any other card on a machine.
>
>
> It has a separate power source for all intents and purposes.  If you
> think of it in QOM terms, what connects the nodes together ultimately
> is the "Vcc" pin that travels across all devices.  The RTC is a little
> special because it has a battery backed CMOS/clock but it's also
> handled specially.

And we fail to emulate it correctly as well, wrt. alarms.

>
> The BMC does not share Vcc.  It's no different than a separate
> physical box.  It just shares a couple buses.

It controls the main power place, reset line, can read VGA and emulate
keyboard, seems pretty well integrated.

>> That is one way to do it.  Figure out the interactions between two
>> different parts in a machine, define an interface for them to
>> communicate, and split them into two processes.  We don't usually do
>> that; I believe your motivation is that the two have different power
>> domains (but then so do NICs with wake-on-LAN support).
>
> The power still comes from the PCI bus.

Maybe.  But it's on when the rest of the machine is off.  So Vcc is not
shared.

>
> Think of something like a blade center.  Each individual blade does
> not have it's own BMC.  There's a single common BMC that provides an
> IPMI interface for all blades.  Yet each blade still sees an IPMI
> interface via a USB rndis device.
>
> You can rip out the memory, PCI devices, etc. from a box while the
> Power is in and the BMC will be unaffected.
>
>>
>>> At any rate, you would have some sort of virtual hardware device that
>>> essentially spoke QMP to the slave instance.  You could just do
>>> virtio-serial and call it a day actually.
>>
>> Sorry I lost you.  Which is the master and which is the slave?
>
> The BMC is the master, system being controlled is the slave.

Ah okay.  It also has to read the VGA output (say via vnc) and supply
keyboard input (via sendkey).

>>> It really boils down to what you are trying to do.  If you want to
>>> just get some piece of software working that expects to do IPMI, the
>>> easiest thing to do is run IPMI in the host and use a USB rndis
>>> interface to interact with it.
>>
>> That would be most strange.  A remote client connecting to the IPMI
>> interface would control the power level of the host, not the guest.
>
> IPMI with a custom backend is what I mean.  That's what I mean by an
> IPMI -> libvirt bridge.  You could build a libvirt client that exposes
> an IPMI interface and when you issue IPMI commands, it translate it to
> libvirt operations.
>
> This can run as a normal process on the host and then network it to
> the guest via an emulated USB rndis device.  Existing software on the
> guest shouldn't be able to tell the difference as long as it doesn't
> try to use I2C to talk to the BMC.

I still like the single process solution, it is more in line with the
rest of qemu and handles live migration better.  But even better would
be not to do this at all, and satisfy the remote management requirements
using the existing tools.

-- 
error compiling committee.c: too many arguments to function

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Anthony Liguori]

On 05/07/2012 10:11 AM, Avi Kivity wrote:
> On 05/07/2012 05:55 PM, Anthony Liguori wrote:
>>>> For all intents and purposes, the BMC/RSA is a separate physical
>>>> machine.
>>>
>>> That's true for any other card on a machine.
>>
>>
>> It has a separate power source for all intents and purposes.  If you
>> think of it in QOM terms, what connects the nodes together ultimately
>> is the "Vcc" pin that travels across all devices.  The RTC is a little
>> special because it has a battery backed CMOS/clock but it's also
>> handled specially.
>
> And we fail to emulate it correctly as well, wrt. alarms.
>
>>
>> The BMC does not share Vcc.  It's no different than a separate
>> physical box.  It just shares a couple buses.
>
> It controls the main power place, reset line, can read VGA and emulate
> keyboard, seems pretty well integrated.

Emulating the keyboard is done through USB.  How the VGA thing works is very 
vendor dependent.  The VGA snooping can happen as part of the display path 
(essentially connected via a VGA cable) or it can be a side-band using a special 
graphics adapter.  I think QEMU VNC emulation is a pretty good analogy actually.

>
>>> That is one way to do it.  Figure out the interactions between two
>>> different parts in a machine, define an interface for them to
>>> communicate, and split them into two processes.  We don't usually do
>>> that; I believe your motivation is that the two have different power
>>> domains (but then so do NICs with wake-on-LAN support).
>>
>> The power still comes from the PCI bus.
>
> Maybe.  But it's on when the rest of the machine is off.  So Vcc is not
> shared.

That's all plumbed through the PCI bus FWIW.

>
>>
>> Think of something like a blade center.  Each individual blade does
>> not have it's own BMC.  There's a single common BMC that provides an
>> IPMI interface for all blades.  Yet each blade still sees an IPMI
>> interface via a USB rndis device.
>>
>> You can rip out the memory, PCI devices, etc. from a box while the
>> Power is in and the BMC will be unaffected.
>>
>>>
>>>> At any rate, you would have some sort of virtual hardware device that
>>>> essentially spoke QMP to the slave instance.  You could just do
>>>> virtio-serial and call it a day actually.
>>>
>>> Sorry I lost you.  Which is the master and which is the slave?
>>
>> The BMC is the master, system being controlled is the slave.
>
> Ah okay.  It also has to read the VGA output (say via vnc) and supply
> keyboard input (via sendkey).

Right, QMP + VNC is a pretty accurate analogy.

>>>> It really boils down to what you are trying to do.  If you want to
>>>> just get some piece of software working that expects to do IPMI, the
>>>> easiest thing to do is run IPMI in the host and use a USB rndis
>>>> interface to interact with it.
>>>
>>> That would be most strange.  A remote client connecting to the IPMI
>>> interface would control the power level of the host, not the guest.
>>
>> IPMI with a custom backend is what I mean.  That's what I mean by an
>> IPMI ->  libvirt bridge.  You could build a libvirt client that exposes
>> an IPMI interface and when you issue IPMI commands, it translate it to
>> libvirt operations.
>>
>> This can run as a normal process on the host and then network it to
>> the guest via an emulated USB rndis device.  Existing software on the
>> guest shouldn't be able to tell the difference as long as it doesn't
>> try to use I2C to talk to the BMC.
>
> I still like the single process solution, it is more in line with the
> rest of qemu and handles live migration better.

Two QEMU processes could be migrated in unison if you really wanted to support 
that...

With qemu-system-mips/sh4 you could probably even run the real BMC software 
stack if you were so inclined :-)

> But even better would
> be not to do this at all, and satisfy the remote management requirements
> using the existing tools.

Right.

Regards,

Anthony Liguori

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Corey Minyard]

I think we are getting a little out of hand here, and we are mixing up 
concepts :).

There are lots of things IPMI *can* do (including serial access, VGA 
snooping, LAN access, etc.) but I don't see any value it that.  The main 
thing here is to emulate the interface to the guest.  OOB management is 
really more appropriately handled with libvirt.  How the BMC integrates 
into the hardware varies a *lot* between systems, but it's really kind 
of irrelevant.  (Well, almost irrelevant, BMCs can provide a direct I2C 
messaging capability, and that may matter.)

A guest can have one (or more) of a number of interfaces (that are all 
fairly bad, unfortunately).  The standard ones are called "KCS", "BT" 
and "SMIC" and they generally are directly on the ISA bus, but are in 
memory on non-x86 boxes (and on some x86 boxes) and sometimes on the PCI 
bus.  Some systems also have interfaces over I2C, but that hasn't really 
caught on.  Others have interfaces over serial ports, and that 
unfortunately has caught on in the ATCA world.  And there are at least 3 
different basic types of serial port interfaces with sub-variants :(.  
I'm not sure what the USB rndis device is, but I'll look at it.  But 
there is no IPMI over USB.

The big things a guest can do are sensor management, watchdog timer, 
reset, and power control.  In complicated IPMI-based systems like ATCA, 
a guest may want to send messages through its local IPMI controller to 
other guest's IPMI controllers or to a central BMC that runs an entire 
chassis of systems.  So that may need to be supported, depending on what 
people want to do and how hard they want to work on it.

My proposal is to start small, with just a local interface, watchdog 
timer, sensors and power control.  But have an architecture that would 
allow external LAN access, tying BMCs in different qemu instances 
together, perhaps serial over IPMI, and other things of that nature.

-corey


On 05/07/2012 10:21 AM, Anthony Liguori wrote:
> On 05/07/2012 10:11 AM, Avi Kivity wrote:
>> On 05/07/2012 05:55 PM, Anthony Liguori wrote:
>>>>> For all intents and purposes, the BMC/RSA is a separate physical
>>>>> machine.
>>>>
>>>> That's true for any other card on a machine.
>>>
>>>
>>> It has a separate power source for all intents and purposes.  If you
>>> think of it in QOM terms, what connects the nodes together ultimately
>>> is the "Vcc" pin that travels across all devices.  The RTC is a little
>>> special because it has a battery backed CMOS/clock but it's also
>>> handled specially.
>>
>> And we fail to emulate it correctly as well, wrt. alarms.
>>
>>>
>>> The BMC does not share Vcc.  It's no different than a separate
>>> physical box.  It just shares a couple buses.
>>
>> It controls the main power place, reset line, can read VGA and emulate
>> keyboard, seems pretty well integrated.
>
> Emulating the keyboard is done through USB.  How the VGA thing works 
> is very vendor dependent.  The VGA snooping can happen as part of the 
> display path (essentially connected via a VGA cable) or it can be a 
> side-band using a special graphics adapter.  I think QEMU VNC 
> emulation is a pretty good analogy actually.
>
>>
>>>> That is one way to do it.  Figure out the interactions between two
>>>> different parts in a machine, define an interface for them to
>>>> communicate, and split them into two processes.  We don't usually do
>>>> that; I believe your motivation is that the two have different power
>>>> domains (but then so do NICs with wake-on-LAN support).
>>>
>>> The power still comes from the PCI bus.
>>
>> Maybe.  But it's on when the rest of the machine is off.  So Vcc is not
>> shared.
>
> That's all plumbed through the PCI bus FWIW.
>
>>
>>>
>>> Think of something like a blade center.  Each individual blade does
>>> not have it's own BMC.  There's a single common BMC that provides an
>>> IPMI interface for all blades.  Yet each blade still sees an IPMI
>>> interface via a USB rndis device.
>>>
>>> You can rip out the memory, PCI devices, etc. from a box while the
>>> Power is in and the BMC will be unaffected.
>>>
>>>>
>>>>> At any rate, you would have some sort of virtual hardware device that
>>>>> essentially spoke QMP to the slave instance.  You could just do
>>>>> virtio-serial and call it a day actually.
>>>>
>>>> Sorry I lost you.  Which is the master and which is the slave?
>>>
>>> The BMC is the master, system being controlled is the slave.
>>
>> Ah okay.  It also has to read the VGA output (say via vnc) and supply
>> keyboard input (via sendkey).
>
> Right, QMP + VNC is a pretty accurate analogy.
>
>>>>> It really boils down to what you are trying to do.  If you want to
>>>>> just get some piece of software working that expects to do IPMI, the
>>>>> easiest thing to do is run IPMI in the host and use a USB rndis
>>>>> interface to interact with it.
>>>>
>>>> That would be most strange.  A remote client connecting to the IPMI
>>>> interface would control the power level of the host, not the guest.
>>>
>>> IPMI with a custom backend is what I mean.  That's what I mean by an
>>> IPMI ->  libvirt bridge.  You could build a libvirt client that exposes
>>> an IPMI interface and when you issue IPMI commands, it translate it to
>>> libvirt operations.
>>>
>>> This can run as a normal process on the host and then network it to
>>> the guest via an emulated USB rndis device.  Existing software on the
>>> guest shouldn't be able to tell the difference as long as it doesn't
>>> try to use I2C to talk to the BMC.
>>
>> I still like the single process solution, it is more in line with the
>> rest of qemu and handles live migration better.
>
> Two QEMU processes could be migrated in unison if you really wanted to 
> support that...
>
> With qemu-system-mips/sh4 you could probably even run the real BMC 
> software stack if you were so inclined :-)
>
>> But even better would
>> be not to do this at all, and satisfy the remote management requirements
>> using the existing tools.
>
> Right.
>
> Regards,
>
> Anthony Liguori

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Dave Allan]

FWIW, the idea of an IPMI interface to VMs was proposed for libvirt
not too long ago.  See:

https://bugzilla.redhat.com/show_bug.cgi?id=815136

Dave

On Mon, May 07, 2012 at 01:07:45PM -0500, Corey Minyard wrote:
> I think we are getting a little out of hand here, and we are mixing
> up concepts :).
> 
> There are lots of things IPMI *can* do (including serial access, VGA
> snooping, LAN access, etc.) but I don't see any value it that.  The
> main thing here is to emulate the interface to the guest.  OOB
> management is really more appropriately handled with libvirt.  How
> the BMC integrates into the hardware varies a *lot* between systems,
> but it's really kind of irrelevant.  (Well, almost irrelevant, BMCs
> can provide a direct I2C messaging capability, and that may matter.)
> 
> A guest can have one (or more) of a number of interfaces (that are
> all fairly bad, unfortunately).  The standard ones are called "KCS",
> "BT" and "SMIC" and they generally are directly on the ISA bus, but
> are in memory on non-x86 boxes (and on some x86 boxes) and sometimes
> on the PCI bus.  Some systems also have interfaces over I2C, but
> that hasn't really caught on.  Others have interfaces over serial
> ports, and that unfortunately has caught on in the ATCA world.  And
> there are at least 3 different basic types of serial port interfaces
> with sub-variants :(.  I'm not sure what the USB rndis device is,
> but I'll look at it.  But there is no IPMI over USB.
> 
> The big things a guest can do are sensor management, watchdog timer,
> reset, and power control.  In complicated IPMI-based systems like
> ATCA, a guest may want to send messages through its local IPMI
> controller to other guest's IPMI controllers or to a central BMC
> that runs an entire chassis of systems.  So that may need to be
> supported, depending on what people want to do and how hard they
> want to work on it.
> 
> My proposal is to start small, with just a local interface, watchdog
> timer, sensors and power control.  But have an architecture that
> would allow external LAN access, tying BMCs in different qemu
> instances together, perhaps serial over IPMI, and other things of
> that nature.
> 
> -corey
> 
> 
> On 05/07/2012 10:21 AM, Anthony Liguori wrote:
> >On 05/07/2012 10:11 AM, Avi Kivity wrote:
> >>On 05/07/2012 05:55 PM, Anthony Liguori wrote:
> >>>>>For all intents and purposes, the BMC/RSA is a separate physical
> >>>>>machine.
> >>>>
> >>>>That's true for any other card on a machine.
> >>>
> >>>
> >>>It has a separate power source for all intents and purposes.  If you
> >>>think of it in QOM terms, what connects the nodes together ultimately
> >>>is the "Vcc" pin that travels across all devices.  The RTC is a little
> >>>special because it has a battery backed CMOS/clock but it's also
> >>>handled specially.
> >>
> >>And we fail to emulate it correctly as well, wrt. alarms.
> >>
> >>>
> >>>The BMC does not share Vcc.  It's no different than a separate
> >>>physical box.  It just shares a couple buses.
> >>
> >>It controls the main power place, reset line, can read VGA and emulate
> >>keyboard, seems pretty well integrated.
> >
> >Emulating the keyboard is done through USB.  How the VGA thing
> >works is very vendor dependent.  The VGA snooping can happen as
> >part of the display path (essentially connected via a VGA cable)
> >or it can be a side-band using a special graphics adapter.  I
> >think QEMU VNC emulation is a pretty good analogy actually.
> >
> >>
> >>>>That is one way to do it.  Figure out the interactions between two
> >>>>different parts in a machine, define an interface for them to
> >>>>communicate, and split them into two processes.  We don't usually do
> >>>>that; I believe your motivation is that the two have different power
> >>>>domains (but then so do NICs with wake-on-LAN support).
> >>>
> >>>The power still comes from the PCI bus.
> >>
> >>Maybe.  But it's on when the rest of the machine is off.  So Vcc is not
> >>shared.
> >
> >That's all plumbed through the PCI bus FWIW.
> >
> >>
> >>>
> >>>Think of something like a blade center.  Each individual blade does
> >>>not have it's own BMC.  There's a single common BMC that provides an
> >>>IPMI interface for all blades.  Yet each blade still sees an IPMI
> >>>interface via a USB rndis device.
> >>>
> >>>You can rip out the memory, PCI devices, etc. from a box while the
> >>>Power is in and the BMC will be unaffected.
> >>>
> >>>>
> >>>>>At any rate, you would have some sort of virtual hardware device that
> >>>>>essentially spoke QMP to the slave instance.  You could just do
> >>>>>virtio-serial and call it a day actually.
> >>>>
> >>>>Sorry I lost you.  Which is the master and which is the slave?
> >>>
> >>>The BMC is the master, system being controlled is the slave.
> >>
> >>Ah okay.  It also has to read the VGA output (say via vnc) and supply
> >>keyboard input (via sendkey).
> >
> >Right, QMP + VNC is a pretty accurate analogy.
> >
> >>>>>It really boils down to what you are trying to do.  If you want to
> >>>>>just get some piece of software working that expects to do IPMI, the
> >>>>>easiest thing to do is run IPMI in the host and use a USB rndis
> >>>>>interface to interact with it.
> >>>>
> >>>>That would be most strange.  A remote client connecting to the IPMI
> >>>>interface would control the power level of the host, not the guest.
> >>>
> >>>IPMI with a custom backend is what I mean.  That's what I mean by an
> >>>IPMI ->  libvirt bridge.  You could build a libvirt client that exposes
> >>>an IPMI interface and when you issue IPMI commands, it translate it to
> >>>libvirt operations.
> >>>
> >>>This can run as a normal process on the host and then network it to
> >>>the guest via an emulated USB rndis device.  Existing software on the
> >>>guest shouldn't be able to tell the difference as long as it doesn't
> >>>try to use I2C to talk to the BMC.
> >>
> >>I still like the single process solution, it is more in line with the
> >>rest of qemu and handles live migration better.
> >
> >Two QEMU processes could be migrated in unison if you really
> >wanted to support that...
> >
> >With qemu-system-mips/sh4 you could probably even run the real BMC
> >software stack if you were so inclined :-)
> >
> >>But even better would
> >>be not to do this at all, and satisfy the remote management requirements
> >>using the existing tools.
> >
> >Right.
> >
> >Regards,
> >
> >Anthony Liguori
> 
> 

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Corey Minyard]

On 05/07/2012 02:45 PM, Dave Allan wrote:
> FWIW, the idea of an IPMI interface to VMs was proposed for libvirt
> not too long ago.  See:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=815136

Well, it wouldn't be to hard to do.  I already have working emulation 
code that does the IPMI LAN interface (including the IPMI 2.0 stuff for 
more reasonable security).  I have a KCS interface and a minimal IPMI 
controller working in KVM, though I'm not quite sure the best final way 
to hook it in.

Configuration is going to be the hardest part, but a minimal 
configuration for providing basic management would be easy.

-corey

> Dave
>
> On Mon, May 07, 2012 at 01:07:45PM -0500, Corey Minyard wrote:
>> I think we are getting a little out of hand here, and we are mixing
>> up concepts :).
>>
>> There are lots of things IPMI *can* do (including serial access, VGA
>> snooping, LAN access, etc.) but I don't see any value it that.  The
>> main thing here is to emulate the interface to the guest.  OOB
>> management is really more appropriately handled with libvirt.  How
>> the BMC integrates into the hardware varies a *lot* between systems,
>> but it's really kind of irrelevant.  (Well, almost irrelevant, BMCs
>> can provide a direct I2C messaging capability, and that may matter.)
>>
>> A guest can have one (or more) of a number of interfaces (that are
>> all fairly bad, unfortunately).  The standard ones are called "KCS",
>> "BT" and "SMIC" and they generally are directly on the ISA bus, but
>> are in memory on non-x86 boxes (and on some x86 boxes) and sometimes
>> on the PCI bus.  Some systems also have interfaces over I2C, but
>> that hasn't really caught on.  Others have interfaces over serial
>> ports, and that unfortunately has caught on in the ATCA world.  And
>> there are at least 3 different basic types of serial port interfaces
>> with sub-variants :(.  I'm not sure what the USB rndis device is,
>> but I'll look at it.  But there is no IPMI over USB.
>>
>> The big things a guest can do are sensor management, watchdog timer,
>> reset, and power control.  In complicated IPMI-based systems like
>> ATCA, a guest may want to send messages through its local IPMI
>> controller to other guest's IPMI controllers or to a central BMC
>> that runs an entire chassis of systems.  So that may need to be
>> supported, depending on what people want to do and how hard they
>> want to work on it.
>>
>> My proposal is to start small, with just a local interface, watchdog
>> timer, sensors and power control.  But have an architecture that
>> would allow external LAN access, tying BMCs in different qemu
>> instances together, perhaps serial over IPMI, and other things of
>> that nature.
>>
>> -corey
>>
>>
>> On 05/07/2012 10:21 AM, Anthony Liguori wrote:
>>> On 05/07/2012 10:11 AM, Avi Kivity wrote:
>>>> On 05/07/2012 05:55 PM, Anthony Liguori wrote:
>>>>>>> For all intents and purposes, the BMC/RSA is a separate physical
>>>>>>> machine.
>>>>>> That's true for any other card on a machine.
>>>>>
>>>>> It has a separate power source for all intents and purposes.  If you
>>>>> think of it in QOM terms, what connects the nodes together ultimately
>>>>> is the "Vcc" pin that travels across all devices.  The RTC is a little
>>>>> special because it has a battery backed CMOS/clock but it's also
>>>>> handled specially.
>>>> And we fail to emulate it correctly as well, wrt. alarms.
>>>>
>>>>> The BMC does not share Vcc.  It's no different than a separate
>>>>> physical box.  It just shares a couple buses.
>>>> It controls the main power place, reset line, can read VGA and emulate
>>>> keyboard, seems pretty well integrated.
>>> Emulating the keyboard is done through USB.  How the VGA thing
>>> works is very vendor dependent.  The VGA snooping can happen as
>>> part of the display path (essentially connected via a VGA cable)
>>> or it can be a side-band using a special graphics adapter.  I
>>> think QEMU VNC emulation is a pretty good analogy actually.
>>>
>>>>>> That is one way to do it.  Figure out the interactions between two
>>>>>> different parts in a machine, define an interface for them to
>>>>>> communicate, and split them into two processes.  We don't usually do
>>>>>> that; I believe your motivation is that the two have different power
>>>>>> domains (but then so do NICs with wake-on-LAN support).
>>>>> The power still comes from the PCI bus.
>>>> Maybe.  But it's on when the rest of the machine is off.  So Vcc is not
>>>> shared.
>>> That's all plumbed through the PCI bus FWIW.
>>>
>>>>> Think of something like a blade center.  Each individual blade does
>>>>> not have it's own BMC.  There's a single common BMC that provides an
>>>>> IPMI interface for all blades.  Yet each blade still sees an IPMI
>>>>> interface via a USB rndis device.
>>>>>
>>>>> You can rip out the memory, PCI devices, etc. from a box while the
>>>>> Power is in and the BMC will be unaffected.
>>>>>
>>>>>>> At any rate, you would have some sort of virtual hardware device that
>>>>>>> essentially spoke QMP to the slave instance.  You could just do
>>>>>>> virtio-serial and call it a day actually.
>>>>>> Sorry I lost you.  Which is the master and which is the slave?
>>>>> The BMC is the master, system being controlled is the slave.
>>>> Ah okay.  It also has to read the VGA output (say via vnc) and supply
>>>> keyboard input (via sendkey).
>>> Right, QMP + VNC is a pretty accurate analogy.
>>>
>>>>>>> It really boils down to what you are trying to do.  If you want to
>>>>>>> just get some piece of software working that expects to do IPMI, the
>>>>>>> easiest thing to do is run IPMI in the host and use a USB rndis
>>>>>>> interface to interact with it.
>>>>>> That would be most strange.  A remote client connecting to the IPMI
>>>>>> interface would control the power level of the host, not the guest.
>>>>> IPMI with a custom backend is what I mean.  That's what I mean by an
>>>>> IPMI ->   libvirt bridge.  You could build a libvirt client that exposes
>>>>> an IPMI interface and when you issue IPMI commands, it translate it to
>>>>> libvirt operations.
>>>>>
>>>>> This can run as a normal process on the host and then network it to
>>>>> the guest via an emulated USB rndis device.  Existing software on the
>>>>> guest shouldn't be able to tell the difference as long as it doesn't
>>>>> try to use I2C to talk to the BMC.
>>>> I still like the single process solution, it is more in line with the
>>>> rest of qemu and handles live migration better.
>>> Two QEMU processes could be migrated in unison if you really
>>> wanted to support that...
>>>
>>> With qemu-system-mips/sh4 you could probably even run the real BMC
>>> software stack if you were so inclined :-)
>>>
>>>> But even better would
>>>> be not to do this at all, and satisfy the remote management requirements
>>>> using the existing tools.
>>> Right.
>>>
>>> Regards,
>>>
>>> Anthony Liguori
>>

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Anthony Liguori]

On 05/07/2012 01:07 PM, Corey Minyard wrote:
> I think we are getting a little out of hand here, and we are mixing up concepts :).
>
> There are lots of things IPMI *can* do (including serial access, VGA snooping,
> LAN access, etc.) but I don't see any value it that. The main thing here is to
> emulate the interface to the guest. OOB management is really more appropriately
> handled with libvirt. How the BMC integrates into the hardware varies a *lot*
> between systems, but it's really kind of irrelevant. (Well, almost irrelevant,
> BMCs can provide a direct I2C messaging capability, and that may matter.)
>
> A guest can have one (or more) of a number of interfaces (that are all fairly
> bad, unfortunately). The standard ones are called "KCS", "BT" and "SMIC" and
> they generally are directly on the ISA bus, but are in memory on non-x86 boxes
> (and on some x86 boxes) and sometimes on the PCI bus. Some systems also have
> interfaces over I2C, but that hasn't really caught on. Others have interfaces
> over serial ports, and that unfortunately has caught on in the ATCA world. And
> there are at least 3 different basic types of serial port interfaces with
> sub-variants :(. I'm not sure what the USB rndis device is, but I'll look at it.
> But there is no IPMI over USB.

USB rndis == USB network adapater.  It's just seen by the machine as IPMI over LAN.

Regards,

Anthony Liguori

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Stefan Hajnoczi]

On Mon, May 7, 2012 at 3:30 PM, Anthony Liguori <anthony@codemonkey.ws> wrote:
> On 05/06/2012 09:39 AM, Avi Kivity wrote:
>>
>> On 05/06/2012 05:35 PM, Anthony Liguori wrote:
> So what's really the use case here?  Would an IPMI -> libvirt bridge get you
> what you need?  I really think that's the best path forward.

I'm still curious about this and didn't see it answered in the thread.
 You mentioned "watchdog timer, sensors and power control" but what
exactly should they expose?

Do you want to expose host sensors - the challenge is that they don't
reflect the hardware that the virtual machine sees?  Or do you want to
have synthetic sensors - which virtual sensors are useful to have?

Is the IPMI watchdog useful, QEMU already supports the i6300esb PCI
watchdog (see qemu -watchdog option documentation)?

Some use cases that illustrate how the guest is going to use IPMI
would be interesting and could help guide the discussion.

Stefan

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Corey Minyard]

[-- Attachment #1: Type: text/plain, Size: 4045 bytes --]

On 05/18/2012 08:08 AM, Stefan Hajnoczi wrote:
> On Mon, May 7, 2012 at 3:30 PM, Anthony Liguori<anthony@codemonkey.ws>  wrote:
>> On 05/06/2012 09:39 AM, Avi Kivity wrote:
>>> On 05/06/2012 05:35 PM, Anthony Liguori wrote:
>> So what's really the use case here?  Would an IPMI ->  libvirt bridge get you
>> what you need?  I really think that's the best path forward.
> I'm still curious about this and didn't see it answered in the thread.
>   You mentioned "watchdog timer, sensors and power control" but what
> exactly should they expose?

Many of our customer use the standard watchdog timer and power controls 
on IPMI.  Basically, the watchdog timer needs to be able to power off, 
power cycle, reset, and send an NMI.  The IPMI watchdog can be 
configured to do all those things.  (The NMI is for a "pretimeout" that 
generally triggers a panic.)

Power control is no big surprise.  It's true that you have ACPI to do 
this, but that's not terribly useful on non-x86 (and non ia64, I 
suppose) systems.

>
> Do you want to expose host sensors - the challenge is that they don't
> reflect the hardware that the virtual machine sees?  Or do you want to
> have synthetic sensors - which virtual sensors are useful to have?

A few sensors, notably the watchdog timer sensor, are not synthetic.  
The others generally are.  Sensors cover things beyond just power and 
temperature.  Important other ones in IPMI deal with the presence of 
FRUs in the system, BIOS/OS state, and intrusion detection.

Two main reasons for synthesized sensors exist.  One, in a legacy 
situation, is to "fool" the management system into thinking everything 
is ok, since it is expecting to see these sensors with specific values.  
You could even possibly reflect the state of real sensors, mapping them 
somehow, if you wanted.

The other reason is for testing.  It's pretty hard to induce some of 
these sensors to go out of range in a real system.  So without 
simulation, you need a hack-ed up management controller in a real system 
to truly test your software.

>
> Is the IPMI watchdog useful, QEMU already supports the i6300esb PCI
> watchdog (see qemu -watchdog option documentation)?

That's only useful if you can simulate an i6300esb.  Probably not 
possible on non-x86.  Plus legacy systems may be expecting the 
capabilities of the IPMI watchdog.

>
> Some use cases that illustrate how the guest is going to use IPMI
> would be interesting and could help guide the discussion.

You are probably right.  I've mentioned a couple above.

One other possible one, that someone else mentioned, is the ability to 
control a VM using standard tools like ipmitool or OpenIPMI over network 
interfaces.  People may already have management systems that are 
designed around IPMI, and it would make a move to virtual machines 
easier.  This was the request mentioned that was in the Redhat database.

Such a capability would require re-thinking things a bit.  You have two 
basic options that I see.  Either have the "management controller" run 
outside qemu and define a simple interface to it, or modify qemu to be 
able to run the management controller internally.

My preference in to run it outside qemu, for the following reasons:

  * It decouples things that are IPMI internals from having to go
    through getting into qemu repositories.
  * The management controller then becomes useful for a number of other
    purposes.  I already have one mostly done that I use for testing. 
    It could be used by other VMs.
  * A full management controller is a fairly big piece of software,
    especially if you include the network access.  It probably doesn't
    belong in qemu.
  * No offense, I don't want to muck around inside qemu to accomplish
    this :).

There are, of course, disadvantages.  Some I can think of:

  * There can be confusion about which versions of two things work together.
  * Overall, it is more complex to get working.
  * There are possible security implications.

So I guess those are points we can talk about...

-corey


[-- Attachment #2: Type: text/html, Size: 5717 bytes --]

Re: [Qemu-devel] Adding an IPMI BMC device to KVM

[Corey Minyard]

On 05/18/2012 08:08 AM, Stefan Hajnoczi wrote:
> On Mon, May 7, 2012 at 3:30 PM, Anthony Liguori<anthony@codemonkey.ws>  wrote:
>> On 05/06/2012 09:39 AM, Avi Kivity wrote:
>>> On 05/06/2012 05:35 PM, Anthony Liguori wrote:
>> So what's really the use case here?  Would an IPMI ->  libvirt bridge get you
>> what you need?  I really think that's the best path forward.
Sorry for sending this twice.  I thought my mailer was set to disable HTML.

Many of our customer use the standard watchdog timer and power controls 
on IPMI.  Basically, the watchdog timer needs to be able to power off, 
power cycle, reset, and send an NMI.  The IPMI watchdog can be 
configured to do all those things.  (The NMI is for a "pretimeout" that 
generally triggers a panic.)

Power control is no big surprise.  It's true that you have ACPI to do 
this, but that's not terribly useful on non-x86 (and non ia64, I 
suppose) systems.

> Do you want to expose host sensors - the challenge is that they don't
> reflect the hardware that the virtual machine sees?  Or do you want to
> have synthetic sensors - which virtual sensors are useful to have?

A few sensors, notably the watchdog timer sensor, are not synthetic.  
The others generally are.  Sensors cover things beyond just power and 
temperature.  Important other ones in IPMI deal with the presence of 
FRUs in the system, BIOS/OS state, and intrusion detection.

Two main reasons for synthesized sensors exist.  One, in a legacy 
situation, is to "fool" the management system into thinking everything 
is ok, since it is expecting to see these sensors with specific values.  
You could even possibly reflect the state of real sensors, mapping them 
somehow, if you wanted.

The other reason is for testing.  It's pretty hard to induce some of 
these sensors to go out of range in a real system.  So without 
simulation, you need a hack-ed up management controller in a real system 
to truly test your software.

> Is the IPMI watchdog useful, QEMU already supports the i6300esb PCI
> watchdog (see qemu -watchdog option documentation)?

That's only useful if you can simulate an i6300esb.  Probably not 
possible on non-x86.  Plus legacy systems may be expecting the 
capabilities of the IPMI watchdog.

> Some use cases that illustrate how the guest is going to use IPMI
> would be interesting and could help guide the discussion.

You are probably right.  I've mentioned a couple above.

One other possible one, that someone else mentioned, is the ability to 
control a VM using standard tools like ipmitool or OpenIPMI over network 
interfaces.  People may already have management systems that are 
designed around IPMI, and it would make a move to virtual machines 
easier.  This was the request mentioned that was in the Redhat database.

Such a capability would require re-thinking things a bit.  You have two 
basic options that I see.  Either have the "management controller" run 
outside qemu and define a simple interface to it, or modify qemu to be 
able to run the management controller internally.

My preference in to run it outside qemu, for the following reasons:

  * It decouples things that are IPMI internals from having to go
    through getting into qemu repositories.
  * The management controller then becomes useful for a number of other
    purposes.  I already have one mostly done that I use for testing. 
    It could be used by other VMs.
  * A full management controller is a fairly big piece of software,
    especially if you include the network access.  It probably doesn't
    belong in qemu.
  * No offense, I don't want to muck around inside qemu to accomplish
    this :).

There are, of course, disadvantages.  Some I can think of:

  * There can be confusion about which versions of two things work together.
  * Overall, it is more complex to get working.
  * There are possible security implications.

So I guess those are points we can talk about...

-corey