How to expose various BMC chip controls ?

How to expose various BMC chip controls ?

[Benjamin Herrenschmidt]

Hi Folks !

I would like to discuss something we need to solve for BMC chips before
we start implementing a solution that you'll reject upstream :-)

So quick recap: the BMC chip is the management controller of your
server, typically some kind of specialized ARM SoC which controls a
variety of things ranging from power supply, fans, inventory, flash, to
having even a built-in GPU connected to the main server processor via
PCIe etc...

As part of the OpenBMC project, we have been contributing (along with
others from Google etc...) a bunch of upstream drivers to support the
BMC chips from Aspeed which are quite popular. There's work in progress
to also upstream other vendor(s) chip support.

For most things, we can write appropriate drivers.

However, there's a range of things for which there is no good fit for
dedicated per-function kernel drivers in the BMC:

The BMC chips have a variety of control bits and pieces (registers)
that affect the host in all sorts of ways. A few semi-random examples:

 - Scratch registers that the host BIOS can read via LPC that contain
system specific details of interest to the BIOS itself. The
content/format is specific to a given BIOS <-> BMC userspace pair.

 - Similar ones related to the built-in VGA (the one the host sees on
PCIe, it doesn't appear as a device on the BMC side). Used to pass some
wiring information to the VGA BIOS and the Linux driver among others.

 - Bits controlling which parts of the BMC the host can access remotely
via either LPC or PCIe. (Visibility of the VGA to the host, of the BMC
PCIe system-controller device, opening of various windows from the host
into the BMC address space etc...)

 - ... and a few more

Those things tend to be fairly system specific. Meaning that what's
written in them (or read from them) and when it's written in these
registers should be under userspace control on the BMC. There isn't
necessarily many registers that need to be exposed that way, here too,
which specific ones is rather system specific.

But /dev/mem isn't a solution :-)

So one simple idea I had, you tell me if it can fly, is to have some
driver for the SoC as a whole (thinking of syscon here...) expose a
selection of these things via sysfs.

The list being potentially system specific, it would be provided to
the kernel via the device-tree (binding tbd) where we would effectively
provide a list of names, register offset, start bit, bitfield size.

Any better idea ? It's a fairly simple problem, and the above solution
would be very little code, but I just don't want us to go down a rabbit
hole if some of you have religious objections to some of it :)

Cheers,
Ben.

Re: How to expose various BMC chip controls ?

[Arnd Bergmann]

On Tue, Apr 10, 2018 at 11:17 AM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> Hi Folks !
>
> I would like to discuss something we need to solve for BMC chips before
> we start implementing a solution that you'll reject upstream :-)
>
> So quick recap: the BMC chip is the management controller of your
> server, typically some kind of specialized ARM SoC which controls a
> variety of things ranging from power supply, fans, inventory, flash, to
> having even a built-in GPU connected to the main server processor via
> PCIe etc...
>
> As part of the OpenBMC project, we have been contributing (along with
> others from Google etc...) a bunch of upstream drivers to support the
> BMC chips from Aspeed which are quite popular. There's work in progress
> to also upstream other vendor(s) chip support.
>
> For most things, we can write appropriate drivers.
>
> However, there's a range of things for which there is no good fit for
> dedicated per-function kernel drivers in the BMC:
>
> The BMC chips have a variety of control bits and pieces (registers)
> that affect the host in all sorts of ways. A few semi-random examples:
>
>  - Scratch registers that the host BIOS can read via LPC that contain
> system specific details of interest to the BIOS itself. The
> content/format is specific to a given BIOS <-> BMC userspace pair.
>
>  - Similar ones related to the built-in VGA (the one the host sees on
> PCIe, it doesn't appear as a device on the BMC side). Used to pass some
> wiring information to the VGA BIOS and the Linux driver among others.
>
>  - Bits controlling which parts of the BMC the host can access remotely
> via either LPC or PCIe. (Visibility of the VGA to the host, of the BMC
> PCIe system-controller device, opening of various windows from the host
> into the BMC address space etc...)
>
>  - ... and a few more
>
> Those things tend to be fairly system specific. Meaning that what's
> written in them (or read from them) and when it's written in these
> registers should be under userspace control on the BMC. There isn't
> necessarily many registers that need to be exposed that way, here too,
> which specific ones is rather system specific.
>
> But /dev/mem isn't a solution :-)
>
> So one simple idea I had, you tell me if it can fly, is to have some
> driver for the SoC as a whole (thinking of syscon here...) expose a
> selection of these things via sysfs.
>
> The list being potentially system specific, it would be provided to
> the kernel via the device-tree (binding tbd) where we would effectively
> provide a list of names, register offset, start bit, bitfield size.
>
> Any better idea ? It's a fairly simple problem, and the above solution
> would be very little code, but I just don't want us to go down a rabbit
> hole if some of you have religious objections to some of it :)

I think the hard part is finding the right balance: traditionally, these
kinds of systems used ad-hoc interfaces like /dev/mem or random
kernel interfaces to do both very generic and very non-generic
thigns.

The initial reaction you get when posting a patch that adds
an ad-hoc interface will (should?) always be a question of "can
this be done using a generic interface?", but that doesn't mean
that the answer is always "yes", we just need to come up with
ideas of how that interface would look like and figure out together
if it's actually worth it or not.

In the cases where an ad-hoc interface is needed, I can see
two options: we can stick something in drivers/soc that exposes
it in the least ugly way, completely specific to that particular
SoC, or we can create a new subsystem from "random BMC
slave-side interfaces" and have it maintained by you all (leave
me out of the picture, except if you are looking for ideas).

In any case, I think it makes most sense to discuss new interfaces
between the people working on the competing BMC implementations,
mainly to see if some functionality can be abstracted in a common
way or not. I believe you are already working with the Nuvoton BMC
people, but I don't know much about the others. Is the
Emulex/Broadcom/Aspeed "Pilot" BMC series used anywhere
that you know of? It seems that Renesas and TI are no longer
active in this space, and that the big server vendors are either
not running Linux (e.g. HP) on their BMC or they have their
own software stack on the generic third-party BMCs (e.g. Dell).
I think Huawei also has their own BMC hardware, but no idea
what they run on it. Do you know of any others?

       Arnd

Re: How to expose various BMC chip controls ?

[Benjamin Herrenschmidt]

On Tue, 2018-04-10 at 13:29 +0200, Arnd Bergmann wrote:

> > Any better idea ? It's a fairly simple problem, and the above solution
> > would be very little code, but I just don't want us to go down a rabbit
> > hole if some of you have religious objections to some of it :)
> 
> I think the hard part is finding the right balance: traditionally, these
> kinds of systems used ad-hoc interfaces like /dev/mem or random
> kernel interfaces to do both very generic and very non-generic
> thigns.

Right. This is always the question, and for a bunch of things we *have*
used either generic interfaces or specialized drivers. There's an open
ended issue with our mailbox driver since we don't use the mailbox
framework, which we need to resolve, but otherwise, we are keen on
using existing interfaces when one exists.

> The initial reaction you get when posting a patch that adds
> an ad-hoc interface will (should?) always be a question of "can
> this be done using a generic interface?", but that doesn't mean
> that the answer is always "yes", we just need to come up with
> ideas of how that interface would look like and figure out together
> if it's actually worth it or not.

So I listed a few examples, I think Joel and Andrew can come up with a
few mores. These things, mostly because they control host-visible
things rather than BMC-visible things tend to simply not fit in
existing interfaces.

They are also sufficiently platform *and* BMC type specific that they
wouldn't fit into some kind of generic "BMC controls" driver either I
suspect.

Basically, what we need a set of named "knobs" that the BMC userspace
code can tweak based on policies set by a given system vendor and the
interactions between their BMC implementation (including OpenBMC
variant) and their BIOS/Firmware.

> In the cases where an ad-hoc interface is needed, I can see
> two options: we can stick something in drivers/soc that exposes
> it in the least ugly way, completely specific to that particular
> SoC, or we can create a new subsystem from "random BMC
> slave-side interfaces" and have it maintained by you all (leave
> me out of the picture, except if you are looking for ideas).

I don't think there enough commonality in these things to create a "BMC
slave interface" class that is anything other than a bunch of sysfs
files.

That's why I ended up with my personal conclusion/preference of just
having the syscon "driver" for the SoC just expose a table of a dozen
or so such knobs, and use the DT to describe them to abstract a bit the
difference between SoC models/revisions and how they are wired up in a
given platform.

> In any case, I think it makes most sense to discuss new interfaces
> between the people working on the competing BMC implementations,
> mainly to see if some functionality can be abstracted in a common
> way or not.

So we have some ideas of what competing BMC *chip* vendors do since
OpenBMC people communicates with other vendors.

We know a lot less about BMC SW stacks, but those are a rat nest of
closed source & GPL stuff that doesn't get upstream (or even released
sometimes ... ahem ...). Having had access to some vendor code in the
past, I can tell you that you really don't want to look at what's going
on in there ;-)

Now, lets be clear, these is really for a few (we hope maybe a handful
or two as described in my examples) knobs that really don't fit
anywhere else. By their very nature they are very specific to a given
BMC HW implementation.

>  I believe you are already working with the Nuvoton BMC
> people, but I don't know much about the others.

Nuvoton and Aspeed own the market as far as I know these days...

>  Is the
> Emulex/Broadcom/Aspeed "Pilot" BMC series used anywhere
> that you know of? 

I don't know.

> It seems that Renesas and TI are no longer
> active in this space, and that the big server vendors are either
> not running Linux (e.g. HP) on their BMC or they have their
> own software stack on the generic third-party BMCs (e.g. Dell).
> I think Huawei also has their own BMC hardware, but no idea
> what they run on it. Do you know of any others?

Nope. That said, I don't expect much commonality for this specific
problem.

Cheers,
Ben.

>        Arnd

Re: How to expose various BMC chip controls ?

[Arnd Bergmann]

On Tue, Apr 10, 2018 at 2:57 PM, Benjamin Herrenschmidt
<benh@kernel.crashing.org> wrote:
> On Tue, 2018-04-10 at 13:29 +0200, Arnd Bergmann wrote:

>> In the cases where an ad-hoc interface is needed, I can see
>> two options: we can stick something in drivers/soc that exposes
>> it in the least ugly way, completely specific to that particular
>> SoC, or we can create a new subsystem from "random BMC
>> slave-side interfaces" and have it maintained by you all (leave
>> me out of the picture, except if you are looking for ideas).
>
> I don't think there enough commonality in these things to create a "BMC
> slave interface" class that is anything other than a bunch of sysfs
> files.
>
> That's why I ended up with my personal conclusion/preference of just
> having the syscon "driver" for the SoC just expose a table of a dozen
> or so such knobs, and use the DT to describe them to abstract a bit the
> difference between SoC models/revisions and how they are wired up in a
> given platform.

The idea with having a subsystem for it would be to group stuff together
in one directory and have that owned by people that care about it,
even if there is little commonality between the things it's used for.

That might be more useful than having it in drivers/misc/ or
drivers/soc/, especially when it comes to recognizing things that
are common between different BMC hardware implementations
or (host) firmware stacks that were initially thought to be more
specific.

>> In any case, I think it makes most sense to discuss new interfaces
>> between the people working on the competing BMC implementations,
>> mainly to see if some functionality can be abstracted in a common
>> way or not.
>
> So we have some ideas of what competing BMC *chip* vendors do since
> OpenBMC people communicates with other vendors.
>
> We know a lot less about BMC SW stacks, but those are a rat nest of
> closed source & GPL stuff that doesn't get upstream (or even released
> sometimes ... ahem ...). Having had access to some vendor code in the
> past, I can tell you that you really don't want to look at what's going
> on in there ;-)

Sure, I really only meant the hardware capabilities here.

      Arnd

Re: How to expose various BMC chip controls ?

[Benjamin Herrenschmidt]

On Tue, 2018-04-10 at 15:06 +0200, Arnd Bergmann wrote:
> On Tue, Apr 10, 2018 at 2:57 PM, Benjamin Herrenschmidt
> <benh@kernel.crashing.org> wrote:
> > On Tue, 2018-04-10 at 13:29 +0200, Arnd Bergmann wrote:
> > > In the cases where an ad-hoc interface is needed, I can see
> > > two options: we can stick something in drivers/soc that exposes
> > > it in the least ugly way, completely specific to that particular
> > > SoC, or we can create a new subsystem from "random BMC
> > > slave-side interfaces" and have it maintained by you all (leave
> > > me out of the picture, except if you are looking for ideas).
> > 
> > I don't think there enough commonality in these things to create a "BMC
> > slave interface" class that is anything other than a bunch of sysfs
> > files.
> > 
> > That's why I ended up with my personal conclusion/preference of just
> > having the syscon "driver" for the SoC just expose a table of a dozen
> > or so such knobs, and use the DT to describe them to abstract a bit the
> > difference between SoC models/revisions and how they are wired up in a
> > given platform.
> 
> The idea with having a subsystem for it would be to group stuff together
> in one directory and have that owned by people that care about it,
> even if there is little commonality between the things it's used for.

Ok, so we could use a subsystem as a grouping method in sysfs, that
does make some amount of sense ;-)

That said, fundamentally, it is going to be exposed by whatever creates
the syscon for the SoC, so yes, a "SoC" driver is probably the right
thing to instanciate that thing, as those knobs are very much SoC
specific. Though I do want to keep the actual definition DT based at
least for Aspeed.

As for finding the commonalities, the connundrum here is that we juts
don't have enough experience of platforms with different kinds of BMCs
to do that (provided there's anything to find).

So we can wait a few years until we do and keep out-of-tree crap
drivers, or just solve the immediate problem in a semi-reasonable way,
and maybe make things evolve as we get more SoC's into the picture.

My issue right now is that we need products out of the door (and not
just we IBM, but everybody using OpenBMC) and we are just carrying
various horrid out of tree hacks to do so, and that needs urgent
fixing.

The main worry I have with my own idea here is that it's too easily
abused for things that shouldn't get in there. But I can't find a way
around that, we'll just have to be vigilant I suppose...

Cheers,
Ben.