libflash questions mostly relating to P9 bringup

libflash questions mostly relating to P9 bringup

[Xo Wang]

Hi folks,

I'm under the impression for P9 bringup we're sticking with BMC-side
MMIO to set up host PNOR access (as in what the host is booting from
over LPC). I see—and agree—that we're not gonna have MTD working by
the time we need to boot early machines:
https://github.com/openbmc/openbmc/issues/553

What does libflash-based PNOR control look like? I mean the hacky way
we're gonna use for the next month. This is how I *think* it works.

booting the host:
1) the SMC SPI flash master auto-detects (?) how to read like the FMC
2) Swiss Army aspeed.c do_x_setup() enables LPC/AHB bridge to the SPI
flash window
Is that it? Doesn't involve MTD or pflash at all and it just works?

updating PNOR from BMC:
1) org.openbmc.control.Flash's update method calls op-flasher
2) org.openbmc.control.Flasher invokes libflash erase and program
3) libflash arch_flash_arm_io.h (?) and friends
  a. know how to talk to the flash controller (BMC_DIRECT)
  b. map AHB, GPIO, and SMC window from /dev/mem
  c. know the size of the host PNOR? (it looks hardcoded)

updating PNOR from host:
1) whatever calls pflash on host
2) host libflash powerpc_io.c and friends poke BMC registers over LPC
  a. write whatever LPC stuff that turns on iLPC2AHB, then use that to
set GPIO and SMC window to LPC for host access
  b. powerpc_io.c also overwrites the LPC/AHB bridge mapping (previous
set by BMC kernel) with its own desired config

Can somebody review that I got all this right? I saw some conflicting
information in https://github.com/openbmc/linux/issues/51

Other questions:
- what's the plan for "booting the host" and "updating PNOR from BMC?"
- is it worth extending the current libflash to read BMC & PNOR size,
window, etc from device tree?
- what does arch_flash_powerpc.c do? It looks like a whole other
access method with a host-side MTD device.

thanks
xo

Re: libflash questions mostly relating to P9 bringup

[Joel Stanley]

On Sat, Sep 24, 2016 at 11:41 AM, Xo Wang <xow@google.com> wrote:
> Hi folks,
>
> I'm under the impression for P9 bringup we're sticking with BMC-side
> MMIO to set up host PNOR access (as in what the host is booting from
> over LPC). I see—and agree—that we're not gonna have MTD working by
> the time we need to boot early machines:
> https://github.com/openbmc/openbmc/issues/553

We have mtd support written and in testing. I plan to land it this week.

I would like us to be able to disable /dev/mem access soon. pflash is
one area where we almost have the bits in place, so I think we should
push to go straight to the correct solution.

>
> What does libflash-based PNOR control look like? I mean the hacky way
> we're gonna use for the next month. This is how I *think* it works.
>
> booting the host:
> 1) the SMC SPI flash master auto-detects (?) how to read like the FMC
> 2) Swiss Army aspeed.c do_x_setup() enables LPC/AHB bridge to the SPI
> flash window
> Is that it? Doesn't involve MTD or pflash at all and it just works?

Yes, I think so.

>
> updating PNOR from BMC:
> 1) org.openbmc.control.Flash's update method calls op-flasher
> 2) org.openbmc.control.Flasher invokes libflash erase and program
> 3) libflash arch_flash_arm_io.h (?) and friends
>   a. know how to talk to the flash controller (BMC_DIRECT)
>   b. map AHB, GPIO, and SMC window from /dev/mem
>   c. know the size of the host PNOR? (it looks hardcoded)

I suspect it uses the ffs header to determine the pnor size, but I do
not know that code well.

>
> updating PNOR from host:
> 1) whatever calls pflash on host
> 2) host libflash powerpc_io.c and friends poke BMC registers over LPC
>   a. write whatever LPC stuff that turns on iLPC2AHB, then use that to
> set GPIO and SMC window to LPC for host access
>   b. powerpc_io.c also overwrites the LPC/AHB bridge mapping (previous
> set by BMC kernel) with its own desired config

Hacky. Not really supported.

>
> Can somebody review that I got all this right? I saw some conflicting
> information in https://github.com/openbmc/linux/issues/51
>
> Other questions:
> - what's the plan for "booting the host" and "updating PNOR from BMC?"

We plan to expose the contents with an in-memory copy of the firmware.

> - is it worth extending the current libflash to read BMC & PNOR size,
> window, etc from device tree?
> - what does arch_flash_powerpc.c do? It looks like a whole other
> access method with a host-side MTD device.
>
> thanks
> xo
> _______________________________________________
> openbmc mailing list
> openbmc@lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/openbmc

Re: libflash questions mostly relating to P9 bringup

[Cyril Bur]

On Tue, 2016-10-11 at 17:55 +1100, Joel Stanley wrote:
> On Sat, Sep 24, 2016 at 11:41 AM, Xo Wang <xow@google.com> wrote:
> > 
> > Hi folks,
> > 
> > I'm under the impression for P9 bringup we're sticking with BMC-
> > side
> > MMIO to set up host PNOR access (as in what the host is booting
> > from
> > over LPC). I see—and agree—that we're not gonna have MTD working by
> > the time we need to boot early machines:
> > https://github.com/openbmc/openbmc/issues/553
> 
> We have mtd support written and in testing. I plan to land it this
> week.
> 
> I would like us to be able to disable /dev/mem access soon. pflash is
> one area where we almost have the bits in place, so I think we should
> push to go straight to the correct solution.
> 
> > 
> > 
> > What does libflash-based PNOR control look like? I mean the hacky
> > way
> > we're gonna use for the next month. This is how I *think* it works.
> > 
> > booting the host:
> > 1) the SMC SPI flash master auto-detects (?) how to read like the
> > FMC
> > 2) Swiss Army aspeed.c do_x_setup() enables LPC/AHB bridge to the
> > SPI
> > flash window
> > Is that it? Doesn't involve MTD or pflash at all and it just works?
> 
> Yes, I think so.
> 
> > 
> > 
> > updating PNOR from BMC:
> > 1) org.openbmc.control.Flash's update method calls op-flasher
> > 2) org.openbmc.control.Flasher invokes libflash erase and program
> > 3) libflash arch_flash_arm_io.h (?) and friends
> >   a. know how to talk to the flash controller (BMC_DIRECT)
> >   b. map AHB, GPIO, and SMC window from /dev/mem
> >   c. know the size of the host PNOR? (it looks hardcoded)
> 
> I suspect it uses the ffs header to determine the pnor size, but I do
> not know that code well.
> 

The ffs header has sizes but you could of course flash an ffs partition
smaller than the total flash and then the ffs size wouldn't be the
total size. If you mean the size of the flash, there are various
mechanisms, in this case (PNOR from BMC) libflash has a mapping between
flash chip ID and total size which it can report up to tools like
pflash.

Ideally this access method will go away in favour of the BMC kernel
exposing the flash through /dev/mtd

In the case where you would want to know this on the host then it
should be in the device-tree and subsequently exposed by the powernv
flash driver (https://github.com/open-power/linux/blob/master/drivers/m
td/devices/powernv_flash.c) though /proc/mtd

> > 
> > 
> > updating PNOR from host:
> > 1) whatever calls pflash on host
> > 2) host libflash powerpc_io.c and friends poke BMC registers over
> > LPC
> >   a. write whatever LPC stuff that turns on iLPC2AHB, then use that
> > to
> > set GPIO and SMC window to LPC for host access
> >   b. powerpc_io.c also overwrites the LPC/AHB bridge mapping
> > (previous
> > set by BMC kernel) with its own desired config
> 
> Hacky. Not really supported.

I should apologise, if we're talking about https://github.com/open-powe
r/skiboot/blob/master/external/pflash/powerpc_io.c, we shouldn't. That
code isn't used anymore and I doubt it will work. I'll send patches to
remove it!

Currently updating PNOR from the host, Joel is correct, it isn't really
supported by openbmc. Of course it can be done through /dev/mtd exposed
by host linux (https://github.com/open-power/linux/blob/master/drivers/
mtd/devices/powernv_flash.c) and backed by a driver in skiboot. I
really hope that all pflash binaries around today know how to do this.

> 
> > 
> > 
> > Can somebody review that I got all this right? I saw some
> > conflicting
> > information in https://github.com/openbmc/linux/issues/51
> > 
> > Other questions:
> > - what's the plan for "booting the host" and "updating PNOR from
> > BMC?"
> 
> We plan to expose the contents with an in-memory copy of the
> firmware.
> 
> > 
> > - is it worth extending the current libflash to read BMC & PNOR
> > size,
> > window, etc from device tree?
> > - what does arch_flash_powerpc.c do? It looks like a whole other
> > access method with a host-side MTD device.

Hopefully I've answered that...

> > 
> > thanks
> > xo
> > _______________________________________________
> > openbmc mailing list
> > openbmc@lists.ozlabs.org
> > https://lists.ozlabs.org/listinfo/openbmc

Re: libflash questions mostly relating to P9 bringup

[Benjamin Herrenschmidt]

On Tue, 2016-10-11 at 18:45 +1100, Cyril Bur wrote:
> > Hacky. Not really supported.
> 
> I should apologise, if we're talking about https://github.com/open-powe
> r/skiboot/blob/master/external/pflash/powerpc_io.c, we shouldn't. That
> code isn't used anymore and I doubt it will work. I'll send patches to
> remove it!
> 
> Currently updating PNOR from the host, Joel is correct, it isn't really
> supported by openbmc. Of course it can be done through /dev/mtd exposed
> by host linux (https://github.com/open-power/linux/blob/master/drivers/
> mtd/devices/powernv_flash.c) and backed by a driver in skiboot. I
> really hope that all pflash binaries around today know how to do this.

I dont know if that was mentioned earlier in the thread but we are
working on a mailbox based protocol for flash access to avoid accessing
the flash controller registers from any other entity than the BMC, in
order to be able to cut off the iLPC->AHB backdoor completely.

This should support having the flash either access directly for reads
or via an in-memory image in the BMC (aka memboot) and writes will be
done to an in-memory window on the LPC and "committed" to flash via
mbox commands.

We can get you details when we have a bit more sorted out on our side,
hopefully in a few days.

Cheers,
Ben.

Fwd: libflash questions mostly relating to P9 bringup

[Brendan Higgins]

Sorry forgot to include the OpenBMC mail list.


---------- Forwarded message ----------
From: Brendan Higgins <brendanhiggins@google.com>
Date: Wed, Oct 19, 2016 at 1:54 PM
Subject: libflash questions mostly relating to P9 bringup
To: cyrilbur@gmail.com, joel@jms.id.au, xow@google.com, benh@kernel.crashing.org
Cc: Brendan Higgins <brendanhiggins@google.com>


> I dont know if that was mentioned earlier in the thread but we are
> working on a mailbox based protocol for flash access to avoid accessing
> the flash controller registers from any other entity than the BMC, in
> order to be able to cut off the iLPC->AHB backdoor completely.
>
> This should support having the flash either access directly for reads
> or via an in-memory image in the BMC (aka memboot) and writes will be
> done to an in-memory window on the LPC and "committed" to flash via
> mbox commands.
>
> We can get you details when we have a bit more sorted out on our side,
> hopefully in a few days.

We are also working on a flash access protocol except ours is via IPMI; this is
was a motivating reason for the 0-N response per request extension discussed
here: https://lists.ozlabs.org/pipermail/openbmc/2016-September/004523.html

The main desire for using IPMI for this is that IPMI is the closest thing we
have to a common communication protocol between the BMC and the primary CPU
across platforms. Some platforms, for example, may not have support for LPC or
eSPI.

Our plan is to implement this as an OEM/Group extension with the payload as a
Protocol Buffer encoded (see
https://developers.google.com/protocol-buffers/docs/encoding) header:

message PacketHeader {
  optional uint32 packet_idx = 1;
  optional uint32 packet_count = 2;
}

followed by a chunk of data with whatever space remains. So the actual message
would look something like:

struct Packet {
  PacketHeader header;
  char data[X];
};

where X is IPMI payload size - PacketHeader size. This is not a valid struct
definition as the fields have variable offsets and size, and is only meant for
illustrative purposes.

The motivation behind the protobuf encoding is that it can be extended in a
backwards compatible way and that unneeded fields (for example, we may only want
to report the packet_count, the number of expected packets, in the first
message) only take up space if they are sent; similarly, numeric fields use
varint encoding so that the value takes up less space if it is small.

The actual flash access message would be reassembled from multiple packets and
could contain different messages, for example:

message RawFlashRequest {
  enum RequestType {
    Read = 0;
    Write = 1;
    Info = 2;
    Checksum = 3;
  }

  optional RequestType type = 1;
  optional bytes partition_guid = 2;
  optional uint32 offset = 3;
  optional uint32 size = 4;
  repeated bytes payload = 5;
}

could be one message type.

This is not intended to be a complete specification, but an overview of what we
are working on and what our needs are. I think it would be preferential if we
could have a common implementation.

Cheers.

---------- Forwarded message ----------
From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date: Wed, Oct 19, 2016 at 5:15 PM
Subject: Re: libflash questions mostly relating to P9 bringup
To: Brendan Higgins <brendanhiggins@google.com>, cyrilbur@gmail.com,
joel@jms.id.au, xow@google.com


On Wed, 2016-10-19 at 13:54 -0700, Brendan Higgins wrote:

> We are also working on a flash access protocol except ours is via IPMI; this is
> was a motivating reason for the 0-N response per request extension discussed
> here: https://lists.ozlabs.org/pipermail/openbmc/2016-September/004523.html
>
> The main desire for using IPMI for this is that IPMI is the closest thing we
> have to a common communication protocol between the BMC and the primary CPU
> across platforms. Some platforms, for example, may not have support for LPC or
> eSPI.

 .../...

We chose do to something a bit different for several reasons.

The main one is probably speed. IPMI is fairly slow and we don't really want
to transport the actual flash data over it.

So the idea is to read/write through the LPC FW space, by mapping it to a
region of BMC memory rather than flash, and send commands via a separate
IO channel (which could be IPMI, though in our current prototype it's not)
to "commit" the modified portions.

It's a bit more convoluted as we plan to support a sliding write window
that might be distinct from the read window as to not require using as
much BMC memory as the size of the flash, but we haven't implemented all
of the "options" yet.

It will also give us memboot for free, ie, the ability to boot from an
image of the flash coming from the network rather than flash which is useful
for development, and using memory rather than flash seems to speed things
up a bit as well.

It might not be portable to different BMCs, so having the IPMI method as
some kind of fallback might still be an options for us though, we should
probably look into it.

The other reason, which granted isn't a terribly good one, is that HostBoot
has their flash layer in their "kernel" while their IPMI stack is in their
"User space", and they are reluctant on changing that.

That being said, we could use the second BT interface along with a massively
stripped down IPMI stack in their kernel to keep the flash path separate from
the main IPMI path if we wanted.

For now, we use the mbox instead which is slightly easier to use than BT and
avoids all "conflicts" with the IPMI traffic.

Antoher advantage in segregating these is that we open the possibility of
assigning the main iBT interface directly to linux instead of OPAL, taking
OPAL out of the path of normal IPMI messages. OPAL <-> BMC would then use
our mbox sideband (but could use the secondary BT if we wanted to, it's just
inconvenient as it doesn't support interrupts in the Aspeed hardware).

Cheers,
Ben.


---------- Forwarded message ----------
From: Brendan Higgins <brendanhiggins@google.com>
Date: Thu, Oct 20, 2016 at 11:41 AM
Subject: Re: libflash questions mostly relating to P9 bringup
To: Benjamin Herrenschmidt <benh@kernel.crashing.org>, Cyril Bur
<cyrilbur@gmail.com>, Joel Stanley <joel@jms.id.au>, Xo Wang
<xow@google.com>, Abhishek Pandit <abhishekpandit@google.com>


> The main one is probably speed. IPMI is fairly slow and we don't really want
>
>  to transport the actual flash data over it.


I do not dispute that; I think we accept that as a consequence, again,
as LPC does not exist on many platforms.

> It's a bit more convoluted as we plan to support a sliding write window
> that might be distinct from the read window as to not require using as
> much BMC memory as the size of the flash, but we haven't implemented all
> of the "options" yet.


I only mentioned the most simple method that we wanted to implement,
RawFlashRequest, which would provide direct flash access, but we were
interested in supporting other methods as well, so if you would be
willing to support other access methods, I see this as being totally
compatible with what we want to do.

> It might not be portable to different BMCs, so having the IPMI method as
> some kind of fallback might still be an options for us though, we should
> probably look into it.


That sounds reasonable to me. As long as we have a common way to do
flash access on all systems we are happy; if we happen to support some
other faster access methods (as long as they share a common API), I
think that would be even better :-)

> For now, we use the mbox instead which is slightly easier to use than BT and
> avoids all "conflicts" with the IPMI traffic.


Yep, that is what we are trying to address with our OEM/Group
extension; it provides an interface that sends arbitrary data over
IPMI. Admittedly, it will make all other IPMI traffic slower; however,
this is unavoidable on platforms that only have one hardware interface
between the BMC and the host CPU.

>
> Antoher advantage in segregating these is that we open the possibility of
> assigning the main iBT interface directly to linux instead of OPAL, taking
> OPAL out of the path of normal IPMI messages. OPAL <-> BMC would then use
> our mbox sideband (but could use the secondary BT if we wanted to, it's just
> inconvenient as it doesn't support interrupts in the Aspeed hardware).


Also true, but still unavoidable if we only have a single hardware interface.

Sounds to me like we do not have any incompatible goals; although it
is true that on our side we are more concerned about a common
implementation that will work on any platform and you guys seem to be
more concerned about performance, I think that just means that we
should provide both transport layers: IPMI and LPC mailbox. I imagine
that a good deal of the actual flash access logic will be the same for
the both of us.

Cheers

Fwd: libflash questions mostly relating to P9 bringup

[Brendan Higgins]

[-- Attachment #1: Type: text/plain, Size: 2802 bytes --]

---------- Forwarded message ---------
From: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Date: Thu, Oct 20, 2016 at 2:06 PM
Subject: Re: libflash questions mostly relating to P9 bringup
To: Brendan Higgins <brendanhiggins@google.com>, Cyril Bur <
cyrilbur@gmail.com>, Joel Stanley <joel@jms.id.au>, Xo Wang <xow@google.com>,
Abhishek Pandit <abhishekpandit@google.com>


On Thu, 2016-10-20 at 11:41 -0700, Brendan Higgins wrote:
> > The main one is probably speed. IPMI is fairly slow and we don't
> > really want
> >  to transport the actual flash data over it.
>
> I do not dispute that; I think we accept that as a consequence,
> again, as LPC does not exist on many platforms.

Ok.

> I only mentioned the most simple method that we wanted to
> implement, RawFlashRequest, which would provide direct flash access,
> but we were interested in supporting other methods as well, so if you
> would be willing to support other access methods, I see this as being
> totally compatible with what we want to do.

Yes, it would work fine to support multiple methods.

> That sounds reasonable to me. As long as we have a common way to do
> flash access on all systems we are happy; if we happen to support
> some other faster access methods (as long as they share a common
> API), I think that would be even better :-)

So the API ... The plan for us is to plumb this behind the existing
OPAL flash APIs which are themselves plumbed into Linux MTD layer.

It's fairly easy inside OPAL to implement different backends for a
given API.

> Yep, that is what we are trying to address with our OEM/Group
> extension; it provides an interface that sends arbitrary data over
> IPMI. Admittedly, it will make all other IPMI traffic slower;
> however, this is unavoidable on platforms that only have one hardware
> interface between the BMC and the host CPU.

Right.

> Also true, but still unavoidable if we only have a single hardware
> interface.
>
> Sounds to me like we do not have any incompatible goals; although it
> is true that on our side we are more concerned about a common
> implementation that will work on any platform and you guys seem to be
> more concerned about performance, I think that just means that we
> should provide both transport layers: IPMI and LPC mailbox. I imagine
> that a good deal of the actual flash access logic will be the same
> for the both of us.

Yup, I think both work. At some point we might want to look at the
implementation details a bit more closely, it would be good to merge
the Zaius support into the main OPAL repo sooner rather than later.

We are still working on the implementation of our mbox based flash
protocol, but once we are done it should be easy for you guys to pick
it up as well if you wish to for your Aspeed based machine(s).

Cheers,
Ben.

[-- Attachment #2: Type: text/html, Size: 4662 bytes --]

Re: libflash questions mostly relating to P9 bringup

[Cyril Bur]

On Tue, 2016-10-11 at 17:55 +1100, Joel Stanley wrote:
> On Sat, Sep 24, 2016 at 11:41 AM, Xo Wang <xow@google.com> wrote:
> > 
> > Hi folks,
> > 
> > I'm under the impression for P9 bringup we're sticking with BMC-
> > side
> > MMIO to set up host PNOR access (as in what the host is booting
> > from
> > over LPC). I see—and agree—that we're not gonna have MTD working by
> > the time we need to boot early machines:
> > https://github.com/openbmc/openbmc/issues/553
> 
> We have mtd support written and in testing. I plan to land it this
> week.
> 
> I would like us to be able to disable /dev/mem access soon. pflash is
> one area where we almost have the bits in place, so I think we should
> push to go straight to the correct solution.
> 
> > 
> > 
> > What does libflash-based PNOR control look like? I mean the hacky
> > way
> > we're gonna use for the next month. This is how I *think* it works.
> > 
> > booting the host:
> > 1) the SMC SPI flash master auto-detects (?) how to read like the
> > FMC
> > 2) Swiss Army aspeed.c do_x_setup() enables LPC/AHB bridge to the
> > SPI
> > flash window
> > Is that it? Doesn't involve MTD or pflash at all and it just works?
> 
> Yes, I think so.
> 
> > 
> > 
> > updating PNOR from BMC:
> > 1) org.openbmc.control.Flash's update method calls op-flasher
> > 2) org.openbmc.control.Flasher invokes libflash erase and program
> > 3) libflash arch_flash_arm_io.h (?) and friends
> >   a. know how to talk to the flash controller (BMC_DIRECT)
> >   b. map AHB, GPIO, and SMC window from /dev/mem
> >   c. know the size of the host PNOR? (it looks hardcoded)
> 
> I suspect it uses the ffs header to determine the pnor size, but I do
> not know that code well.
> 

The ffs header has sizes but you could of course flash an ffs partition
smaller than the total flash and then the ffs size wouldn't be the
total size. If you mean the size of the flash, there are various
mechanisms, in this case (PNOR from BMC) libflash has a mapping between
flash chip ID and total size which it can report up to tools like
pflash.

Ideally this access method will go away in favour of the BMC kernel
exposing the flash through /dev/mtd

In the case where you would want to know this on the host then it
should be in the device-tree and subsequently exposed by the powernv
flash driver (https://github.com/open-power/linux/blob/master/drivers/m
td/devices/powernv_flash.c) though /proc/mtd

> > 
> > 
> > updating PNOR from host:
> > 1) whatever calls pflash on host
> > 2) host libflash powerpc_io.c and friends poke BMC registers over
> > LPC
> >   a. write whatever LPC stuff that turns on iLPC2AHB, then use that
> > to
> > set GPIO and SMC window to LPC for host access
> >   b. powerpc_io.c also overwrites the LPC/AHB bridge mapping
> > (previous
> > set by BMC kernel) with its own desired config
> 
> Hacky. Not really supported.

Firstly I should apologise, if we're talking about https://github.com/o
pen-power/skiboot/blob/master/external/pflash/powerpc_io.c, we
shouldn't. That code isn't used anymore and I doubt it will work. I'll
send patches to remove it!

Currently updating PNOR from the host, Joel is correct, it isn't really
supported by openbmc. Of course it can be done through /dev/mtd exposed
by host linux (https://github.com/open-power/linux/blob/master/drivers/
mtd/devices/powernv_flash.c) and backed by a driver in skiboot. I
really hope that all pflash binaries around today know how to do this.

> 
> > 
> > 
> > Can somebody review that I got all this right? I saw some
> > conflicting
> > information in https://github.com/openbmc/linux/issues/51
> > 
> > Other questions:
> > - what's the plan for "booting the host" and "updating PNOR from
> > BMC?"
> 
> We plan to expose the contents with an in-memory copy of the
> firmware.
> 
> > 
> > - is it worth extending the current libflash to read BMC & PNOR
> > size,
> > window, etc from device tree?
> > - what does arch_flash_powerpc.c do? It looks like a whole other
> > access method with a host-side MTD device.

Hopefully I've answered that...

> > 
> > thanks
> > xo
> > _______________________________________________
> > openbmc mailing list
> > openbmc@lists.ozlabs.org
> > https://lists.ozlabs.org/listinfo/openbmc