IPMI SOL performance

IPMI SOL performance

[Tom Joseph]

Hello,

I wanted to discuss about an issue related to IPMI SOL. Petitboot has a 
menu to select the boot option, it has a timeout after which the default 
boot option is selected and boot continues.The observation is that when 
verbose debug traces are enabled in the hostboot, by the time petitboot 
options are printed on the SOL console the menu timeout expires and the 
user is not able to change the boot option.

The same issue is not hit by the ssh console on port 2200 and does not 
miss the timeout of the petitboot menu option.

IPMI uses UDP protocol and the maximum size of the  SOL payload per 
packet according to the specification is 255 bytes. Also the ack for 
each of the IPMI packet requires a command lookup
and execution. Compared with IPMI SOL, ssh supports 64kb payloads and 
ack does not need an application lookup, so the round-trip timing is 
faster. IPMI specification for SOL is written with UART speed as 
reference, but the VUART pushes the data at a much faster rate.

One suggestion that I came across support an OEM payload size for SOL, 
which is higher than 255 bytes, to get a better throughput. Please do 
share other ideas to tackle this problem and improve performance.

Regards,
Tom

Re: IPMI SOL performance

[Tom Joseph]

   Related issue -  https://github.com/openbmc/openbmc/issues/2958

On Monday 19 March 2018 04:43 PM, Tom Joseph wrote:
> Hello,
>
> I wanted to discuss about an issue related to IPMI SOL. Petitboot has 
> a menu to select the boot option, it has a timeout after which the 
> default boot option is selected and boot continues.The observation is 
> that when verbose debug traces are enabled in the hostboot, by the 
> time petitboot options are printed on the SOL console the menu timeout 
> expires and the user is not able to change the boot option.
>
> The same issue is not hit by the ssh console on port 2200 and does not 
> miss the timeout of the petitboot menu option.
>
> IPMI uses UDP protocol and the maximum size of the  SOL payload per 
> packet according to the specification is 255 bytes. Also the ack for 
> each of the IPMI packet requires a command lookup
> and execution. Compared with IPMI SOL, ssh supports 64kb payloads and 
> ack does not need an application lookup, so the round-trip timing is 
> faster. IPMI specification for SOL is written with UART speed as 
> reference, but the VUART pushes the data at a much faster rate.
>
> One suggestion that I came across support an OEM payload size for SOL, 
> which is higher than 255 bytes, to get a better throughput. Please do 
> share other ideas to tackle this problem and improve performance.
>
> Regards,
> Tom
>

Re: IPMI SOL performance

[Emily Shaffer]

[-- Attachment #1: Type: text/plain, Size: 1638 bytes --]

On Mon, Mar 19, 2018 at 4:18 AM Tom Joseph <tomjose@linux.vnet.ibm.com>
wrote:

>
>    Related issue -  https://github.com/openbmc/openbmc/issues/2958
>
> On Monday 19 March 2018 04:43 PM, Tom Joseph wrote:
> > Hello,
> >
> > I wanted to discuss about an issue related to IPMI SOL. Petitboot has
> > a menu to select the boot option, it has a timeout after which the
> > default boot option is selected and boot continues.The observation is
> > that when verbose debug traces are enabled in the hostboot, by the
> > time petitboot options are printed on the SOL console the menu timeout
> > expires and the user is not able to change the boot option.
> >
> > The same issue is not hit by the ssh console on port 2200 and does not
> > miss the timeout of the petitboot menu option.
> >
> > IPMI uses UDP protocol and the maximum size of the  SOL payload per
> > packet according to the specification is 255 bytes. Also the ack for
> > each of the IPMI packet requires a command lookup
> > and execution. Compared with IPMI SOL, ssh supports 64kb payloads and
> > ack does not need an application lookup, so the round-trip timing is
> > faster. IPMI specification for SOL is written with UART speed as
> > reference, but the VUART pushes the data at a much faster rate.
> >
> > One suggestion that I came across support an OEM payload size for SOL,
> > which is higher than 255 bytes, to get a better throughput. Please do
> > share other ideas to tackle this problem and improve performance.
> >
> > Regards,
> > Tom
> >
>
> Is it a possibility to just increase the petitboot timeout? Do you have an
idea of how much we are missing it by?

[-- Attachment #2: Type: text/html, Size: 2170 bytes --]

Re: IPMI SOL performance

[Stewart Smith]

Emily Shaffer <emilyshaffer@google.com> writes:
>> Is it a possibility to just increase the petitboot timeout? Do you have an
> idea of how much we are missing it by?

Worst case (may be different currently due to improvements in ipmi
stack) was about 10-15 minutes.

We have different issues on the SSH console though, we can end up
dropping chunks of console output under load (BMC CPU load or high
console usage).

I think Jeremy can relive the trauma of heading into the TTY layer, but
I wonder if the solution here has something to do with having a
end-to-end flow control story.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: IPMI SOL performance

[Emily Shaffer]

[-- Attachment #1: Type: text/plain, Size: 1120 bytes --]

On Mon, Mar 19, 2018 at 9:27 AM Stewart Smith <stewart@linux.vnet.ibm.com>
wrote:

> Emily Shaffer <emilyshaffer@google.com> writes:
> >> Is it a possibility to just increase the petitboot timeout? Do you have
> an
> > idea of how much we are missing it by?
>
> Worst case (may be different currently due to improvements in ipmi
> stack) was about 10-15 minutes.

Yikes.

>


> We have different issues on the SSH console though, we can end up
> dropping chunks of console output under load (BMC CPU load or high
> console usage).
>
> I think Jeremy can relive the trauma of heading into the TTY layer, but
> I wonder if the solution here has something to do with having a
> end-to-end flow control story.
>
> --
> Stewart Smith
> OPAL Architect, IBM.
>
>
I was looking in the spec and I saw that the packet length for SOL is
configurable in the session header, but I didn't find the layout for the
header.  It's definitely not configurable past 255B?

It really doesn't sound like IPMI is well-suited to this task.  Tom, can
you post a proposal for the OEM command you'd like to see, or if you've
tried one internally?

[-- Attachment #2: Type: text/html, Size: 1813 bytes --]

Re: IPMI SOL performance

[Tom Joseph]

[-- Attachment #1: Type: text/plain, Size: 1775 bytes --]



On Tuesday 20 March 2018 12:17 AM, Emily Shaffer wrote:
>
>
> On Mon, Mar 19, 2018 at 9:27 AM Stewart Smith 
> <stewart@linux.vnet.ibm.com <mailto:stewart@linux.vnet.ibm.com>> wrote:
>
>     Emily Shaffer <emilyshaffer@google.com
>     <mailto:emilyshaffer@google.com>> writes:
>     >> Is it a possibility to just increase the petitboot timeout? Do
>     you have an
>     > idea of how much we are missing it by?
>
>     Worst case (may be different currently due to improvements in ipmi
>     stack) was about 10-15 minutes.
>
> Yikes.
>
>
>     We have different issues on the SSH console though, we can end up
>     dropping chunks of console output under load (BMC CPU load or high
>     console usage).
>
>     I think Jeremy can relive the trauma of heading into the TTY
>     layer, but
>     I wonder if the solution here has something to do with having a
>     end-to-end flow control story.
>
>     --
>     Stewart Smith
>     OPAL Architect, IBM.
>
>
> I was looking in the spec and I saw that the packet length for SOL is 
> configurable in the session header, but I didn't find the layout for 
> the header.  It's definitely not configurable past 255B?

Even though character data field is a variable length field, the 
accepted character count in the SOL payload is a single byte. It is 
based on the accepted character count that console acknowledges to BMC 
and offset is changed. That is the 255 character limitation mentioned.
>
> It really doesn't sound like IPMI is well-suited to this task.  Tom, 
> can you post a proposal for the OEM command you'd like to see, or if 
> you've tried one internally?
I haven't tried this option, the OEM option is to bump the accepted 
character count field. It will need changes on the clients (like ipmitool).


[-- Attachment #2: Type: text/html, Size: 3572 bytes --]

Re: IPMI SOL performance

[Emily Shaffer]

[-- Attachment #1: Type: text/plain, Size: 2297 bytes --]

On Mon, Mar 19, 2018 at 12:04 PM Tom Joseph <tomjose@linux.vnet.ibm.com>
wrote:

>
>
> On Tuesday 20 March 2018 12:17 AM, Emily Shaffer wrote:
>
>
>
> On Mon, Mar 19, 2018 at 9:27 AM Stewart Smith <stewart@linux.vnet.ibm.com>
> wrote:
>
>> Emily Shaffer <emilyshaffer@google.com> writes:
>> >> Is it a possibility to just increase the petitboot timeout? Do you
>> have an
>> > idea of how much we are missing it by?
>>
>> Worst case (may be different currently due to improvements in ipmi
>> stack) was about 10-15 minutes.
>
> Yikes.
>
>>
>
>
>> We have different issues on the SSH console though, we can end up
>> dropping chunks of console output under load (BMC CPU load or high
>> console usage).
>>
>> I think Jeremy can relive the trauma of heading into the TTY layer, but
>> I wonder if the solution here has something to do with having a
>> end-to-end flow control story.
>>
>> --
>> Stewart Smith
>> OPAL Architect, IBM.
>>
>>
> I was looking in the spec and I saw that the packet length for SOL is
> configurable in the session header, but I didn't find the layout for the
> header.  It's definitely not configurable past 255B?
>
>
> Even though character data field is a variable length field, the accepted
> character count in the SOL payload is a single byte. It is based on the
> accepted character count that console acknowledges to BMC and offset is
> changed. That is the 255 character limitation mentioned.
>
>
> It really doesn't sound like IPMI is well-suited to this task.  Tom, can
> you post a proposal for the OEM command you'd like to see, or if you've
> tried one internally?
>
> I haven't tried this option, the OEM option is to bump the accepted
> character count field. It will need changes on the clients (like ipmitool).
>
> I don't know that this is the right approach, as it would break anybody
trying to use OpenBMC with their own copy of ipmitool.  I'd rather see an
entirely separate OEM command that we can tailor to our needs (for example,
if we are expecting ACK/NACK with SOL in the form of another IPMI command,
does it make more sense to just use TCP in an OEM command instead), either
leaving the slow per-spec approach up as best-effort, or adding some
warning to anyone trying to use it that we've added an improved OEM
command.  Just my two cents...

[-- Attachment #2: Type: text/html, Size: 4373 bytes --]

Re: IPMI SOL performance

[Kun Yi]

[-- Attachment #1: Type: text/plain, Size: 2702 bytes --]

SOL performance issue aside, there is also an IPMI command for selecting
the boot device. Is that command currently supported in the client in any
way? That might be a more reliable solution than relying on console alone.


On Mon, Mar 19, 2018 at 12:21 PM Emily Shaffer <emilyshaffer@google.com>
wrote:

>
>
> On Mon, Mar 19, 2018 at 12:04 PM Tom Joseph <tomjose@linux.vnet.ibm.com>
> wrote:
>
>>
>>
>> On Tuesday 20 March 2018 12:17 AM, Emily Shaffer wrote:
>>
>>
>>
>> On Mon, Mar 19, 2018 at 9:27 AM Stewart Smith <stewart@linux.vnet.ibm.com>
>> wrote:
>>
>>> Emily Shaffer <emilyshaffer@google.com> writes:
>>> >> Is it a possibility to just increase the petitboot timeout? Do you
>>> have an
>>> > idea of how much we are missing it by?
>>>
>>> Worst case (may be different currently due to improvements in ipmi
>>> stack) was about 10-15 minutes.
>>
>> Yikes.
>>
>>>
>>
>>
>>> We have different issues on the SSH console though, we can end up
>>> dropping chunks of console output under load (BMC CPU load or high
>>> console usage).
>>>
>>> I think Jeremy can relive the trauma of heading into the TTY layer, but
>>> I wonder if the solution here has something to do with having a
>>> end-to-end flow control story.
>>>
>>> --
>>> Stewart Smith
>>> OPAL Architect, IBM.
>>>
>>>
>> I was looking in the spec and I saw that the packet length for SOL is
>> configurable in the session header, but I didn't find the layout for the
>> header.  It's definitely not configurable past 255B?
>>
>>
>> Even though character data field is a variable length field, the accepted
>> character count in the SOL payload is a single byte. It is based on the
>> accepted character count that console acknowledges to BMC and offset is
>> changed. That is the 255 character limitation mentioned.
>>
>>
>> It really doesn't sound like IPMI is well-suited to this task.  Tom, can
>> you post a proposal for the OEM command you'd like to see, or if you've
>> tried one internally?
>>
>> I haven't tried this option, the OEM option is to bump the accepted
>> character count field. It will need changes on the clients (like ipmitool).
>>
>> I don't know that this is the right approach, as it would break anybody
> trying to use OpenBMC with their own copy of ipmitool.  I'd rather see an
> entirely separate OEM command that we can tailor to our needs (for example,
> if we are expecting ACK/NACK with SOL in the form of another IPMI command,
> does it make more sense to just use TCP in an OEM command instead), either
> leaving the slow per-spec approach up as best-effort, or adding some
> warning to anyone trying to use it that we've added an improved OEM
> command.  Just my two cents...
>
>


-- 
Regards,
Kun

[-- Attachment #2: Type: text/html, Size: 5122 bytes --]

Re: IPMI SOL performance

[Stewart Smith]

Kun Yi <kunyi@google.com> writes:
> SOL performance issue aside, there is also an IPMI command for selecting
> the boot device. Is that command currently supported in the client in any
> way? That might be a more reliable solution than relying on console
> alone.

There is, and any reasonable automation kit is going to do that, except
for when it's something like my firmware test suite that executes a
bunch of commands in the petitboot shell environment, for which there's
no way to set "exit to shell" via IPMI, so we just use expect on the
console. We've worked around this in op-test by just using the ssh
console instead (and try not to pump too much data through it too
quickly so we don't hit the bug where we lose console output).

It'd be nice to be able to run the test suite over IPMI though, if only
for completeness.

-- 
Stewart Smith
OPAL Architect, IBM.

Re: IPMI SOL performance

[Rob Lippert]

[-- Attachment #1: Type: text/plain, Size: 3191 bytes --]

Another related issue: https://github.com/openbmc/obmc-console/issues/10

We've worked around this problem internally with that patch mentioned in
the issue but this has the side effect of making the SSH-based console
output just as slowly as the real UART (115200 bps).
The obmc console server can provide "pushback" to the host console by not
draining the host virtual UART until all clients have consumed/ack'ed the
data but I'm not sure this is implemented yet.
But this will make your boot really slow so is a bit of a tradeoff.  Also
some BIOS/firmware I've seen has timeouts on how long it will wait for the
serial port FIFO to become available which can result in dropped characters
if your clients are really slow.

-Rob

On Mon, Mar 19, 2018 at 12:08 PM, Emily Shaffer <emilyshaffer@google.com>
wrote:

>
>
> On Mon, Mar 19, 2018 at 12:04 PM Tom Joseph <tomjose@linux.vnet.ibm.com>
> wrote:
>
>>
>>
>> On Tuesday 20 March 2018 12:17 AM, Emily Shaffer wrote:
>>
>>
>>
>> On Mon, Mar 19, 2018 at 9:27 AM Stewart Smith <stewart@linux.vnet.ibm.com>
>> wrote:
>>
>>> Emily Shaffer <emilyshaffer@google.com> writes:
>>> >> Is it a possibility to just increase the petitboot timeout? Do you
>>> have an
>>> > idea of how much we are missing it by?
>>>
>>> Worst case (may be different currently due to improvements in ipmi
>>> stack) was about 10-15 minutes.
>>
>> Yikes.
>>
>>>
>>
>>
>>> We have different issues on the SSH console though, we can end up
>>> dropping chunks of console output under load (BMC CPU load or high
>>> console usage).
>>>
>>> I think Jeremy can relive the trauma of heading into the TTY layer, but
>>> I wonder if the solution here has something to do with having a
>>> end-to-end flow control story.
>>>
>>> --
>>> Stewart Smith
>>> OPAL Architect, IBM.
>>>
>>>
>> I was looking in the spec and I saw that the packet length for SOL is
>> configurable in the session header, but I didn't find the layout for the
>> header.  It's definitely not configurable past 255B?
>>
>>
>> Even though character data field is a variable length field, the accepted
>> character count in the SOL payload is a single byte. It is based on the
>> accepted character count that console acknowledges to BMC and offset is
>> changed. That is the 255 character limitation mentioned.
>>
>>
>> It really doesn't sound like IPMI is well-suited to this task.  Tom, can
>> you post a proposal for the OEM command you'd like to see, or if you've
>> tried one internally?
>>
>> I haven't tried this option, the OEM option is to bump the accepted
>> character count field. It will need changes on the clients (like ipmitool).
>>
>> I don't know that this is the right approach, as it would break anybody
> trying to use OpenBMC with their own copy of ipmitool.  I'd rather see an
> entirely separate OEM command that we can tailor to our needs (for example,
> if we are expecting ACK/NACK with SOL in the form of another IPMI command,
> does it make more sense to just use TCP in an OEM command instead), either
> leaving the slow per-spec approach up as best-effort, or adding some
> warning to anyone trying to use it that we've added an improved OEM
> command.  Just my two cents...
>
>

[-- Attachment #2: Type: text/html, Size: 5715 bytes --]

Re: IPMI SOL performance

[Jeremy Kerr]

Hi Tom,

> I wanted to discuss about an issue related to IPMI SOL.

Cool!

Given that:

> The same issue is not hit by the ssh console on port 2200 and does not
> miss the timeout of the petitboot menu option.

- and that we don't see the same behaviour with other SoL
implementations (ie, different BMC firmware on similar hardware), it
sounds to me like this might be due to our IPMI implementation?

Have you confirmed that we're making the best use of the SoL payloads?
(ie, we're not using smaller TX buffers for any reason).

> IPMI uses UDP protocol and the maximum size of the  SOL payload per
> packet according to the specification is 255 bytes. Also the ack for
> each of the IPMI packet requires a command lookup
> and execution. Compared with IPMI SOL, ssh supports 64kb payloads and
> ack does not need an application lookup, so the round-trip timing is
> faster. IPMI specification for SOL is written with UART speed as
> reference, but the VUART pushes the data at a much faster rate.

The console multiplexer will only push data as fast as *all* consumers
can consume it. As a bit of background: obmc-console-server will read
from the VUART into a global 16kB ringbuffer, and write to each of the
consumers (ie, the sockets that ssh and IPMI SoL daemons connect to).

If those writes block (for any one of those consumers), the ringbuffer
tail is not advanced. Only once the slowest consumer has unblocked and
consumed data, we advance the tail.

Once the ringbuffer is full though, we stop reading from the VUART until
there is more space available (ie, the slowest consumer has caught up).

This means that we have 16kB of space for bursty data, after which the
daemon will slow down reads from the VUART device.

So it sounds like our IPMI SoL implementation is slower than other BMC
implementations, as we don't see the delay on other platforms. Can we
address that?

As others have mentioned, there's currently an issue with the VUART
hardware or driver dropping serial data on high load, but that's
somewhat separate from IPMI performance. I'm currently working on that
now :)

Regards,


Jeremy

Re: IPMI SOL performance

[Tom Joseph]

On Tuesday 20 March 2018 06:05 AM, Jeremy Kerr wrote:
> Hi Tom,
>
>> I wanted to discuss about an issue related to IPMI SOL.
> Cool!
>
> Given that:
>
>> The same issue is not hit by the ssh console on port 2200 and does not
>> miss the timeout of the petitboot menu option.
> - and that we don't see the same behaviour with other SoL
> implementations (ie, different BMC firmware on similar hardware), it
> sounds to me like this might be due to our IPMI implementation?

> Have you confirmed that we're making the best use of the SoL payloads?
> (ie, we're not using smaller TX buffers for any reason).

The accepted character count field in the SOL payload is what the 
client(ipmitool) uses to acknowledge to BMC.
Since this is a 1 byte field, so the maximum SOL character data is 
restricted to 255.
>
>> IPMI uses UDP protocol and the maximum size of the  SOL payload per
>> packet according to the specification is 255 bytes. Also the ack for
>> each of the IPMI packet requires a command lookup
>> and execution. Compared with IPMI SOL, ssh supports 64kb payloads and
>> ack does not need an application lookup, so the round-trip timing is
>> faster. IPMI specification for SOL is written with UART speed as
>> reference, but the VUART pushes the data at a much faster rate.
> The console multiplexer will only push data as fast as *all* consumers
> can consume it. As a bit of background: obmc-console-server will read
> from the VUART into a global 16kB ringbuffer, and write to each of the
> consumers (ie, the sockets that ssh and IPMI SoL daemons connect to).
>
> If those writes block (for any one of those consumers), the ringbuffer
> tail is not advanced. Only once the slowest consumer has unblocked and
> consumed data, we advance the tail.
Is it guaranteed that the Petitboot menu option will be shown only after 
the obmc-console-server,
reads it from the VUART? When does the petitboot menu timeout begin?
>
> Once the ringbuffer is full though, we stop reading from the VUART until
> there is more space available (ie, the slowest consumer has caught up).
>
> This means that we have 16kB of space for bursty data, after which the
> daemon will slow down reads from the VUART device.
>
> So it sounds like our IPMI SoL implementation is slower than other BMC
> implementations, as we don't see the delay on other platforms. Can we
> address that?
This is good piece of information. IPMI server on BMC listens on 
"\0obmc-console" socket and reads from the socket
  on a EPOLLIN. The read console data is updated in a buffer (double 
ended queue) and sent to the ipmitool
based on the flow rate. Since the console data is read into the IPMI 
buffer( and sending it to the client at the speed of client)
is the reason that obmc-console-server does not slow down.

One possible option is to put a limit on the buffer size and read from 
the socket only when the buffer is within that limits. The trade-off
would be ssh based console would output at the speed of IPMI.

> As others have mentioned, there's currently an issue with the VUART
> hardware or driver dropping serial data on high load, but that's
> somewhat separate from IPMI performance. I'm currently working on that
> now :)
>
> Regards,
>
>
> Jeremy
>

Re: IPMI SOL performance

[Jeremy Kerr]

Hi Tom,

> The accepted character count field in the SOL payload is what the
> client(ipmitool) uses to acknowledge to BMC.
> Since this is a 1 byte field, so the maximum SOL character data is
> restricted to 255.

Yep, but are you sure you're sending packets of the maximum size?

> Is it guaranteed that the Petitboot menu option will be shown only after
> the obmc-console-server,
> reads it from the VUART?

Not at all. The petitboot process has no idea what's happening on the
BMC. Same with all other console-based processes on the host system.

> When does the petitboot menu timeout begin?

As soon as the first default-bootable option is found.

> One possible option is to put a limit on the buffer size and read from
> the socket only when the buffer is within that limits.

Yep, there's no real need for the IPMI SoL infrastructure to do any
output buffering at all. If your maximum packet size is 255 bytes, just
read a maximum of 255 bytes from the obmc-console socket.

However, we should still see what's slowing down the IPMI SoL data.
10-15 minutes for it to drain is *very* slow.

> The trade-off
> would be ssh based console would output at the speed of IPMI.

That would only be a concern if:

 - both IPMI and SSH clients are connected to the unix domain socket at
   the same time (does the IPMI SoL process connect to the obmc-console
   socket if no clients are connected over the network?)

   and

 - we've filled up the 16kB burst-buffer in the obmc-console-server
   process

So, I'd say we're fine there.

Regards,


Jeremy