A question about phosphor-sel-logger

A question about phosphor-sel-logger

[Will Liang (梁永鉉)]

Hi,

Our BMC is using package "phosphor-sel-logger" to support IPMI SEL. it saved the event data in the journal and the id is 1-65534. 
It means it can save 65534 records. Actually, we found the file size of single journal file is around 8MB(the default of journal file is 64MB, it will be separated by 8 files).
We cannot show all SELs(65534 records) in IPMI. it might be a bug here if we don't limit the maximum.

Is it necessary to limit the number of SELs to smaller?

BRs,
Will

Re: A question about phosphor-sel-logger

[Bills, Jason M]

On 5/1/2019 7:36 PM, Will Liang (梁永鉉) wrote:
> Hi,
> 
> Our BMC is using package "phosphor-sel-logger" to support IPMI SEL. it saved the event data in the journal and the id is 1-65534.
> It means it can save 65534 records. Actually, we found the file size of single journal file is around 8MB(the default of journal file is 64MB, it will be separated by 8 files).
> We cannot show all SELs(65534 records) in IPMI. it might be a bug here if we don't limit the maximum.
> 
> Is it necessary to limit the number of SELs to smaller?

Hi Will,

The reason that value is left at 65534 is because in the journal we 
don't have a way to determine how many total SEL entries are possible or 
how much space is left.

With the journal set at 64MB we could see more than 30000 entries.  When 
I made the journal persistent at 6MB, we could see anywhere between 
1200-2000 entries.

We could choose a smaller number for IPMI to report, but it will be 
arbitrary because the journal size can be changed without notifying IPMI.

However, because of the size (1200 entries in 6MB) and other limitations 
with a persistent journal, I'm experimenting with using rsyslog to log 
SEL entries into a syslog file.  This will allow the journal to remain 
volatile while persisting only the logs that we need to.  It will also 
have a more consistent size, so it will be possible to fix the size to a 
smaller number more accurately as you suggest.

You can see my current rsyslog changes here: 
https://gerrit.openbmc-project.xyz/q/topic:"rsyslog+SEL"+(status:open%20OR%20status:merged)

If you are interested, I will also share my rsyslog configuration, but I 
was planning to test it out internally for a while to make sure it will 
work so I don't cause thrash if it has to change again.

Thanks,
-Jason

> 
> BRs,
> Will
> 

RE: A question about phosphor-sel-logger

[Will Liang (梁永鉉)]

> 
> On 5/1/2019 7:36 PM, Will Liang (梁永鉉) wrote:
> > Hi,
> >
> > Our BMC is using package "phosphor-sel-logger" to support IPMI SEL. it
> saved the event data in the journal and the id is 1-65534.
> > It means it can save 65534 records. Actually, we found the file size of single
> journal file is around 8MB(the default of journal file is 64MB, it will be
> separated by 8 files).
> > We cannot show all SELs(65534 records) in IPMI. it might be a bug here if we
> don't limit the maximum.
> >
> > Is it necessary to limit the number of SELs to smaller?
> 
> Hi Will,
> 
> The reason that value is left at 65534 is because in the journal we don't have a
> way to determine how many total SEL entries are possible or how much space
> is left.
> 
> With the journal set at 64MB we could see more than 30000 entries.  When I
> made the journal persistent at 6MB, we could see anywhere between
> 1200-2000 entries.
> 
> We could choose a smaller number for IPMI to report, but it will be arbitrary
> because the journal size can be changed without notifying IPMI.
> 
> However, because of the size (1200 entries in 6MB) and other limitations with
> a persistent journal, I'm experimenting with using rsyslog to log SEL entries
> into a syslog file.  This will allow the journal to remain volatile while
> persisting only the logs that we need to.  It will also have a more consistent
> size, so it will be possible to fix the size to a smaller number more accurately
> as you suggest.
> 
> You can see my current rsyslog changes here:
> https://gerrit.openbmc-project.xyz/q/topic:"rsyslog+SEL"+(status:open%20OR%
> 20status:merged)
> 
> If you are interested, I will also share my rsyslog configuration, but I was
> planning to test it out internally for a while to make sure it will work so I don't
> cause thrash if it has to change again.

Hi Jason,

I am very thankful that you reply me quickly and it's great to hear that you are working on this issue.
If you can share your rsyslog configuration with me that will be a good news for me, I can trace it first.

According to the above URL, is it something I missed, I did not find the "add sel" to sys log function?
I wander to know how you add sel to sys log. Or you want to share it after internal testing.

Thanks,
Will

> Thanks,
> -Jason
> >
> > BRs,
> > Will
> >

Re: A question about phosphor-sel-logger

[Bills, Jason M]

On 5/2/2019 8:39 PM, Will Liang (梁永鉉) wrote:
> Hi Jason,
> 
> I am very thankful that you reply me quickly and it's great to hear that you are working on this issue.
> If you can share your rsyslog configuration with me that will be a good news for me, I can trace it first.
I will share my rsyslog configuration file below.  It requires the 
imjournal module to be installed which can be done by adding 
'PACKAGECONFIG_append = " imjournal"' to an rsyslog_%.bbappend file.

> 
> According to the above URL, is it something I missed, I did not find the "add sel" to sys log function?

If the imjournal rsyslog module works well enough, then the "add sel" 
function won't need to change.  SEL entries will still be added to the 
journal and the imjournal module will read the journal to find new SEL 
entries for rsyslog to save to the SEL log file.

> I wander to know how you add sel to sys log. Or you want to share it after internal testing.
> 
I'm willing to share whatever you are interested in, but it may change 
if internal testing finds issues.  I'm planning to push the entire 
solution once we complete initial internal testing.

For clearing the log to work, you need to add "Reload" support to the 
rsyslog service file.  I do this by adding this line to an override 
.conf file:
ExecReload=/bin/kill -HUP $MAINPID

Here is my rsyslog.conf file which configures the entries the imjournal 
module is looking for and how to format them in the syslog files:
# if you experience problems, check
# http://www.rsyslog.com/troubleshoot for assistance

# rsyslog v3: load input modules
# If you do not load inputs, nothing happens!
# You may need to set the module load path if modules are not found.
#
# Ported from debian's sysklogd.conf

# Journal-style logging
# Limit to no more than 2000 entries in one minute and enable the
# journal workaround to avoid duplicate entries
module(load="imjournal" StateFile="/var/log/state"
                         RateLimit.Interval="60"
                         RateLimit.Burst="2000"
                         WorkAroundJournalBug="on")

# Template for IPMI SEL messages
# "<timestamp> <ID>,<Type>,<EventData>,[<Generator ID>,<Path>,<Direction>]"
template(name="IPMISELTemplate" type="list") {
     property(name="timereported" dateFormat="rfc3339")
     constant(value=" ")
     property(name="$!IPMI_SEL_RECORD_ID")
     constant(value=",")
     property(name="$!IPMI_SEL_RECORD_TYPE")
     constant(value=",")
     property(name="$!IPMI_SEL_DATA")
     constant(value=",")
     property(name="$!IPMI_SEL_GENERATOR_ID")
     constant(value=",")
     property(name="$!IPMI_SEL_SENSOR_PATH")
     constant(value=",")
     property(name="$!IPMI_SEL_EVENT_DIR")
     constant(value="\n")
}

# Template for Redfish messages
# "<timestamp> <MessageId>,<MessageArgs>"
template(name="RedfishTemplate" type="list") {
     property(name="timereported" dateFormat="rfc3339")
     constant(value=" ")
     property(name="$!REDFISH_MESSAGE_ID")
     constant(value=",")
     property(name="$!REDFISH_MESSAGE_ARGS")
     constant(value="\n")
}

# If the journal entry has the IPMI SEL MESSAGE_ID, save as IPMI SEL
if ($!MESSAGE_ID == "b370836ccf2f4850ac5bee185b77893a") then {
    action(type="omfile" file="/var/log/ipmi_sel" 
template="IPMISELTemplate")
}

# If the journal entry has a Redfish MessageId, save as a Redfish event
if ($!REDFISH_MESSAGE_ID != "") then {
    action(type="omfile" file="/var/log/redfish" template="RedfishTemplate")
}

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf