journal log messages are missing with latest version

journal log messages are missing with latest version

[Yong Li]

[-- Attachment #1: Type: text/plain, Size: 497 bytes --]

 

Hi Patrick,

 

I encountered a problem after upgrade to the latest opnebmc version:

 

These log messages in ipmid are missing with the "journalctl -f".

 

My tests shows that the blow change is related, after I revert it, I can get
these log messages.

 

Did you meet such issue, or any suggestions/comments to narrow down this
issue?

https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c
88d6a3b9a5c4d49

add sdjournal interface to inject tests

 

Thanks,

Yong


[-- Attachment #2: Type: text/html, Size: 3227 bytes --]

Re: journal log messages are missing with latest version

[Patrick Venture]

On Thu, Dec 20, 2018 at 4:38 PM Yong Li <yong.b.li@linux.intel.com> wrote:
>
>
>
> Hi Patrick,
>
>
>
> I encountered a problem after upgrade to the latest opnebmc version:
>
>
>
> These log messages in ipmid are missing with the “journalctl -f”.

Which log messages?

>
>
>
> My tests shows that the blow change is related, after I revert it, I can get these log messages.
>
>
>
> Did you meet such issue, or any suggestions/comments to narrow down this issue?
>
> https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49
>
> add sdjournal interface to inject tests

There were some follow-on patches for phosphor-logging that you
shouldn't need -- it's

What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ?
(so I know what version of that is in use by you).

Checking the HEAD version of
https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
I see it's including the proper libs to get the full phosphor-logging
objects.

>
>
>
> Thanks,
>
> Yong

Re: journal log messages are missing with latest version

[Matt Spinler]

On 2019-01-02 13:09, Patrick Venture wrote:
> On Thu, Dec 20, 2018 at 4:38 PM Yong Li <yong.b.li@linux.intel.com> 
> wrote:
>> 
>> 
>> 
>> Hi Patrick,
>> 
>> 
>> 
>> I encountered a problem after upgrade to the latest opnebmc version:
>> 
>> 
>> 
>> These log messages in ipmid are missing with the “journalctl -f”.
> 
> Which log messages?


This showed up on all of our systems too, anything logged using 
phosphor-logging
is missing from the journal.  I even ended up using a git bisect to also 
narrow
it down to:

phosphor-logging: srcrev bump 30047bf964..31552c05e2

     Patrick Venture (5):
           add sdjournal interface to inject tests
           test: use only one main library
           bugfix: add fixes to mock class
           bugfix: rename method to match implementation
           test: add test to force building of sdjournal mock


> 
>> 
>> 
>> 
>> My tests shows that the blow change is related, after I revert it, I 
>> can get these log messages.
>> 
>> 
>> 
>> Did you meet such issue, or any suggestions/comments to narrow down 
>> this issue?
>> 
>> https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49
>> 
>> add sdjournal interface to inject tests
> 
> There were some follow-on patches for phosphor-logging that you
> shouldn't need -- it's
> 
> What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ?
> (so I know what version of that is in use by you).
> 
> Checking the HEAD version of
> https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
> I see it's including the proper libs to get the full phosphor-logging
> objects.
> 
>> 
>> 
>> 
>> Thanks,
>> 
>> Yong

Re: journal log messages are missing with latest version

[Patrick Venture]

On Thu, Jan 3, 2019 at 10:07 AM Matt Spinler
<mspinler@linux.vnet.ibm.com> wrote:
>
> On 2019-01-02 13:09, Patrick Venture wrote:
> > On Thu, Dec 20, 2018 at 4:38 PM Yong Li <yong.b.li@linux.intel.com>
> > wrote:
> >>
> >>
> >>
> >> Hi Patrick,
> >>
> >>
> >>
> >> I encountered a problem after upgrade to the latest opnebmc version:
> >>
> >>
> >>
> >> These log messages in ipmid are missing with the “journalctl -f”.
> >
> > Which log messages?
>
>
> This showed up on all of our systems too, anything logged using
> phosphor-logging
> is missing from the journal.  I even ended up using a git bisect to also
> narrow
> it down to:
>
> phosphor-logging: srcrev bump 30047bf964..31552c05e2
>
>      Patrick Venture (5):
>            add sdjournal interface to inject tests
>            test: use only one main library
>            bugfix: add fixes to mock class
>            bugfix: rename method to match implementation
>            test: add test to force building of sdjournal mock
>

William, you've been digging into sdbus a lot recently, can you take a
deep look at the patchset above and see what I may have messed up?
The phosphor-logging unit-tests were broken until I got it "working."
Which lead me to believe that it's not correct -- and the daemons that
have unit-tests that hit the log message are seeing the calls.

>
> >
> >>
> >>
> >>
> >> My tests shows that the blow change is related, after I revert it, I
> >> can get these log messages.
> >>
> >>
> >>
> >> Did you meet such issue, or any suggestions/comments to narrow down
> >> this issue?
> >>
> >> https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49
> >>
> >> add sdjournal interface to inject tests
> >
> > There were some follow-on patches for phosphor-logging that you
> > shouldn't need -- it's
> >
> > What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ?
> > (so I know what version of that is in use by you).
> >
> > Checking the HEAD version of
> > https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
> > I see it's including the proper libs to get the full phosphor-logging
> > objects.
> >
> >>
> >>
> >>
> >> Thanks,
> >>
> >> Yong
>

Re: journal log messages are missing with latest version

[Patrick Venture]

On Thu, Jan 3, 2019 at 11:19 AM Patrick Venture <venture@google.com> wrote:
>
> On Thu, Jan 3, 2019 at 10:07 AM Matt Spinler
> <mspinler@linux.vnet.ibm.com> wrote:
> >
> > On 2019-01-02 13:09, Patrick Venture wrote:
> > > On Thu, Dec 20, 2018 at 4:38 PM Yong Li <yong.b.li@linux.intel.com>
> > > wrote:
> > >>
> > >>
> > >>
> > >> Hi Patrick,
> > >>
> > >>
> > >>
> > >> I encountered a problem after upgrade to the latest opnebmc version:
> > >>
> > >>
> > >>
> > >> These log messages in ipmid are missing with the “journalctl -f”.
> > >
> > > Which log messages?
> >
> >
> > This showed up on all of our systems too, anything logged using
> > phosphor-logging
> > is missing from the journal.  I even ended up using a git bisect to also
> > narrow
> > it down to:
> >
> > phosphor-logging: srcrev bump 30047bf964..31552c05e2
> >
> >      Patrick Venture (5):
> >            add sdjournal interface to inject tests
> >            test: use only one main library
> >            bugfix: add fixes to mock class
> >            bugfix: rename method to match implementation
> >            test: add test to force building of sdjournal mock
> >
>
> William, you've been digging into sdbus a lot recently, can you take a
> deep look at the patchset above and see what I may have messed up?
> The phosphor-logging unit-tests were broken until I got it "working."
> Which lead me to believe that it's not correct -- and the daemons that
> have unit-tests that hit the log message are seeing the calls.

From reading more into the var_args pass-through, that may be the
issue.  One cannot pass the var args down directly like this.  I
propose to initially snip out this extra call and then revisit so that
it'll resume working.

The problem is that sd_journal_send doesn't expect a va_list argument,
but rather strictly var args, which aren't the same.

>
> >
> > >
> > >>
> > >>
> > >>
> > >> My tests shows that the blow change is related, after I revert it, I
> > >> can get these log messages.
> > >>
> > >>
> > >>
> > >> Did you meet such issue, or any suggestions/comments to narrow down
> > >> this issue?
> > >>
> > >> https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49
> > >>
> > >> add sdjournal interface to inject tests
> > >
> > > There were some follow-on patches for phosphor-logging that you
> > > shouldn't need -- it's
> > >
> > > What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ?
> > > (so I know what version of that is in use by you).
> > >
> > > Checking the HEAD version of
> > > https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
> > > I see it's including the proper libs to get the full phosphor-logging
> > > objects.
> > >
> > >>
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Yong
> >

Re: journal log messages are missing with latest version

[Patrick Venture]

On Thu, Jan 3, 2019 at 2:28 PM Patrick Venture <venture@google.com> wrote:
>
> On Thu, Jan 3, 2019 at 11:19 AM Patrick Venture <venture@google.com> wrote:
> >
> > On Thu, Jan 3, 2019 at 10:07 AM Matt Spinler
> > <mspinler@linux.vnet.ibm.com> wrote:
> > >
> > > On 2019-01-02 13:09, Patrick Venture wrote:
> > > > On Thu, Dec 20, 2018 at 4:38 PM Yong Li <yong.b.li@linux.intel.com>
> > > > wrote:
> > > >>
> > > >>
> > > >>
> > > >> Hi Patrick,
> > > >>
> > > >>
> > > >>
> > > >> I encountered a problem after upgrade to the latest opnebmc version:
> > > >>
> > > >>
> > > >>
> > > >> These log messages in ipmid are missing with the “journalctl -f”.
> > > >
> > > > Which log messages?
> > >
> > >
> > > This showed up on all of our systems too, anything logged using
> > > phosphor-logging
> > > is missing from the journal.  I even ended up using a git bisect to also
> > > narrow
> > > it down to:
> > >
> > > phosphor-logging: srcrev bump 30047bf964..31552c05e2
> > >
> > >      Patrick Venture (5):
> > >            add sdjournal interface to inject tests
> > >            test: use only one main library
> > >            bugfix: add fixes to mock class
> > >            bugfix: rename method to match implementation
> > >            test: add test to force building of sdjournal mock
> > >
> >
> > William, you've been digging into sdbus a lot recently, can you take a
> > deep look at the patchset above and see what I may have messed up?
> > The phosphor-logging unit-tests were broken until I got it "working."
> > Which lead me to believe that it's not correct -- and the daemons that
> > have unit-tests that hit the log message are seeing the calls.
>
> From reading more into the var_args pass-through, that may be the
> issue.  One cannot pass the var args down directly like this.  I
> propose to initially snip out this extra call and then revisit so that
> it'll resume working.

https://gerrit.openbmc-project.xyz/17054
>
> The problem is that sd_journal_send doesn't expect a va_list argument,
> but rather strictly var args, which aren't the same.
>
> >
> > >
> > > >
> > > >>
> > > >>
> > > >>
> > > >> My tests shows that the blow change is related, after I revert it, I
> > > >> can get these log messages.
> > > >>
> > > >>
> > > >>
> > > >> Did you meet such issue, or any suggestions/comments to narrow down
> > > >> this issue?
> > > >>
> > > >> https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49
> > > >>
> > > >> add sdjournal interface to inject tests
> > > >
> > > > There were some follow-on patches for phosphor-logging that you
> > > > shouldn't need -- it's
> > > >
> > > > What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ?
> > > > (so I know what version of that is in use by you).
> > > >
> > > > Checking the HEAD version of
> > > > https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
> > > > I see it's including the proper libs to get the full phosphor-logging
> > > > objects.
> > > >
> > > >>
> > > >>
> > > >>
> > > >> Thanks,
> > > >>
> > > >> Yong
> > >

Re: journal log messages are missing with latest version

[Li, Yong B]

Thanks Patrick for your mails.

Will test https://gerrit.openbmc-project.xyz/17054, and update the results.

On 1/4/2019 6:33 AM, Patrick Venture wrote:
> On Thu, Jan 3, 2019 at 2:28 PM Patrick Venture <venture@google.com> wrote:
>> On Thu, Jan 3, 2019 at 11:19 AM Patrick Venture <venture@google.com> wrote:
>>> On Thu, Jan 3, 2019 at 10:07 AM Matt Spinler
>>> <mspinler@linux.vnet.ibm.com> wrote:
>>>> On 2019-01-02 13:09, Patrick Venture wrote:
>>>>> On Thu, Dec 20, 2018 at 4:38 PM Yong Li <yong.b.li@linux.intel.com>
>>>>> wrote:
>>>>>>
>>>>>>
>>>>>> Hi Patrick,
>>>>>>
>>>>>>
>>>>>>
>>>>>> I encountered a problem after upgrade to the latest opnebmc version:
>>>>>>
>>>>>>
>>>>>>
>>>>>> These log messages in ipmid are missing with the “journalctl -f”.
>>>>> Which log messages?
>>>>
>>>> This showed up on all of our systems too, anything logged using
>>>> phosphor-logging
>>>> is missing from the journal.  I even ended up using a git bisect to also
>>>> narrow
>>>> it down to:
>>>>
>>>> phosphor-logging: srcrev bump 30047bf964..31552c05e2
>>>>
>>>>       Patrick Venture (5):
>>>>             add sdjournal interface to inject tests
>>>>             test: use only one main library
>>>>             bugfix: add fixes to mock class
>>>>             bugfix: rename method to match implementation
>>>>             test: add test to force building of sdjournal mock
>>>>
>>> William, you've been digging into sdbus a lot recently, can you take a
>>> deep look at the patchset above and see what I may have messed up?
>>> The phosphor-logging unit-tests were broken until I got it "working."
>>> Which lead me to believe that it's not correct -- and the daemons that
>>> have unit-tests that hit the log message are seeing the calls.
>>  From reading more into the var_args pass-through, that may be the
>> issue.  One cannot pass the var args down directly like this.  I
>> propose to initially snip out this extra call and then revisit so that
>> it'll resume working.
> https://gerrit.openbmc-project.xyz/17054
>> The problem is that sd_journal_send doesn't expect a va_list argument,
>> but rather strictly var args, which aren't the same.
>>
>>>>>>
>>>>>>
>>>>>> My tests shows that the blow change is related, after I revert it, I
>>>>>> can get these log messages.
>>>>>>
>>>>>>
>>>>>>
>>>>>> Did you meet such issue, or any suggestions/comments to narrow down
>>>>>> this issue?
>>>>>>
>>>>>> https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49
>>>>>>
>>>>>> add sdjournal interface to inject tests
>>>>> There were some follow-on patches for phosphor-logging that you
>>>>> shouldn't need -- it's
>>>>>
>>>>> What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ?
>>>>> (so I know what version of that is in use by you).
>>>>>
>>>>> Checking the HEAD version of
>>>>> https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
>>>>> I see it's including the proper libs to get the full phosphor-logging
>>>>> objects.
>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Yong

Re: journal log messages are missing with latest version

[Patrick Venture]

On Thu, Jan 3, 2019 at 5:20 PM Li, Yong B <yong.b.li@linux.intel.com> wrote:
>
> Thanks Patrick for your mails.
>
> Will test https://gerrit.openbmc-project.xyz/17054, and update the results.
>
> On 1/4/2019 6:33 AM, Patrick Venture wrote:
> > On Thu, Jan 3, 2019 at 2:28 PM Patrick Venture <venture@google.com> wrote:
> >> On Thu, Jan 3, 2019 at 11:19 AM Patrick Venture <venture@google.com> wrote:
> >>> On Thu, Jan 3, 2019 at 10:07 AM Matt Spinler
> >>> <mspinler@linux.vnet.ibm.com> wrote:
> >>>> On 2019-01-02 13:09, Patrick Venture wrote:
> >>>>> On Thu, Dec 20, 2018 at 4:38 PM Yong Li <yong.b.li@linux.intel.com>
> >>>>> wrote:
> >>>>>>
> >>>>>>
> >>>>>> Hi Patrick,
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I encountered a problem after upgrade to the latest opnebmc version:
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> These log messages in ipmid are missing with the “journalctl -f”.
> >>>>> Which log messages?
> >>>>
> >>>> This showed up on all of our systems too, anything logged using
> >>>> phosphor-logging
> >>>> is missing from the journal.  I even ended up using a git bisect to also
> >>>> narrow
> >>>> it down to:
> >>>>
> >>>> phosphor-logging: srcrev bump 30047bf964..31552c05e2
> >>>>
> >>>>       Patrick Venture (5):
> >>>>             add sdjournal interface to inject tests
> >>>>             test: use only one main library
> >>>>             bugfix: add fixes to mock class
> >>>>             bugfix: rename method to match implementation
> >>>>             test: add test to force building of sdjournal mock
> >>>>
> >>> William, you've been digging into sdbus a lot recently, can you take a
> >>> deep look at the patchset above and see what I may have messed up?
> >>> The phosphor-logging unit-tests were broken until I got it "working."
> >>> Which lead me to believe that it's not correct -- and the daemons that
> >>> have unit-tests that hit the log message are seeing the calls.
> >>  From reading more into the var_args pass-through, that may be the
> >> issue.  One cannot pass the var args down directly like this.  I
> >> propose to initially snip out this extra call and then revisit so that
> >> it'll resume working.
> > https://gerrit.openbmc-project.xyz/17054
> >> The problem is that sd_journal_send doesn't expect a va_list argument,
> >> but rather strictly var args, which aren't the same.

I generally try to avoid copying implementations, but an approach to
this that would work and allow testing would be to use
sd_journal_sendv() and have the interface call walk the parameters and
build a list of iovec structures and then pass that.  This is
effectively what is done by sd_journal_send, which just processes the
input and sends to sd_journal_sendv.  We could also pass to
sd_journal_print which takes a va_list, however, the behavior isn't
identical for structured log entries.

So my question is, if we lift the code from systemd and make it work,
that'll presumably add another license piece to phosphor-logging ---
and I'm not really sure how that works.

> >>
> >>>>>>
> >>>>>>
> >>>>>> My tests shows that the blow change is related, after I revert it, I
> >>>>>> can get these log messages.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> Did you meet such issue, or any suggestions/comments to narrow down
> >>>>>> this issue?
> >>>>>>
> >>>>>> https://github.com/openbmc/phosphor-logging/commit/59a6b1f27e83066baa6f3711c88d6a3b9a5c4d49
> >>>>>>
> >>>>>> add sdjournal interface to inject tests
> >>>>> There were some follow-on patches for phosphor-logging that you
> >>>>> shouldn't need -- it's
> >>>>>
> >>>>> What SHA is for your https://github.com/openbmc/phosphor-host-ipmid ?
> >>>>> (so I know what version of that is in use by you).
> >>>>>
> >>>>> Checking the HEAD version of
> >>>>> https://github.com/openbmc/phosphor-host-ipmid/blob/master/Makefile.am,
> >>>>> I see it's including the proper libs to get the full phosphor-logging
> >>>>> objects.
> >>>>>
> >>>>>>
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Yong
>