Custom structured data from logger utility

Custom structured data from logger utility

[Priya Ahuja]

Hi,

I am using version 2.20.1 logger utility to send logs to rsyslogd from
bash scripts. I want to send logs in RFC5424 format with custom
structured data fields, example: [sdid@enterprise-number level=INFO
id=1234]. I understand 2.20.1 does not support 5424 format but I
looked at the logger.c code in the latest version and seems structured
data fields are set to [timeQuality tzKnown=\"1\" isSynced=\"1\"
syncAccuracy=\"%ld\"]. Is there a way/plan to format custom structured
data?

Thanks,
Priya Ahuja

Re: Custom structured data from logger utility

[Sami Kerola]

On 19 September 2015 at 00:25, Priya Ahuja <priahuja321@gmail.com> wrote:
> I am using version 2.20.1 logger utility to send logs to rsyslogd from
> bash scripts. I want to send logs in RFC5424 format with custom
> structured data fields, example: [sdid@enterprise-number level=INFO
> id=1234]. I understand 2.20.1 does not support 5424 format but I
> looked at the logger.c code in the latest version and seems structured
> data fields are set to [timeQuality tzKnown=\"1\" isSynced=\"1\"
> syncAccuracy=\"%ld\"]. Is there a way/plan to format custom structured
> data?

Hi Priya,

I am in favor of more control for structured data items.

I guess custom data could be implemented the same way as it's done with
journal, e.g., with io vectors.  I have not had a look of the code what sort
of changes would be needed in order to achieve that nice way.  Quite frankly
my time has been lately taken by various other things than contributions.
Perhaps someone else could take the lead doing in writing these changes,
would you for example fancy doing that?

-- 
Sami Kerola
http://www.iki.fi/kerolasa/

Re: Custom structured data from logger utility

[Karel Zak]

On Sat, Sep 19, 2015 at 09:50:35AM +0100, Sami Kerola wrote:
> I guess custom data could be implemented the same way as it's done with
> journal, e.g., with io vectors.

Well, for --journal we read the data from stdin. I'm not sure if this
is the right way for rfc5424 where we already use stdin to read the
message body. Maybe we can add extra option 

    --structured-data name=value

where you can specify the option more than once.

> Perhaps someone else could take the lead doing in writing these changes,
> would you for example fancy doing that?

I'll try it... ;-)

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Custom structured data from logger utility

[Karel Zak]

On Fri, Sep 18, 2015 at 04:25:29PM -0700, Priya Ahuja wrote:
> I am using version 2.20.1 logger utility to send logs to rsyslogd from
> bash scripts. I want to send logs in RFC5424 format with custom
> structured data fields, example: [sdid@enterprise-number level=INFO
> id=1234]. I understand 2.20.1 does not support 5424 format but I
> looked at the logger.c code in the latest version and seems structured
> data fields are set to [timeQuality tzKnown=\"1\" isSynced=\"1\"
> syncAccuracy=\"%ld\"]. Is there a way/plan to format custom structured
> data?

Implemented, for example:
 
     logger --rfc5424 --sd-id zoo@123                \
                      --sd-param tiger=\"hungry\"    \
                      --sd-param zebra=\"running\"   \
                      --sd-id manager@123            \
                      --sd-param onMeeting=\"yes\"   \
                      "this is message"
    
produces:
  
     <13>1 2015-10-01T14:07:59.168662+02:00 ws kzak - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="218616"][zoo@123 tiger="hungry" zebra="running"][manager@123 onMeeting="yes"] this is message


The implementation also allows to specify standardized SD-ELEMENTs like
origin, meta and timeQuality (then build-in timeQuality is disabled).

It would be nice to test it against some advanced syslog.

    Karel

-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com

Re: Custom structured data from logger utility

[Priya Ahuja]

Hi Karel,

Thanks for the changes. Is there a reason for keeping pre-set
structured data elements like timeQuality, tzKnown if custom data is
provided? Also if the SD-ID is the same, it would make more sense
merging the structured data into one, what do you think?

Thanks,
Priya Ahuja

On Thu, Oct 1, 2015 at 5:58 AM, Karel Zak <kzak@redhat.com> wrote:
> On Fri, Sep 18, 2015 at 04:25:29PM -0700, Priya Ahuja wrote:
>> I am using version 2.20.1 logger utility to send logs to rsyslogd from
>> bash scripts. I want to send logs in RFC5424 format with custom
>> structured data fields, example: [sdid@enterprise-number level=INFO
>> id=1234]. I understand 2.20.1 does not support 5424 format but I
>> looked at the logger.c code in the latest version and seems structured
>> data fields are set to [timeQuality tzKnown=\"1\" isSynced=\"1\"
>> syncAccuracy=\"%ld\"]. Is there a way/plan to format custom structured
>> data?
>
> Implemented, for example:
>
>      logger --rfc5424 --sd-id zoo@123                \
>                       --sd-param tiger=\"hungry\"    \
>                       --sd-param zebra=\"running\"   \
>                       --sd-id manager@123            \
>                       --sd-param onMeeting=\"yes\"   \
>                       "this is message"
>
> produces:
>
>      <13>1 2015-10-01T14:07:59.168662+02:00 ws kzak - - [timeQuality tzKnown="1" isSynced="1" syncAccuracy="218616"][zoo@123 tiger="hungry" zebra="running"][manager@123 onMeeting="yes"] this is message
>
>
> The implementation also allows to specify standardized SD-ELEMENTs like
> origin, meta and timeQuality (then build-in timeQuality is disabled).
>
> It would be nice to test it against some advanced syslog.
>
>     Karel
>
> --
>  Karel Zak  <kzak@redhat.com>
>  http://karelzak.blogspot.com

Re: Custom structured data from logger utility

[Karel Zak]

On Fri, Oct 02, 2015 at 11:43:30AM -0700, Priya Ahuja wrote:
> Thanks for the changes. Is there a reason for keeping pre-set
> structured data elements like timeQuality, tzKnown if custom data is
> provided?

I don't think uses want to generate things like timeQuality by shell 
on logger command line. IMHO "--sd-id timeQuality" will be used very
rarely (but it's possible and supported use case).

The another thing is that we already released logger with build-in
timeQuality and we have to keep it for backward compatibility.

> Also if the SD-ID is the same, it would make more sense
> merging the structured data into one, what do you think?

You have to use unique SD-ID and use it only once. It's pretty simple
semantic, I'm not sure if need something more complicated/complex.
(Or do you mean something else?)

For example:

 --sd-id zoo@123                \
 --sd-param tiger=\"hungry\"    \
 --sd-param zebra=\"running\"   \
 --sd-id manager@123            \
 --sd-param onMeeting=\"yes\" 


sd-id command line option works as separator between the elements, so
it generates two SD-ELEMENTS:

  [zoo@123 tiger="hungry" zebra="running"]
  [manager@123 onMeeting="yes"]


 Karel


-- 
 Karel Zak  <kzak@redhat.com>
 http://karelzak.blogspot.com