[PATCH] systemd: Instruct the journal to always be volatile

[PATCH] systemd: Instruct the journal to always be volatile

[Randy Witt]

When the storage mode for the journal is "auto" if /var/log/journal
exists then the journal will flush to /var/log/journal assuming that
/var/log/journal is persistent.

However /var/log -> /var/volatile/log in poky, so even though
/var/log/journal exists, it is still volatile.

Since this can cause ordering issues due to /var/volatile needing to be mounted before the journal actually writes to it, just specify that the journal
should always be volatile and never try to write to "persistent"
storage. The journal will exist in /run/log/journal only.

This also disables the "After" of the journal on var-volatile.mount
since the ordering is no longer necessary when the journal is only
stored in /run/log/journal.

[Yocto #7388]

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
---
 meta/recipes-core/systemd/systemd/journald-volatile.conf | 6 ------
 meta/recipes-core/systemd/systemd_219.bb                 | 4 ++--
 2 files changed, 2 insertions(+), 8 deletions(-)
 delete mode 100644 meta/recipes-core/systemd/systemd/journald-volatile.conf

diff --git a/meta/recipes-core/systemd/systemd/journald-volatile.conf b/meta/recipes-core/systemd/systemd/journald-volatile.conf
deleted file mode 100644
index b11e160..0000000
--- a/meta/recipes-core/systemd/systemd/journald-volatile.conf
+++ /dev/null
@@ -1,6 +0,0 @@
-# If /var/volatile is a mount point then make sure to mount it before
-# the journal starts. This is because base-files creates a symlink
-# /var/log -> /var/volatile/log. And if the journal starts before the mount
-# happens, the journal will appear empty until restarted.
-[Unit]
-After=var-volatile.mount
diff --git a/meta/recipes-core/systemd/systemd_219.bb b/meta/recipes-core/systemd/systemd_219.bb
index d5eed08..f3bb0c4 100644
--- a/meta/recipes-core/systemd/systemd_219.bb
+++ b/meta/recipes-core/systemd/systemd_219.bb
@@ -48,7 +48,6 @@ SRC_URI = "git://anongit.freedesktop.org/systemd/systemd;branch=master;protocol=
            file://00-create-volatile.conf \
            file://init \
            file://run-ptest \
-           file://journald-volatile.conf \
           "
 
 S = "${WORKDIR}/git"
@@ -145,7 +144,6 @@ do_install() {
 	install -m 0644 ${WORKDIR}/*.rules ${D}${sysconfdir}/udev/rules.d/
 
 	install -m 0644 ${WORKDIR}/00-create-volatile.conf ${D}${sysconfdir}/tmpfiles.d/
-	install -D -m 0644 ${WORKDIR}/journald-volatile.conf ${D}${systemd_unitdir}/system/systemd-journald.service.d/journald-volatile.conf
 
 	if ${@bb.utils.contains('DISTRO_FEATURES','sysvinit','true','false',d)}; then
 		install -d ${D}${sysconfdir}/init.d
@@ -170,6 +168,8 @@ do_install() {
 
 	# Enable journal to forward message to syslog daemon
 	sed -i -e 's/.*ForwardToSyslog.*/ForwardToSyslog=yes/' ${D}${sysconfdir}/systemd/journald.conf
+	# Make the journal volatile by default.
+	sed -i -e 's/.*Storage.*/Storage=volatile/' ${D}${sysconfdir}/systemd/journald.conf
 	# its needed in 216 upstream has fixed it with 919699ec301ea507edce4a619141ed22e789ac0d
 	# don't order journal flushing afte remote-fs.target
 	sed -i -e 's/ remote-fs.target$//' ${D}${systemd_unitdir}/system/systemd-journal-flush.service
-- 
1.9.3

Re: [PATCH] systemd: Instruct the journal to always be volatile

[Khem Raj]

> On Mar 11, 2015, at 2:28 PM, Randy Witt <randy.e.witt@linux.intel.com> wrote:
> 
> When the storage mode for the journal is "auto" if /var/log/journal
> exists then the journal will flush to /var/log/journal assuming that
> /var/log/journal is persistent.
> 
> However /var/log -> /var/volatile/log in poky, so even though
> /var/log/journal exists, it is still volatile.
> 
> Since this can cause ordering issues due to /var/volatile needing to be mounted before the journal actually writes to it, just specify that the journal
> should always be volatile and never try to write to "persistent"
> storage. The journal will exist in /run/log/journal only.
> 
> This also disables the "After" of the journal on var-volatile.mount
> since the ordering is no longer necessary when the journal is only
> stored in /run/log/journal.
> 
> [Yocto #7388]


This is not right. What if I want persistent logs.? the options is just gone. Both cases should work. /var/volatile should only be used
when doing ro-rfs its not FHS specified anyway. Lets not go with this patch.

> 
> Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
> ---
> meta/recipes-core/systemd/systemd/journald-volatile.conf | 6 ------
> meta/recipes-core/systemd/systemd_219.bb                 | 4 ++--
> 2 files changed, 2 insertions(+), 8 deletions(-)
> delete mode 100644 meta/recipes-core/systemd/systemd/journald-volatile.conf
> 
> diff --git a/meta/recipes-core/systemd/systemd/journald-volatile.conf b/meta/recipes-core/systemd/systemd/journald-volatile.conf
> deleted file mode 100644
> index b11e160..0000000
> --- a/meta/recipes-core/systemd/systemd/journald-volatile.conf
> +++ /dev/null
> @@ -1,6 +0,0 @@
> -# If /var/volatile is a mount point then make sure to mount it before
> -# the journal starts. This is because base-files creates a symlink
> -# /var/log -> /var/volatile/log. And if the journal starts before the mount
> -# happens, the journal will appear empty until restarted.
> -[Unit]
> -After=var-volatile.mount
> diff --git a/meta/recipes-core/systemd/systemd_219.bb b/meta/recipes-core/systemd/systemd_219.bb
> index d5eed08..f3bb0c4 100644
> --- a/meta/recipes-core/systemd/systemd_219.bb
> +++ b/meta/recipes-core/systemd/systemd_219.bb
> @@ -48,7 +48,6 @@ SRC_URI = "git://anongit.freedesktop.org/systemd/systemd;branch=master;protocol=
>            file://00-create-volatile.conf \
>            file://init \
>            file://run-ptest \
> -           file://journald-volatile.conf \
>           "
> 
> S = "${WORKDIR}/git"
> @@ -145,7 +144,6 @@ do_install() {
> 	install -m 0644 ${WORKDIR}/*.rules ${D}${sysconfdir}/udev/rules.d/
> 
> 	install -m 0644 ${WORKDIR}/00-create-volatile.conf ${D}${sysconfdir}/tmpfiles.d/
> -	install -D -m 0644 ${WORKDIR}/journald-volatile.conf ${D}${systemd_unitdir}/system/systemd-journald.service.d/journald-volatile.conf
> 
> 	if ${@bb.utils.contains('DISTRO_FEATURES','sysvinit','true','false',d)}; then
> 		install -d ${D}${sysconfdir}/init.d
> @@ -170,6 +168,8 @@ do_install() {
> 
> 	# Enable journal to forward message to syslog daemon
> 	sed -i -e 's/.*ForwardToSyslog.*/ForwardToSyslog=yes/' ${D}${sysconfdir}/systemd/journald.conf
> +	# Make the journal volatile by default.
> +	sed -i -e 's/.*Storage.*/Storage=volatile/' ${D}${sysconfdir}/systemd/journald.conf
> 	# its needed in 216 upstream has fixed it with 919699ec301ea507edce4a619141ed22e789ac0d
> 	# don't order journal flushing afte remote-fs.target
> 	sed -i -e 's/ remote-fs.target$//' ${D}${systemd_unitdir}/system/systemd-journal-flush.service
> -- 
> 1.9.3
> 
> -- 
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH] systemd: Instruct the journal to always be volatile

[randy.e.witt]

>
>> On Mar 11, 2015, at 2:28 PM, Randy Witt <randy.e.witt@linux.intel.com>
>> wrote:
>>
>> When the storage mode for the journal is "auto" if /var/log/journal
>> exists then the journal will flush to /var/log/journal assuming that
>> /var/log/journal is persistent.
>>
>> However /var/log -> /var/volatile/log in poky, so even though
>> /var/log/journal exists, it is still volatile.
>>
>> Since this can cause ordering issues due to /var/volatile needing to be
>> mounted before the journal actually writes to it, just specify that the
>> journal
>> should always be volatile and never try to write to "persistent"
>> storage. The journal will exist in /run/log/journal only.
>>
>> This also disables the "After" of the journal on var-volatile.mount
>> since the ordering is no longer necessary when the journal is only
>> stored in /run/log/journal.
>>
>> [Yocto #7388]
>
>
> This is not right. What if I want persistent logs.? the options is just
> gone. Both cases should work. /var/volatile should only be used
> when doing ro-rfs its not FHS specified anyway. Lets not go with this
> patch.
The distro could still change it with a bbappend or additional
configuration files just like other settings. We don't have an option for
a persistent journal so I'm curious, how would both cases work currently?

Currently if a persistent journal is desired, the /var/log symlink has to
be changed to point to persistent storage, or /var/volatile has to be
dropped from /etc/fstab. These are also configuration changes.

The lesser of the two evils seems to be making it so that the journal
works in the volatile case, which is all we've ever supported in poky. If
a user wanted a non-volatile journal it was left to them to do.

I agree that the journal should be persistent, but I also agree with
Richard that seems to be too big of a change in behavior when we're so
close to the release point.

>>
>> Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
>> ---
>> meta/recipes-core/systemd/systemd/journald-volatile.conf | 6 ------
>> meta/recipes-core/systemd/systemd_219.bb                 | 4 ++--
>> 2 files changed, 2 insertions(+), 8 deletions(-)
>> delete mode 100644
>> meta/recipes-core/systemd/systemd/journald-volatile.conf
>>
>> diff --git a/meta/recipes-core/systemd/systemd/journald-volatile.conf
>> b/meta/recipes-core/systemd/systemd/journald-volatile.conf
>> deleted file mode 100644
>> index b11e160..0000000
>> --- a/meta/recipes-core/systemd/systemd/journald-volatile.conf
>> +++ /dev/null
>> @@ -1,6 +0,0 @@
>> -# If /var/volatile is a mount point then make sure to mount it before
>> -# the journal starts. This is because base-files creates a symlink
>> -# /var/log -> /var/volatile/log. And if the journal starts before the
>> mount
>> -# happens, the journal will appear empty until restarted.
>> -[Unit]
>> -After=var-volatile.mount
>> diff --git a/meta/recipes-core/systemd/systemd_219.bb
>> b/meta/recipes-core/systemd/systemd_219.bb
>> index d5eed08..f3bb0c4 100644
>> --- a/meta/recipes-core/systemd/systemd_219.bb
>> +++ b/meta/recipes-core/systemd/systemd_219.bb
>> @@ -48,7 +48,6 @@ SRC_URI =
>> "git://anongit.freedesktop.org/systemd/systemd;branch=master;protocol=
>>            file://00-create-volatile.conf \
>>            file://init \
>>            file://run-ptest \
>> -           file://journald-volatile.conf \
>>           "
>>
>> S = "${WORKDIR}/git"
>> @@ -145,7 +144,6 @@ do_install() {
>> 	install -m 0644 ${WORKDIR}/*.rules ${D}${sysconfdir}/udev/rules.d/
>>
>> 	install -m 0644 ${WORKDIR}/00-create-volatile.conf
>> ${D}${sysconfdir}/tmpfiles.d/
>> -	install -D -m 0644 ${WORKDIR}/journald-volatile.conf
>> ${D}${systemd_unitdir}/system/systemd-journald.service.d/journald-volatile.conf
>>
>> 	if
>> ${@bb.utils.contains('DISTRO_FEATURES','sysvinit','true','false',d)};
>> then
>> 		install -d ${D}${sysconfdir}/init.d
>> @@ -170,6 +168,8 @@ do_install() {
>>
>> 	# Enable journal to forward message to syslog daemon
>> 	sed -i -e 's/.*ForwardToSyslog.*/ForwardToSyslog=yes/'
>> ${D}${sysconfdir}/systemd/journald.conf
>> +	# Make the journal volatile by default.
>> +	sed -i -e 's/.*Storage.*/Storage=volatile/'
>> ${D}${sysconfdir}/systemd/journald.conf
>> 	# its needed in 216 upstream has fixed it with
>> 919699ec301ea507edce4a619141ed22e789ac0d
>> 	# don't order journal flushing afte remote-fs.target
>> 	sed -i -e 's/ remote-fs.target$//'
>> ${D}${systemd_unitdir}/system/systemd-journal-flush.service
>> --
>> 1.9.3
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>
>

Re: [PATCH] systemd: Instruct the journal to always be volatile

[Khem Raj]

On Wed, Mar 11, 2015 at 9:18 PM,  <randy.e.witt@linux.intel.com> wrote:
>>
>>> On Mar 11, 2015, at 2:28 PM, Randy Witt <randy.e.witt@linux.intel.com>
>>> wrote:
>>>
>>> When the storage mode for the journal is "auto" if /var/log/journal
>>> exists then the journal will flush to /var/log/journal assuming that
>>> /var/log/journal is persistent.
>>>
>>> However /var/log -> /var/volatile/log in poky, so even though
>>> /var/log/journal exists, it is still volatile.
>>>
>>> Since this can cause ordering issues due to /var/volatile needing to be
>>> mounted before the journal actually writes to it, just specify that the
>>> journal
>>> should always be volatile and never try to write to "persistent"
>>> storage. The journal will exist in /run/log/journal only.
>>>
>>> This also disables the "After" of the journal on var-volatile.mount
>>> since the ordering is no longer necessary when the journal is only
>>> stored in /run/log/journal.
>>>
>>> [Yocto #7388]
>>
>>
>> This is not right. What if I want persistent logs.? the options is just
>> gone. Both cases should work. /var/volatile should only be used
>> when doing ro-rfs its not FHS specified anyway. Lets not go with this
>> patch.
> The distro could still change it with a bbappend or additional
> configuration files just like other settings. We don't have an option for
> a persistent journal so I'm curious, how would both cases work currently?
>

No they dont and I have posted a patch which potentially should.

> Currently if a persistent journal is desired, the /var/log symlink has to
> be changed to point to persistent storage, or /var/volatile has to be
> dropped from /etc/fstab. These are also configuration changes.
>
> The lesser of the two evils seems to be making it so that the journal
> works in the volatile case, which is all we've ever supported in poky. If
> a user wanted a non-volatile journal it was left to them to do.

poky is not only the distro thats fed with OE-Core there are several
open ones and several
which are maintained by companies

>
> I agree that the journal should be persistent, but I also agree with
> Richard that seems to be too big of a change in behavior when we're so
> close to the release point.

what makes you think the change you posted is not intrusive, we have a
regression compared to last release,
this patch does not fix the regression, it infact pacifies some errors
which are new and creates yet another behavior.  think of people who
will be migrating from 1.7 to 1.8  problems should be solved in best
possible way.
I am fine with whatever is accepted solution. I have no interest in
push one or the other
I do have a solution which fixes my problems therefore I am less
concerned. Carrying a couple of local reverts it not a big of a deal.

-Khem

Re: [PATCH] systemd: Instruct the journal to always be volatile

[Otavio Salvador]

On Thu, Mar 12, 2015 at 12:09 AM, Khem Raj <raj.khem@gmail.com> wrote:
>
>> On Mar 11, 2015, at 2:28 PM, Randy Witt <randy.e.witt@linux.intel.com> wrote:
>>
>> When the storage mode for the journal is "auto" if /var/log/journal
>> exists then the journal will flush to /var/log/journal assuming that
>> /var/log/journal is persistent.
>>
>> However /var/log -> /var/volatile/log in poky, so even though
>> /var/log/journal exists, it is still volatile.
>>
>> Since this can cause ordering issues due to /var/volatile needing to be mounted before the journal actually writes to it, just specify that the journal
>> should always be volatile and never try to write to "persistent"
>> storage. The journal will exist in /run/log/journal only.
>>
>> This also disables the "After" of the journal on var-volatile.mount
>> since the ordering is no longer necessary when the journal is only
>> stored in /run/log/journal.
>>
>> [Yocto #7388]
>
>
> This is not right. What if I want persistent logs.? the options is just gone. Both cases should work. /var/volatile should only be used
> when doing ro-rfs its not FHS specified anyway. Lets not go with this patch.

Agreed.

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH] systemd: Instruct the journal to always be volatile

[Richard Purdie]

On Thu, 2015-03-12 at 04:18 -0300, Otavio Salvador wrote:
> On Thu, Mar 12, 2015 at 12:09 AM, Khem Raj <raj.khem@gmail.com> wrote:
> >
> >> On Mar 11, 2015, at 2:28 PM, Randy Witt <randy.e.witt@linux.intel.com> wrote:
> >>
> >> When the storage mode for the journal is "auto" if /var/log/journal
> >> exists then the journal will flush to /var/log/journal assuming that
> >> /var/log/journal is persistent.
> >>
> >> However /var/log -> /var/volatile/log in poky, so even though
> >> /var/log/journal exists, it is still volatile.
> >>
> >> Since this can cause ordering issues due to /var/volatile needing to be mounted before the journal actually writes to it, just specify that the journal
> >> should always be volatile and never try to write to "persistent"
> >> storage. The journal will exist in /run/log/journal only.
> >>
> >> This also disables the "After" of the journal on var-volatile.mount
> >> since the ordering is no longer necessary when the journal is only
> >> stored in /run/log/journal.
> >>
> >> [Yocto #7388]
> >
> >
> > This is not right. What if I want persistent logs.? the options is just gone. Both cases should work. /var/volatile should only be used
> > when doing ro-rfs its not FHS specified anyway. Lets not go with this patch.
> 
> Agreed.

This isn't quite so simple as "agreed". Are we saying that we want to
rework volatiles handling after feature freeze?

My view on this is that yes, we do need to do something about it, but
right now during stabilisation is not the right time. Particularly when
the patches proposed aren't even tested with sysvinit.

So I'm interested in patches which make things work better without
making fundamental changes now. We can look at changing volatiles with a
properly thought out plan in the next development cycle.

Making the systemd configuration match the rest of the default system
setup would seem to be sensible. We should make it possible for others
to override that if their setup isn't volatile.

Cheers,

Richard

Re: [PATCH] systemd: Instruct the journal to always be volatile

[Otavio Salvador]

On Thu, Mar 12, 2015 at 5:04 AM, Richard Purdie
<richard.purdie@linuxfoundation.org> wrote:
> On Thu, 2015-03-12 at 04:18 -0300, Otavio Salvador wrote:
>> On Thu, Mar 12, 2015 at 12:09 AM, Khem Raj <raj.khem@gmail.com> wrote:
>> >
>> >> On Mar 11, 2015, at 2:28 PM, Randy Witt <randy.e.witt@linux.intel.com> wrote:
>> >>
>> >> When the storage mode for the journal is "auto" if /var/log/journal
>> >> exists then the journal will flush to /var/log/journal assuming that
>> >> /var/log/journal is persistent.
>> >>
>> >> However /var/log -> /var/volatile/log in poky, so even though
>> >> /var/log/journal exists, it is still volatile.
>> >>
>> >> Since this can cause ordering issues due to /var/volatile needing to be mounted before the journal actually writes to it, just specify that the journal
>> >> should always be volatile and never try to write to "persistent"
>> >> storage. The journal will exist in /run/log/journal only.
>> >>
>> >> This also disables the "After" of the journal on var-volatile.mount
>> >> since the ordering is no longer necessary when the journal is only
>> >> stored in /run/log/journal.
>> >>
>> >> [Yocto #7388]
>> >
>> >
>> > This is not right. What if I want persistent logs.? the options is just gone. Both cases should work. /var/volatile should only be used
>> > when doing ro-rfs its not FHS specified anyway. Lets not go with this patch.
>>
>> Agreed.
>
> This isn't quite so simple as "agreed". Are we saying that we want to
> rework volatiles handling after feature freeze?
>
> My view on this is that yes, we do need to do something about it, but
> right now during stabilisation is not the right time. Particularly when
> the patches proposed aren't even tested with sysvinit.
>
> So I'm interested in patches which make things work better without
> making fundamental changes now. We can look at changing volatiles with a
> properly thought out plan in the next development cycle.
>
> Making the systemd configuration match the rest of the default system
> setup would seem to be sensible. We should make it possible for others
> to override that if their setup isn't volatile.

I am saying we should then test the patch (from Khem) and see if it is
valid for merge. Khem being quite active contributor I am sure will be
willing to help to fix any issue for merging (and regression, if any).
I think having the volatile support working is very important and as
we are not releasing tomorrow ...

-- 
Otavio Salvador                             O.S. Systems
http://www.ossystems.com.br        http://code.ossystems.com.br
Mobile: +55 (53) 9981-7854            Mobile: +1 (347) 903-9750

Re: [PATCH] systemd: Instruct the journal to always be volatile

[Khem Raj]

> On Mar 12, 2015, at 1:04 AM, Richard Purdie <richard.purdie@linuxfoundation.org> wrote:
> 
> On Thu, 2015-03-12 at 04:18 -0300, Otavio Salvador wrote:
>> On Thu, Mar 12, 2015 at 12:09 AM, Khem Raj <raj.khem@gmail.com> wrote:
>>> 
>>>> On Mar 11, 2015, at 2:28 PM, Randy Witt <randy.e.witt@linux.intel.com> wrote:
>>>> 
>>>> When the storage mode for the journal is "auto" if /var/log/journal
>>>> exists then the journal will flush to /var/log/journal assuming that
>>>> /var/log/journal is persistent.
>>>> 
>>>> However /var/log -> /var/volatile/log in poky, so even though
>>>> /var/log/journal exists, it is still volatile.
>>>> 
>>>> Since this can cause ordering issues due to /var/volatile needing to be mounted before the journal actually writes to it, just specify that the journal
>>>> should always be volatile and never try to write to "persistent"
>>>> storage. The journal will exist in /run/log/journal only.
>>>> 
>>>> This also disables the "After" of the journal on var-volatile.mount
>>>> since the ordering is no longer necessary when the journal is only
>>>> stored in /run/log/journal.
>>>> 
>>>> [Yocto #7388]
>>> 
>>> 
>>> This is not right. What if I want persistent logs.? the options is just gone. Both cases should work. /var/volatile should only be used
>>> when doing ro-rfs its not FHS specified anyway. Lets not go with this patch.
>> 
>> Agreed.
> 
> This isn't quite so simple as "agreed". Are we saying that we want to
> rework volatiles handling after feature freeze?
> 

> My view on this is that yes, we do need to do something about it, but
> right now during stabilisation is not the right time. Particularly when
> the patches proposed aren't even tested with sysvinit.

I usually send out patches early to take advantage of some community testing especially Yocto auto builders, Since not everyone has hefty machines
to do all kind of builds for such changes. Meanwhile, keep doing it in serial order.

> 
> So I'm interested in patches which make things work better without
> making fundamental changes now. We can look at changing volatiles with a
> properly thought out plan in the next development cycle.

Whatever is decided I have already said thats fine, so lets just stop this thread here.