[PATCH 0/2] Less intrusive journal fix

[PATCH 0/2] Less intrusive journal fix

[Randy Witt]

The following changes since commit 4bd886fb0081954879aaa2f574f3c7f6ced745d5:

  systemd: fix firmware path to match linux-firmware (2015-03-23 11:27:21 +0000)

are available in the git repository at:

  git://git.yoctoproject.org/poky-contrib rewitt/empty-var-volatile

for you to fetch changes up to 34642f136f3830f8de88f816b9818ce855f9f129:

  image.bbclass: Empty /var/volatile if it is a mount point (2015-03-23 16:53:45 -0700)

----------------------------------------------------------------
Randy Witt (2):
      Revert "systemd: Fix the problem of an empty journal on boot"
      image.bbclass: Empty /var/volatile if it is a mount point

 meta/classes/image.bbclass                               | 16 ++++++++++++++++
 meta/recipes-core/systemd/systemd/journald-volatile.conf |  6 ------
 meta/recipes-core/systemd/systemd_219.bb                 |  2 --
 3 files changed, 16 insertions(+), 8 deletions(-)
 delete mode 100644 meta/recipes-core/systemd/systemd/journald-volatile.conf

[PATCH 1/2] Revert "systemd: Fix the problem of an empty journal on boot"

[Randy Witt]

This reverts commit e75c95ceca34879698b1ea229a2101dc1ccc12fc. The fix
was reported to cause long waits for some users.

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
---
 meta/recipes-core/systemd/systemd/journald-volatile.conf | 6 ------
 meta/recipes-core/systemd/systemd_219.bb                 | 2 --
 2 files changed, 8 deletions(-)
 delete mode 100644 meta/recipes-core/systemd/systemd/journald-volatile.conf

diff --git a/meta/recipes-core/systemd/systemd/journald-volatile.conf b/meta/recipes-core/systemd/systemd/journald-volatile.conf
deleted file mode 100644
index b11e160..0000000
--- a/meta/recipes-core/systemd/systemd/journald-volatile.conf
+++ /dev/null
@@ -1,6 +0,0 @@
-# If /var/volatile is a mount point then make sure to mount it before
-# the journal starts. This is because base-files creates a symlink
-# /var/log -> /var/volatile/log. And if the journal starts before the mount
-# happens, the journal will appear empty until restarted.
-[Unit]
-After=var-volatile.mount
diff --git a/meta/recipes-core/systemd/systemd_219.bb b/meta/recipes-core/systemd/systemd_219.bb
index f2d3c13..43c4f95 100644
--- a/meta/recipes-core/systemd/systemd_219.bb
+++ b/meta/recipes-core/systemd/systemd_219.bb
@@ -51,7 +51,6 @@ SRC_URI = "git://anongit.freedesktop.org/systemd/systemd;branch=master \
            file://00-create-volatile.conf \
            file://init \
            file://run-ptest \
-           file://journald-volatile.conf \
           "
 
 S = "${WORKDIR}/git"
@@ -153,7 +152,6 @@ do_install() {
 	install -m 0644 ${WORKDIR}/*.rules ${D}${sysconfdir}/udev/rules.d/
 
 	install -m 0644 ${WORKDIR}/00-create-volatile.conf ${D}${sysconfdir}/tmpfiles.d/
-	install -D -m 0644 ${WORKDIR}/journald-volatile.conf ${D}${systemd_unitdir}/system/systemd-journald.service.d/journald-volatile.conf
 
 	if ${@bb.utils.contains('DISTRO_FEATURES','sysvinit','true','false',d)}; then
 		install -d ${D}${sysconfdir}/init.d
-- 
1.9.3

[PATCH 2/2] image.bbclass: Empty /var/volatile if it is a mount point

[Randy Witt]

If /var/volatile is a mount point it shouldn't contain any files before
mount time. If files are there, they will no longer be able to be accessed
once the tmpfs gets mounted at /var/volatile.

This problem can be seen for instance when systemd creates
/var/volatile/log/journal as part of its package installation. It then
assumes the journal is persistent even though /var/volatile/log/journal
goes away shortly thereafter.

This change makes sure that there are no files in /var/volatile if it is
to be used as a mount point.

[Yocto #7388]

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
---
 meta/classes/image.bbclass | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 89eb5f3..dfedf9d 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -181,6 +181,8 @@ POSTINST_LOGFILE ?= "${localstatedir}/log/postinstall.log"
 SYSTEMD_DEFAULT_TARGET ?= '${@bb.utils.contains("IMAGE_FEATURES", "x11-base", "graphical.target", "multi-user.target", d)}'
 ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("DISTRO_FEATURES", "systemd", "set_systemd_default_target; ", "", d)}'
 
+ROOTFS_POSTPROCESS_COMMAND += 'empty_var_volatile;'
+
 # some default locales
 IMAGE_LINGUAS ?= "de-de fr-fr en-gb"
 
@@ -377,6 +379,20 @@ set_systemd_default_target () {
 	fi
 }
 
+# If /var/volatile is not empty, we have seen problems where programs such as the
+# journal make assumptions based on the contents of /var/volatile. The journal
+# would then write to /var/volatile before it was mounted, thus hiding the
+# items previously written.
+#
+# This change is to attempt to fix those types of issues in a way that doesn't
+# affect users that may not be using /var/volatile.
+empty_var_volatile () {
+	match=`awk '$1 !~ "#" && $2 ~ /\/var\/volatile/{print $2}' ${IMAGE_ROOTFS}/etc/fstab 2> /dev/null`
+	if [ -n "$match" ]; then
+		find ${IMAGE_ROOTFS}/var/volatile -mindepth 1 -delete
+	fi
+}
+
 # Turn any symbolic /sbin/init link into a file
 remove_init_link () {
 	if [ -h ${IMAGE_ROOTFS}/sbin/init ]; then
-- 
1.9.3

Re: [PATCH 2/2] image.bbclass: Empty /var/volatile if it is a mount point

[Khem Raj]

On Mon, Mar 23, 2015 at 5:28 PM, Randy Witt
<randy.e.witt@linux.intel.com> wrote:
> If /var/volatile is a mount point it shouldn't contain any files before
> mount time. If files are there, they will no longer be able to be accessed
> once the tmpfs gets mounted at /var/volatile.

why not use copy-bind when mounting it second time as tmpfs ?

>
> This problem can be seen for instance when systemd creates
> /var/volatile/log/journal as part of its package installation. It then
> assumes the journal is persistent even though /var/volatile/log/journal
> goes away shortly thereafter.
>
> This change makes sure that there are no files in /var/volatile if it is
> to be used as a mount point.
>
> [Yocto #7388]
>
> Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
> ---
>  meta/classes/image.bbclass | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
>
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 89eb5f3..dfedf9d 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -181,6 +181,8 @@ POSTINST_LOGFILE ?= "${localstatedir}/log/postinstall.log"
>  SYSTEMD_DEFAULT_TARGET ?= '${@bb.utils.contains("IMAGE_FEATURES", "x11-base", "graphical.target", "multi-user.target", d)}'
>  ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("DISTRO_FEATURES", "systemd", "set_systemd_default_target; ", "", d)}'
>
> +ROOTFS_POSTPROCESS_COMMAND += 'empty_var_volatile;'
> +
>  # some default locales
>  IMAGE_LINGUAS ?= "de-de fr-fr en-gb"
>
> @@ -377,6 +379,20 @@ set_systemd_default_target () {
>         fi
>  }
>
> +# If /var/volatile is not empty, we have seen problems where programs such as the
> +# journal make assumptions based on the contents of /var/volatile. The journal
> +# would then write to /var/volatile before it was mounted, thus hiding the
> +# items previously written.
> +#
> +# This change is to attempt to fix those types of issues in a way that doesn't
> +# affect users that may not be using /var/volatile.
> +empty_var_volatile () {
> +       match=`awk '$1 !~ "#" && $2 ~ /\/var\/volatile/{print $2}' ${IMAGE_ROOTFS}/etc/fstab 2> /dev/null`
> +       if [ -n "$match" ]; then
> +               find ${IMAGE_ROOTFS}/var/volatile -mindepth 1 -delete
> +       fi
> +}
> +
>  # Turn any symbolic /sbin/init link into a file
>  remove_init_link () {
>         if [ -h ${IMAGE_ROOTFS}/sbin/init ]; then
> --
> 1.9.3
>
> --
> _______________________________________________
> Openembedded-core mailing list
> Openembedded-core@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/openembedded-core

Re: [PATCH 2/2] image.bbclass: Empty /var/volatile if it is a mount point

[randy.e.witt]

> On Mon, Mar 23, 2015 at 5:28 PM, Randy Witt
> <randy.e.witt@linux.intel.com> wrote:
>> If /var/volatile is a mount point it shouldn't contain any files before
>> mount time. If files are there, they will no longer be able to be
>> accessed
>> once the tmpfs gets mounted at /var/volatile.
>
> why not use copy-bind when mounting it second time as tmpfs ?
>

Khem, could you elaborate? I'm not sure I know the mechanism to which you
are referring.

>>
>> This problem can be seen for instance when systemd creates
>> /var/volatile/log/journal as part of its package installation. It then
>> assumes the journal is persistent even though /var/volatile/log/journal
>> goes away shortly thereafter.
>>
>> This change makes sure that there are no files in /var/volatile if it is
>> to be used as a mount point.
>>
>> [Yocto #7388]
>>
>> Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
>> ---
>>  meta/classes/image.bbclass | 16 ++++++++++++++++
>>  1 file changed, 16 insertions(+)
>>
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 89eb5f3..dfedf9d 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -181,6 +181,8 @@ POSTINST_LOGFILE ?=
>> "${localstatedir}/log/postinstall.log"
>>  SYSTEMD_DEFAULT_TARGET ?= '${@bb.utils.contains("IMAGE_FEATURES",
>> "x11-base", "graphical.target", "multi-user.target", d)}'
>>  ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("DISTRO_FEATURES",
>> "systemd", "set_systemd_default_target; ", "", d)}'
>>
>> +ROOTFS_POSTPROCESS_COMMAND += 'empty_var_volatile;'
>> +
>>  # some default locales
>>  IMAGE_LINGUAS ?= "de-de fr-fr en-gb"
>>
>> @@ -377,6 +379,20 @@ set_systemd_default_target () {
>>         fi
>>  }
>>
>> +# If /var/volatile is not empty, we have seen problems where programs
>> such as the
>> +# journal make assumptions based on the contents of /var/volatile. The
>> journal
>> +# would then write to /var/volatile before it was mounted, thus hiding
>> the
>> +# items previously written.
>> +#
>> +# This change is to attempt to fix those types of issues in a way that
>> doesn't
>> +# affect users that may not be using /var/volatile.
>> +empty_var_volatile () {
>> +       match=`awk '$1 !~ "#" && $2 ~ /\/var\/volatile/{print $2}'
>> ${IMAGE_ROOTFS}/etc/fstab 2> /dev/null`
>> +       if [ -n "$match" ]; then
>> +               find ${IMAGE_ROOTFS}/var/volatile -mindepth 1 -delete
>> +       fi
>> +}
>> +
>>  # Turn any symbolic /sbin/init link into a file
>>  remove_init_link () {
>>         if [ -h ${IMAGE_ROOTFS}/sbin/init ]; then
>> --
>> 1.9.3
>>
>> --
>> _______________________________________________
>> Openembedded-core mailing list
>> Openembedded-core@lists.openembedded.org
>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>

Re: [PATCH 2/2] image.bbclass: Empty /var/volatile if it is a mount point

[Khem Raj]

> On Mar 24, 2015, at 8:46 AM, randy.e.witt@linux.intel.com wrote:
> 
>> On Mon, Mar 23, 2015 at 5:28 PM, Randy Witt
>> <randy.e.witt@linux.intel.com> wrote:
>>> If /var/volatile is a mount point it shouldn't contain any files before
>>> mount time. If files are there, they will no longer be able to be
>>> accessed
>>> once the tmpfs gets mounted at /var/volatile.
>> 
>> why not use copy-bind when mounting it second time as tmpfs ?
>> 
> 
> Khem, could you elaborate? I'm not sure I know the mechanism to which you
> are referring.

copy bind will copy the contents from old place to new mount. Then hopefully you do not lose the content
that was already there before remount happens see VOLATILE_BINDS implementation.

> 
>>> 
>>> This problem can be seen for instance when systemd creates
>>> /var/volatile/log/journal as part of its package installation. It then
>>> assumes the journal is persistent even though /var/volatile/log/journal
>>> goes away shortly thereafter.
>>> 
>>> This change makes sure that there are no files in /var/volatile if it is
>>> to be used as a mount point.
>>> 
>>> [Yocto #7388]
>>> 
>>> Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
>>> ---
>>> meta/classes/image.bbclass | 16 ++++++++++++++++
>>> 1 file changed, 16 insertions(+)
>>> 
>>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>>> index 89eb5f3..dfedf9d 100644
>>> --- a/meta/classes/image.bbclass
>>> +++ b/meta/classes/image.bbclass
>>> @@ -181,6 +181,8 @@ POSTINST_LOGFILE ?=
>>> "${localstatedir}/log/postinstall.log"
>>> SYSTEMD_DEFAULT_TARGET ?= '${@bb.utils.contains("IMAGE_FEATURES",
>>> "x11-base", "graphical.target", "multi-user.target", d)}'
>>> ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("DISTRO_FEATURES",
>>> "systemd", "set_systemd_default_target; ", "", d)}'
>>> 
>>> +ROOTFS_POSTPROCESS_COMMAND += 'empty_var_volatile;'
>>> +
>>> # some default locales
>>> IMAGE_LINGUAS ?= "de-de fr-fr en-gb"
>>> 
>>> @@ -377,6 +379,20 @@ set_systemd_default_target () {
>>>        fi
>>> }
>>> 
>>> +# If /var/volatile is not empty, we have seen problems where programs
>>> such as the
>>> +# journal make assumptions based on the contents of /var/volatile. The
>>> journal
>>> +# would then write to /var/volatile before it was mounted, thus hiding
>>> the
>>> +# items previously written.
>>> +#
>>> +# This change is to attempt to fix those types of issues in a way that
>>> doesn't
>>> +# affect users that may not be using /var/volatile.
>>> +empty_var_volatile () {
>>> +       match=`awk '$1 !~ "#" && $2 ~ /\/var\/volatile/{print $2}'
>>> ${IMAGE_ROOTFS}/etc/fstab 2> /dev/null`
>>> +       if [ -n "$match" ]; then
>>> +               find ${IMAGE_ROOTFS}/var/volatile -mindepth 1 -delete
>>> +       fi
>>> +}
>>> +
>>> # Turn any symbolic /sbin/init link into a file
>>> remove_init_link () {
>>>        if [ -h ${IMAGE_ROOTFS}/sbin/init ]; then
>>> --
>>> 1.9.3
>>> 
>>> --
>>> _______________________________________________
>>> Openembedded-core mailing list
>>> Openembedded-core@lists.openembedded.org
>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>> 
> 

Re: [PATCH 2/2] image.bbclass: Empty /var/volatile if it is a mount point

[Randy Witt]

On 03/24/2015 08:53 AM, Khem Raj wrote:
>
>> On Mar 24, 2015, at 8:46 AM, randy.e.witt@linux.intel.com wrote:
>>
>>> On Mon, Mar 23, 2015 at 5:28 PM, Randy Witt
>>> <randy.e.witt@linux.intel.com> wrote:
>>>> If /var/volatile is a mount point it shouldn't contain any files before
>>>> mount time. If files are there, they will no longer be able to be
>>>> accessed
>>>> once the tmpfs gets mounted at /var/volatile.
>>>
>>> why not use copy-bind when mounting it second time as tmpfs ?
>>>
>>
>> Khem, could you elaborate? I'm not sure I know the mechanism to which you
>> are referring.
>
> copy bind will copy the contents from old place to new mount. Then hopefully you do not lose the content
> that was already there before remount happens see VOLATILE_BINDS implementation.
>

I like this idea, but considering that /var/volatile/log/journal is within the 
mountpoint I'm not sure how it could work. The contents will already be hidden 
by the mount of /var/volatile before the files can be copied.

Perhaps I am misunderstanding?

>>
>>>>
>>>> This problem can be seen for instance when systemd creates
>>>> /var/volatile/log/journal as part of its package installation. It then
>>>> assumes the journal is persistent even though /var/volatile/log/journal
>>>> goes away shortly thereafter.
>>>>
>>>> This change makes sure that there are no files in /var/volatile if it is
>>>> to be used as a mount point.
>>>>
>>>> [Yocto #7388]
>>>>
>>>> Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
>>>> ---
>>>> meta/classes/image.bbclass | 16 ++++++++++++++++
>>>> 1 file changed, 16 insertions(+)
>>>>
>>>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>>>> index 89eb5f3..dfedf9d 100644
>>>> --- a/meta/classes/image.bbclass
>>>> +++ b/meta/classes/image.bbclass
>>>> @@ -181,6 +181,8 @@ POSTINST_LOGFILE ?=
>>>> "${localstatedir}/log/postinstall.log"
>>>> SYSTEMD_DEFAULT_TARGET ?= '${@bb.utils.contains("IMAGE_FEATURES",
>>>> "x11-base", "graphical.target", "multi-user.target", d)}'
>>>> ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("DISTRO_FEATURES",
>>>> "systemd", "set_systemd_default_target; ", "", d)}'
>>>>
>>>> +ROOTFS_POSTPROCESS_COMMAND += 'empty_var_volatile;'
>>>> +
>>>> # some default locales
>>>> IMAGE_LINGUAS ?= "de-de fr-fr en-gb"
>>>>
>>>> @@ -377,6 +379,20 @@ set_systemd_default_target () {
>>>>         fi
>>>> }
>>>>
>>>> +# If /var/volatile is not empty, we have seen problems where programs
>>>> such as the
>>>> +# journal make assumptions based on the contents of /var/volatile. The
>>>> journal
>>>> +# would then write to /var/volatile before it was mounted, thus hiding
>>>> the
>>>> +# items previously written.
>>>> +#
>>>> +# This change is to attempt to fix those types of issues in a way that
>>>> doesn't
>>>> +# affect users that may not be using /var/volatile.
>>>> +empty_var_volatile () {
>>>> +       match=`awk '$1 !~ "#" && $2 ~ /\/var\/volatile/{print $2}'
>>>> ${IMAGE_ROOTFS}/etc/fstab 2> /dev/null`
>>>> +       if [ -n "$match" ]; then
>>>> +               find ${IMAGE_ROOTFS}/var/volatile -mindepth 1 -delete
>>>> +       fi
>>>> +}
>>>> +
>>>> # Turn any symbolic /sbin/init link into a file
>>>> remove_init_link () {
>>>>         if [ -h ${IMAGE_ROOTFS}/sbin/init ]; then
>>>> --
>>>> 1.9.3
>>>>
>>>> --
>>>> _______________________________________________
>>>> Openembedded-core mailing list
>>>> Openembedded-core@lists.openembedded.org
>>>> http://lists.openembedded.org/mailman/listinfo/openembedded-core
>>>
>>
>

[PATCH v2] image.bbclass: Empty /var/volatile if it is a mount point

[Randy Witt]

If /var/volatile is a mount point it shouldn't contain any files before
mount time. If files are there, they will no longer be able to be accessed
once the tmpfs gets mounted at /var/volatile.

This problem can be seen for instance when systemd creates
/var/volatile/log/journal as part of its package installation. It then
assumes the journal is persistent even though /var/volatile/log/journal
goes away shortly thereafter.

This change makes sure that there are no files in /var/volatile if it is
to be used as a mount point.

[Yocto #7388]

Signed-off-by: Randy Witt <randy.e.witt@linux.intel.com>
---
 meta/classes/image.bbclass | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 89eb5f3..d7e0734 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -181,6 +181,8 @@ POSTINST_LOGFILE ?= "${localstatedir}/log/postinstall.log"
 SYSTEMD_DEFAULT_TARGET ?= '${@bb.utils.contains("IMAGE_FEATURES", "x11-base", "graphical.target", "multi-user.target", d)}'
 ROOTFS_POSTPROCESS_COMMAND += '${@bb.utils.contains("DISTRO_FEATURES", "systemd", "set_systemd_default_target; ", "", d)}'
 
+ROOTFS_POSTPROCESS_COMMAND += 'empty_var_volatile;'
+
 # some default locales
 IMAGE_LINGUAS ?= "de-de fr-fr en-gb"
 
@@ -377,6 +379,22 @@ set_systemd_default_target () {
 	fi
 }
 
+# If /var/volatile is not empty, we have seen problems where programs such as the
+# journal make assumptions based on the contents of /var/volatile. The journal
+# would then write to /var/volatile before it was mounted, thus hiding the
+# items previously written.
+#
+# This change is to attempt to fix those types of issues in a way that doesn't
+# affect users that may not be using /var/volatile.
+empty_var_volatile () {
+	if [ -e ${IMAGE_ROOTFS}/etc/fstab ]; then
+		match=`awk '$1 !~ "#" && $2 ~ /\/var\/volatile/{print $2}' ${IMAGE_ROOTFS}/etc/fstab 2> /dev/null`
+		if [ -n "$match" ]; then
+			find ${IMAGE_ROOTFS}/var/volatile -mindepth 1 -delete
+		fi
+	fi
+}
+
 # Turn any symbolic /sbin/init link into a file
 remove_init_link () {
 	if [ -h ${IMAGE_ROOTFS}/sbin/init ]; then
-- 
1.9.3