Issue: /var/tmp does not persist due to /var/volatile being tmpfs in fstab #scarthgap

Issue: /var/tmp does not persist due to /var/volatile being tmpfs in fstab #scarthgap

[Vish]

[-- Attachment #1: Type: text/plain, Size: 1603 bytes --]

The current Yocto basefiles fstab mounts /var/volatile as tmpfs, which results in the loss of all files stored in /var/tmp upon reboot.

fstab in base-files:

https://git.yoctoproject.org/poky/tree/meta/recipes-core/base-files/base-files/fstab?h=scarthgap&id=3cc8c3520cedb66500757f843f757646da69ab30#n7

Since /var/volatile is a tmpfs, its contents are lost on reboot.

This commit might have introduced the issue - s ince /var/volatile/tmp is inside a tmpfs mount, it gets wiped on reboot, meaning /var/tmp is also cleared.
Commit - https://git.yoctoproject.org/poky/commit/meta/recipes-core/systemd/systemd/00-create-volatile.conf?h=scarthgap&id=3cc8c3520cedb66500757f843f757646da69ab30 ( https://git.yoctoproject.org/poky/commit/meta/recipes-core/systemd/systemd/00-create-volatile.conf?h=scarthgap&id=3cc8c3520cedb66500757f843f757646da69ab30 )

This commit links /var/tmp to /var/volatile/tmp
*Expected Behavior*

* /var/tmp should retain files across reboots, as per standard Linux behavior.
* Using a symlink to a volatile tmpfs directory contradicts this expectation.

*Impact*

* Any application expecting persistent temporary storage in /var/tmp will lose data after a reboot.
* This behavior may break software relying on the standard /var/tmp persistence.

Should /var/tmp continue following standard Linux behavior by *not being linked to /var/volatile/tmp* , or is this change intentional for specific use cases? If persistence is desired, this approach seems problematic.

Would appreciate input from the maintainers on whether this is a bug or intended behavior.

[-- Attachment #2: Type: text/html, Size: 3860 bytes --]

RE: [OE-core] Issue: /var/tmp does not persist due to /var/volatile being tmpfs in fstab #scarthgap

[Chen, Qi]

[-- Attachment #1: Type: text/plain, Size: 2350 bytes --]

This is configurable.

meta/conf/bitbake.conf:FILESYSTEM_PERMS_TABLES ?=  "files/fs-perms.txt \
meta/conf/bitbake.conf-                             files/fs-perms-volatile-log.txt \
meta/conf/bitbake.conf-                             files/fs-perms-volatile-tmp.txt"

You can remove files/fs-perms-volatile-tmp.txt from FILESYSTEM_PERMS_TABLES, and /var/tmp will be persistent.

Regards,
Qi

From: openembedded-core@lists.openembedded.org <openembedded-core@lists.openembedded.org> On Behalf Of Vish via lists.openembedded.org
Sent: Tuesday, March 18, 2025 11:17 AM
To: openembedded-core@lists.openembedded.org
Subject: [OE-core] Issue: /var/tmp does not persist due to /var/volatile being tmpfs in fstab #scarthgap

The current Yocto basefiles fstab mounts /var/volatile as tmpfs, which results in the loss of all files stored in /var/tmp upon reboot.

fstab in base-files:

https://git.yoctoproject.org/poky/tree/meta/recipes-core/base-files/base-files/fstab?h=scarthgap&id=3cc8c3520cedb66500757f843f757646da69ab30#n7

Since /var/volatile is a tmpfs, its contents are lost on reboot.

This commit might have introduced the issue - since /var/volatile/tmp is inside a tmpfs mount, it gets wiped on reboot, meaning /var/tmp is also cleared.
Commit - https://git.yoctoproject.org/poky/commit/meta/recipes-core/systemd/systemd/00-create-volatile.conf?h=scarthgap&id=3cc8c3520cedb66500757f843f757646da69ab30 <https://git.yoctoproject.org/poky/commit/meta/recipes-core/systemd/systemd/00-create-volatile.conf?h=scarthgap&id=3cc8c3520cedb66500757f843f757646da69ab30>

This commit links /var/tmp to /var/volatile/tmp

Expected Behavior

  *   /var/tmp should retain files across reboots, as per standard Linux behavior.
  *   Using a symlink to a volatile tmpfs directory contradicts this expectation.


Impact

  *   Any application expecting persistent temporary storage in /var/tmp will lose data after a reboot.
  *   This behavior may break software relying on the standard /var/tmp persistence.



Should /var/tmp continue following standard Linux behavior by not being linked to /var/volatile/tmp, or is this change intentional for specific use cases? If persistence is desired, this approach seems problematic.

Would appreciate input from the maintainers on whether this is a bug or intended behavior.

[-- Attachment #2: Type: text/html, Size: 12421 bytes --]

Re: Issue: /var/tmp does not persist due to /var/volatile being tmpfs in fstab #scarthgap

[Vish]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

Thank you for the reply.

In Scarthgap LTS implementation is different, but I got the idea.
https://git.yoctoproject.org/poky/tree/meta/conf/bitbake.conf?h=scarthgap#n387

I believe default behavior should be same as before to persist /var/temp over reboot and these changes should only be required for any customization. Also somewhat difficult for us to upgrade CI/CD with this config.

Can we know the reasons for original changes here?
https://git.yoctoproject.org/poky/commit/meta/recipes-core/systemd/systemd/00-create-volatile.conf?h=scarthgap&id=3cc8c3520cedb66500757f843f757646da69ab30

I reviewed commit message but not giving much details. Considering default base-files and as persistence is required, this newer approach seems be an issue.

We recently migrated to Scarthgap and noticed the issues with our system. This can be issue for other developers as well whoever is planning to migrate to Scarthgap and embedded devices in the field. This commit requires all build systems to adopt changes bitbake.conf which is not preferable .

What is the usecases targeted with the changes? I believe the commit should be reviewed again and potentially reverted considering default behavior with base-files and 00-create-volatile.conf.

Regards,
VS.

[-- Attachment #2: Type: text/html, Size: 1921 bytes --]

Re: [OE-core] Issue: /var/tmp does not persist due to /var/volatile being tmpfs in fstab #scarthgap

[Richard Purdie]

On Tue, 2025-03-18 at 09:11 -0700, Vish via lists.openembedded.org wrote:
> Thank you for the reply. 
>  
> In Scarthgap LTS implementation is different, but I got the idea. 
> https://git.yoctoproject.org/poky/tree/meta/conf/bitbake.conf?h=scarthgap#n387
>  
> I believe default behavior should be same as before to persist
> /var/temp over reboot and these changes should only be required for
> any customization. Also somewhat difficult for us to upgrade CI/CD
> with this config. 
>  
> Can we know the reasons for original changes here?
> https://git.yoctoproject.org/poky/commit/meta/recipes-core/systemd/systemd/00-create-volatile.conf?h=scarthgap&id=3cc8c3520cedb66500757f843f757646da69ab30
>  
> I reviewed commit message but not giving much details. Considering
> default base-files and as persistence is required, this newer
> approach seems be an issue. 
>  
> We recently migrated to Scarthgap and noticed the issues with our
> system. This can be issue for other developers as well whoever is
> planning to migrate to Scarthgap and embedded devices in the field.
> This commit requires all build systems to adopt changes bitbake.conf
> which is not preferable .  
>  
> What is the usecases targeted with the changes? I believe the commit
> should be reviewed again and potentially reverted considering default
> behavior with base-files and 00-create-volatile.conf.

scarthgap has been released for a while now so making a change like
this well into the stable release cycle would be a problem in its own
right. I therefore don't think this can be reverted now regardless of
whether it should have happened or not.

Cheers,

Richard

Re: Issue: /var/tmp does not persist due to /var/volatile being tmpfs in fstab #scarthgap

[Vish]

[-- Attachment #1: Type: text/plain, Size: 848 bytes --]

Thanks for the reply.

Do maintainers recognize this as an issue with having this as default option? There are chances that similar to us other devs/orgs are planning migration in phases to Scarthgap considering Kirkstone getting updates, so this could potentially become issue for others as well. We do have sizable amount of edge systems in production with various CI, and this change broke an important feature and took some time for us to find root cause.

I believe correcting is still a proper thing to do and can be planned during any releases.
Additional reference -
https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s15.html ( https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s15.html )

Also, it would be good to know what scenarios were considered with this change, probably we are missing some context.

Regards,
VS.

[-- Attachment #2: Type: text/html, Size: 1220 bytes --]

Re: [OE-core] Issue: /var/tmp does not persist due to /var/volatile being tmpfs in fstab #scarthgap

[Alexander Kanavin]

You can probably propose a patch to change the default in master, but
this wouldn't be eligible for a backport to scarthgap. Once they're
out, stable releases simply can't change behaviour like this.

Also it would be nice if you introduce yourself and your company.

Alex



On Wed, 19 Mar 2025 at 00:24, Vish via lists.openembedded.org
<vs.yocto=outlook.com@lists.openembedded.org> wrote:
>
> Thanks for the reply.
>
> Do maintainers recognize this as an issue with having this as default option? There are chances that similar to us other devs/orgs are planning migration in phases to Scarthgap considering Kirkstone getting updates, so this could potentially become issue for others as well. We do have sizable amount of edge systems in production with various CI, and this change broke an important feature and took some time for us to find root cause.
>
> I believe correcting is still a proper thing to do and can be planned during any releases.
> Additional reference -
> https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s15.html
>
> Also, it would be good to know what scenarios were considered with this change, probably we are missing some context.
>
>
> Regards,
> VS.
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#213259): https://lists.openembedded.org/g/openembedded-core/message/213259
> Mute This Topic: https://lists.openembedded.org/mt/111764066/1686489
> Mute #scarthgap:https://lists.openembedded.org/g/openembedded-core/mutehashtag/scarthgap
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>

Re: [OE-core] Issue: /var/tmp does not persist due to /var/volatile being tmpfs in fstab #scarthgap

[Richard Purdie]

On Tue, 2025-03-18 at 16:24 -0700, Vish via lists.openembedded.org wrote:
> Thanks for the reply. 
>  
> Do maintainers recognize this as an issue with having this as default
> option? There are chances that similar to us other devs/orgs are
> planning migration in phases to Scarthgap considering Kirkstone
> getting updates, so this could potentially become issue for others as
> well.

I recognise there is an issue when behaviours change. We can't change
this in that release now but I would happily see patches/updates to the
migration guide and release notes making the issue clear.

> We do have sizable amount of edge systems in production with various
> CI, and this change broke an important feature and took some time for
> us to find root cause.

Sorry it was a pain. If you are able to check the state of master
periodically, it may help head some of these kinds of issues off.

> I believe correcting is still a proper thing to do and can be planned
> during any releases.
> 
> Additional reference -
> 
> 
> https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch05s15.html 
>  
> Also, it would be good to know what scenarios were considered with
> this change, probably we are missing some context.

Traditionally, OpenEmbedded was used in resource constrained embedded
systems and all "tmp" space was transient, not persistent. This
configuration is in keeping with the project's history.

Our approach isn't entirely against FHS, it is just not quite what
other distros do in our default config. There are other areas we do
differ from FHS too.

On this specific one, we support two init systems and I think the
configuration was mismatched between the two and the configuration
options were hard to get right and not working in all cases. There were
various patches over the last few years trying to make the docs,
options and init systems match up and be configurable. I suspect
somewhere on the way things changed in a way that wasn't realised.

The project is crying out for maintainer and review help. We do the
best we can with what we have, like most open source projects. An
implication of this is that where patches are submitted and nobody
objects and I don't see/spot any problem with them, they can be merged
even if there is an issue we didn't realise. We do the best we can.

How we react when things are merged depends on the circumstances.
Scarthgap has been out for nearly a year so a major behaviour change is
not really an option now. Even for master, I think the situation is
relatively well understood and more consistent now so any change would
need to demonstrate that isn't the case and that a change would improve
things rather than further confuse people.

I don't know who you work for but if you are able to help support the
project through patch review, recipe maintenance or anything else that
be nice. Alternatively, project membership does give us funding which
allows us to help in a different way too if that is an option. If you
don't do any of these things...

Hope that helps give some of the context.

Cheers,

Richard

Re: Issue: /var/tmp does not persist due to /var/volatile being tmpfs in fstab #scarthgap

[Vish]

[-- Attachment #1: Type: text/plain, Size: 409 bytes --]

Thank you everyone for responses. We have proceeded as Niko suggested with appending config for now. (Also, will definitely introduce myself/company outside of this forum.)

Thank you very much Richard for the detailed context and history. And I absolutely admire all the efforts that you and community putting in maintaining this massive project.

Appreciating all valuable inputs here.

Regards,
VS

[-- Attachment #2: Type: text/html, Size: 553 bytes --]

Re: [OE-core] Issue: /var/tmp does not persist due to /var/volatile being tmpfs in fstab #scarthgap

[Niko Mauno]

On 3/18/25 18:11, Vish via lists.openembedded.org wrote:
> Can we know the reasons for original changes here?
> https://git.yoctoproject.org/poky/commit/meta/recipes-core/systemd/ 
> systemd/00-create-volatile.conf? 
> h=scarthgap&id=3cc8c3520cedb66500757f843f757646da69ab30 <https:// 
> git.yoctoproject.org/poky/commit/meta/recipes-core/systemd/systemd/00- 
> create-volatile.conf? 
> h=scarthgap&id=3cc8c3520cedb66500757f843f757646da69ab30>
> I reviewed commit message but not giving much details. Considering 
> default base-files and as persistence is required, this newer approach 
> seems be an issue.

Hi Vish, that commit should only have the effect to avoid seeing an 
associated error line in systemd journal ("/var/tmp" already exists and 
is not a directory). The error line was stemming from the fact that in 
Yocto Scarthgap base-files creates /var/tmp as symlink to 
/var/volatile/tmp 
(https://git.yoctoproject.org/poky/tree/meta/recipes-core/base-files/base-files_3.0.14.bb?h=scarthgap#n87) 
which contradicts systemd's default runtime expectations.

The base-files provided /var/tmp -> /var/volatile/tmp symlink has been 
prevalent convention in Yocto for a very long time. (If you want to have 
/var/tmp as directory instead in Scarthgap I believe you can e.g. append 
base-files do_install() task to remove the symlink and replace it with 
directory.)

BR, Niko