v4.13 overlay fs: sometimes fails when creating or moving files

v4.13 overlay fs: sometimes fails when creating or moving files

[Jordi Pujol Palomer]

[-- Attachment #1: Type: text/plain, Size: 378 bytes --]

Hello,

Found some errors when doing a remaster of a filesystem image in Linux v
4.13,
the filesystem is mounted in overlay and then modifying the
system into the chroot.
The upgrade process fails when renaming or moving files.
Sometimes the same process runs right and sometimes wrong, 

screen dump to chroot.txt

# mount-list > mount-list.txt

Thanks,

Jordi Pujol i Palomer

[-- Attachment #2: chroot.txt --]
[-- Type: text/plain, Size: 3640 bytes --]

# uname -a
Linux pcjordi 4.13.0-2-haswell-lnet-amd64 #1 SMP PREEMPT Thu Sep 7 04:30:57 CEST 2017 x86_64 GNU/Linux

root@pcjordi:LneT-LXQT-amd64# /home/JPLive/git/live-net-remaster/lnet-remaster -pr
The Original source file or dir is 00filesystem.squashfs
The destination directory is /home/JPLive/LneT-LXQT-amd64
Mounting overlay image.Mounting image 00filesystem.squashfs in overlay root.Done.
Chroot to modify the system.

Now populating the root profile with useful commands...
Warning: Before exit, verify that there are no running processes into this shell.
(LneT-LXQT-amd64)root@pcjordi:/# aptitude full-upgrade
The following packages will be upgraded: 
  claws-mail claws-mail-multi-notifier dirmngr gnupg gnupg-l10n gnupg-utils gpg gpg-agent gpg-wks-client gpg-wks-server 
  gpgconf gpgsm gpgv hunspell-ca libmpfr4 sudo 
The following packages are RECOMMENDED but will NOT be installed:
  aspell-am aspell-ar aspell-ar-large aspell-bg aspell-br aspell-ca aspell-cs aspell-cy aspell-da aspell-de 
  aspell-de-1901 aspell-el aspell-en aspell-eo aspell-eo-cx7 aspell-es aspell-et aspell-eu aspell-fa aspell-fo aspell-fr 
  aspell-ga aspell-gl-minimos aspell-he aspell-hr aspell-hsb aspell-hu aspell-hy aspell-is aspell-it aspell-kk aspell-ku 
  aspell-lt aspell-lv aspell-nl aspell-pl aspell-pt-br aspell-pt-pt aspell-ro aspell-ru aspell-sk aspell-sl aspell-sv 
  aspell-tl aspell-uk aspell-uz claws-mail-i18n 
16 packages upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/10.4 MB of archives. After unpacking 18.4 kB will be used.
Do you want to continue? [Y/n/?] y
(Reading database ... 53176 files and directories currently installed.)
Preparing to unpack .../00-gpg-wks-client_2.2.0-3_amd64.deb ...
Unpacking gpg-wks-client (2.2.0-3) over (2.2.0-1) ...
Preparing to unpack .../01-dirmngr_2.2.0-3_amd64.deb ...
Unpacking dirmngr (2.2.0-3) over (2.2.0-1) ...
Preparing to unpack .../02-gnupg-l10n_2.2.0-3_all.deb ...
Unpacking gnupg-l10n (2.2.0-3) over (2.2.0-1) ...
Preparing to unpack .../03-gpg_2.2.0-3_amd64.deb ...
Unpacking gpg (2.2.0-3) over (2.2.0-1) ...
Preparing to unpack .../04-gnupg-utils_2.2.0-3_amd64.deb ...
Unpacking gnupg-utils (2.2.0-3) over (2.2.0-1) ...
Preparing to unpack .../05-gpg-agent_2.2.0-3_amd64.deb ...
Unpacking gpg-agent (2.2.0-3) over (2.2.0-1) ...
Preparing to unpack .../06-gpgsm_2.2.0-3_amd64.deb ...
Unpacking gpgsm (2.2.0-3) over (2.2.0-1) ...
Preparing to unpack .../07-gpgconf_2.2.0-3_amd64.deb ...
Unpacking gpgconf (2.2.0-3) over (2.2.0-1) ...
Preparing to unpack .../08-gnupg_2.2.0-3_amd64.deb ...
Unpacking gnupg (2.2.0-3) over (2.2.0-1) ...
Preparing to unpack .../09-gpg-wks-server_2.2.0-3_amd64.deb ...
Unpacking gpg-wks-server (2.2.0-3) over (2.2.0-1) ...
Preparing to unpack .../10-gpgv_2.2.0-3_amd64.deb ...
Unpacking gpgv (2.2.0-3) over (2.2.0-1) ...
dpkg: error: error removing old backup file '/var/lib/dpkg/status-old': Stale file handle
E: Sub-process /usr/bin/dpkg returned an error code (2)
dpkg: error: error removing old backup file '/var/lib/dpkg/status-old': Stale file handle
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. 
E: Could not regain the system lock!  (Perhaps another apt or dpkg is running?)
E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem. 
                                         
W: Could not lock the cache file; this usually means that dpkg or another apt tool is already installing packages.  Opening in read-only mode; any changes you make to the states of packages will NOT be preserved!
(LneT-LXQT-amd64)root@pcjordi:/# 

[-- Attachment #3: mount-list.txt --]
[-- Type: text/plain, Size: 4114 bytes --]

sysfs	/sys	sysfs	nodev,noexec,nosuid,relatime,rw	sysfs	0:17
proc	/proc	proc	nodev,noexec,nosuid,relatime,rw	proc	0:5
udev	/dev	devtmpfs	mode=755,nosuid,nr_inodes=978527,relatime,rw,size=3914108k	udev	0:6
devpts	/dev/pts	devpts	gid=5,mode=620,noexec,nosuid,ptmxmode=000,relatime,rw	devpts	0:18
tmpfs	/run	tmpfs	mode=755,noexec,nosuid,relatime,rw,size=786156k	tmpfs	0:19
tmpfs	/lnet	tmpfs	mode=755,relatime,rw,size=7861540k	tmpfs	0:20
/dev/sda1	/lnet/image	ext2	errors=continue,noacl,noatime,nouser_xattr,ro	/dev/sda1	8:1
/lnet/image/LneTXFCE64/00filesystem.squashfs	/lnet/00filesystem.squashfs	squashfs	noatime,ro,loop=/dev/loop0	/dev/loop0	7:0
/lnet/image/LneTXFCE64/10interfaces.squashfs	/lnet/10interfaces.squashfs	squashfs	noatime,ro,loop=/dev/loop1	/dev/loop1	7:1
/dev/sda2	/lnet/cowroot	ext4	data=ordered,relatime,rw	/dev/sda2	8:2
overlay	/	overlay	index=off,lowerdir=/lnet/00filesystem.squashfs:/lnet/10interfaces.squashfs,relatime,rw,upperdir=/lnet/cowroot/LneTXFCECOW/upper,workdir=/lnet/cowroot/LneTXFCECOW/work	overlay	0:21
tmpfs	/tmp	tmpfs	nodev,nosuid,relatime,rw,size=7861540k,mode=1777	tmpfs	0:22
tmpfs	/var/log	tmpfs	mode=755,nosuid,relatime,rw,size=7861540k	tmpfs	0:23
tmpfs	/var/tmp	tmpfs	nodev,nosuid,relatime,rw,size=7861540k,mode=1777	tmpfs	0:24
securityfs	/sys/kernel/security	securityfs	nodev,noexec,nosuid,relatime,rw	securityfs	0:7
tmpfs	/dev/shm	tmpfs	nodev,nosuid,rw,mode=1777	tmpfs	0:25
tmpfs	/run/lock	tmpfs	nodev,noexec,nosuid,relatime,rw,size=5120k,mode=1777	tmpfs	0:26
tmpfs	/sys/fs/cgroup	tmpfs	mode=755,nodev,noexec,nosuid,ro	tmpfs	0:27
cgroup	/sys/fs/cgroup/unified	cgroup2	nodev,noexec,nosuid,relatime,rw	cgroup	0:28
cgroup	/sys/fs/cgroup/systemd	cgroup	name=systemd,nodev,noexec,nosuid,relatime,rw,xattr	cgroup	0:29
pstore	/sys/fs/pstore	pstore	nodev,noexec,nosuid,relatime,rw	pstore	0:30
cgroup	/sys/fs/cgroup/cpu	cgroup	cpu,nodev,noexec,nosuid,relatime,rw	cgroup	0:31
systemd-1	/proc/sys/fs/binfmt_misc	autofs	direct,fd=32,maxproto=5,minproto=5,pgrp=1,relatime,rw,timeout=0	systemd-1	0:32
/lnet/image/LneTXFCE64	/boot	none	ro,bind	/dev/sda1	8:1
/lnet/cowroot/LneTXFCEhome	/home	none	rw,bind	/dev/sda2	8:2
mqueue	/dev/mqueue	mqueue	relatime,rw	mqueue	0:16
debugfs	/sys/kernel/debug	debugfs	relatime,rw	debugfs	0:8
hugetlbfs	/dev/hugepages	hugetlbfs	pagesize=2M,relatime,rw	hugetlbfs	0:33
/lnet/image/grub	/boot/grub	none	ro,bind	/dev/sda1	8:1
/lnet/image/grub	/lnet/image/LneTXFCE64/grub	none	ro,bind	/dev/sda1	8:1
/lnet/cowroot/JPLive	/home/JPLive	none	rw,bind	/dev/sda2	8:2
/lnet/cowroot/JPLive	/lnet/cowroot/LneTXFCEhome/JPLive	none	rw,bind	/dev/sda2	8:2
/lnet/cowroot/LneTXFCEhome/root	/root	none	rw,bind	/dev/sda2	8:2
/lnet/cowroot/JPLive/apt-cache-amd64	/var/cache/apt	none	rw,bind	/dev/sda2	8:2
/lnet/cowroot/JPLive/var-spool	/var/spool	none	rw,bind	/dev/sda2	8:2
/lnet/cowroot/JPLive/apt-lib-amd64	/var/lib/apt	none	rw,bind	/dev/sda2	8:2
tmpfs	/run/user/1001	tmpfs	gid=100,mode=700,nodev,nosuid,relatime,rw,size=786152k,uid=1001	tmpfs	0:38
/home/JPLive/LneT-LXQT-amd64/iso/LneT-LXQT/00filesystem.squashfs	/tmp/live-net-remaster-JGtNY3/image/00filesystem.squashfs	squashfs	relatime,ro,loop=/dev/loop2	/dev/loop2	7:2
overlay	/tmp/live-net-remaster-JGtNY3/chroot	overlay	index=off,lowerdir=/tmp/live-net-remaster-JGtNY3/image/00filesystem.squashfs,relatime,rw,upperdir=/home/tmp.IxhAZV2w0q/upper,workdir=/home/tmp.IxhAZV2w0q/work	overlay	0:37
/dev/	/tmp/live-net-remaster-JGtNY3/chroot/dev	none	rw,bind	udev	0:6
/dev/pts/	/tmp/live-net-remaster-JGtNY3/chroot/dev/pts	none	rw,bind	devpts	0:18
/dev/pts/	/dev/pts	none	rw,bind	devpts	0:18
/sys/	/tmp/live-net-remaster-JGtNY3/chroot/sys	none	rw,bind	sysfs	0:17
/tmp/	/tmp/live-net-remaster-JGtNY3/chroot/tmp	none	rw,bind	tmpfs	0:22
/lnet/cowroot/JPLive	/tmp/live-net-remaster-JGtNY3/chroot/home/JPLive	none	rw,bind	/dev/sda2	8:2
/proc/	/tmp/live-net-remaster-JGtNY3/chroot/proc	none	rw,bind	none	0:5
/lnet/cowroot/JPLive/apt-cache-amd64	/tmp/live-net-remaster-JGtNY3/chroot/var/cache/apt	none	rw,bind	/dev/sda2	8:2
/lnet/cowroot/JPLive/apt-lib-amd64	/tmp/live-net-remaster-JGtNY3/chroot/var/lib/apt	none	rw,bind	/dev/sda2	8:2

Re: v4.13 overlay fs: sometimes fails when creating or moving files

[Amir Goldstein]

On Fri, Sep 8, 2017 at 3:37 PM, Jordi Pujol Palomer
<jordipujolp@gmail.com> wrote:
> Hello,
>
> Found some errors when doing a remaster of a filesystem image in Linux v
> 4.13,
> the filesystem is mounted in overlay and then modifying the
> system into the chroot.
> The upgrade process fails when renaming or moving files.
> Sometimes the same process runs right and sometimes wrong,
>

This sounds a lot like https://bugzilla.redhat.com/show_bug.cgi?id=1213602
A long standing issue with a simple workaround (see link above)

Is the problem you are hitting new to v4.13?

Thanks
Amir.

Re: v4.13 overlay fs: sometimes fails when creating or moving files

[Jordi Pujol Palomer]

On Fri, 8 Sep 2017 17:20:50 +0300
Amir Goldstein <amir73il@gmail.com> wrote:

> This sounds a lot like
> https://bugzilla.redhat.com/show_bug.cgi?id=1213602 A long standing
> issue with a simple workaround (see link above)
> 
the problem also arises when executing 
touch /var/lib/dpkg/*
before upgrading,
maybe the issue is not the same,

> Is the problem you are hitting new to v4.13?
Yes, this problem is new,
therefore my computer continues running 4.12.11 by now.

Thanks,

Jordi Pujol i Palomer

Re: v4.13 overlay fs: sometimes fails when creating or moving files

[Amir Goldstein]

On Fri, Sep 8, 2017 at 6:55 PM, Jordi Pujol <jordipujolp@gmail.com> wrote:
>
> On Fri, 8 Sep 2017 17:20:50 +0300
> Amir Goldstein <amir73il@gmail.com> wrote:
>
> > This sounds a lot like
> > https://bugzilla.redhat.com/show_bug.cgi?id=1213602 A long standing
> > issue with a simple workaround (see link above)
> >
> the problem also arises when executing
> touch /var/lib/dpkg/*
> before upgrading,
> maybe the issue is not the same,
>
> The first time replacing the file does OK, but consecutive replacements fail.
> Maybe an unitialized variable internally in overlayfs.
> > Is the problem you are hitting new to v4.13?
> Yes, this problem is new,
> therefore my computer continues running 4.12.11 by now.
>
I see. Do you see any overlay fs warning in dmesg?
Please send them as well.
I did not understand if this happens during file create
rename delete if you understand the scenario and can send a simple
reproducer script that would be great.

Thanks,
Amir.

Re: v4.13 overlay fs: sometimes fails when creating or moving files

[Jordi Pujol]

On Fri, Sep 8, 2017 at 9:31 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> I see. Do you see any overlay fs warning in dmesg?
> Please send them as well.

The only overlay message is not an error,

kernel: ^[[0;1;39moverlayfs: fs on
'/tmp/live-net-remaster-ZpapXn/image/00filesystem.squashfs' does n\
ot support file handles, falling back to index=off.

> I did not understand if this happens during file create
> rename delete if you understand the scenario and can send a simple
> reproducer script that would be great.

after executing
touch /var/lib/dpkg/*
the first upgrade fails also,

I am not sure, but will try to do an script,

Re: v4.13 overlay fs: sometimes fails when creating or moving files

[Jordi Pujol]

This error disappears after installing 4.13.1, but other problems have
arised when unmounting some kind of filesystems.

Maybe it was not an overlayfs error.

On Fri, Sep 8, 2017 at 9:31 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> On Fri, Sep 8, 2017 at 6:55 PM, Jordi Pujol <jordipujolp@gmail.com> wrote:
>>
>> On Fri, 8 Sep 2017 17:20:50 +0300
>> Amir Goldstein <amir73il@gmail.com> wrote:
>>
>> > This sounds a lot like
>> > https://bugzilla.redhat.com/show_bug.cgi?id=1213602 A long standing
>> > issue with a simple workaround (see link above)
>> >
>> the problem also arises when executing
>> touch /var/lib/dpkg/*
>> before upgrading,
>> maybe the issue is not the same,
>>
>> The first time replacing the file does OK, but consecutive replacements fail.
>> Maybe an unitialized variable internally in overlayfs.
>> > Is the problem you are hitting new to v4.13?
>> Yes, this problem is new,
>> therefore my computer continues running 4.12.11 by now.
>>
> I see. Do you see any overlay fs warning in dmesg?
> Please send them as well.
> I did not understand if this happens during file create
> rename delete if you understand the scenario and can send a simple
> reproducer script that would be great.
>
> Thanks,
> Amir.

Re: v4.13 overlay fs: sometimes fails when creating or moving files

[Amir Goldstein]

On Sun, Sep 10, 2017 at 5:10 PM, Jordi Pujol <jordipujolp@gmail.com> wrote:
> This error disappears after installing 4.13.1, but other problems have
> arised when unmounting some kind of filesystems.
>

That is strage, because I see nothing fs related in v4.13..v4.13.1
which umount problems are you referring to?

> Maybe it was not an overlayfs error.
>

Ehm.. maybe, but considering the changes in overlayfs for v4.13
I cannot rule out that we missed some subtle race with overlay inode
cache that would change behavior.

Also, I did not test much with lower fs that does not support file handles,
and that is the case in your setup, so there might be a bug hiding there.

Please keep us posted if the errors make a comeback.

Thanks,
Amir.

Re: v4.13 overlay fs: sometimes fails when creating or moving files

[Jordi Pujol Palomer]

On Sun, 10 Sep 2017 22:21:09 +0300
Amir Goldstein <amir73il@gmail.com> wrote:

> That is strage, because I see nothing fs related in v4.13..v4.13.1
> which umount problems are you referring to?

I will document it later and open a new thread.

> Please keep us posted if the errors make a comeback.

OK, I agree

Thanks,

Jordi Pujol i Palomer

Re: v4.13 overlay fs: sometimes fails when creating or moving files

[Amir Goldstein]

On Sun, Sep 10, 2017 at 10:21 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> On Sun, Sep 10, 2017 at 5:10 PM, Jordi Pujol <jordipujolp@gmail.com> wrote:
>> This error disappears after installing 4.13.1, but other problems have
>> arised when unmounting some kind of filesystems.
>>
>
> That is strage, because I see nothing fs related in v4.13..v4.13.1
> which umount problems are you referring to?
>
>> Maybe it was not an overlayfs error.
>>
>
> Ehm.. maybe, but considering the changes in overlayfs for v4.13
> I cannot rule out that we missed some subtle race with overlay inode
> cache that would change behavior.
>
> Also, I did not test much with lower fs that does not support file handles,
> and that is the case in your setup, so there might be a bug hiding there.
>

Sure enough, testing with squashfs as lower layer found a bug very
much like you described.
Bug is easy to reproduce by trying to hardlink a lower file.
Harder to reproduce with rename, because it requires that dentry
of a copied up file is evicted from cache after rename, but that overlay
inode is still in cache.

You are welcome to try this fix if problem is reproducing for you
or verify the simpler hardlink regression.

Thanks,
Amir.

--- a/fs/overlayfs/inode.c
+++ b/fs/overlayfs/inode.c
@@ -581,8 +581,7 @@ static bool ovl_verify_inode(struct inode *inode,
struct dentry *lowerdentry,
 {
        struct inode *lowerinode = lowerdentry ? d_inode(lowerdentry) : NULL;

-       /* Lower (origin) inode must match, even if NULL */
-       if (ovl_inode_lower(inode) != lowerinode)
+       if (lowerdentry && ovl_inode_lower(inode) != lowerinode)
                return false;

Re: v4.13 overlay fs: sometimes fails when creating or moving files

[Jordi Pujol Palomer]

On Mon, 11 Sep 2017 15:19:44 +0300
Amir Goldstein <amir73il@gmail.com> wrote:

> You are welcome to try this fix if problem is reproducing for you
> or verify the simpler hardlink regression.

I have tried it in varied scenarios, a Live system with persistence
and a chroot to an image filesystem.
In all cases overlayfs performs well without that related errors.

Thanks,

Jordi Pujol i Palomer