overlayfs regression

overlayfs regression

[Christian Brauner]

Hey Miklosz,
hey Amir,

We've been observing regressions in our containers test-suite with
commit:

Author: Miklos Szeredi <mszeredi@redhat.com>
Date:   Tue Mar 17 15:04:22 2020 +0100

    ovl: separate detection of remote upper layer from stacked overlay

    Following patch will allow remote as upper layer, but not overlay stacked
    on upper layer.  Separate the two concepts.

    This patch is doesn't change behavior.

    Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>

It suddenly consistently reports:
[2422.695340] overlayfs: filesystem on '/home/lxcunpriv/.local/share/lxc/c2/overlay/delta' not supported as upperdir
in dmesg where it used to work fine for basically 6 years when we added
that test. The test creates a container c2 that uses the rootfs of
another container c1 (normal directory on an ext4 filesystem). Here you
can see the full mount options:

Invalid argument - Failed to mount "/home/lxcunpriv/.local/share/lxc/c1/rootfs" on "/usr/lib/x86_64-linux-gnu/lxc" with options "upperdir=/home/lxcunpriv/.local/share/lxc/c2/overlay/delta,lowerdir=/home/lxcunpriv/.local/share/lxc/c1/rootfs,workdir=/home/lxcunpriv/.local/share/lxc/c2/overlay/work"

Thanks!
Christian

Re: overlayfs regression

[Amir Goldstein]

On Wed, Jun 24, 2020 at 5:48 PM Christian Brauner
<christian.brauner@ubuntu.com> wrote:
>
> Hey Miklosz,
> hey Amir,
>
> We've been observing regressions in our containers test-suite with
> commit:
>
> Author: Miklos Szeredi <mszeredi@redhat.com>
> Date:   Tue Mar 17 15:04:22 2020 +0100
>
>     ovl: separate detection of remote upper layer from stacked overlay
>
>     Following patch will allow remote as upper layer, but not overlay stacked
>     on upper layer.  Separate the two concepts.
>
>     This patch is doesn't change behavior.
>
>     Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
>

Are you sure this is the offending commit?
Look at it. It is really moving a bit of code around and should not
change logic.
There are several other commits in 5.7 that could have gone wrong...

> It suddenly consistently reports:
> [2422.695340] overlayfs: filesystem on '/home/lxcunpriv/.local/share/lxc/c2/overlay/delta' not supported as upperdir
> in dmesg where it used to work fine for basically 6 years when we added
> that test. The test creates a container c2 that uses the rootfs of
> another container c1 (normal directory on an ext4 filesystem). Here you
> can see the full mount options:
>
> Invalid argument - Failed to mount "/home/lxcunpriv/.local/share/lxc/c1/rootfs" on "/usr/lib/x86_64-linux-gnu/lxc" with options "upperdir=/home/lxcunpriv/.local/share/lxc/c2/overlay/delta,lowerdir=/home/lxcunpriv/.local/share/lxc/c1/rootfs,workdir=/home/lxcunpriv/.local/share/lxc/c2/overlay/work"
>

/home/lxcunpriv/.local/share/lxc/c2/overlay/delta' is ext4?

and it fails the test because (path->dentry->d_flags & DCACHE_OP_REAL)?
It the only thing special about that path is that it is not in root mount ns?

That is strange...

Thanks,
Amir.

Re: overlayfs regression

[Christian Brauner]

On Wed, Jun 24, 2020 at 06:25:55PM +0300, Amir Goldstein wrote:
> On Wed, Jun 24, 2020 at 5:48 PM Christian Brauner
> <christian.brauner@ubuntu.com> wrote:
> >
> > Hey Miklosz,
> > hey Amir,
> >
> > We've been observing regressions in our containers test-suite with
> > commit:
> >
> > Author: Miklos Szeredi <mszeredi@redhat.com>
> > Date:   Tue Mar 17 15:04:22 2020 +0100
> >
> >     ovl: separate detection of remote upper layer from stacked overlay
> >
> >     Following patch will allow remote as upper layer, but not overlay stacked
> >     on upper layer.  Separate the two concepts.
> >
> >     This patch is doesn't change behavior.
> >
> >     Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
> >
> 
> Are you sure this is the offending commit?
> Look at it. It is really moving a bit of code around and should not
> change logic.
> There are several other commits in 5.7 that could have gone wrong...

Yeah, most likely. I can do a bisect but it might take a little until I
get around to it. Is that ok?

> 
> > It suddenly consistently reports:
> > [2422.695340] overlayfs: filesystem on '/home/lxcunpriv/.local/share/lxc/c2/overlay/delta' not supported as upperdir
> > in dmesg where it used to work fine for basically 6 years when we added
> > that test. The test creates a container c2 that uses the rootfs of
> > another container c1 (normal directory on an ext4 filesystem). Here you
> > can see the full mount options:
> >
> > Invalid argument - Failed to mount "/home/lxcunpriv/.local/share/lxc/c1/rootfs" on "/usr/lib/x86_64-linux-gnu/lxc" with options "upperdir=/home/lxcunpriv/.local/share/lxc/c2/overlay/delta,lowerdir=/home/lxcunpriv/.local/share/lxc/c1/rootfs,workdir=/home/lxcunpriv/.local/share/lxc/c2/overlay/work"
> >
> 
> /home/lxcunpriv/.local/share/lxc/c2/overlay/delta' is ext4?

Yeah, so what we do is:

lower -> /foo/c1/rootfs

then we create the directory for the new container:

mkdir_p(/foo/c2)
mkdir_p(/foo/c2/rootfs)
mkdir_p(/foo/c2/overlay)
mkdir_p(/foo/c2/overlay/delta)
mkdir_p(/foo/c2/overlay/workdir)

ensuring that delta + work are on the same mount and that workdir is
empty. The we perform the mount you see above.

Christian

Re: overlayfs regression

[Amir Goldstein]

On Wed, Jun 24, 2020 at 6:35 PM Christian Brauner
<christian.brauner@ubuntu.com> wrote:
>
> On Wed, Jun 24, 2020 at 06:25:55PM +0300, Amir Goldstein wrote:
> > On Wed, Jun 24, 2020 at 5:48 PM Christian Brauner
> > <christian.brauner@ubuntu.com> wrote:
> > >
> > > Hey Miklosz,
> > > hey Amir,
> > >
> > > We've been observing regressions in our containers test-suite with
> > > commit:
> > >
> > > Author: Miklos Szeredi <mszeredi@redhat.com>
> > > Date:   Tue Mar 17 15:04:22 2020 +0100
> > >
> > >     ovl: separate detection of remote upper layer from stacked overlay
> > >
> > >     Following patch will allow remote as upper layer, but not overlay stacked
> > >     on upper layer.  Separate the two concepts.
> > >
> > >     This patch is doesn't change behavior.
> > >
> > >     Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
> > >
> >
> > Are you sure this is the offending commit?
> > Look at it. It is really moving a bit of code around and should not
> > change logic.
> > There are several other commits in 5.7 that could have gone wrong...
>
> Yeah, most likely. I can do a bisect but it might take a little until I
> get around to it. Is that ok?
>

ok.
I thought you pointed to a commit that you bisected the regression to.
I guess not.

Thanks,
Amir.

Re: overlayfs regression

[Seth Forshee]

On Wed, Jun 24, 2020 at 07:24:24PM +0300, Amir Goldstein wrote:
> On Wed, Jun 24, 2020 at 6:35 PM Christian Brauner
> <christian.brauner@ubuntu.com> wrote:
> >
> > On Wed, Jun 24, 2020 at 06:25:55PM +0300, Amir Goldstein wrote:
> > > On Wed, Jun 24, 2020 at 5:48 PM Christian Brauner
> > > <christian.brauner@ubuntu.com> wrote:
> > > >
> > > > Hey Miklosz,
> > > > hey Amir,
> > > >
> > > > We've been observing regressions in our containers test-suite with
> > > > commit:
> > > >
> > > > Author: Miklos Szeredi <mszeredi@redhat.com>
> > > > Date:   Tue Mar 17 15:04:22 2020 +0100
> > > >
> > > >     ovl: separate detection of remote upper layer from stacked overlay
> > > >
> > > >     Following patch will allow remote as upper layer, but not overlay stacked
> > > >     on upper layer.  Separate the two concepts.
> > > >
> > > >     This patch is doesn't change behavior.
> > > >
> > > >     Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
> > > >
> > >
> > > Are you sure this is the offending commit?
> > > Look at it. It is really moving a bit of code around and should not
> > > change logic.
> > > There are several other commits in 5.7 that could have gone wrong...
> >
> > Yeah, most likely. I can do a bisect but it might take a little until I
> > get around to it. Is that ok?
> >
> 
> ok.
> I thought you pointed to a commit that you bisected the regression to.
> I guess not.

I think this is only an Ubuntu problem, it looks like someone did some
bad conflict resoluation when rebasing a sauce patch onto 5.7.

Seth

Re: overlayfs regression

[Christian Brauner]

On Thu, Jun 25, 2020 at 08:19:43AM -0500, Seth Forshee wrote:
> On Wed, Jun 24, 2020 at 07:24:24PM +0300, Amir Goldstein wrote:
> > On Wed, Jun 24, 2020 at 6:35 PM Christian Brauner
> > <christian.brauner@ubuntu.com> wrote:
> > >
> > > On Wed, Jun 24, 2020 at 06:25:55PM +0300, Amir Goldstein wrote:
> > > > On Wed, Jun 24, 2020 at 5:48 PM Christian Brauner
> > > > <christian.brauner@ubuntu.com> wrote:
> > > > >
> > > > > Hey Miklosz,
> > > > > hey Amir,
> > > > >
> > > > > We've been observing regressions in our containers test-suite with
> > > > > commit:
> > > > >
> > > > > Author: Miklos Szeredi <mszeredi@redhat.com>
> > > > > Date:   Tue Mar 17 15:04:22 2020 +0100
> > > > >
> > > > >     ovl: separate detection of remote upper layer from stacked overlay
> > > > >
> > > > >     Following patch will allow remote as upper layer, but not overlay stacked
> > > > >     on upper layer.  Separate the two concepts.
> > > > >
> > > > >     This patch is doesn't change behavior.
> > > > >
> > > > >     Signed-off-by: Miklos Szeredi <mszeredi@redhat.com>
> > > > >
> > > >
> > > > Are you sure this is the offending commit?
> > > > Look at it. It is really moving a bit of code around and should not
> > > > change logic.
> > > > There are several other commits in 5.7 that could have gone wrong...
> > >
> > > Yeah, most likely. I can do a bisect but it might take a little until I
> > > get around to it. Is that ok?
> > >
> > 
> > ok.
> > I thought you pointed to a commit that you bisected the regression to.
> > I guess not.
> 
> I think this is only an Ubuntu problem, it looks like someone did some
> bad conflict resoluation when rebasing a sauce patch onto 5.7.

Yeah, I just saw this as well.

Sorry for the noise.
Christian