* overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"
@ 2016-09-06 10:46 Rafał Miłecki
2016-09-06 11:59 ` Richard Weinberger
0 siblings, 1 reply; 6+ messages in thread
From: Rafał Miłecki @ 2016-09-06 10:46 UTC (permalink / raw)
To: Miklos Szeredi, linux-unionfs, Richard Weinberger,
Artem Bityutskiy, Adrian Hunter, linux-mtd
Cc: lede-dev
Hi,
In LEDE/OpenWrt we use UBI with two volumes: "rootfs" and "rootfs_data". With
rootfs mounted at / we mount rootfs_data volume at /overlay and then use
overlayfs to combine these two.
1) Before:
/dev/root on / type squashfs (ro,relatime)
/dev/ubi0_1 on /overlay type ubifs (rw,noatime)
2) Mounting overlayfs
mount("overlayfs:/overlay", "/mnt", "overlay", MS_NOATIME, "lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work")
3) After:
/dev/root on / type squashfs (ro,relatime)
/dev/ubi0_1 on /overlay type ubifs (rw,noatime)
overlayfs:/overlay on /mnt type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
It works fine as long as there aren't unexpected power cuts. I can easily break
it in a well-reproducible way by doing:
date > bar.txt && rm foo.sh
sleep 2s
[power cut]
On the next boot LEDE/OpenWrt script calls the same mount function but it
results in:
[ 10.017235] overlayfs: failed to create directory /overlay/work/work (errno: 17); mounting read-only
I tracked that error and got a following forward trace for it:
ovl_fill_super
ovl_workdir_create
ovl_cleanup
vfs_rmdir
ubifs_rmdir
check_dir_empty
In other words mounting overlayfs fails because ubifs refuses to remove
work/work as this directory is not empty. On a clean boot /overlay/work/work is
empty and everything works fine but after a power cut I can see
/overlay/work/work/#c752f2a8 file.
I don't really know anything about overlayfs and its workdir, I just expected
overlayfs to recover from such situation on its own. FWIW after a power cut and
reboot my bar.txt is an empty file and foo.sh still exists.
Does it sound like an overlayfs issue to you? Can you think of some way of
fixing this?
--
Rafał
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"
2016-09-06 10:46 overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only" Rafał Miłecki
@ 2016-09-06 11:59 ` Richard Weinberger
2016-09-06 12:50 ` Miklos Szeredi
0 siblings, 1 reply; 6+ messages in thread
From: Richard Weinberger @ 2016-09-06 11:59 UTC (permalink / raw)
To: Rafał Miłecki, Miklos Szeredi, linux-unionfs,
Artem Bityutskiy, Adrian Hunter, linux-mtd
Cc: lede-dev
Rafał,
On 06.09.2016 12:46, Rafał Miłecki wrote:
> Hi,
>
> In LEDE/OpenWrt we use UBI with two volumes: "rootfs" and "rootfs_data". With
> rootfs mounted at / we mount rootfs_data volume at /overlay and then use
> overlayfs to combine these two.
>
> 1) Before:
> /dev/root on / type squashfs (ro,relatime)
> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
>
> 2) Mounting overlayfs
> mount("overlayfs:/overlay", "/mnt", "overlay", MS_NOATIME, "lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work")
>
> 3) After:
> /dev/root on / type squashfs (ro,relatime)
> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
> overlayfs:/overlay on /mnt type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
>
> It works fine as long as there aren't unexpected power cuts. I can easily break
> it in a well-reproducible way by doing:
> date > bar.txt && rm foo.sh
> sleep 2s
> [power cut]
>
> On the next boot LEDE/OpenWrt script calls the same mount function but it
> results in:
> [ 10.017235] overlayfs: failed to create directory /overlay/work/work (errno: 17); mounting read-only
>
> I tracked that error and got a following forward trace for it:
> ovl_fill_super
> ovl_workdir_create
> ovl_cleanup
> vfs_rmdir
> ubifs_rmdir
> check_dir_empty
>
> In other words mounting overlayfs fails because ubifs refuses to remove
> work/work as this directory is not empty. On a clean boot /overlay/work/work is
> empty and everything works fine but after a power cut I can see
> /overlay/work/work/#c752f2a8 file.
>
> I don't really know anything about overlayfs and its workdir, I just expected
> overlayfs to recover from such situation on its own. FWIW after a power cut and
> reboot my bar.txt is an empty file and foo.sh still exists.
>
> Does it sound like an overlayfs issue to you? Can you think of some way of
> fixing this?
overlayfs does not issue a recursive rm on workdir, AFAICT you are supposed to cleanup
workdir yourself before mounting.
Miklos?
Thanks,
//richard
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"
2016-09-06 11:59 ` Richard Weinberger
@ 2016-09-06 12:50 ` Miklos Szeredi
2016-09-06 12:56 ` Rafał Miłecki
0 siblings, 1 reply; 6+ messages in thread
From: Miklos Szeredi @ 2016-09-06 12:50 UTC (permalink / raw)
To: Richard Weinberger
Cc: Rafał Miłecki, linux-unionfs@vger.kernel.org,
Artem Bityutskiy, Adrian Hunter, linux-mtd, lede-dev
On Tue, Sep 6, 2016 at 1:59 PM, Richard Weinberger <richard@nod.at> wrote:
> Rafał,
>
> On 06.09.2016 12:46, Rafał Miłecki wrote:
>> Hi,
>>
>> In LEDE/OpenWrt we use UBI with two volumes: "rootfs" and "rootfs_data". With
>> rootfs mounted at / we mount rootfs_data volume at /overlay and then use
>> overlayfs to combine these two.
>>
>> 1) Before:
>> /dev/root on / type squashfs (ro,relatime)
>> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
>>
>> 2) Mounting overlayfs
>> mount("overlayfs:/overlay", "/mnt", "overlay", MS_NOATIME, "lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work")
>>
>> 3) After:
>> /dev/root on / type squashfs (ro,relatime)
>> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
>> overlayfs:/overlay on /mnt type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
>>
>> It works fine as long as there aren't unexpected power cuts. I can easily break
>> it in a well-reproducible way by doing:
>> date > bar.txt && rm foo.sh
>> sleep 2s
>> [power cut]
>>
>> On the next boot LEDE/OpenWrt script calls the same mount function but it
>> results in:
>> [ 10.017235] overlayfs: failed to create directory /overlay/work/work (errno: 17); mounting read-only
>>
>> I tracked that error and got a following forward trace for it:
>> ovl_fill_super
>> ovl_workdir_create
>> ovl_cleanup
>> vfs_rmdir
>> ubifs_rmdir
>> check_dir_empty
>>
>> In other words mounting overlayfs fails because ubifs refuses to remove
>> work/work as this directory is not empty. On a clean boot /overlay/work/work is
>> empty and everything works fine but after a power cut I can see
>> /overlay/work/work/#c752f2a8 file.
>>
>> I don't really know anything about overlayfs and its workdir, I just expected
>> overlayfs to recover from such situation on its own. FWIW after a power cut and
>> reboot my bar.txt is an empty file and foo.sh still exists.
>>
>> Does it sound like an overlayfs issue to you? Can you think of some way of
>> fixing this?
>
> overlayfs does not issue a recursive rm on workdir, AFAICT you are supposed to cleanup
> workdir yourself before mounting.
> Miklos?
Fixed in 4.8-rc5 by
eea2fb4851e9 ("ovl: proper cleanup of workdir")
I think it's better if kernel does the cleanup, but if it's easier to
upgrade userspace, then that's a valid workaround.
Thanks,
Miklos
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"
2016-09-06 12:50 ` Miklos Szeredi
@ 2016-09-06 12:56 ` Rafał Miłecki
2016-09-06 15:53 ` Rafał Miłecki
0 siblings, 1 reply; 6+ messages in thread
From: Rafał Miłecki @ 2016-09-06 12:56 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Richard Weinberger, linux-unionfs@vger.kernel.org,
Artem Bityutskiy, Adrian Hunter, linux-mtd@lists.infradead.org,
LEDE Development List
On 6 September 2016 at 14:50, Miklos Szeredi <miklos@szeredi.hu> wrote:
> On Tue, Sep 6, 2016 at 1:59 PM, Richard Weinberger <richard@nod.at> wrote:
>> Rafał,
>>
>> On 06.09.2016 12:46, Rafał Miłecki wrote:
>>> Hi,
>>>
>>> In LEDE/OpenWrt we use UBI with two volumes: "rootfs" and "rootfs_data". With
>>> rootfs mounted at / we mount rootfs_data volume at /overlay and then use
>>> overlayfs to combine these two.
>>>
>>> 1) Before:
>>> /dev/root on / type squashfs (ro,relatime)
>>> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
>>>
>>> 2) Mounting overlayfs
>>> mount("overlayfs:/overlay", "/mnt", "overlay", MS_NOATIME, "lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work")
>>>
>>> 3) After:
>>> /dev/root on / type squashfs (ro,relatime)
>>> /dev/ubi0_1 on /overlay type ubifs (rw,noatime)
>>> overlayfs:/overlay on /mnt type overlay (ro,noatime,lowerdir=/,upperdir=/overlay/upper,workdir=/overlay/work)
>>>
>>> It works fine as long as there aren't unexpected power cuts. I can easily break
>>> it in a well-reproducible way by doing:
>>> date > bar.txt && rm foo.sh
>>> sleep 2s
>>> [power cut]
>>>
>>> On the next boot LEDE/OpenWrt script calls the same mount function but it
>>> results in:
>>> [ 10.017235] overlayfs: failed to create directory /overlay/work/work (errno: 17); mounting read-only
>>>
>>> I tracked that error and got a following forward trace for it:
>>> ovl_fill_super
>>> ovl_workdir_create
>>> ovl_cleanup
>>> vfs_rmdir
>>> ubifs_rmdir
>>> check_dir_empty
>>>
>>> In other words mounting overlayfs fails because ubifs refuses to remove
>>> work/work as this directory is not empty. On a clean boot /overlay/work/work is
>>> empty and everything works fine but after a power cut I can see
>>> /overlay/work/work/#c752f2a8 file.
>>>
>>> I don't really know anything about overlayfs and its workdir, I just expected
>>> overlayfs to recover from such situation on its own. FWIW after a power cut and
>>> reboot my bar.txt is an empty file and foo.sh still exists.
>>>
>>> Does it sound like an overlayfs issue to you? Can you think of some way of
>>> fixing this?
>>
>> overlayfs does not issue a recursive rm on workdir, AFAICT you are supposed to cleanup
>> workdir yourself before mounting.
>> Miklos?
>
> Fixed in 4.8-rc5 by
>
> eea2fb4851e9 ("ovl: proper cleanup of workdir")
>
> I think it's better if kernel does the cleanup, but if it's easier to
> upgrade userspace, then that's a valid workaround.
Oh, this should solve my problem! It seems it didn't git stable
kernels yet, but I can backport it on my own for now.
Thanks!
P.S.
Interesting part:
> While this has never been reported, it is possible to get a non-empty
> "work" dir from a previous mount of overlayfs in case of crash in the
> middle of an operation using the work directory.
Funny to hit never-reported problem right after someone (Miklos) fixed it ;)
--
Rafał
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"
2016-09-06 12:56 ` Rafał Miłecki
@ 2016-09-06 15:53 ` Rafał Miłecki
2016-09-07 8:28 ` Miklos Szeredi
0 siblings, 1 reply; 6+ messages in thread
From: Rafał Miłecki @ 2016-09-06 15:53 UTC (permalink / raw)
To: Miklos Szeredi
Cc: Richard Weinberger, linux-unionfs@vger.kernel.org,
Artem Bityutskiy, Adrian Hunter, linux-mtd@lists.infradead.org,
LEDE Development List
On 6 September 2016 at 14:56, Rafał Miłecki <zajec5@gmail.com> wrote:
> On 6 September 2016 at 14:50, Miklos Szeredi <miklos@szeredi.hu> wrote:
>> On Tue, Sep 6, 2016 at 1:59 PM, Richard Weinberger <richard@nod.at> wrote:
>>> overlayfs does not issue a recursive rm on workdir, AFAICT you are supposed to cleanup
>>> workdir yourself before mounting.
>>> Miklos?
>>
>> Fixed in 4.8-rc5 by
>>
>> eea2fb4851e9 ("ovl: proper cleanup of workdir")
>>
>> I think it's better if kernel does the cleanup, but if it's easier to
>> upgrade userspace, then that's a valid workaround.
>
> Oh, this should solve my problem! It seems it didn't git stable
> kernels yet, but I can backport it on my own for now.
I tried backporting it to 4.4, but it appears it won't be that
trivial. Your patch ("ovl: proper cleanup of workdir") uses e.g.
inode_lock_nested which is available in 4.5+. Commit that added this
function: 5955102c9984f ("wrappers for ->i_mutex access") wasn't
marked for stable, so I guess your patch won't go into 4.4 at all.
I don't expect any magic solution for this, just saying, leaving a
note for reference.
--
Rafał
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only"
2016-09-06 15:53 ` Rafał Miłecki
@ 2016-09-07 8:28 ` Miklos Szeredi
0 siblings, 0 replies; 6+ messages in thread
From: Miklos Szeredi @ 2016-09-07 8:28 UTC (permalink / raw)
To: Rafał Miłecki
Cc: Richard Weinberger, linux-unionfs@vger.kernel.org,
Artem Bityutskiy, Adrian Hunter, linux-mtd@lists.infradead.org,
LEDE Development List
On Tue, Sep 6, 2016 at 5:53 PM, Rafał Miłecki <zajec5@gmail.com> wrote:
> On 6 September 2016 at 14:56, Rafał Miłecki <zajec5@gmail.com> wrote:
>> On 6 September 2016 at 14:50, Miklos Szeredi <miklos@szeredi.hu> wrote:
>>> On Tue, Sep 6, 2016 at 1:59 PM, Richard Weinberger <richard@nod.at> wrote:
>>>> overlayfs does not issue a recursive rm on workdir, AFAICT you are supposed to cleanup
>>>> workdir yourself before mounting.
>>>> Miklos?
>>>
>>> Fixed in 4.8-rc5 by
>>>
>>> eea2fb4851e9 ("ovl: proper cleanup of workdir")
>>>
>>> I think it's better if kernel does the cleanup, but if it's easier to
>>> upgrade userspace, then that's a valid workaround.
>>
>> Oh, this should solve my problem! It seems it didn't git stable
>> kernels yet, but I can backport it on my own for now.
>
> I tried backporting it to 4.4, but it appears it won't be that
> trivial. Your patch ("ovl: proper cleanup of workdir") uses e.g.
> inode_lock_nested which is available in 4.5+. Commit that added this
> function: 5955102c9984f ("wrappers for ->i_mutex access") wasn't
> marked for stable, so I guess your patch won't go into 4.4 at all.
>
> I don't expect any magic solution for this, just saying, leaving a
> note for reference.
Conversion is trivial, in fact:
#define inode_lock(inode) mutex_lock(&(inode)->i_mutex)
#define inode_unlock(inode) mutex_unlock(&(inode)->i_mutex)
#define inode_trylock(inode) mutex_trylock(&(inode)->i_mutex)
#define inode_lock_nested(inode, subclass)
mutex_lock_nested(&(inode)->i_mutex, subclass)
And Greg promised to put these defines into -stable for easier
backport experience.
Thanks,
Miklos
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2016-09-07 8:28 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2016-09-06 10:46 overlayfs + ubifs: power cut results in "failed to create directory /overlay/work/work (errno: 17); mounting read-only" Rafał Miłecki
2016-09-06 11:59 ` Richard Weinberger
2016-09-06 12:50 ` Miklos Szeredi
2016-09-06 12:56 ` Rafał Miłecki
2016-09-06 15:53 ` Rafał Miłecki
2016-09-07 8:28 ` Miklos Szeredi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).