ubifs+overlayfs

ubifs+overlayfs

[Pushpal Sidhu]

Hi,

It seems that overlayfs requires RENAME_WHITEOUT and RENAME_EXCHANGE.
However, these two flags aren't currently supported in ubifs (or
anything other than ext4, f2fs, and xfs). This causes any atomic
renames to fail (e.g. using 'mv' command).

I'm currently trying to implement these flags, but I'm having some
issues as I'm pretty vanilla when it comes to implementation details
on ubifs. I'm taking inspiration from patches [1] and [2], but I'm
kind of coming up short. Is there anyone currently working on this? If
not, I would certainly appreciate any advice for implementing these
features.

What I'm doing: I'm attempting to create a 'whiteout' object from
within the ubifs_rename function. However, I noticed that it calls
ubifs_jnl_rename, which is where I believe I should try to implement
this. I haven't touched RENAME_EXCHANGE yet.

[1] https://dev.openwrt.org/browser/trunk/target/linux/generic/patches-4.4/110-jffs2-use-.rename2-and-add-RENAME_WHITEOUT-support.patch
[2] https://dev.openwrt.org/browser/trunk/target/linux/generic/patches-4.4/111-jffs2-add-RENAME_EXCHANGE-support.patch

Thanks,
- Pushpal

Re: ubifs+overlayfs

[Richard Weinberger]

Cc'ing linux-mtd.

On Sat, Feb 20, 2016 at 1:02 AM, Pushpal Sidhu <psidhu@gateworks.com> wrote:
> It seems that overlayfs requires RENAME_WHITEOUT and RENAME_EXCHANGE.
> However, these two flags aren't currently supported in ubifs (or
> anything other than ext4, f2fs, and xfs). This causes any atomic
> renames to fail (e.g. using 'mv' command).

Hmm, right. I always thought overlayfs does checks at mount time.
But apparently I was wrong.
Thanks for pointing this out.

> I'm currently trying to implement these flags, but I'm having some
> issues as I'm pretty vanilla when it comes to implementation details
> on ubifs. I'm taking inspiration from patches [1] and [2], but I'm
> kind of coming up short. Is there anyone currently working on this? If
> not, I would certainly appreciate any advice for implementing these
> features.

Not to my knowledge. But you can use my O_TMPFILE patch as start.
See linux-mtd mailinglist.
If you have questions, just ask.
As using overlayfs is common on embedded stuff we definitely need that on UBIFS.
Just in case I'm putting that also on my todo list.

> What I'm doing: I'm attempting to create a 'whiteout' object from
> within the ubifs_rename function. However, I noticed that it calls
> ubifs_jnl_rename, which is where I believe I should try to implement
> this. I haven't touched RENAME_EXCHANGE yet.
>
> [1] https://dev.openwrt.org/browser/trunk/target/linux/generic/patches-4.4/110-jffs2-use-.rename2-and-add-RENAME_WHITEOUT-support.patch
> [2] https://dev.openwrt.org/browser/trunk/target/linux/generic/patches-4.4/111-jffs2-add-RENAME_EXCHANGE-support.patch

Hmm, these patches seem wrong to me.
Unless I'm very mistaken it looks like every whiteout creates an busy
orphaned inode.

-- 
Thanks,
//richard

Re: ubifs+overlayfs

[Dave Chinner]

On Sun, Feb 21, 2016 at 03:27:18PM +0100, Richard Weinberger wrote:
> Cc'ing linux-mtd.
> 
> On Sat, Feb 20, 2016 at 1:02 AM, Pushpal Sidhu <psidhu@gateworks.com> wrote:
> > It seems that overlayfs requires RENAME_WHITEOUT and RENAME_EXCHANGE.
> > However, these two flags aren't currently supported in ubifs (or
> > anything other than ext4, f2fs, and xfs). This causes any atomic
> > renames to fail (e.g. using 'mv' command).
> 
> Hmm, right. I always thought overlayfs does checks at mount time.
> But apparently I was wrong.
> Thanks for pointing this out.

It doesn't check any of the requirements it has from the underlying
filesystem. e.g. overlay requires DT_CHR for detecting whiteouts,
but it doesn't check if the upper filesystem actually supports
DT_CHR or not, and hence will fail to detect whiteouts correctly on
such filesystems. I've been pushing people to fix this, so maybe
patches will appear in the not too distant future...

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: ubifs+overlayfs

[Vivek Goyal]

On Mon, Feb 22, 2016 at 08:19:21AM +1100, Dave Chinner wrote:
> On Sun, Feb 21, 2016 at 03:27:18PM +0100, Richard Weinberger wrote:
> > Cc'ing linux-mtd.
> > 
> > On Sat, Feb 20, 2016 at 1:02 AM, Pushpal Sidhu <psidhu@gateworks.com> wrote:
> > > It seems that overlayfs requires RENAME_WHITEOUT and RENAME_EXCHANGE.
> > > However, these two flags aren't currently supported in ubifs (or
> > > anything other than ext4, f2fs, and xfs). This causes any atomic
> > > renames to fail (e.g. using 'mv' command).
> > 
> > Hmm, right. I always thought overlayfs does checks at mount time.
> > But apparently I was wrong.
> > Thanks for pointing this out.
> 
> It doesn't check any of the requirements it has from the underlying
> filesystem. e.g. overlay requires DT_CHR for detecting whiteouts,
> but it doesn't check if the upper filesystem actually supports
> DT_CHR or not, and hence will fail to detect whiteouts correctly on
> such filesystems. I've been pushing people to fix this, so maybe
> patches will appear in the not too distant future...

Dave pushed me to write the patch to detect whether upper filesystem
supports d_type or not :-). I will cleanup what I have and post for
review.

Thanks
Vivek

Re: ubifs+overlayfs

[Pushpal Sidhu]

Hi,

On Sun, Feb 21, 2016 at 6:27 AM, Richard Weinberger
<richard.weinberger@gmail.com> wrote:
> Cc'ing linux-mtd.
>
> On Sat, Feb 20, 2016 at 1:02 AM, Pushpal Sidhu <psidhu@gateworks.com> wrote:
>> It seems that overlayfs requires RENAME_WHITEOUT and RENAME_EXCHANGE.
>> However, these two flags aren't currently supported in ubifs (or
>> anything other than ext4, f2fs, and xfs). This causes any atomic
>> renames to fail (e.g. using 'mv' command).
>
> Hmm, right. I always thought overlayfs does checks at mount time.
> But apparently I was wrong.
> Thanks for pointing this out.

No problem.

>> I'm currently trying to implement these flags, but I'm having some
>> issues as I'm pretty vanilla when it comes to implementation details
>> on ubifs. I'm taking inspiration from patches [1] and [2], but I'm
>> kind of coming up short. Is there anyone currently working on this? If
>> not, I would certainly appreciate any advice for implementing these
>> features.
>
> Not to my knowledge. But you can use my O_TMPFILE patch as start.
> See linux-mtd mailinglist.
> If you have questions, just ask.
> As using overlayfs is common on embedded stuff we definitely need that on UBIFS.
> Just in case I'm putting that also on my todo list.

Great, that patch looks like it would help. If I can't totally figure
out a solution on my own, I'll let you know since you can probably
type out the implementation to these flags much faster than I am
currently able to.

>> What I'm doing: I'm attempting to create a 'whiteout' object from
>> within the ubifs_rename function. However, I noticed that it calls
>> ubifs_jnl_rename, which is where I believe I should try to implement
>> this. I haven't touched RENAME_EXCHANGE yet.
>>
>> [1] https://dev.openwrt.org/browser/trunk/target/linux/generic/patches-4.4/110-jffs2-use-.rename2-and-add-RENAME_WHITEOUT-support.patch
>> [2] https://dev.openwrt.org/browser/trunk/target/linux/generic/patches-4.4/111-jffs2-add-RENAME_EXCHANGE-support.patch
>
> Hmm, these patches seem wrong to me.
> Unless I'm very mistaken it looks like every whiteout creates an busy
> orphaned inode.

Great, I will only look at implementations currently in the kernel then.

Thanks,
- Pushpal

> --
> Thanks,
> //richard