Re: constant st_ino/st_dev for non-samefs case

Re: constant st_ino/st_dev for non-samefs case

[Amir Goldstein]

On Wed, May 17, 2017 at 3:46 PM, Chandan Rajendra
<chandan@linux.vnet.ibm.com> wrote:
> On Wednesday, May 17, 2017 5:56:31 PM IST Amir Goldstein wrote:
>> On Wed, May 17, 2017 at 2:42 PM, Chandan Rajendra

>> There is actually one task that should be very simple to do and can also
>> bring large benefit for many users.
>>
>> In his pull request for kernel 4.12 Miklos writes:
>> "The biggest part of this is making st_dev/st_ino on the overlay behave like a
>> normal filesystem ... future work will move the general case towards more sane
>> behavior."
>> https://marc.info/?l=linux-unionfs&m=149442365202823&w=2
>>
>> The work towards constant st_dev/st_ino for the general case is not within my
>> immediate scope of interest, but it shouldn't be hard. The basic idea was
>> explained by  Miklos here:
>> https://marc.info/?l=linux-unionfs&m=149259338809700&w=2
>>
> Amir, I will work on this task. Thanks for the guidance.
>

Hi Chandan,

Were you able to figure out the scope of the task?
Or didn't get around to it yet?

I forgot to mention that I have written tests to cover constant inode
across copy up.
If you have never used unionmount-testsuite before, now would be a good time to
start.

You should pull my master branch from github:
https://github.com/amir73il/unionmount-testsuite

Just do:
# cd unionmount-testsuite
# sudo ./run --ov

It takes only ~13 sec for the basic test run to compete on my laptop.

Then, you should remove the relaxing of constant ino check for non samefs:
https://github.com/amir73il/unionmount-testsuite/commit/958b737a0a886b6949964d51dbf8e0786ea52d02#diff-2d2badb93764e2d3fa25662025095014R496

And you will hit the ino check error, for example:

# sudo ./run --ov rename-file
***
*** ./run --ov --ts=0 rename-file
***
TEST rename-file.py:10: Rename file and rename back
 ./run --rename /mnt/a/foo100 /mnt/a/no_foo100
/mnt/a/no_foo100: inode number wrong (got 35985, want 35584)


Please let me know if you intend to work on this soon, because I may be
working on some parts of this task for a different feature.

Thanks,
Amir.

Re: constant st_ino/st_dev for non-samefs case

[Chandan Rajendra]

On Tuesday, May 23, 2017 6:27:42 PM IST Amir Goldstein wrote:
> On Wed, May 17, 2017 at 3:46 PM, Chandan Rajendra
> <chandan@linux.vnet.ibm.com> wrote:
> > On Wednesday, May 17, 2017 5:56:31 PM IST Amir Goldstein wrote:
> >> On Wed, May 17, 2017 at 2:42 PM, Chandan Rajendra
> 
> >> There is actually one task that should be very simple to do and can also
> >> bring large benefit for many users.
> >>
> >> In his pull request for kernel 4.12 Miklos writes:
> >> "The biggest part of this is making st_dev/st_ino on the overlay behave like a
> >> normal filesystem ... future work will move the general case towards more sane
> >> behavior."
> >> https://marc.info/?l=linux-unionfs&m=149442365202823&w=2
> >>
> >> The work towards constant st_dev/st_ino for the general case is not within my
> >> immediate scope of interest, but it shouldn't be hard. The basic idea was
> >> explained by  Miklos here:
> >> https://marc.info/?l=linux-unionfs&m=149259338809700&w=2
> >>
> > Amir, I will work on this task. Thanks for the guidance.
> >
> 
> Hi Chandan,
> 
> Were you able to figure out the scope of the task?
> Or didn't get around to it yet?

Hi Amir,

I spent time understanding current overlayfs code. I am now starting to
work on the problem statement.

I should be able to post the patches to the mailing list by the end of 
this week.

> 
> I forgot to mention that I have written tests to cover constant inode
> across copy up.
> If you have never used unionmount-testsuite before, now would be a good time to
> start.
> 
> You should pull my master branch from github:
> https://github.com/amir73il/unionmount-testsuite
> 
> Just do:
> # cd unionmount-testsuite
> # sudo ./run --ov
> 
> It takes only ~13 sec for the basic test run to compete on my laptop.
> 
> Then, you should remove the relaxing of constant ino check for non samefs:
> https://github.com/amir73il/unionmount-testsuite/commit/958b737a0a886b6949964d51dbf8e0786ea52d02#diff-2d2badb93764e2d3fa25662025095014R496
> 
> And you will hit the ino check error, for example:
> 
> # sudo ./run --ov rename-file
> ***
> *** ./run --ov --ts=0 rename-file
> ***
> TEST rename-file.py:10: Rename file and rename back
>  ./run --rename /mnt/a/foo100 /mnt/a/no_foo100
> /mnt/a/no_foo100: inode number wrong (got 35985, want 35584)
> 
> 
> Please let me know if you intend to work on this soon, because I may be
> working on some parts of this task for a different feature.
> 
> Thanks,
> Amir.
> 
> 


-- 
chandan

Re: constant st_ino/st_dev for non-samefs case

[Amir Goldstein]

On Tue, May 23, 2017 at 4:19 PM, Chandan Rajendra
<chandan@linux.vnet.ibm.com> wrote:
> On Tuesday, May 23, 2017 6:27:42 PM IST Amir Goldstein wrote:
>> On Wed, May 17, 2017 at 3:46 PM, Chandan Rajendra
>> <chandan@linux.vnet.ibm.com> wrote:
>> > On Wednesday, May 17, 2017 5:56:31 PM IST Amir Goldstein wrote:
>> >> On Wed, May 17, 2017 at 2:42 PM, Chandan Rajendra
>>
>> >> There is actually one task that should be very simple to do and can also
>> >> bring large benefit for many users.
>> >>
>> >> In his pull request for kernel 4.12 Miklos writes:
>> >> "The biggest part of this is making st_dev/st_ino on the overlay behave like a
>> >> normal filesystem ... future work will move the general case towards more sane
>> >> behavior."
>> >> https://marc.info/?l=linux-unionfs&m=149442365202823&w=2
>> >>
>> >> The work towards constant st_dev/st_ino for the general case is not within my
>> >> immediate scope of interest, but it shouldn't be hard. The basic idea was
>> >> explained by  Miklos here:
>> >> https://marc.info/?l=linux-unionfs&m=149259338809700&w=2
>> >>
>> > Amir, I will work on this task. Thanks for the guidance.
>> >
>>
>> Hi Chandan,
>>
>> Were you able to figure out the scope of the task?
>> Or didn't get around to it yet?
>
> Hi Amir,
>
> I spent time understanding current overlayfs code. I am now starting to
> work on the problem statement.
>
> I should be able to post the patches to the mailing list by the end of
> this week.
>
...
>>
>> Please let me know if you intend to work on this soon, because I may be
>> working on some parts of this task for a different feature.

Chandan,

My apologies for biting into your task, but as I wrote, I needed some parts
for another feature, so implemented partial non-samefs support [1].
At least one of the 2 "relax same fs" patches was not as trivial as I indented
for your task to be.

You still need to implement this part:
"The only extra thing needed compared to the samefs case is the allocation
of dummy device numbers for lower layers."

I also fixes unionmount-testsuite to check constant inode for non samefs [2].

The problem now is that all the test pass, although, as I wrote, I only
implemented partial support for non samefs. This means that the test coverage
for constant st_ino/st_dev is insufficient to validate the results of your task.
Which is good, because it will give you an opportunity to enhance the tests :)

It would be great if you could add validation to the fact that the
overlay st_ino/
st_dev pair is different from the underlying inode st_ino/st_dev.
That is not the case with upstream overlayfs and that is not the case with
my partial work on non same fs constant st_ino.

A new overlay xfstest, along the lines of overlay/017 that validates
the uniqueness of st_ino/st_dev would also be a good idea.

Thanks,
Amir.

[1] https://github.com/amir73il/linux/commits/ovl-constino
[2] https://github.com/amir73il/unionmount-testsuite/commits/ovl-constino

Re: constant st_ino/st_dev for non-samefs case

[Chandan Rajendra]

On Wednesday, May 24, 2017 2:03:29 AM IST Amir Goldstein wrote:
> On Tue, May 23, 2017 at 4:19 PM, Chandan Rajendra
> <chandan@linux.vnet.ibm.com> wrote:
> > On Tuesday, May 23, 2017 6:27:42 PM IST Amir Goldstein wrote:
> >> On Wed, May 17, 2017 at 3:46 PM, Chandan Rajendra
> >> <chandan@linux.vnet.ibm.com> wrote:
> >> > On Wednesday, May 17, 2017 5:56:31 PM IST Amir Goldstein wrote:
> >> >> On Wed, May 17, 2017 at 2:42 PM, Chandan Rajendra
> >>
> >> >> There is actually one task that should be very simple to do and can also
> >> >> bring large benefit for many users.
> >> >>
> >> >> In his pull request for kernel 4.12 Miklos writes:
> >> >> "The biggest part of this is making st_dev/st_ino on the overlay behave like a
> >> >> normal filesystem ... future work will move the general case towards more sane
> >> >> behavior."
> >> >> https://marc.info/?l=linux-unionfs&m=149442365202823&w=2
> >> >>
> >> >> The work towards constant st_dev/st_ino for the general case is not within my
> >> >> immediate scope of interest, but it shouldn't be hard. The basic idea was
> >> >> explained by  Miklos here:
> >> >> https://marc.info/?l=linux-unionfs&m=149259338809700&w=2
> >> >>
> >> > Amir, I will work on this task. Thanks for the guidance.
> >> >
> >>
> >> Hi Chandan,
> >>
> >> Were you able to figure out the scope of the task?
> >> Or didn't get around to it yet?
> >
> > Hi Amir,
> >
> > I spent time understanding current overlayfs code. I am now starting to
> > work on the problem statement.
> >
> > I should be able to post the patches to the mailing list by the end of
> > this week.
> >
> ...
> >>
> >> Please let me know if you intend to work on this soon, because I may be
> >> working on some parts of this task for a different feature.
> 
> Chandan,
> 
> My apologies for biting into your task, but as I wrote, I needed some parts
> for another feature, so implemented partial non-samefs support [1].
> At least one of the 2 "relax same fs" patches was not as trivial as I indented
> for your task to be.
> 
> You still need to implement this part:
> "The only extra thing needed compared to the samefs case is the allocation
> of dummy device numbers for lower layers."
> 
> I also fixes unionmount-testsuite to check constant inode for non samefs [2].
> 
> The problem now is that all the test pass, although, as I wrote, I only
> implemented partial support for non samefs. This means that the test coverage
> for constant st_ino/st_dev is insufficient to validate the results of your task.
> Which is good, because it will give you an opportunity to enhance the tests :)
> 
> It would be great if you could add validation to the fact that the
> overlay st_ino/
> st_dev pair is different from the underlying inode st_ino/st_dev.
> That is not the case with upstream overlayfs and that is not the case with
> my partial work on non same fs constant st_ino.
> 
> A new overlay xfstest, along the lines of overlay/017 that validates
> the uniqueness of st_ino/st_dev would also be a good idea.

Amir, Thanks for informing me. I will base my work on top of your patches.

> 
> Thanks,
> Amir.
> 
> [1] https://github.com/amir73il/linux/commits/ovl-constino
> [2] https://github.com/amir73il/unionmount-testsuite/commits/ovl-constino
> 
> 


-- 
chandan

Re: constant st_ino/st_dev for non-samefs case

[Amir Goldstein]

On Wed, May 24, 2017 at 3:59 PM, Amir Goldstein <amir73il@gmail.com> wrote:
> On Wed, May 24, 2017 at 11:51 AM, Chandan Rajendra
> <chandan@linux.vnet.ibm.com> wrote:
>>
>> Amir, Thanks for informing me. I will base my work on top of your patches.
>>
>
> Chandan,
>
> FYI, just found a nasty bug in one the patches under the "relax same fs" patches
> I referred you to, so if you already pulled my ovl-constino branch,
> please re-fetch it.
> New head is:
> 931488f1acd1 ovl: consistent st_ino/d_ino
>

FYI, fixed another minor issue. Had to relax WARN_ON_ONCE in readdir
for non-samefs.

Force pushed ovl-constino branch.

New head is:
02b67e4c4e73 ovl: consistent st_ino/d_ino

Cheers,
Amir.