Re: Reiser4 Upstream Git Repositories on GitHub

[Edward Shishkin <edward.shishkin@gmail.com>]

On 09/28/2016 11:50 PM, Ivan Shapovalov wrote:
> On 2016-09-28 at 21:58 +0200, Edward Shishkin wrote:
>> On 09/28/2016 05:03 PM, Ivan Shapovalov wrote:
>>> On 2016-09-28 at 16:44 +0200, Edward Shishkin wrote:
>>>> On 09/28/2016 03:56 PM, Edward Shishkin wrote:
>>>>> On 09/28/2016 12:36 PM, Ivan Shapovalov wrote:
>>>>>> On 2016-09-28 at 12:17 +0200, Edward Shishkin wrote:
>>>>>>> On 09/27/2016 11:51 PM, Ivan Shapovalov wrote:
>>>>>>>> On 2016-09-27 at 23:47 +0200, Edward Shishkin wrote:
>>>>>>>>> On 09/27/2016 08:36 PM, Ivan Shapovalov wrote:
>>>>>>>>>> On 2016-09-27 at 16:13 +0200, Edward Shishkin wrote:
>>>>>>>>>>> On 09/27/2016 04:43 AM, Ivan Shapovalov wrote:
>>>>>>>>>>>> On 2016-09-27 at 00:37 +0200, Edward Shishkin
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> On 09/27/2016 12:05 AM, Ivan Shapovalov wrote:
>>>>>>>>>>>>>> On 2016-09-24 at 22:16 +0200, Edward Shishkin
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>> Hello everyone,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I have set up the updated Namesys
>>>>>>>>>>>>>>> repositories
>>>>>>>>>>>>>>> at my
>>>>>>>>>>>>>>> Github
>>>>>>>>>>>>>>> page.
>>>>>>>>>>>>>>> Those repositories are supposed to contain
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> latest
>>>>>>>>>>>>>>> updates
>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>> the (stable) master branch and in other
>>>>>>>>>>>>>>> (experimental)
>>>>>>>>>>>>>>> branches
>>>>>>>>>>>>>>> that
>>>>>>>>>>>>>>> I'll announce.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 1) https://github.com/edward6/reiser4
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> This is a "standalone" reiser4 tree, which
>>>>>>>>>>>>>>> doesn't
>>>>>>>>>>>>>>> include
>>>>>>>>>>>>>>> specific
>>>>>>>>>>>>>>> changes of Linux kernel needed for reiser4
>>>>>>>>>>>>>>> port. Such
>>>>>>>>>>>>>>> changes
>>>>>>>>>>>>>>> can
>>>>>>>>>>>>>>> be
>>>>>>>>>>>>>>> found at the project's page on Sourceforge:
>>>>>>>>>>>>>>> https://sourceforge.net/projects/reiser4/
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> An example of work with the standalone
>>>>>>>>>>>>>>> reiser4
>>>>>>>>>>>>>>> tree:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> . Patch the respective kernel with the
>>>>>>>>>>>>>>> latest
>>>>>>>>>>>>>>> available
>>>>>>>>>>>>>>> stuff
>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>           Sourceforge;
>>>>>>>>>>>>>>> . cd to the "fs" directory;
>>>>>>>>>>>>>>> . delete the directory reiser4;
>>>>>>>>>>>>>>> . instead of the deleted stuff clone the
>>>>>>>>>>>>>>> standalone
>>>>>>>>>>>>>>> reiser4
>>>>>>>>>>>>>>>           repository from Github;
>>>>>>>>>>>>>>> . build and install as usual.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2) Libaal and Reiser4progs:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> https://github.com/edward6/libaal
>>>>>>>>>>>>>>> https://github.com/edward6/reiser4progs
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Before building Libaal and Reiser4progs
>>>>>>>>>>>>>>> execute
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>> ./prepare
>>>>>>>>>>>>>>> script,
>>>>>>>>>>>>>>> which will create files needed for build
>>>>>>>>>>>>>>> process.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Edward.
>>>>>>>>>>>>>> Wow, finally.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Maybe we could avoid that "all changes for 10
>>>>>>>>>>>>>> years"
>>>>>>>>>>>>>> commit?
>>>>>>>>>>>>> Hi Ivan,
>>>>>>>>>>>>> Sorry, don't have a time to granulate it.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I tried to keep track of all patches since
>>>>>>>>>>>>>> 3.2...
>>>>>>>>>>>>> There will be "all changes for 6 years" commit.
>>>>>>>>>>>>> Is it much better?
>>>>>>>>>>>> So well, I finished splitting off all known diffs
>>>>>>>>>>>> from that
>>>>>>>>>>>> big
>>>>>>>>>>>> commit.
>>>>>>>>>>>> Tt was 12k(+)/8k(-), now it is 7k(+)/7k(-).
>>>>>>>>>>>>
>>>>>>>>>>>> The updated branch is here: https://github.com/in
>>>>>>>>>>>> telf
>>>>>>>>>>>> x/reis
>>>>>>>>>>>> er4
>>>>>>>>>>>> (unfortunately, not fast-forward).
>>>>>>>>>>>>
>>>>>>>>>>>> Moreover, my tree has accumulated quite a few
>>>>>>>>>>>> differences
>>>>>>>>>>>> from
>>>>>>>>>>>> your
>>>>>>>>>>>> one. I've dropped trivial discrepancies
>>>>>>>>>>>> (comments,
>>>>>>>>>>>> formatting
>>>>>>>>>>>> etc.)
>>>>>>>>>>>> and put the larger ones in separate branches:
>>>>>>>>>>>>
>>>>>>>>>>>> 1. https://github.com/intelfx/reiser4/tree/differ
>>>>>>>>>>>> ence
>>>>>>>>>>>> s/enot
>>>>>>>>>>>> ty
>>>>>>>>>>>>          (unsupported ioctls return -ENOTTY, not
>>>>>>>>>>>> -ENOSYS)
>>>>>>>>>>>>
>>>>>>>>>>>> 2. https://github.com/intelfx/reiser4/tree/differ
>>>>>>>>>>>> ence
>>>>>>>>>>>> s/migr
>>>>>>>>>>>> atep
>>>>>>>>>>>> age
>>>>>>>>>>>>          (the ->migratepage() implementation,
>>>>>>>>>>>> which I
>>>>>>>>>>>> still do
>>>>>>>>>>>> not
>>>>>>>>>>>> completely
>>>>>>>>>>>>           understand, but it works)
>>>>>>>>>>>>
>>>>>>>>>>>> 3. https://github.com/intelfx/reiser4/tree/differ
>>>>>>>>>>>> ence
>>>>>>>>>>>> s/rena
>>>>>>>>>>>> meat
>>>>>>>>>>>> 2
>>>>>>>>>>>>          (renameat2(RENAME_NOREPLACE)
>>>>>>>>>>>> implementation,
>>>>>>>>>>>> which
>>>>>>>>>>>> you
>>>>>>>>>>>> haven't
>>>>>>>>>>>>           merged somewhy)
>>>>>>>>>>>>
>>>>>>>>>>>> 4. https://github.com/intelfx/reiser4/tree/differ
>>>>>>>>>>>> ence
>>>>>>>>>>>> s/adju
>>>>>>>>>>>> st-t
>>>>>>>>>>>> o-3.
>>>>>>>>>>>> 15
>>>>>>>>>>>>          (part of porting to 3.15 which, again,
>>>>>>>>>>>> you
>>>>>>>>>>>> haven't
>>>>>>>>>>>> merged
>>>>>>>>>>>> somewhy)
>>>>>>>>>>>>
>>>>>>>>>>>> These branches are on top of that granular
>>>>>>>>>>>> "master".
>>>>>>>>>>>> Anyway, please take a look.
>>>>>>>>>>> It was definitely useful work,
>>>>>>>>>>> I'll look at those differences..
>>>>>>>>>> Maybe you could also consider rebasing things on top
>>>>>>>>>> of
>>>>>>>>>> that
>>>>>>>>>> extracted
>>>>>>>>>> granular history?
>>>>>>>>>>
>>>>>>>>> Interesting idea, but I am not able to estimate
>>>>>>>>> complexity of such rebasing for now.
>>>>>>>>>
>>>>>>>> While we do not have any forks and can afford non-fast-
>>>>>>>> forward
>>>>>>>> updates
>>>>>>>> of master, the complexity is almost nil.
>>>>>>>>
>>>>>>>> To rebase your format41 branch, one can do this:
>>>>>>>>
>>>>>>>> git rebase --preserve-merges --onto
>>>>>>>> 3c7b3c5802e20381496f641fe64b6c1573228c6e
>>>>>>>> 8a896fd48ed35c7dfa0188f9b7f4cbdfd469cacb format41
>>>>>>>>
>>>>>>>> where 8a896fd is merge-base of format41 and master (that
>>>>>>>> big
>>>>>>>> commit),
>>>>>>>> and 3c7b3c5 is the corresponding commit of the
>>>>>>>> synthesized
>>>>>>>> history.
>>>>>>>>
>>>>>>>> Both commits have identical file content (`git diff
>>>>>>>> 8a896fd
>>>>>>>> 3c7b3c5`
>>>>>>>> yields empty output).
>>>>>>> OK, everything went smoothly,
>>>>>>> Thanks for scrupulously archiving things!
>>>>>>>
>>>>>>> Edward.
>>>>>> Great. (In fact, I intended this to be `git push -f`-ed, not
>>>>>> `git
>>>>>> merge`-ed with original history, but well, git blame now does
>>>>>> the
>>>>>> right
>>>>>> thing, so it's good enough as it is.)
>>>>>>
>>>>>> We do not use github pull requests and still send formatted
>>>>>> patch
>>>>>> series to the ML, correct?
>>>>>>
>>>>> Yup, everything to the list, as usual..
>>>> BTW, your fstrim-scanner is the first candidate to scrub ;)
>>>> Actually, I think about a common multi-functional scanner, with 3
>>>> modes:
>>>> 1) discard only (handle only free blocks);
>>>> 2) scrub only (handle only busy blocks);
>>>> 3) combined (scan the whole partition; for free blocks call
>>>> discard,
>>>>        for busy ones call scrub).
>>>> Any ideas?
>>>>
>>>> Thanks,
>>>> Edward.
>>>> PS: We have an own ioctl number: 0xCD inherited from
>>>> ReiserFS(v3).
>>> I still have to finish the erase unit detection (which has
>>> completely
>>> stalled) to merge all this work. Moreover:
>>>
>>> For the fstrim, we have dropped all locking and serialization
>>> issues
>>> and declared that fstrim is best-effort: if it misses some blocks
>>> due
>>> to concurrent transactions allocating and freeing blocks, it
>>> doesn't
>>> matter.
>>>
>>> For the scrub, this won't fly...
>> Indeed, the requirements to fstrim and scrub are different,
>> but, as I remember, the last decision was to not miss:
>> http://marc.info/?l=reiserfs-devel&m=141391883022745&w=2
>> so everything will fly just perfectly..
>>
>> Edward.
> This is different thing, it's about grabbing space in bigger chunks...
> If a concurrent transaction allocates some space and frees some space,
> we don't care, because it will then be discarded "online".
>
> But in case of the scrub, how do we protect from the storage tree
> changing right beneath us?

Yup, it seems that the idea of common scanner is dead.
It should be an independent tool. I think, we need to simply scan the
storage tree, do whatever is needed for each node, and make it dirty.

Edward.