Re: [LSF/MM/BPF TOPIC]: Challenges and Ideas in Transitioning EXT* and other FS to iomap

Re: [LSF/MM/BPF TOPIC]: Challenges and Ideas in Transitioning EXT* and other FS to iomap

[Ritesh Harjani]

"Ritesh Harjani (IBM)" <ritesh.list@gmail.com> writes:

++ linux-ext4

> In continuation from last year's efforts on conversion of ext* filesystems to iomap,
> I would like to propose an LSFMM session on the said topic. Last year's session
> was mainly centered around documentation discussion around iomap (so that it can help others
> converting their filesystems to iomap), and I think we now have a kernelnewbies page [1] 
> which can provide good details on how one can start transitioning their filesystem to iomap
> interface.
>
> Note, ext2/ext4 filesystems direct-io path now utilizes iomap where ext2
> DIO conversion happened last year during LSFMM [2] [3]. I have also submitted patches
> for ext2 buffered-io path for regular files to move to iomap and thereby enabling
> large folio support to it. Along similar lines there are also patches around EXT4
> buffered-io conversion to iomap.
>
> Some of the challenges
> =======================
> 1. For EXT2 directory handling which uses page cache and buffer heads, moving that path to 
>    iomap has challenges with writeback path since iomap also uses folio->private to keep some 
>    of its internal state (iomap_folio_state).
> 2. One other thing which was pointed out by Matthew is the BH_Boundary handling currently missing
>    in iomap. This can lead to non-optimized data I/O patterns causing performance penalty. 
> 3. Filesystems need a mechanism to validate cached logical->physical block translations 
>    in iomap writeback code (can this be lifted to common code?)
> 4. Another missing piece from iomap is the metadata handling for filesystems. There is no
>    interface which iomap provides that the FS can utilize to move away from buffer heads
>    for its metadata operations. It can be argued that it is not the responsibility of iomap, however
>    filesystems do need a mechanism for their metadata handling operations.
>
> Proposal
> =========
> In this talk I would like to discuss about the efforts, challenges & the lessons learnt in doing the conversion of
> ext2's DIO and buffered-io paths to iomap, which might help others in conversion of their filesystem.
> I would also like to have a discussion on the current open challenges we have in converting ext2 (buffered-io path) 
> and discuss on what ideas people have, which we can consider for transitioning ext* and other filesystems to iomap. 
>
> PS: As we speak, I am in the process of rebasing ext2 bufferred-io path to latest upstream kernel. 
> It's mostly done and I am also looking into some of the open problems listed by community. 
>
>
> References
> ============
> [1]: https://kernelnewbies.org/KernelProjects/iomap
> [2]: https://lore.kernel.org/linux-ext4/cover.1682069716.git.ritesh.list@gmail.com/
> [3]: https://lwn.net/Articles/935934/
> [4]: https://lore.kernel.org/linux-ext4/cover.1700505907.git.ritesh.list@gmail.com/

Re: [LSF/MM/BPF TOPIC]: Challenges and Ideas in Transitioning EXT* and other FS to iomap

[Ritesh Harjani]

Ritesh Harjani (IBM) <ritesh.list@gmail.com> writes:

> "Ritesh Harjani (IBM)" <ritesh.list@gmail.com> writes:
>
> ++ linux-ext4
>
>> In continuation from last year's efforts on conversion of ext* filesystems to iomap,
>> I would like to propose an LSFMM session on the said topic. Last year's session
>> was mainly centered around documentation discussion around iomap (so that it can help others
>> converting their filesystems to iomap), and I think we now have a kernelnewbies page [1] 
>> which can provide good details on how one can start transitioning their filesystem to iomap
>> interface.
>>
>> Note, ext2/ext4 filesystems direct-io path now utilizes iomap where ext2
>> DIO conversion happened last year during LSFMM [2] [3]. I have also submitted patches
>> for ext2 buffered-io path for regular files to move to iomap and thereby enabling
>> large folio support to it. Along similar lines there are also patches around EXT4
>> buffered-io conversion to iomap.
>>
>> Some of the challenges
>> =======================
>> 1. For EXT2 directory handling which uses page cache and buffer heads, moving that path to 
>>    iomap has challenges with writeback path since iomap also uses folio->private to keep some 
>>    of its internal state (iomap_folio_state).
>> 2. One other thing which was pointed out by Matthew is the BH_Boundary handling currently missing
>>    in iomap. This can lead to non-optimized data I/O patterns causing performance penalty. 
>> 3. Filesystems need a mechanism to validate cached logical->physical block translations 
>>    in iomap writeback code (can this be lifted to common code?)
>> 4. Another missing piece from iomap is the metadata handling for filesystems. There is no
>>    interface which iomap provides that the FS can utilize to move away from buffer heads
>>    for its metadata operations. It can be argued that it is not the responsibility of iomap, however
>>    filesystems do need a mechanism for their metadata handling operations.
>>
>> Proposal
>> =========
>> In this talk I would like to discuss about the efforts, challenges & the lessons learnt in doing the conversion of
>> ext2's DIO and buffered-io paths to iomap, which might help others in conversion of their filesystem.
>> I would also like to have a discussion on the current open challenges we have in converting ext2 (buffered-io path) 
>> and discuss on what ideas people have, which we can consider for transitioning ext* and other filesystems to iomap. 
>>
>> PS: As we speak, I am in the process of rebasing ext2 bufferred-io path to latest upstream kernel. 
>> It's mostly done and I am also looking into some of the open problems listed by community. 


I have rebased the RFC over the latest upstream and implemented the seq
counter approach that was roughly discussed in the RFC patch here [1].
Please find the latest tree at my github [2], in case if anyone is
interested in checking it out.

Currently I am running some tests on this tree as we speak. Post that
will look into BH_Boundary problem.

[1]: https://lore.kernel.org/all/8734wnj53k.fsf@doe.com/
[2]: https://github.com/riteshharjani/linux/commits/ext2-iomap-lsfmm-rfcv2/

-ritesh

>>
>>
>> References
>> ============
>> [1]: https://kernelnewbies.org/KernelProjects/iomap
>> [2]: https://lore.kernel.org/linux-ext4/cover.1682069716.git.ritesh.list@gmail.com/
>> [3]: https://lwn.net/Articles/935934/
>> [4]: https://lore.kernel.org/linux-ext4/cover.1700505907.git.ritesh.list@gmail.com/