[LSF/MM/BPF TOPIC]: Challenges and Ideas in Transitioning EXT* and other FS to iomap

[LSF/MM/BPF TOPIC]: Challenges and Ideas in Transitioning EXT* and other FS to iomap

[Ritesh Harjani (IBM)]

In continuation from last year's efforts on conversion of ext* filesystems to iomap,
I would like to propose an LSFMM session on the said topic. Last year's session
was mainly centered around documentation discussion around iomap (so that it can help others
converting their filesystems to iomap), and I think we now have a kernelnewbies page [1] 
which can provide good details on how one can start transitioning their filesystem to iomap
interface.

Note, ext2/ext4 filesystems direct-io path now utilizes iomap where ext2
DIO conversion happened last year during LSFMM [2] [3]. I have also submitted patches
for ext2 buffered-io path for regular files to move to iomap and thereby enabling
large folio support to it. Along similar lines there are also patches around EXT4
buffered-io conversion to iomap.

Some of the challenges
=======================
1. For EXT2 directory handling which uses page cache and buffer heads, moving that path to 
   iomap has challenges with writeback path since iomap also uses folio->private to keep some 
   of its internal state (iomap_folio_state).
2. One other thing which was pointed out by Matthew is the BH_Boundary handling currently missing
   in iomap. This can lead to non-optimized data I/O patterns causing performance penalty. 
3. Filesystems need a mechanism to validate cached logical->physical block translations 
   in iomap writeback code (can this be lifted to common code?)
4. Another missing piece from iomap is the metadata handling for filesystems. There is no
   interface which iomap provides that the FS can utilize to move away from buffer heads
   for its metadata operations. It can be argued that it is not the responsibility of iomap, however
   filesystems do need a mechanism for their metadata handling operations.

Proposal
=========
In this talk I would like to discuss about the efforts, challenges & the lessons learnt in doing the conversion of
ext2's DIO and buffered-io paths to iomap, which might help others in conversion of their filesystem.
I would also like to have a discussion on the current open challenges we have in converting ext2 (buffered-io path) 
and discuss on what ideas people have, which we can consider for transitioning ext* and other filesystems to iomap. 

PS: As we speak, I am in the process of rebasing ext2 bufferred-io path to latest upstream kernel. 
It's mostly done and I am also looking into some of the open problems listed by community. 


References
============
[1]: https://kernelnewbies.org/KernelProjects/iomap
[2]: https://lore.kernel.org/linux-ext4/cover.1682069716.git.ritesh.list@gmail.com/
[3]: https://lwn.net/Articles/935934/
[4]: https://lore.kernel.org/linux-ext4/cover.1700505907.git.ritesh.list@gmail.com/

Re: [LSF/MM/BPF TOPIC]: Challenges and Ideas in Transitioning EXT* and other FS to iomap

[Ritesh Harjani]

"Ritesh Harjani (IBM)" <ritesh.list@gmail.com> writes:

++ linux-ext4

> In continuation from last year's efforts on conversion of ext* filesystems to iomap,
> I would like to propose an LSFMM session on the said topic. Last year's session
> was mainly centered around documentation discussion around iomap (so that it can help others
> converting their filesystems to iomap), and I think we now have a kernelnewbies page [1] 
> which can provide good details on how one can start transitioning their filesystem to iomap
> interface.
>
> Note, ext2/ext4 filesystems direct-io path now utilizes iomap where ext2
> DIO conversion happened last year during LSFMM [2] [3]. I have also submitted patches
> for ext2 buffered-io path for regular files to move to iomap and thereby enabling
> large folio support to it. Along similar lines there are also patches around EXT4
> buffered-io conversion to iomap.
>
> Some of the challenges
> =======================
> 1. For EXT2 directory handling which uses page cache and buffer heads, moving that path to 
>    iomap has challenges with writeback path since iomap also uses folio->private to keep some 
>    of its internal state (iomap_folio_state).
> 2. One other thing which was pointed out by Matthew is the BH_Boundary handling currently missing
>    in iomap. This can lead to non-optimized data I/O patterns causing performance penalty. 
> 3. Filesystems need a mechanism to validate cached logical->physical block translations 
>    in iomap writeback code (can this be lifted to common code?)
> 4. Another missing piece from iomap is the metadata handling for filesystems. There is no
>    interface which iomap provides that the FS can utilize to move away from buffer heads
>    for its metadata operations. It can be argued that it is not the responsibility of iomap, however
>    filesystems do need a mechanism for their metadata handling operations.
>
> Proposal
> =========
> In this talk I would like to discuss about the efforts, challenges & the lessons learnt in doing the conversion of
> ext2's DIO and buffered-io paths to iomap, which might help others in conversion of their filesystem.
> I would also like to have a discussion on the current open challenges we have in converting ext2 (buffered-io path) 
> and discuss on what ideas people have, which we can consider for transitioning ext* and other filesystems to iomap. 
>
> PS: As we speak, I am in the process of rebasing ext2 bufferred-io path to latest upstream kernel. 
> It's mostly done and I am also looking into some of the open problems listed by community. 
>
>
> References
> ============
> [1]: https://kernelnewbies.org/KernelProjects/iomap
> [2]: https://lore.kernel.org/linux-ext4/cover.1682069716.git.ritesh.list@gmail.com/
> [3]: https://lwn.net/Articles/935934/
> [4]: https://lore.kernel.org/linux-ext4/cover.1700505907.git.ritesh.list@gmail.com/

Re: [LSF/MM/BPF TOPIC]: Challenges and Ideas in Transitioning EXT* and other FS to iomap

[Ritesh Harjani]

Ritesh Harjani (IBM) <ritesh.list@gmail.com> writes:

> "Ritesh Harjani (IBM)" <ritesh.list@gmail.com> writes:
>
> ++ linux-ext4
>
>> In continuation from last year's efforts on conversion of ext* filesystems to iomap,
>> I would like to propose an LSFMM session on the said topic. Last year's session
>> was mainly centered around documentation discussion around iomap (so that it can help others
>> converting their filesystems to iomap), and I think we now have a kernelnewbies page [1] 
>> which can provide good details on how one can start transitioning their filesystem to iomap
>> interface.
>>
>> Note, ext2/ext4 filesystems direct-io path now utilizes iomap where ext2
>> DIO conversion happened last year during LSFMM [2] [3]. I have also submitted patches
>> for ext2 buffered-io path for regular files to move to iomap and thereby enabling
>> large folio support to it. Along similar lines there are also patches around EXT4
>> buffered-io conversion to iomap.
>>
>> Some of the challenges
>> =======================
>> 1. For EXT2 directory handling which uses page cache and buffer heads, moving that path to 
>>    iomap has challenges with writeback path since iomap also uses folio->private to keep some 
>>    of its internal state (iomap_folio_state).
>> 2. One other thing which was pointed out by Matthew is the BH_Boundary handling currently missing
>>    in iomap. This can lead to non-optimized data I/O patterns causing performance penalty. 
>> 3. Filesystems need a mechanism to validate cached logical->physical block translations 
>>    in iomap writeback code (can this be lifted to common code?)
>> 4. Another missing piece from iomap is the metadata handling for filesystems. There is no
>>    interface which iomap provides that the FS can utilize to move away from buffer heads
>>    for its metadata operations. It can be argued that it is not the responsibility of iomap, however
>>    filesystems do need a mechanism for their metadata handling operations.
>>
>> Proposal
>> =========
>> In this talk I would like to discuss about the efforts, challenges & the lessons learnt in doing the conversion of
>> ext2's DIO and buffered-io paths to iomap, which might help others in conversion of their filesystem.
>> I would also like to have a discussion on the current open challenges we have in converting ext2 (buffered-io path) 
>> and discuss on what ideas people have, which we can consider for transitioning ext* and other filesystems to iomap. 
>>
>> PS: As we speak, I am in the process of rebasing ext2 bufferred-io path to latest upstream kernel. 
>> It's mostly done and I am also looking into some of the open problems listed by community. 


I have rebased the RFC over the latest upstream and implemented the seq
counter approach that was roughly discussed in the RFC patch here [1].
Please find the latest tree at my github [2], in case if anyone is
interested in checking it out.

Currently I am running some tests on this tree as we speak. Post that
will look into BH_Boundary problem.

[1]: https://lore.kernel.org/all/8734wnj53k.fsf@doe.com/
[2]: https://github.com/riteshharjani/linux/commits/ext2-iomap-lsfmm-rfcv2/

-ritesh

>>
>>
>> References
>> ============
>> [1]: https://kernelnewbies.org/KernelProjects/iomap
>> [2]: https://lore.kernel.org/linux-ext4/cover.1682069716.git.ritesh.list@gmail.com/
>> [3]: https://lwn.net/Articles/935934/
>> [4]: https://lore.kernel.org/linux-ext4/cover.1700505907.git.ritesh.list@gmail.com/