Re: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

Re: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Yangtao Li]

+cc rust-for-linux

在 2025/5/28 07:39, Viacheslav Dubeyko 写道:
> Hi Adrian, Yangtao,
> 
> One idea crossed my mind recently. And this is about re-writing HFS/HFS+ in
> Rust. It could be interesting direction but I am not sure how reasonable it
> could be. From one point of view, HFS/HFS+ are not critical subsystems and we
> can afford some experiments. From another point of view, we have enough issues
> in the HFS/HFS+ code and, maybe, re-working HFS/HFS+ can make the code more
> stable.
> 
> I don't think that it's a good idea to implement the complete re-writing of the
> whole driver at once. However, we need a some unification and generalization of
> HFS/HFS+ code patterns in the form of re-usable code by both drivers. This re-
> usable code can be represented as by C code as by Rust code. And we can
> introduce this generalized code in the form of C and Rust at the same time. So,
> we can re-write HFS/HFS+ code gradually step by step. My point here that we
> could have C code and Rust code for generalized functionality of HFS/HFS+ and
> Kconfig would define which code will be compiled and used, finally.
> 
> How do you feel about this? And can we afford such implementation efforts?

It must be a crazy idea! Honestly, I'm a fan of new things.
If there is a clear path, I don't mind moving in that direction.

It seems that downstream already has rust implementations of puzzle and 
ext2 file systems. If I understand correctly, there is currently a lack 
of support for vfs and various infrastructure.

I'm not an expert on Rust, so it would be great if some Rust people 
could share their opinions.

Thx,
Yangtao

RE: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Viacheslav Dubeyko]

On Wed, 2025-05-28 at 20:40 +0800, Yangtao Li wrote:
> +cc rust-for-linux
> 
> 在 2025/5/28 07:39, Viacheslav Dubeyko 写道:
> > Hi Adrian, Yangtao,
> > 
> > One idea crossed my mind recently. And this is about re-writing HFS/HFS+ in
> > Rust. It could be interesting direction but I am not sure how reasonable it
> > could be. From one point of view, HFS/HFS+ are not critical subsystems and we
> > can afford some experiments. From another point of view, we have enough issues
> > in the HFS/HFS+ code and, maybe, re-working HFS/HFS+ can make the code more
> > stable.
> > 
> > I don't think that it's a good idea to implement the complete re-writing of the
> > whole driver at once. However, we need a some unification and generalization of
> > HFS/HFS+ code patterns in the form of re-usable code by both drivers. This re-
> > usable code can be represented as by C code as by Rust code. And we can
> > introduce this generalized code in the form of C and Rust at the same time. So,
> > we can re-write HFS/HFS+ code gradually step by step. My point here that we
> > could have C code and Rust code for generalized functionality of HFS/HFS+ and
> > Kconfig would define which code will be compiled and used, finally.
> > 
> > How do you feel about this? And can we afford such implementation efforts?
> 
> It must be a crazy idea! Honestly, I'm a fan of new things.
> If there is a clear path, I don't mind moving in that direction.
> 

Why don't try even some crazy way. :)

> It seems that downstream already has rust implementations of puzzle and 
> ext2 file systems. If I understand correctly, there is currently a lack 
> of support for vfs and various infrastructure.
> 

Yes, Rust implementation in kernel is slightly complicated topic. And I don't
suggest to implement the whole HFS/HFS+ driver at once. My idea is to start from
introduction of small Rust module that can implement some subset of HFS/HFS+
functionality that can be called by C code. It could look like a library that
HFS/HFS+ drivers can re-use. And we can have C and Rust "library" and people can
select what they would like to compile (C or Rust implementation).

> I'm not an expert on Rust, so it would be great if some Rust people 
> could share their opinions.
> 
> 

I hope that Rust people would like the idea. :)

Thanks,
Slava.

Re: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Benno Lossin]

Andreas Hindborg will most likely reply with some more info in the near
future, but I'll drop some of my thoughts.

On Wed May 28, 2025 at 6:16 PM CEST, Viacheslav Dubeyko wrote:
> On Wed, 2025-05-28 at 20:40 +0800, Yangtao Li wrote:
>> +cc rust-for-linux
>> 
>> 在 2025/5/28 07:39, Viacheslav Dubeyko 写道:
>> > Hi Adrian, Yangtao,
>> > 
>> > One idea crossed my mind recently. And this is about re-writing HFS/HFS+ in
>> > Rust. It could be interesting direction but I am not sure how reasonable it
>> > could be. From one point of view, HFS/HFS+ are not critical subsystems and we
>> > can afford some experiments. From another point of view, we have enough issues
>> > in the HFS/HFS+ code and, maybe, re-working HFS/HFS+ can make the code more
>> > stable.
>> > 
>> > I don't think that it's a good idea to implement the complete re-writing of the
>> > whole driver at once. However, we need a some unification and generalization of
>> > HFS/HFS+ code patterns in the form of re-usable code by both drivers. This re-
>> > usable code can be represented as by C code as by Rust code. And we can
>> > introduce this generalized code in the form of C and Rust at the same time. So,
>> > we can re-write HFS/HFS+ code gradually step by step. My point here that we
>> > could have C code and Rust code for generalized functionality of HFS/HFS+ and
>> > Kconfig would define which code will be compiled and used, finally.
>> > 
>> > How do you feel about this? And can we afford such implementation efforts?
>> 
>> It must be a crazy idea! Honestly, I'm a fan of new things.
>> If there is a clear path, I don't mind moving in that direction.
>> 
>
> Why don't try even some crazy way. :)

There are different paths that can be taken. One of the easiest would be
to introduce a rust reference driver [1] for HFS. The default config
option would still be the C driver so it doesn't break users (& still
allows all supported architectures), but it allows you to experiment
using Rust. Eventually, you could remove the C driver when ggc_rs is
mature enough or only keep the C one around for the obscure
architectures.

If you don't want to break the duplicate drivers rule, then I can expand
a bit on the other options, but honestly, they aren't that great:

There are some subsystems that go for a library approach: extract some
self-contained piece of functionality and move it to Rust code and then
call that from C. I personally don't really like this approach, as it
makes it hard to separate the safety boundary, create proper
abstractions & write idiomatic Rust code.

[1]: https://rust-for-linux.com/rust-reference-drivers

>> It seems that downstream already has rust implementations of puzzle and 
>> ext2 file systems. If I understand correctly, there is currently a lack 
>> of support for vfs and various infrastructure.
>> 
>
> Yes, Rust implementation in kernel is slightly complicated topic. And I don't
> suggest to implement the whole HFS/HFS+ driver at once. My idea is to start from
> introduction of small Rust module that can implement some subset of HFS/HFS+
> functionality that can be called by C code. It could look like a library that
> HFS/HFS+ drivers can re-use. And we can have C and Rust "library" and people can
> select what they would like to compile (C or Rust implementation).

One good path forward using the reference driver would be to first
create a read-only version. That was the plan that Wedson followed with
ext2 (and IIRC also ext4? I might misremember). It apparently makes the
initial implementation easier (I have no experience with filesystems)
and thus works better as a PoC.

>> I'm not an expert on Rust, so it would be great if some Rust people 
>> could share their opinions.
>> 
>
> I hope that Rust people would like the idea. :)

I'm sure that several Rust folks would be interested in getting their
hands dirty helping with writing abstractions and/or the driver itself.

I personally am more on the Rust side of things, so I could help make
the abstractions feel idiomatic and ergonomic.

Feel free to ask any follow up questions. Hope this helps!

---
Cheers,
Benno

Re: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Miguel Ojeda]

On Thu, Jun 19, 2025 at 9:35 PM Benno Lossin <lossin@kernel.org> wrote:
>
> There are some subsystems that go for a library approach: extract some
> self-contained piece of functionality and move it to Rust code and then
> call that from C. I personally don't really like this approach, as it
> makes it hard to separate the safety boundary, create proper
> abstractions & write idiomatic Rust code.

Yeah, that approach works best when the interface surface is
small/simple enough relative to the functionality, e.g. the QR code
case we have in the kernel already.

So there are some use cases of the approach (codecs have also been
discussed as another one). But going, in the extreme, "function by
function" replacing C with Rust and having two-way calls everywhere
isn't good, and it isn't the goal.

Instead, we aim to write safe Rust APIs ("abstractions") and then
ideally having pure Rust modules that take advantage of those.
Sometimes you may want to keep certain pieces in C, but still use them
from the Rust module until you cover those too eventually.

> One good path forward using the reference driver would be to first
> create a read-only version. That was the plan that Wedson followed with
> ext2 (and IIRC also ext4? I might misremember). It apparently makes the
> initial implementation easier (I have no experience with filesystems)
> and thus works better as a PoC.

Yeah, my understanding is that a read-only version would be easier.
Performance is another axis too.

It would be nice to see eventually a 100% safe code Rust filesystem,
even if read-only and "slow". That could already have use cases.

(Wedson was planning read-writes ones too.)

(Thanks Benno!)

Cheers,
Miguel

RE: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Viacheslav Dubeyko]

On Thu, 2025-06-19 at 22:22 +0200, Miguel Ojeda wrote:
> On Thu, Jun 19, 2025 at 9:35 PM Benno Lossin <lossin@kernel.org> wrote:
> > 
> > There are some subsystems that go for a library approach: extract some
> > self-contained piece of functionality and move it to Rust code and then
> > call that from C. I personally don't really like this approach, as it
> > makes it hard to separate the safety boundary, create proper
> > abstractions & write idiomatic Rust code.
> 
> Yeah, that approach works best when the interface surface is
> small/simple enough relative to the functionality, e.g. the QR code
> case we have in the kernel already.
> 
> So there are some use cases of the approach (codecs have also been
> discussed as another one). But going, in the extreme, "function by
> function" replacing C with Rust and having two-way calls everywhere
> isn't good, and it isn't the goal.
> 
> Instead, we aim to write safe Rust APIs ("abstractions") and then
> ideally having pure Rust modules that take advantage of those.
> Sometimes you may want to keep certain pieces in C, but still use them
> from the Rust module until you cover those too eventually.
> 

Yeah, I completely agree. But I would like to implement the step-by-step
approach. At first, introduce a Rust "library" that will absorb a "duplicated"
functionality of HFS/HFS+. And C code of HFS/HFS+ will re-use the functionality
of Rust "library". Then, elaborate "abstractions" (probably, HFS/HFS+ metadata
primitives) and test it in Rust "library" by calling from C code. And, finally,
completely switch to Rust implementation.

> > One good path forward using the reference driver would be to first
> > create a read-only version. That was the plan that Wedson followed with
> > ext2 (and IIRC also ext4? I might misremember). It apparently makes the
> > initial implementation easier (I have no experience with filesystems)
> > and thus works better as a PoC.
> 
> Yeah, my understanding is that a read-only version would be easier.
> Performance is another axis too.
> 
> It would be nice to see eventually a 100% safe code Rust filesystem,
> even if read-only and "slow". That could already have use cases.
> 

Frankly speaking, I don't see how Read-Only version can be easier. :) Because,
even Read-Only version requires to operate by file system's metadata. And it's
around 80% - 90% of the whole file system driver functionality. From my point of
view, it is much easier to convert every metadata structure implementation step
by step into Rust.

Thanks,
Slava.

Re: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Benno Lossin]

On Thu Jun 19, 2025 at 11:48 PM CEST, Viacheslav Dubeyko wrote:
> On Thu, 2025-06-19 at 22:22 +0200, Miguel Ojeda wrote:
>> On Thu, Jun 19, 2025 at 9:35 PM Benno Lossin <lossin@kernel.org> wrote:
>> > 
>> > There are some subsystems that go for a library approach: extract some
>> > self-contained piece of functionality and move it to Rust code and then
>> > call that from C. I personally don't really like this approach, as it
>> > makes it hard to separate the safety boundary, create proper
>> > abstractions & write idiomatic Rust code.
>> 
>> Yeah, that approach works best when the interface surface is
>> small/simple enough relative to the functionality, e.g. the QR code
>> case we have in the kernel already.
>> 
>> So there are some use cases of the approach (codecs have also been
>> discussed as another one). But going, in the extreme, "function by
>> function" replacing C with Rust and having two-way calls everywhere
>> isn't good, and it isn't the goal.
>> 
>> Instead, we aim to write safe Rust APIs ("abstractions") and then
>> ideally having pure Rust modules that take advantage of those.
>> Sometimes you may want to keep certain pieces in C, but still use them
>> from the Rust module until you cover those too eventually.
>> 
>
> Yeah, I completely agree. But I would like to implement the step-by-step
> approach. At first, introduce a Rust "library" that will absorb a "duplicated"
> functionality of HFS/HFS+. And C code of HFS/HFS+ will re-use the functionality
> of Rust "library". Then, elaborate "abstractions" (probably, HFS/HFS+ metadata
> primitives) and test it in Rust "library" by calling from C code. And, finally,
> completely switch to Rust implementation.

So the difficult part is the back and forth between Rust and C. You'll
have to write lots of glue code to make it work and that's why we don't
recommend it.

You can of course still build the driver step-by-step (and I'd recommend
doing that). For example the Nova driver is being built this way.

---
Cheers,
Benno

Re: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Miguel Ojeda]

On Thu, Jun 19, 2025 at 11:49 PM Viacheslav Dubeyko
<Slava.Dubeyko@ibm.com> wrote:
>
> But I would like to implement the step-by-step approach.

Like Benno mentions, it is hard to say how it will look with your
description, i.e. how you plan to "cut" things.

On one hand, it sounds like you don't plan to write VFS abstractions
-- did you check Wedson's work?

    https://lore.kernel.org/rust-for-linux/20240514131711.379322-1-wedsonaf@gmail.com/

i.e. it sounds like you want to replace parts of e.g. HFS with Rust
code while still going through C interfaces at some places inside HFS,
and that has downsides.

On the other hand, you mention "abstractions" around VFS concepts too,
so you may have something else in mind.

> Frankly speaking, I don't see how Read-Only version can be easier. :) Because,
> even Read-Only version requires to operate by file system's metadata. And it's
> around 80% - 90% of the whole file system driver functionality. From my point of
> view, it is much easier to convert every metadata structure implementation step
> by step into Rust.

Well, apart from having to write more operations/code, as soon as
there may be writers, you have to take care of that possibility in
everything you do, no?

Worst of all, I imagine you have to test (and generally treat the
project) way more carefully, because now your users could lose real
data.

Cheers,
Miguel

RE: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Viacheslav Dubeyko]

On Fri, 2025-06-20 at 10:17 +0200, Miguel Ojeda wrote:
> On Thu, Jun 19, 2025 at 11:49 PM Viacheslav Dubeyko
> <Slava.Dubeyko@ibm.com> wrote:
> > 
> > But I would like to implement the step-by-step approach.
> 
> Like Benno mentions, it is hard to say how it will look with your
> description, i.e. how you plan to "cut" things.
> 
> On one hand, it sounds like you don't plan to write VFS abstractions
> -- did you check Wedson's work?
> 

I don't have plans to re-write the whole Linux kernel. :) Because, any
particular file system driver needs to interact with VFS, memory subsystem and
block layer. So, pure Rust implementation means that everything should be re-
written in Rust. But it's mission impossible any time soon. :)

>     https://lore.kernel.org/rust-for-linux/20240514131711.379322-1-wedsonaf@gmail.com/  
> 
> i.e. it sounds like you want to replace parts of e.g. HFS with Rust
> code while still going through C interfaces at some places inside HFS,
> and that has downsides.

> On the other hand, you mention "abstractions" around VFS concepts too,
> so you may have something else in mind.
> 

I am considering of re-writing HFS/HFS+ in Rust but still surviving in the C
implemented environment. I don't think that even VFS will be completely re-
written in Rust and adopted any time soon. So, I am talking only about HFS/HFS+
"abstractions" now.

> > Frankly speaking, I don't see how Read-Only version can be easier. :) Because,
> > even Read-Only version requires to operate by file system's metadata. And it's
> > around 80% - 90% of the whole file system driver functionality. From my point of
> > view, it is much easier to convert every metadata structure implementation step
> > by step into Rust.
> 
> Well, apart from having to write more operations/code, as soon as
> there may be writers, you have to take care of that possibility in
> everything you do, no?
> 
> Worst of all, I imagine you have to test (and generally treat the
> project) way more carefully, because now your users could lose real
> data.
> 
> 

Even if you have bugs in read path only, then end user will not have access to
data. As a result, end-user will think that data is lost anyway because the data
cannot be accessed.

Any modification or bug fix in file system driver could result in real data loss
even for C implementation. So, of course, we need to be really careful with re-
writing file system driver in Rust. But if we have valid implementation in C
then we need to follow the functional logic and make the Rust implementation
consistent with logic in C. Also, we have xfstests that can help to check that
functionality works correctly. Of course, xfstests cannot guarantee 100%
correctness of file system operations.

So, frankly speaking, I don't see big difference between Read-Only or Read-Write
functionality. Because, both functionalities should be correct and could result
in inconsistent state of file system's in-core metadata or what end-user can
access or see, finally.

Also, anyway, C implementation and Rust implementation needs to co-exist, at
minimum, for some time, I assume.

Thanks,
Slava.

Re: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Miguel Ojeda]

On Fri, Jun 20, 2025 at 8:10 PM Viacheslav Dubeyko
<Slava.Dubeyko@ibm.com> wrote:
>
> I don't have plans to re-write the whole Linux kernel. :) Because, any
> particular file system driver needs to interact with VFS, memory subsystem and
> block layer. So, pure Rust implementation means that everything should be re-
> written in Rust. But it's mission impossible any time soon. :)

There is no need to do everything at once, but avoiding to write/use
safe abstractions completely for common facilities is not the goal.

But, to be clear, we are not suggesting rewriting VFS or others in Rust.

As for "mission impossible", a couple filesystems were prototyped on
top of the (read-only) VFS abstractions I linked. I am not saying the
abstractions are complete or that it will be easy or that HFS (not
even HFS+) is comparable to those or anything (I don't know), but you
could check them out  / try them to get a feeling for the approach.

Others may be able to tell you more about their experience with those.

> I am considering of re-writing HFS/HFS+ in Rust but still surviving in the C
> implemented environment. I don't think that even VFS will be completely re-
> written in Rust and adopted any time soon. So, I am talking only about HFS/HFS+
> "abstractions" now.

Again, nobody suggested rewriting VFS etc.. It is about abstracting
(using) existing C infrastructure.

Please look at the patches I linked above.

> Even if you have bugs in read path only, then end user will not have access to
> data. As a result, end-user will think that data is lost anyway because the data
> cannot be accessed.

That is not what I meant. If you have bugs in your read path (or,
generally, in the kernel), then you will have a big problem in both
scenarios. That is not a difference.

And, yes, you can have systems that end up losing data when fed with
untrusted/garbage data, or a user could discard their original drive
after thinking a copy was correct, and things like that.

But when you also promise to do writes properly, that introduces
complexity and new use cases -- you will have users saving new data
and expecting that to be kept no matter what, you could corrupt
existing unrelated data, and so on.

> Also, anyway, C implementation and Rust implementation needs to co-exist, at
> minimum, for some time, I assume.

That depends on you as a maintainer, as well as what architectures
your current users use and so on.

But there are cases where Rust-only functionality is fine (which is
easier to justify too).

Thanks!

Cheers,
Miguel

RE: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Viacheslav Dubeyko]

On Thu, 2025-06-19 at 21:33 +0200, Benno Lossin wrote:
> Andreas Hindborg will most likely reply with some more info in the near
> future, but I'll drop some of my thoughts.
> 
> On Wed May 28, 2025 at 6:16 PM CEST, Viacheslav Dubeyko wrote:
> > On Wed, 2025-05-28 at 20:40 +0800, Yangtao Li wrote:
> > > +cc rust-for-linux
> > > 
> > > 在 2025/5/28 07:39, Viacheslav Dubeyko 写道:
> > > > Hi Adrian, Yangtao,
> > > > 
> > > > One idea crossed my mind recently. And this is about re-writing HFS/HFS+ in
> > > > Rust. It could be interesting direction but I am not sure how reasonable it
> > > > could be. From one point of view, HFS/HFS+ are not critical subsystems and we
> > > > can afford some experiments. From another point of view, we have enough issues
> > > > in the HFS/HFS+ code and, maybe, re-working HFS/HFS+ can make the code more
> > > > stable.
> > > > 
> > > > I don't think that it's a good idea to implement the complete re-writing of the
> > > > whole driver at once. However, we need a some unification and generalization of
> > > > HFS/HFS+ code patterns in the form of re-usable code by both drivers. This re-
> > > > usable code can be represented as by C code as by Rust code. And we can
> > > > introduce this generalized code in the form of C and Rust at the same time. So,
> > > > we can re-write HFS/HFS+ code gradually step by step. My point here that we
> > > > could have C code and Rust code for generalized functionality of HFS/HFS+ and
> > > > Kconfig would define which code will be compiled and used, finally.
> > > > 
> > > > How do you feel about this? And can we afford such implementation efforts?
> > > 
> > > It must be a crazy idea! Honestly, I'm a fan of new things.
> > > If there is a clear path, I don't mind moving in that direction.
> > > 
> > 
> > Why don't try even some crazy way. :)
> 
> There are different paths that can be taken. One of the easiest would be
> to introduce a rust reference driver [1] for HFS. The default config
> option would still be the C driver so it doesn't break users (& still
> allows all supported architectures), but it allows you to experiment
> using Rust. Eventually, you could remove the C driver when ggc_rs is
> mature enough or only keep the C one around for the obscure
> architectures.
> 

Yeah, makes sense to me. It's one of the possible way. And I would like to have
as C as Rust implementation of driver as the first step. But it's hard enough to
implement everything at once. So, I would like to follow the step by step
approach.

> If you don't want to break the duplicate drivers rule, then I can expand
> a bit on the other options, but honestly, they aren't that great:
> 
> There are some subsystems that go for a library approach: extract some
> self-contained piece of functionality and move it to Rust code and then
> call that from C. I personally don't really like this approach, as it
> makes it hard to separate the safety boundary, create proper
> abstractions & write idiomatic Rust code.
> 

This is what I am considering as the first step. As far as I can see, HFS and
HFS+ have "duplicated" functionality with some peculiarities on every side. So,
I am considering to have something like Rust "library" that can absorb this
"duplicated" fuctionality at first. As a result, HFS and HFS+ C code can re-use
the Rust "library" at first. Finally, the whole driver(s) could be converted
into the Rust implementation. 

> [1]: https://rust-for-linux.com/rust-reference-drivers  
> 

Thanks for sharing this.

> > > It seems that downstream already has rust implementations of puzzle and 
> > > ext2 file systems. If I understand correctly, there is currently a lack 
> > > of support for vfs and various infrastructure.
> > > 
> > 
> > Yes, Rust implementation in kernel is slightly complicated topic. And I don't
> > suggest to implement the whole HFS/HFS+ driver at once. My idea is to start from
> > introduction of small Rust module that can implement some subset of HFS/HFS+
> > functionality that can be called by C code. It could look like a library that
> > HFS/HFS+ drivers can re-use. And we can have C and Rust "library" and people can
> > select what they would like to compile (C or Rust implementation).
> 
> One good path forward using the reference driver would be to first
> create a read-only version. That was the plan that Wedson followed with
> ext2 (and IIRC also ext4? I might misremember). It apparently makes the
> initial implementation easier (I have no experience with filesystems)
> and thus works better as a PoC.
> 

I see your point but even Read-Only functionality is too much. :) Because, it
needs to implement 80% - 90% functionality of metadata management even for Read-
Only case. And I would like to make the whole thing done by small working steps.
This is why I would like: (1) start from Rust "library", (2) move metadata
management into Rust "library" gradually, (3) convert the whole driver into Rust
implementation.

> > > I'm not an expert on Rust, so it would be great if some Rust people 
> > > could share their opinions.
> > > 
> > 
> > I hope that Rust people would like the idea. :)
> 
> I'm sure that several Rust folks would be interested in getting their
> hands dirty helping with writing abstractions and/or the driver itself.
> 

Sounds great! :) I really need some help and advice.

> I personally am more on the Rust side of things, so I could help make
> the abstractions feel idiomatic and ergonomic.
> 
> Feel free to ask any follow up questions. Hope this helps!
> 

Sounds interesting! Let me prepare my questions. :) So, HFS/HFS+ have
superblock, bitmap, b-trees, extent records, catalog records. It sounds to me
like candidates for abstractions. Am I correct here? Are we understand
abstraction at the same way? :)

Thanks,
Slava.

Re: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Benno Lossin]

On Thu Jun 19, 2025 at 11:39 PM CEST, Viacheslav Dubeyko wrote:
> On Thu, 2025-06-19 at 21:33 +0200, Benno Lossin wrote:
>> Andreas Hindborg will most likely reply with some more info in the near
>> future, but I'll drop some of my thoughts.
>> 
>> On Wed May 28, 2025 at 6:16 PM CEST, Viacheslav Dubeyko wrote:
>> > On Wed, 2025-05-28 at 20:40 +0800, Yangtao Li wrote:
>> > > +cc rust-for-linux
>> > > 
>> > > 在 2025/5/28 07:39, Viacheslav Dubeyko 写道:
>> > > > Hi Adrian, Yangtao,
>> > > > 
>> > > > One idea crossed my mind recently. And this is about re-writing HFS/HFS+ in
>> > > > Rust. It could be interesting direction but I am not sure how reasonable it
>> > > > could be. From one point of view, HFS/HFS+ are not critical subsystems and we
>> > > > can afford some experiments. From another point of view, we have enough issues
>> > > > in the HFS/HFS+ code and, maybe, re-working HFS/HFS+ can make the code more
>> > > > stable.
>> > > > 
>> > > > I don't think that it's a good idea to implement the complete re-writing of the
>> > > > whole driver at once. However, we need a some unification and generalization of
>> > > > HFS/HFS+ code patterns in the form of re-usable code by both drivers. This re-
>> > > > usable code can be represented as by C code as by Rust code. And we can
>> > > > introduce this generalized code in the form of C and Rust at the same time. So,
>> > > > we can re-write HFS/HFS+ code gradually step by step. My point here that we
>> > > > could have C code and Rust code for generalized functionality of HFS/HFS+ and
>> > > > Kconfig would define which code will be compiled and used, finally.
>> > > > 
>> > > > How do you feel about this? And can we afford such implementation efforts?
>> > > 
>> > > It must be a crazy idea! Honestly, I'm a fan of new things.
>> > > If there is a clear path, I don't mind moving in that direction.
>> > > 
>> > 
>> > Why don't try even some crazy way. :)
>> 
>> There are different paths that can be taken. One of the easiest would be
>> to introduce a rust reference driver [1] for HFS. The default config
>> option would still be the C driver so it doesn't break users (& still
>> allows all supported architectures), but it allows you to experiment
>> using Rust. Eventually, you could remove the C driver when ggc_rs is
>> mature enough or only keep the C one around for the obscure
>> architectures.
>> 
>
> Yeah, makes sense to me. It's one of the possible way. And I would like to have
> as C as Rust implementation of driver as the first step. But it's hard enough to
> implement everything at once. So, I would like to follow the step by step
> approach.
>
>> If you don't want to break the duplicate drivers rule, then I can expand
>> a bit on the other options, but honestly, they aren't that great:
>> 
>> There are some subsystems that go for a library approach: extract some
>> self-contained piece of functionality and move it to Rust code and then
>> call that from C. I personally don't really like this approach, as it
>> makes it hard to separate the safety boundary, create proper
>> abstractions & write idiomatic Rust code.
>> 
>
> This is what I am considering as the first step. As far as I can see, HFS and
> HFS+ have "duplicated" functionality with some peculiarities on every side. So,
> I am considering to have something like Rust "library" that can absorb this
> "duplicated" fuctionality at first. As a result, HFS and HFS+ C code can re-use
> the Rust "library" at first. Finally, the whole driver(s) could be converted
> into the Rust implementation. 

I'd of course have to see the concrete code, but this sounds a lot like
calling back and forth between C and Rust. Which will most likely be
painful. But it did work for the QR code generator, so we'll see.

>> [1]: https://rust-for-linux.com/rust-reference-drivers  
>> 
>
> Thanks for sharing this.
>
>> > > It seems that downstream already has rust implementations of puzzle and 
>> > > ext2 file systems. If I understand correctly, there is currently a lack 
>> > > of support for vfs and various infrastructure.
>> > > 
>> > 
>> > Yes, Rust implementation in kernel is slightly complicated topic. And I don't
>> > suggest to implement the whole HFS/HFS+ driver at once. My idea is to start from
>> > introduction of small Rust module that can implement some subset of HFS/HFS+
>> > functionality that can be called by C code. It could look like a library that
>> > HFS/HFS+ drivers can re-use. And we can have C and Rust "library" and people can
>> > select what they would like to compile (C or Rust implementation).
>> 
>> One good path forward using the reference driver would be to first
>> create a read-only version. That was the plan that Wedson followed with
>> ext2 (and IIRC also ext4? I might misremember). It apparently makes the
>> initial implementation easier (I have no experience with filesystems)
>> and thus works better as a PoC.
>> 
>
> I see your point but even Read-Only functionality is too much. :) Because, it
> needs to implement 80% - 90% functionality of metadata management even for Read-
> Only case. And I would like to make the whole thing done by small working steps.
> This is why I would like: (1) start from Rust "library", (2) move metadata
> management into Rust "library" gradually, (3) convert the whole driver into Rust
> implementation.

I personally don't know how this argument works, I only cited it in case
it is useful to people with domain knowledge :)

>> > > I'm not an expert on Rust, so it would be great if some Rust people 
>> > > could share their opinions.
>> > > 
>> > 
>> > I hope that Rust people would like the idea. :)
>> 
>> I'm sure that several Rust folks would be interested in getting their
>> hands dirty helping with writing abstractions and/or the driver itself.
>> 
>
> Sounds great! :) I really need some help and advice.
>
>> I personally am more on the Rust side of things, so I could help make
>> the abstractions feel idiomatic and ergonomic.
>> 
>> Feel free to ask any follow up questions. Hope this helps!
>> 
>
> Sounds interesting! Let me prepare my questions. :) So, HFS/HFS+ have
> superblock, bitmap, b-trees, extent records, catalog records. It sounds to me
> like candidates for abstractions. Am I correct here? Are we understand
> abstraction at the same way? :)

Yes! Everything that is used by other drivers/subsystems are usual
candidates for abstractions.

Essentially an abstraction is a rustified version of the C API. For
example, `Mutex<T>` is generic over the contained value, uses guards and
only allows access to the inner value if the mutex is locked.

Abstractions can take a pretty different form from the C API when it's
possible to make certain undesired uses of the API impossible through
Rust's type system or other features (in the case of the mutex, making
it impossible to access a value without locking it). Though they can
also be pretty simple if the C API is straightforward (this of course
depends on the concrete API).

Their purpose is to encapsulate the C API and expose its functionality
to safe Rust (note that calling any C function is considered `unsafe` in
Rust).

Calling C functions directly from Rust driver code (ie without going
through an abstraction) is not something that we want to allow (of
course there might be some exceptional cases where it is needed
temporarily). And thus everything that you use should have an
abstraction (this might include driver-specific abstractions that
effectively are also part of the driver, but encapsulate the C parts,
when you have converted the driver fully to rust, you probably won't
need any of them).

---
Cheers,
Benno

RE: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Viacheslav Dubeyko]

On Fri, 2025-06-20 at 00:24 +0200, Benno Lossin wrote:
> On Thu Jun 19, 2025 at 11:39 PM CEST, Viacheslav Dubeyko wrote:
> > On Thu, 2025-06-19 at 21:33 +0200, Benno Lossin wrote:
> > > Andreas Hindborg will most likely reply with some more info in the near
> > > future, but I'll drop some of my thoughts.
> > > 
> > > On Wed May 28, 2025 at 6:16 PM CEST, Viacheslav Dubeyko wrote:
> > > > On Wed, 2025-05-28 at 20:40 +0800, Yangtao Li wrote:
> > > > > +cc rust-for-linux
> > > > > 
> > > > > 在 2025/5/28 07:39, Viacheslav Dubeyko 写道:
> > > > > > Hi Adrian, Yangtao,
> > > > > > 
> > > > > > One idea crossed my mind recently. And this is about re-writing HFS/HFS+ in
> > > > > > Rust. It could be interesting direction but I am not sure how reasonable it
> > > > > > could be. From one point of view, HFS/HFS+ are not critical subsystems and we
> > > > > > can afford some experiments. From another point of view, we have enough issues
> > > > > > in the HFS/HFS+ code and, maybe, re-working HFS/HFS+ can make the code more
> > > > > > stable.
> > > > > > 
> > > > > > I don't think that it's a good idea to implement the complete re-writing of the
> > > > > > whole driver at once. However, we need a some unification and generalization of
> > > > > > HFS/HFS+ code patterns in the form of re-usable code by both drivers. This re-
> > > > > > usable code can be represented as by C code as by Rust code. And we can
> > > > > > introduce this generalized code in the form of C and Rust at the same time. So,
> > > > > > we can re-write HFS/HFS+ code gradually step by step. My point here that we
> > > > > > could have C code and Rust code for generalized functionality of HFS/HFS+ and
> > > > > > Kconfig would define which code will be compiled and used, finally.
> > > > > > 
> > > > > > How do you feel about this? And can we afford such implementation efforts?
> > > > > 
> > > > > It must be a crazy idea! Honestly, I'm a fan of new things.
> > > > > If there is a clear path, I don't mind moving in that direction.
> > > > > 
> > > > 
> > > > Why don't try even some crazy way. :)
> > > 
> > > There are different paths that can be taken. One of the easiest would be
> > > to introduce a rust reference driver [1] for HFS. The default config
> > > option would still be the C driver so it doesn't break users (& still
> > > allows all supported architectures), but it allows you to experiment
> > > using Rust. Eventually, you could remove the C driver when ggc_rs is
> > > mature enough or only keep the C one around for the obscure
> > > architectures.
> > > 
> > 
> > Yeah, makes sense to me. It's one of the possible way. And I would like to have
> > as C as Rust implementation of driver as the first step. But it's hard enough to
> > implement everything at once. So, I would like to follow the step by step
> > approach.
> > 
> > > If you don't want to break the duplicate drivers rule, then I can expand
> > > a bit on the other options, but honestly, they aren't that great:
> > > 
> > > There are some subsystems that go for a library approach: extract some
> > > self-contained piece of functionality and move it to Rust code and then
> > > call that from C. I personally don't really like this approach, as it
> > > makes it hard to separate the safety boundary, create proper
> > > abstractions & write idiomatic Rust code.
> > > 
> > 
> > This is what I am considering as the first step. As far as I can see, HFS and
> > HFS+ have "duplicated" functionality with some peculiarities on every side. So,
> > I am considering to have something like Rust "library" that can absorb this
> > "duplicated" fuctionality at first. As a result, HFS and HFS+ C code can re-use
> > the Rust "library" at first. Finally, the whole driver(s) could be converted
> > into the Rust implementation. 
> 
> I'd of course have to see the concrete code, but this sounds a lot like
> calling back and forth between C and Rust. Which will most likely be
> painful. But it did work for the QR code generator, so we'll see.
> 
> > > [1]: https://rust-for-linux.com/rust-reference-drivers    
> > > 
> > 
> > Thanks for sharing this.
> > 
> > > > > It seems that downstream already has rust implementations of puzzle and 
> > > > > ext2 file systems. If I understand correctly, there is currently a lack 
> > > > > of support for vfs and various infrastructure.
> > > > > 
> > > > 
> > > > Yes, Rust implementation in kernel is slightly complicated topic. And I don't
> > > > suggest to implement the whole HFS/HFS+ driver at once. My idea is to start from
> > > > introduction of small Rust module that can implement some subset of HFS/HFS+
> > > > functionality that can be called by C code. It could look like a library that
> > > > HFS/HFS+ drivers can re-use. And we can have C and Rust "library" and people can
> > > > select what they would like to compile (C or Rust implementation).
> > > 
> > > One good path forward using the reference driver would be to first
> > > create a read-only version. That was the plan that Wedson followed with
> > > ext2 (and IIRC also ext4? I might misremember). It apparently makes the
> > > initial implementation easier (I have no experience with filesystems)
> > > and thus works better as a PoC.
> > > 
> > 
> > I see your point but even Read-Only functionality is too much. :) Because, it
> > needs to implement 80% - 90% functionality of metadata management even for Read-
> > Only case. And I would like to make the whole thing done by small working steps.
> > This is why I would like: (1) start from Rust "library", (2) move metadata
> > management into Rust "library" gradually, (3) convert the whole driver into Rust
> > implementation.
> 
> I personally don't know how this argument works, I only cited it in case
> it is useful to people with domain knowledge :)
> 
> > > > > I'm not an expert on Rust, so it would be great if some Rust people 
> > > > > could share their opinions.
> > > > > 
> > > > 
> > > > I hope that Rust people would like the idea. :)
> > > 
> > > I'm sure that several Rust folks would be interested in getting their
> > > hands dirty helping with writing abstractions and/or the driver itself.
> > > 
> > 
> > Sounds great! :) I really need some help and advice.
> > 
> > > I personally am more on the Rust side of things, so I could help make
> > > the abstractions feel idiomatic and ergonomic.
> > > 
> > > Feel free to ask any follow up questions. Hope this helps!
> > > 
> > 
> > Sounds interesting! Let me prepare my questions. :) So, HFS/HFS+ have
> > superblock, bitmap, b-trees, extent records, catalog records. It sounds to me
> > like candidates for abstractions. Am I correct here? Are we understand
> > abstraction at the same way? :)
> 
> Yes! Everything that is used by other drivers/subsystems are usual
> candidates for abstractions.
> 
> Essentially an abstraction is a rustified version of the C API. For
> example, `Mutex<T>` is generic over the contained value, uses guards and
> only allows access to the inner value if the mutex is locked.
> 
> Abstractions can take a pretty different form from the C API when it's
> possible to make certain undesired uses of the API impossible through
> Rust's type system or other features (in the case of the mutex, making
> it impossible to access a value without locking it). Though they can
> also be pretty simple if the C API is straightforward (this of course
> depends on the concrete API).
> 
> Their purpose is to encapsulate the C API and expose its functionality
> to safe Rust (note that calling any C function is considered `unsafe` in
> Rust).
> 
> Calling C functions directly from Rust driver code (ie without going
> through an abstraction) is not something that we want to allow (of
> course there might be some exceptional cases where it is needed
> temporarily). And thus everything that you use should have an
> abstraction (this might include driver-specific abstractions that
> effectively are also part of the driver, but encapsulate the C parts,
> when you have converted the driver fully to rust, you probably won't
> need any of them).
> 
> 

So, any file system driver is not isolated thing and it interacts with VFS and
memory subsystem through a set of callbacks:

struct file_operations;
struct inode_operations;
struct dentry_operations;
struct address_space_operations;
struct super_operations;
struct fs_context_operations;
struct file_system_type;

Nowadays, VFS and memory subsystems are C implemented functionality. And I don't
think that it will be changed any time soon. So, even file system driver will be
completely re-written in Rust, then it should be ready to be called from C code.
Moreover, file system driver needs to interact with block layer that is written
in C too. So, glue code is inevitable right now. How bad and inefficient could
be using the glue code? Could you please share some example?

As far as I can see, it is possible to generalize HFS/HFS+ drivers to these
"abstractions":
(1) block bitmap
    - allocate_blocks()
    - free_blocks()
(2) Catalog tree
    - find()
    - create()
    - delete()
    - rename()
(3) Extents tree
    - get_block()
    - extend_file()
    - free_fork()
    - truncate_file()
(4) B-tree
    - open()
    - close()
    - write()
(5) B-tree node
    - find()
    - create()
    - clear()
    - copy()
    - move()
    - read()
    - write()

It's not all abstractions and the possible API doesn't look yet complete and
reasonable. But it is the basic blocks of HFS and HFS+ drivers. I think it needs
to dive deeper in the current code state to elaborate the vision of other
abstractions.

Thanks,
Slava.

Re: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Miguel Ojeda]

On Fri, Jun 20, 2025 at 7:50 PM Viacheslav Dubeyko
<Slava.Dubeyko@ibm.com> wrote:
>
> Nowadays, VFS and memory subsystems are C implemented functionality. And I don't
> think that it will be changed any time soon. So, even file system driver will be
> completely re-written in Rust, then it should be ready to be called from C code.

That is fine and expected.

> Moreover, file system driver needs to interact with block layer that is written
> in C too. So, glue code is inevitable right now. How bad and inefficient could
> be using the glue code? Could you please share some example?

Please take a look the proposed VFS abstractions, the filesystems that
were prototyped on top of them, and generally other Rust code we have.

As for "how bad", the key is that every time you go through a C
signature, you need to constrain yourself to what C can encode (which
is not much), use unsafe code and other interop issues. Thus you want
to avoid having to go back and forth all the time.

Thus, the idea is to write the filesystem in Rust using abstractions
that shield you from that.

Cc'ing other potentially interested/related people.

Cheers,
Miguel

RE: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Viacheslav Dubeyko]

On Fri, 2025-06-20 at 20:11 +0200, Miguel Ojeda wrote:
> On Fri, Jun 20, 2025 at 7:50 PM Viacheslav Dubeyko
> <Slava.Dubeyko@ibm.com> wrote:
> > 
> > Nowadays, VFS and memory subsystems are C implemented functionality. And I don't
> > think that it will be changed any time soon. So, even file system driver will be
> > completely re-written in Rust, then it should be ready to be called from C code.
> 
> That is fine and expected.
> 
> > Moreover, file system driver needs to interact with block layer that is written
> > in C too. So, glue code is inevitable right now. How bad and inefficient could
> > be using the glue code? Could you please share some example?
> 
> Please take a look the proposed VFS abstractions, the filesystems that
> were prototyped on top of them, and generally other Rust code we have.
> 
> As for "how bad", the key is that every time you go through a C
> signature, you need to constrain yourself to what C can encode (which
> is not much), use unsafe code and other interop issues. Thus you want
> to avoid having to go back and forth all the time.
> 
> Thus, the idea is to write the filesystem in Rust using abstractions
> that shield you from that.
> 
> Cc'ing other potentially interested/related people.
> 

I completely see your point. But let's consider allegory of home construction.
Usually, we need to start from foundation, then we need to raise the walls, and
so on. The file system's metadata is the foundation and if I would like to re-
write the file system driver, then I need to start from metadata. It means that
it makes sense to re-write, for example, bitmap or b-tree functionality and to
check that it works completely functionally correct in the C implemented
environment. Then, potentially, I could switch on bitmap implementation in Rust.
This is the vision of step-by-step implementation. And I completely OK with glue
code and inefficiency on the first steps because I need to prepare the file
system "foundation" and "walls". Also, I would like to manage the complexity of
implementation and bug fix. It means that I would like to isolate the bugs in
HFS/HFS+ layer. I can trust to C implementation of VFS but I cannot trust to
Rust implementation of VFS. So, I prefer to re-write HFS/HFS+ functionality in
Rust by using the C implemented environment at first. Because, from my point of
view, it is the way to manage complexity and to isolate bugs by HFS/HFS+ layer
only. And when everything will be in Rust, then it will be possible to switch on
complete Rust environment.

Thanks,
Slava.

Re: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Benno Lossin]

On Sun Jun 22, 2025 at 12:38 AM CEST, Viacheslav Dubeyko wrote:
> On Fri, 2025-06-20 at 20:11 +0200, Miguel Ojeda wrote:
>> On Fri, Jun 20, 2025 at 7:50 PM Viacheslav Dubeyko
>> <Slava.Dubeyko@ibm.com> wrote:
>> > 
>> > Nowadays, VFS and memory subsystems are C implemented functionality. And I don't
>> > think that it will be changed any time soon. So, even file system driver will be
>> > completely re-written in Rust, then it should be ready to be called from C code.
>> 
>> That is fine and expected.
>> 
>> > Moreover, file system driver needs to interact with block layer that is written
>> > in C too. So, glue code is inevitable right now. How bad and inefficient could
>> > be using the glue code? Could you please share some example?
>> 
>> Please take a look the proposed VFS abstractions, the filesystems that
>> were prototyped on top of them, and generally other Rust code we have.
>> 
>> As for "how bad", the key is that every time you go through a C
>> signature, you need to constrain yourself to what C can encode (which
>> is not much), use unsafe code and other interop issues. Thus you want
>> to avoid having to go back and forth all the time.
>> 
>> Thus, the idea is to write the filesystem in Rust using abstractions
>> that shield you from that.
>> 
>> Cc'ing other potentially interested/related people.
>> 
>
> I completely see your point. But let's consider allegory of home construction.
> Usually, we need to start from foundation, then we need to raise the walls, and
> so on. The file system's metadata is the foundation and if I would like to re-
> write the file system driver, then I need to start from metadata. It means that
> it makes sense to re-write, for example, bitmap or b-tree functionality and to
> check that it works completely functionally correct in the C implemented
> environment. Then, potentially, I could switch on bitmap implementation in Rust.
> This is the vision of step-by-step implementation. And I completely OK with glue
> code and inefficiency on the first steps because I need to prepare the file
> system "foundation" and "walls". Also, I would like to manage the complexity of
> implementation and bug fix. It means that I would like to isolate the bugs in
> HFS/HFS+ layer. I can trust to C implementation of VFS but I cannot trust to
> Rust implementation of VFS. So, I prefer to re-write HFS/HFS+ functionality in
> Rust by using the C implemented environment at first. Because, from my point of
> view, it is the way to manage complexity and to isolate bugs by HFS/HFS+ layer
> only. And when everything will be in Rust, then it will be possible to switch on
> complete Rust environment.

Ah maybe this is where the misunderstanding originates: we're not
talking about reimplementing bitmap or b-trees in Rust. We build
abstractions that call into the C side and use the existing
implementations. The abstractions make them available for the Rust side
to use them safely & efficiently.

In the case of bitmaps, there already is someone working on it, see [1].

In your metaphor, our recommendation is use adapters (the abstractions)
to go from the existing house frame (C) to the new interior (Rust). And
to avoid having mixed interior.

[1]: https://lore.kernel.org/all/20250620082954.540955-1-bqe@google.com

---
Cheers,
Benno

Re: [RFC] Should we consider to re-write HFS/HFS+ in Rust?

[Miguel Ojeda]

On Sun, Jun 22, 2025 at 12:39 AM Viacheslav Dubeyko
<Slava.Dubeyko@ibm.com> wrote:
>
> I completely see your point.

I think we are going in circles.

Yes, we understand the engineering value of taking small steps and
replacing small bits each time. That is not the issue -- we haven't
argued anything against that.

If you are talking about replacing code that is local/private to HFS
and that does not have any other users (like a custom data structure),
then it may or may not be a good idea, depending on what is it and how
you do it. It is hard to say without knowing more or seeing an
example.

But if you are talking about using C APIs shared with others, then it
is different, because the goal of the project is to provide safe
abstractions for modules to use, rather than have every module call C
directly (or have everyone reimplement their own abstractions
internally, and possibly unsoundly).

Those "abstractions" are not about reimplementing existing C code, but
about allowing Rust callers to use the existing C code _safely_. That
is a big part of the reason for using Rust in the kernel, and why we
are asking you to take a look at the existing abstractions and the VFS
ones.

Now, there can be exceptions, as usual in the kernel, and we can
discuss them, of course. But that is the general principle.

Cheers,
Miguel