Re: Fwd: Adding crates

[Kent Overstreet <kent.overstreet@gmail.com>]

On Wed, Mar 30, 2022 at 06:31:28AM +0200, Greg KH wrote:
> On Wed, Mar 30, 2022 at 11:03:50AM +1100, Chris Suter wrote:
> > Hi Miguel,
> > 
> > On Wed, Mar 30, 2022 at 2:47 AM Miguel Ojeda
> > <miguel.ojeda.sandonis@gmail.com> wrote:
> > >
> > > On Tue, Mar 29, 2022 at 9:49 AM Chris Suter <chris@sutes.me> wrote:
> > > >
> > > > I'm interested in quite a few crates but the first and perhaps most
> > > > significant would be futures. I'm not necessarily looking for a
> > > > "generic" solution at this point, any solution would do — just enough
> > > > to help me make a little progress and then maybe I could contribute
> > > > back later. I can probably figure it out myself, but I was hoping to
> > > > save some time.
> > >
> > > If you are just looking to experiment with some third-party crates,
> > > then as mentioned in the other thread, what you could do is copy and
> > > adapt the code. For instance, if the crates you need do not use Cargo
> > > build scripts, this may be as simple as copying the sources of the
> > > Rust modules into your kernel module (which is a crate), so that you
> > > do not need to deal with build systems.
> > 
> > I'm working with an existing code base and there are quite a few
> > crates I need to bring in or find replacements for. I'm hoping to
> > minimise or isolate the changes I need to make to the existing code
> > base. Obviously, some changes are going to be unavoidable.
> 
> What type of existing Rust codebase should be ported to be inside the
> kernel?

Greg, I think your point of view made sense in the past, but the world is
changing. We used to not want to share code between kernel space and userspace
as much in the first place, and in C it has historically been impractical, but
consider the following:

 - We already have large swaths of code being imported from userspace into the
   kernel as-is - e.g. zstd, and I bet I could find a lot more if I went
   digging.

   Right now we do this by just importing the code wholesale, but when changes
   end up needing to be applied for it to work in the kernel, this is a
   maintenance nightmare when we then need to re-merge from upstream.

   It would be far better to get the changes needed for use in the kernel merged
   with upstream, and then just use upstream directly like any other package
   (with version pinning, so that when upstream changes, we review those changes
   before updating the version the kernel depends on). And Rust's conditional
   compilation & generics story is _much_ better than what we've had in the past
   in C and C++, meaning writing Rust code that works in both userspace and
   kernelspace is much saner than in the past.

 - People are also starting to want to take kernel code and use it in userspace
   more often. In filesystem land, where I want, you need to do this or else you
   have massive duplication (& corresponding maintenance nightmare) between
   kernelspace and e.g. userspace fsck. In other areas, people just want to be
   able to write and run tests in userspace - e.g. xarrays, maple trees, and
   this is something we should encourage!

   But, when the kernel space & userspace environments are completely different,
   this is a massive undertaking. For bcachefs I had to write shim
   implementations for a _ton_ of code, most of which was quick and dirty
   throwaway code that's a pain in the ass to work with and that I would really
   like to delete.

The solution to problems like these are to stop thinking that kernelspace and
userspace _have_ to be completely different beasts - they really don't have to
be! and start encouraging different thinking and tooling improvements that will
make our lives easier.

In the kernel we also reinvent the wheel a _lot_ with our core data structures.
Some of the stuff we have is awesome, some is... amazing in some ways but
terrifying to have to deal with when it breaks (rhashtables, I'm looking at
you), but often the glaringly obvious basic stuff is missing - why do we have
nothing like CCAN darray? never mind, I finally just wrote one...

But code sharing for core data structures is finally starting to look tractable:
kernel side GFP flags is getting deprecated in favour of
memalloc_no*_save/restore (thank you Willy!) - that's a big one. And the Rust
people, to their credit, have been taking seriously kernel land's needs to
always check for memory allocation failure - Rust is explicitly targeted at bare
metal, kernel stuff!

Anyways. Just try and keep an open mind...