New cachefilesd implementation

New cachefilesd implementation

[Max Kellermann]

Hi,

the venerable cachefilesd has caused us numerous problems. We have a
rather large cache partition (several terabytes containing hundreds of
millions of cached files), and cachefiled's culling would take a
thousand times longer than it would take the kernel to refill the
space that was freed. cachefilesd had been running for months at 100%
CPU with no visible progress. This made our fscache unusable.

This problem is caused by cachefilesd's bad choice of data structure:
it maintains a sorted array of object pointers, making
insert_into_cull_table() extremely slow and CPU consuming. I figured
reimplementing it from scratch in a saner language was faster than
fixing the old C code base. So I did just that, with C++ and io_uring.

This is it, after barely one day of hacking: https://github.com/CM4all/cash
It is not yet complete (hard-coded configuration, no graveyard
cleanup, bad error handling) and the code sure could be prettier, but
it's already running on our production servers and has cleaned up the
mess that cachefilesd has left behind.

Max

Re: New cachefilesd implementation

[David Howells]

Max Kellermann <max.kellermann@ionos.com> wrote:

> the venerable cachefilesd has caused us numerous problems. We have a
> rather large cache partition (several terabytes containing hundreds of
> millions of cached files), and cachefiled's culling would take a
> thousand times longer than it would take the kernel to refill the
> space that was freed. cachefilesd had been running for months at 100%
> CPU with no visible progress. This made our fscache unusable.
> 
> This problem is caused by cachefilesd's bad choice of data structure:
> it maintains a sorted array of object pointers, making
> insert_into_cull_table() extremely slow and CPU consuming. I figured
> reimplementing it from scratch in a saner language was faster than
> fixing the old C code base. So I did just that, with C++ and io_uring.
>
> This is it, after barely one day of hacking: https://github.com/CM4all/cash
> It is not yet complete (hard-coded configuration, no graveyard
> cleanup, bad error handling) and the code sure could be prettier, but
> it's already running on our production servers and has cleaned up the
> mess that cachefilesd has left behind.

Nice:-)

Now, would you be able to clone this:

	https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/cachefilesd.git/

Then rip out the existing code and put in your code?  There's some stuff in
there that is necessary to run on SELinux-based systems and it would make it
a lot easier for me just to replace the current cachefilesd package by just
pulling yours directly.

Also, you are using a simplified BSD licence - can you make it joint BSD/GPL?

David

Re: New cachefilesd implementation

[Max Kellermann]

On Fri, Nov 29, 2024 at 5:54 PM David Howells <dhowells@redhat.com> wrote:
> Now, would you be able to clone this:
>
>         https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/cachefilesd.git/
>
> Then rip out the existing code and put in your code?

You mean create a merge commit from your repository? This would make
it easier for people to "upgrade" by using a simple "git pull" on
their existing clone. Is that what you want?

> There's some stuff in
> there that is necessary to run on SELinux-based systems and it would make it
> a lot easier for me just to replace the current cachefilesd package by just
> pulling yours directly.

I wrote my implementation only for our own use (it solves an important
problem we had), but if you want to retire yours and make mine the
successor ("cachefilesd v2"?), I'd be willing to integrate all
existing features. And give up the silly project name ;-) (naming is
hard)

> Also, you are using a simplified BSD licence - can you make it joint BSD/GPL?

Yes.

Re: New cachefilesd implementation

[David Howells]

Max Kellermann <max.kellermann@ionos.com> wrote:

> On Fri, Nov 29, 2024 at 5:54 PM David Howells <dhowells@redhat.com> wrote:
> > Now, would you be able to clone this:
> >
> >         https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/cachefilesd.git/
> >
> > Then rip out the existing code and put in your code?
> 
> You mean create a merge commit from your repository? This would make
> it easier for people to "upgrade" by using a simple "git pull" on
> their existing clone. Is that what you want?

Yep.  Otherwise we have to go through the package replacement processes in
various distributions to replace cachefilesd with your one of a different
name.

> > There's some stuff in
> > there that is necessary to run on SELinux-based systems and it would make it
> > a lot easier for me just to replace the current cachefilesd package by just
> > pulling yours directly.
> 
> I wrote my implementation only for our own use (it solves an important
> problem we had), but if you want to retire yours and make mine the
> successor ("cachefilesd v2"?), I'd be willing to integrate all
> existing features. And give up the silly project name ;-) (naming is
> hard)

Yep.  If it does much the same thing, but more efficiently and faster, then
why not?

> > Also, you are using a simplified BSD licence - can you make it joint
> > BSD/GPL?
> 
> Yes.

Thanks:-)

David

Re: New cachefilesd implementation

[Max Kellermann]

On Fri, Nov 29, 2024 at 5:54 PM David Howells <dhowells@redhat.com> wrote:
> Now, would you be able to clone this:
>
>         https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/cachefilesd.git/
>
> Then rip out the existing code and put in your code?  There's some stuff in
> there that is necessary to run on SELinux-based systems and it would make it
> a lot easier for me just to replace the current cachefilesd package by just
> pulling yours directly.

Here:

https://github.com/CM4all/cash/tree/cachefilesd2

Please check if that's how you envisioned it. License is also changed
to BSD2 or GPL2+.
RFC! - it's not yet finalized, and not yet ready for merging.

Meanwhile, I have implemented lots of further optimizations. Now
almost all I/O (except for readdir()) is done with io_uring -
including writes to /dev/cachefiles, which required some tricky
machinery around fchdir(), because /dev/cachefiles accepts only file
names in the current working directory.

Some things are missing completely, like logging and daemonization. I
believe that daemonization is an obsolete feature and the init system
should take care for daemonization.

This implementation has been running on our production servers for
more than a week now and the problems we had with the old cachefilesd
are gone now. It's as fast as it can be, thanks to io_uring, and is
more secure: it drops CAP_SYS_ADMIN after opening /dev/cachefiles, and
has no other capabilities.

Max

Re: New cachefilesd implementation

[Max Kellermann]

On Fri, Dec 6, 2024 at 3:06 PM Max Kellermann <max.kellermann@ionos.com> wrote:
>
> On Fri, Nov 29, 2024 at 5:54 PM David Howells <dhowells@redhat.com> wrote:
> > Now, would you be able to clone this:
> >
> >         https://git.kernel.org/pub/scm/linux/kernel/git/dhowells/cachefilesd.git/
> >
> > Then rip out the existing code and put in your code?  There's some stuff in
> > there that is necessary to run on SELinux-based systems and it would make it
> > a lot easier for me just to replace the current cachefilesd package by just
> > pulling yours directly.
>
> Here:
>
> https://github.com/CM4all/cash/tree/cachefilesd2
>
> Please check if that's how you envisioned it. License is also changed
> to BSD2 or GPL2+.
> RFC! - it's not yet finalized, and not yet ready for merging.

David, please don't forget this.

Today, I have merged our latest version (0.3 -> 0.5) into the
"cachefilesd2" branch. The new version limits the io_uring
concurrency, because previously, our daemon could lock up the kernel
inside io_uring for exceeding the cgroup memory limit.

Max