PSA: Avoid Debian

PSA: Avoid Debian

[Kent Overstreet]

Debian (as well as Fedora) currently have a broken policy of switching
Rust dependencies to system packages - which are frequently out of date,
and cause real breakage.

As a result, updates that fix multiple critical bugs aren't getting
packaged.

(Beyond that, Debian is for some reason shipping a truly ancient
bcachefs-tools in stable, for reasons I still cannot fathom, which I've
gotten multiple bug reports over as well).

If you're running bcachefs, you'll want to be on a more modern distro -
or building bcachefs-tools yourself.

If you are building bcachefs-tools yourself, be aware that the mount
helper does not get run unless you install it into /usr (not
/usr/local).

Re: PSA: Avoid Debian

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 2459 bytes --]

On 8/7/24 12:34 AM, Kent Overstreet wrote:
> Debian (as well as Fedora) currently have a broken policy of switching
> Rust dependencies to system packages - which are frequently out of date,
> and cause real breakage.
> 
> As a result, updates that fix multiple critical bugs aren't getting
> packaged.
> 
> (Beyond that, Debian is for some reason shipping a truly ancient
> bcachefs-tools in stable, for reasons I still cannot fathom, which I've
> gotten multiple bug reports over as well).


I can help you fathom the reason.

The inherent purpose of Debian is to not provide updates to software,
since the general class of "computer code" contains the general risk of
"regressions", including "existing config files need to be updated, and
that regresses my ability to run a box unattended for 10 years without
ssh'ing in even once".

It is therefore the stated purpose of Debian that:

- if you managed to get a working installation you do not need new
  versions of the software to maintain a holding pattern in your ability
  to get the same work done today that you were able to get done
  yesterday -- whereas upgrading to new features can break that holding
  pattern.

- if you did NOT manage to get a working installation, you do not need
  that new stuff, and should stop attempting to have an "off-topic
  conversation that misses the purpose of Debian"


If you want updates to software, you upgrade debian, rather than
upgrading software. If you want modern software, you either use Debian
sid rather than stable (bcachefs-tools 1.9.1 may not be the most recent
version but it's very much not "truly ancient") or you do the sensible
thing and use a reasonable distro.

"Reasonable distro" meaning a distro that, when faced with the choice
between:

- allow people who installed the distro in 2017 to pretend it is still
  2017 and keep using their system the way they did back then

- improve the user experience for users

choose the latter.

In short, it is foolish to criticize Debian for shipping ancient
software since the answer is "the purpose of Debian is so that users can
use ancient software".

...

No comment on their switching rust dependencies to system packages.
Possibly the problem can be mitigated though -- is it possible for Rust
build systems to declare a minimum required version of dependencies, the
way pkg-config for C / C++ can do?


-- 
Eli Schwartz


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: PSA: Avoid Debian

[Kent Overstreet]

On Wed, Aug 07, 2024 at 11:01:12AM GMT, Eli Schwartz wrote:
> On 8/7/24 12:34 AM, Kent Overstreet wrote:
> > Debian (as well as Fedora) currently have a broken policy of switching
> > Rust dependencies to system packages - which are frequently out of date,
> > and cause real breakage.
> > 
> > As a result, updates that fix multiple critical bugs aren't getting
> > packaged.
> > 
> > (Beyond that, Debian is for some reason shipping a truly ancient
> > bcachefs-tools in stable, for reasons I still cannot fathom, which I've
> > gotten multiple bug reports over as well).
> 
> 
> I can help you fathom the reason.
> 
> The inherent purpose of Debian is to not provide updates to software,
> since the general class of "computer code" contains the general risk of
> "regressions", including "existing config files need to be updated, and
> that regresses my ability to run a box unattended for 10 years without
> ssh'ing in even once".
> 
> It is therefore the stated purpose of Debian that:
> 
> - if you managed to get a working installation you do not need new
>   versions of the software to maintain a holding pattern in your ability
>   to get the same work done today that you were able to get done
>   yesterday -- whereas upgrading to new features can break that holding
>   pattern.
> 
> - if you did NOT manage to get a working installation, you do not need
>   that new stuff, and should stop attempting to have an "off-topic
>   conversation that misses the purpose of Debian"
> 
> 
> If you want updates to software, you upgrade debian, rather than
> upgrading software. If you want modern software, you either use Debian
> sid rather than stable (bcachefs-tools 1.9.1 may not be the most recent
> version but it's very much not "truly ancient") or you do the sensible
> thing and use a reasonable distro.
> 
> "Reasonable distro" meaning a distro that, when faced with the choice
> between:
> 
> - allow people who installed the distro in 2017 to pretend it is still
>   2017 and keep using their system the way they did back then
> 
> - improve the user experience for users
> 
> choose the latter.
> 
> In short, it is foolish to criticize Debian for shipping ancient
> software since the answer is "the purpose of Debian is so that users can
> use ancient software".

This is holding up _bugfix releases_.

Anyone would run screaming from a distro that didn't ship updates at
all.

Re: PSA: Avoid Debian

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 1272 bytes --]

On 8/7/24 12:01 PM, Kent Overstreet wrote:
> This is holding up _bugfix releases_.
> 
> Anyone would run screaming from a distro that didn't ship updates at
> all.


(What if I said that lots of people *do* run screaming from Debian?)


You have to manually negotiate for those, to avoid the risk of
accidentally shipping an updated bugfix release that breaks their
spacebar heating: https://xkcd.com/1172/

Obviously, lots of software does get bugfix releases, because someone
negotiates the Debian political landscape to advocate for that bugfix
release.

When software cannot be updated by default because it might break
someone's workflow, the natural result of sometimes needing an update is
that people who want updates are pitted adversarially against people who
do not want updates -- you need to plead your case and get permission
and, well, fight for your right to receive a bugfix.

...

Has anyone volunteered to be the political advocate for bcachefs-tools
bugfix releases in Debian?


If not, then I should amend my earlier advice to also include "do not
use Debian if you don't like having to deal with the political
ramifications of using the most politically bureaucratic distro of all
time". :)


-- 
Eli Schwartz


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: PSA: Avoid Debian

[Kent Overstreet]

On Wed, Aug 07, 2024 at 12:09:53PM GMT, Eli Schwartz wrote:
> On 8/7/24 12:01 PM, Kent Overstreet wrote:
> > This is holding up _bugfix releases_.
> > 
> > Anyone would run screaming from a distro that didn't ship updates at
> > all.
> 
> 
> (What if I said that lots of people *do* run screaming from Debian?)

Heh.

Personally, in _general_, I feel quite affectionate towards debian; I've
been running it for 20 years, and there's a lot to like about it and a
lot of good stuff they've done.

But lately a _lot_ of the bug reports I'm seeing have "I was running an
old broken Debian package" as a root cause or additional complicating
factor.

And considering that this is due to something we discussed months? a
year? ago, and they're still insisting on broken policies, I am growing
_increasingly_ pissed off about it.

(Personally, this is pushing me to migrate my infrastructure to NixOS
sooner rather than later...)

> You have to manually negotiate for those, to avoid the risk of
> accidentally shipping an updated bugfix release that breaks their
> spacebar heating: https://xkcd.com/1172/

Which isn't remotely feasible; I have a lot of distros packaging
bcachefs, and I don't have time to devote to interactions like this with
all of them. This is Debian wanting to think they're special, assuming
that they can dominate with their policies - but that's not a winning
long term strategy, that's just going to result in them being left
behind.

The only honest way of influencing other people, and the only way that
works long term, is with the quality of your ideas. "But this is our
policy and you just have to abide" - nope. Even if people don't react
right away, they see that and take note and start maing other plans.

> When software cannot be updated by default because it might break
> someone's workflow, the natural result of sometimes needing an update is
> that people who want updates are pitted adversarially against people who
> do not want updates -- you need to plead your case and get permission
> and, well, fight for your right to receive a bugfix.

I've put a _lot_ of work into making sure bcachefs updates are as
painless as they can be, with e.g. seamless upgrade and downgrade paths,
and ways of dealing with version mismatch between tools/kernel/ondisk
filesystem.

Because we _have to be able to ship our work_, and in a timely manner.
Our systems get steadily more complicated year by year, decade by
decade, as we build up more processes and tooling around the whole
business of writing and shipping code. Making progress in our work
requires shipping code and iterating, so if we can't and we let the
political process bullshit it's death by a thousand cuts and work slowly
grinds to a halt.

> Has anyone volunteered to be the political advocate for bcachefs-tools
> bugfix releases in Debian?

No, and nor would I recommend anyone else for that kind of bullshit,
make-work job.

The real issue here is just that Debian needs to figure out how to have
some flexibility, recognize when policies aren't working, and develop a
better and more practical minded attitude.

So they can stop wasting my time with this stupid bullshit and I can get
back to real work.

Re: PSA: Avoid Debian

[Carl E. Thompson]

Whether or not the concept of Debian is a good idea probably isn't a constructive discussion for this list.

The problem here is that what was essentially an _alpha_ piece of software for what at the time was essentially an _alpha_ filesystem was allowed into the _stable_ release of Debian at all. Whoever on the Debian side allowed its inclusion dropped the ball.

Carl


> On 2024-08-07 10:29 AM PDT Kent Overstreet <kent.overstreet@linux.dev> wrote:
> 
>  
> On Wed, Aug 07, 2024 at 12:09:53PM GMT, Eli Schwartz wrote:
> > On 8/7/24 12:01 PM, Kent Overstreet wrote:
> > > This is holding up _bugfix releases_.
> > > 
> > > Anyone would run screaming from a distro that didn't ship updates at
> > > all.
> > 
> > 
> > (What if I said that lots of people *do* run screaming from Debian?)
> 
> Heh.
> 
> Personally, in _general_, I feel quite affectionate towards debian; I've
> been running it for 20 years, and there's a lot to like about it and a
> lot of good stuff they've done.
> 
> But lately a _lot_ of the bug reports I'm seeing have "I was running an
> old broken Debian package" as a root cause or additional complicating
> factor.
> 
> And considering that this is due to something we discussed months? a
> year? ago, and they're still insisting on broken policies, I am growing
> _increasingly_ pissed off about it.
> 
> (Personally, this is pushing me to migrate my infrastructure to NixOS
> sooner rather than later...)
> 
> > You have to manually negotiate for those, to avoid the risk of
> > accidentally shipping an updated bugfix release that breaks their
> > spacebar heating: https://xkcd.com/1172/
> 
> Which isn't remotely feasible; I have a lot of distros packaging
> bcachefs, and I don't have time to devote to interactions like this with
> all of them. This is Debian wanting to think they're special, assuming
> that they can dominate with their policies - but that's not a winning
> long term strategy, that's just going to result in them being left
> behind.
> 
> The only honest way of influencing other people, and the only way that
> works long term, is with the quality of your ideas. "But this is our
> policy and you just have to abide" - nope. Even if people don't react
> right away, they see that and take note and start maing other plans.
> 
> > When software cannot be updated by default because it might break
> > someone's workflow, the natural result of sometimes needing an update is
> > that people who want updates are pitted adversarially against people who
> > do not want updates -- you need to plead your case and get permission
> > and, well, fight for your right to receive a bugfix.
> 
> I've put a _lot_ of work into making sure bcachefs updates are as
> painless as they can be, with e.g. seamless upgrade and downgrade paths,
> and ways of dealing with version mismatch between tools/kernel/ondisk
> filesystem.
> 
> Because we _have to be able to ship our work_, and in a timely manner.
> Our systems get steadily more complicated year by year, decade by
> decade, as we build up more processes and tooling around the whole
> business of writing and shipping code. Making progress in our work
> requires shipping code and iterating, so if we can't and we let the
> political process bullshit it's death by a thousand cuts and work slowly
> grinds to a halt.
> 
> > Has anyone volunteered to be the political advocate for bcachefs-tools
> > bugfix releases in Debian?
> 
> No, and nor would I recommend anyone else for that kind of bullshit,
> make-work job.
> 
> The real issue here is just that Debian needs to figure out how to have
> some flexibility, recognize when policies aren't working, and develop a
> better and more practical minded attitude.
> 
> So they can stop wasting my time with this stupid bullshit and I can get
> back to real work.

Re: PSA: Avoid Debian Stable

[Martin Steigerwald]

I amended the subject line a bit. I am not sure whether Debian Testing or 
Debian Unstable should also be avoided.

And by the way, you probably have the same issue with Ubuntu, unless they 
update bcachefs-tools to new versions in stable releases.

Carl E. Thompson - 07.08.24, 19:43:17 CEST:
> Whether or not the concept of Debian is a good idea probably isn't a
> constructive discussion for this list.
> 
> The problem here is that what was essentially an _alpha_ piece of
> software for what at the time was essentially an _alpha_ filesystem was
> allowed into the _stable_ release of Debian at all. Whoever on the
> Debian side allowed its inclusion dropped the ball.

An option would be to at least ask for a backport.

Whether that is feasible with system based Rust packages in Debian 12 aka 
Bookworm is another story. Or probably there might also be a way to ask 
for removal of the outdated bcachefs-tools package. I agree it has been 
too early to include it in Debian 12. There are users who expect 1. stable 
and 2. most current software. I.e. expect they could use up-to-date 
BCacheFS in Debian 12. But the development process of Debian does not 
really cater for that. Either you have quite recent software, but face 
temporary instabilities or you have stable and soon to be outdated 
software.

One idea to use system based packages is to fix security issues once, 
instead of recompiling a lot of packages that use their own version of 
some Rust dependency.

For Debian 13 I expect BCacheFS tools to be stable enough for inclusion 
into stable release. However… that would still not solve the issue with 
fixing bugs in there during stable support – unless there would be some 
kind of LTS branch of BCacheFS tools by then. Critical fixes probably could 
be backported, that is the usual approach for Debian packages in stable, 
but this also does not work with Firefox ESR after a certain point. They 
update it to the newest ESR once the support for the old one is dropped by 
Mozilla. Cause it would be too much work to backport all security fixes.

Best would probably be to have a bcachefs-tools backport even for Debian 
13. If possible.

Best,
-- 
Martin

Re: PSA: Avoid Debian Stable

[Kent Overstreet]

On Wed, Aug 07, 2024 at 08:58:23PM GMT, Martin Steigerwald wrote:
> I amended the subject line a bit. I am not sure whether Debian Testing or 
> Debian Unstable should also be avoided.

Unstable as well - unstable is stuck at 1.9.1 and tools is up to 1.9.4,
and it's their policy of switching rust dependencies to distro packages
that broke the build.

Re: PSA: Avoid Debian Stable

[Martin Steigerwald]

Kent Overstreet - 07.08.24, 21:55:09 CEST:
> On Wed, Aug 07, 2024 at 08:58:23PM GMT, Martin Steigerwald wrote:
> > I amended the subject line a bit. I am not sure whether Debian Testing
> > or Debian Unstable should also be avoided.
> 
> Unstable as well - unstable is stuck at 1.9.1 and tools is up to 1.9.4,
> and it's their policy of switching rust dependencies to distro packages
> that broke the build.

Experimental meanwhile has 1.9.4.

But I do find this quite sad and concerning:

https://jonathancarter.org/2024/08/29/orphaning-bcachefs-tools-in-debian/

I do not completely agree with removing it from Debian Unstable aka Sid at 
this time, but if upstream development continues like this until too short 
before Debian 13 aka Trixie release I can somewhat understand. However it 
would still be good to have it there for people who use Debian Sid to 
test.

I more and more come to the conclusion myself that BCacheFS might be just 
a tad bit too much of a moving target for me at the moment.

BTRFS can be used just fine in Debian Stable meanwhile. But it took quite a 
while to get there. Version of btrfs-progs in Unstable is available as a 
backport for Debian stable. As far as I understand this cannot be done 
with BCacheFS tools without putting all the dependencies as is into the 
package and violating the principle to package a library dependency in one 
place and be able to update it for security updates in one place for all 
the applications and tools depending on it to benefit from them in one go.

I hope that at one point it can be like this for BCacheFS as well. And I 
am fine with this point being several years away.

Of course distributions with relaxed policies may not have a problem with 
packaging all the dependencies within a bcache-tools package. But this is 
not Debian or Ubuntu… or quite a lot of derivatives. I bet this is also 
not RHEL or SLES and derivatives of those.

-- 
Martin

Re: PSA: Avoid Debian Stable

[Kent Overstreet]

On Tue, Sep 03, 2024 at 11:37:32PM GMT, Martin Steigerwald wrote:
> Kent Overstreet - 07.08.24, 21:55:09 CEST:
> > On Wed, Aug 07, 2024 at 08:58:23PM GMT, Martin Steigerwald wrote:
> > > I amended the subject line a bit. I am not sure whether Debian Testing
> > > or Debian Unstable should also be avoided.
> > 
> > Unstable as well - unstable is stuck at 1.9.1 and tools is up to 1.9.4,
> > and it's their policy of switching rust dependencies to distro packages
> > that broke the build.
> 
> Experimental meanwhile has 1.9.4.
> 
> But I do find this quite sad and concerning:
> 
> https://jonathancarter.org/2024/08/29/orphaning-bcachefs-tools-in-debian/
> 
> I do not completely agree with removing it from Debian Unstable aka Sid at 
> this time, but if upstream development continues like this until too short 
> before Debian 13 aka Trixie release I can somewhat understand. However it 
> would still be good to have it there for people who use Debian Sid to 
> test.

Upstream development continues how...?

I would like Debian people to really consider if they might be
overreaching with their policies too much.

For now, we really need to be able to update bcachefs-tools in a timely
manner, because

- bugs happen: when updates stopped and debian users were stuck on
  1.9.1, that left users with filesystems they weren't able to mount -
  their machines were down - because of the bug with passing mount
  options, they weren't able to mount degraded.

  And I must repeat, I was the one fielding those bug reports, not the
  Debian package maintainer. I don't want Debian creating a mess and
  leaving me to deal with the aftermath.

- because -tools needs to stay up-to-date with on disk format features
  in the kernel

> I more and more come to the conclusion myself that BCacheFS might be just 
> a tad bit too much of a moving target for me at the moment.
> 
> BTRFS can be used just fine in Debian Stable meanwhile. But it took quite a 
> while to get there. Version of btrfs-progs in Unstable is available as a 
> backport for Debian stable. As far as I understand this cannot be done 
> with BCacheFS tools without putting all the dependencies as is into the 
> package and violating the principle to package a library dependency in one 
> place and be able to update it for security updates in one place for all 
> the applications and tools depending on it to benefit from them in one go.

I still have not yet heard a single good reason for this policy.

The rust dependencies are statically linked, and that's not going to
change because dynamic linking that works with the full Rust typesystem
is, simply, a _really_ hard problem that's going to take a massive
enginering effort to solve. Swift was able to do it - but I'm quite
familiar with what it took to do that, and they were only able to
because Apple invested significantly into it.

So, given that they're statically linked, why?

It can't be for security updates, because to update a dependency we
_still_ have to update and spin a new release of bcachefs-tools. And
that's not such a big deal, because nodays there's bots that notify me
if I need to do that.

So it seems to me that Debian people just aren't thinking things
through.

Re: PSA: Avoid Debian Stable

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 6799 bytes --]

On 9/3/24 6:15 PM, Kent Overstreet wrote:
> I still have not yet heard a single good reason for this policy.
> 
> The rust dependencies are statically linked, and that's not going to
> change because dynamic linking that works with the full Rust typesystem
> is, simply, a _really_ hard problem that's going to take a massive
> enginering effort to solve. Swift was able to do it - but I'm quite
> familiar with what it took to do that, and they were only able to
> because Apple invested significantly into it.
> 
> So, given that they're statically linked, why?
> 
> It can't be for security updates, because to update a dependency we
> _still_ have to update and spin a new release of bcachefs-tools. And
> that's not such a big deal, because nodays there's bots that notify me
> if I need to do that.
> 
> So it seems to me that Debian people just aren't thinking things
> through.


There's an interesting conversation to be had here about distfiles,
duplication, and offline building.

Probably the biggest single requirement here is the offline building.
It's not enough to download the bcachefs-tools source tarball and build
that, since you also need crates downloaded from crates.io and added to
a local registry. Most (sane) distros build packages with the network
disabled for security reasons, because you want to verify that all the
code you are building is code which was included in the original source
checksumming which is part of the build manifest (and the build manifest
in turn is probably PGP-signed by the distro developer that processed
the update).

And the Debian packaging format involves converting every package into
an "orig" tarball with the debian build recipes installed inside as
debian/ and then running the dpkg package building tools from inside
there. The key to a successful packaging experience is somehow getting
to run the build tools from that source tarball.

The packaging formats used by most other distros work differently: a
package can have multiple (checksummed) source tarballs and one of the
build phases is unpacking each listed distfile. Alpine, Arch, CRUX,
Fedora, Gentoo, Void, all list urls to download the source(s) from and
the checksums to validate against as part of the build recipe, and
unpack those into a temporary directory.

So for example that means that Gentoo can list 90 different crate
dependencies, plus one primary bcachefs tarball, and download each of
those files into /var/cache/distfiles, then unpack them into a fake
cargo registry. Of course, this is pretty slow if you don't have
parallel downloads -- not usually a problem for most packages that have
only one or two sources, but there you have it. And they can be shared
between multiple packages, or multiple versions of bcachefs-tools,
assuming that packages use exactly identical versions.

In theory, one can also specify a different crate version to use instead
of basing the versions on Cargo.lock. The update tool (pycargoebuild)
always builds the list based on Cargo.lock, though.

Now you have a 90-line package recipe, so have fun with that! :) Of
course, since any solution for bcachefs-tools is logically consistent
with handling other rust software, keep in mind that some packages have
1,000+ crates. Even more fun!


Back to Debian. Since each package can only have one source
*.orig.tar.xz, I would assume that Debian has basically two choices as a
result of their packaging format:

- package each crate as a system dependency
- repack each program source after running `cargo vendor`, and upload
  *that* as the canonical source code

Option 2 tends to not become great, e.g. due to the complete inability
to deduplicate between packages. Option 1 is entirely workable, *iff*
you assume that each crate and each crate *version* forms a unique
package *name* within Debian. For example, byteorder version 1.5.0 would
become a Debian package "librust-byteorder-1.5.0-dev (1.5.0)", and then
you could install as many versions of the "byteorder" crate as you like,
and depend on whichever one you need. Except... oops! Debian does this a
bit different. They depend on librust-byteorder-1-dev (>= 1.3), because
the package name is based on semver and the package restriction is >=
your Cargo.toml

[dependencies]
byteorder = "1.3"


One possible reason why they bind the name to the semver might be
because they, as noted, "have" to install the resulting -dev package to
/usr/share/cargo/registry/byteorder-1.3.0 or whatever, and by using a
global registry it's possibly impractical for *cargo* to determine the
"correct" version to use when both byteorder-1.3.0 and byteorder-1.5.0
are there.


Since the rust ecosystem takes great pride in semver and cargo is
designed to work this way (Cargo.lock even when checked into git is not
necessarily respected by ordinary commands) that probably feels like a
reasonable solution for the average use case of the rust ecosystem
within Debian. And it has some advantages, such as the ability to have
packages depend on only their direct dependencies, and have *those*
depend in turn on their dependencies, for great simplification of the
manifest of crates needed. Or, provide a patch to a crate *once* and
have all packages using that crate respect the patch. Anyone who thinks
that distros shouldn't ever patch software is unfortunately a fool. :)
It tends to be especially urgent for users of alt-arches where software
wasn't tested on e.g. mips or ppc64 or sparc or hppa and miserably
fails, while the maintainers have disappeared, but there are other fun
examples such as software with a build.rs that builds its own private
copy of a C library and should use the system one instead. Clowns that
insist on building their own private openssl version and ignoring the
system copy are the bane of computing.

Again, this is all about why a linux distro might have a *general
policy* for handling rust crates in one place per crate.

I suppose it's pretty unfortunate for rust software that isn't
compatible with the ^1.2.3 style versions described in Cargo.toml,
though that is a bit of an easy fix -- upgrade them.

It's unclear to me if bcachefs-tools was having bugs on Debian due to
following the strictures of Cargo.toml, or due to the patch "Relax build
dependencies to match what's available in Debian":

https://salsa.debian.org/debian/bcachefs-tools/-/blob/master/debian/patches/relax-build-dependencies

e.g.

-rustix = { version = "0.38.34", features = ["termios"] }
+rustix = { version = ">= 0.35 , < 1", features = ["termios"] }


Because Debian Stable has 0.35.12 and Debian Testing has 0.38.32 and
neither of those are sufficient.


-- 
Eli Schwartz


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: PSA: Avoid Debian Stable

[Kent Overstreet]

On Tue, Sep 03, 2024 at 07:44:52PM GMT, Eli Schwartz wrote:
> On 9/3/24 6:15 PM, Kent Overstreet wrote:
> > I still have not yet heard a single good reason for this policy.
> > 
> > The rust dependencies are statically linked, and that's not going to
> > change because dynamic linking that works with the full Rust typesystem
> > is, simply, a _really_ hard problem that's going to take a massive
> > enginering effort to solve. Swift was able to do it - but I'm quite
> > familiar with what it took to do that, and they were only able to
> > because Apple invested significantly into it.
> > 
> > So, given that they're statically linked, why?
> > 
> > It can't be for security updates, because to update a dependency we
> > _still_ have to update and spin a new release of bcachefs-tools. And
> > that's not such a big deal, because nodays there's bots that notify me
> > if I need to do that.
> > 
> > So it seems to me that Debian people just aren't thinking things
> > through.
> 
> 
> There's an interesting conversation to be had here about distfiles,
> duplication, and offline building.
> 
> Probably the biggest single requirement here is the offline building.
> It's not enough to download the bcachefs-tools source tarball and build
> that, since you also need crates downloaded from crates.io and added to
> a local registry. Most (sane) distros build packages with the network
> disabled for security reasons, because you want to verify that all the
> code you are building is code which was included in the original source
> checksumming which is part of the build manifest (and the build manifest
> in turn is probably PGP-signed by the distro developer that processed
> the update).
> 
> And the Debian packaging format involves converting every package into
> an "orig" tarball with the debian build recipes installed inside as
> debian/ and then running the dpkg package building tools from inside
> there. The key to a successful packaging experience is somehow getting
> to run the build tools from that source tarball.
> 
> The packaging formats used by most other distros work differently: a
> package can have multiple (checksummed) source tarballs and one of the
> build phases is unpacking each listed distfile. Alpine, Arch, CRUX,
> Fedora, Gentoo, Void, all list urls to download the source(s) from and
> the checksums to validate against as part of the build recipe, and
> unpack those into a temporary directory.
> 
> So for example that means that Gentoo can list 90 different crate
> dependencies, plus one primary bcachefs tarball, and download each of
> those files into /var/cache/distfiles, then unpack them into a fake
> cargo registry. Of course, this is pretty slow if you don't have
> parallel downloads -- not usually a problem for most packages that have
> only one or two sources, but there you have it. And they can be shared
> between multiple packages, or multiple versions of bcachefs-tools,
> assuming that packages use exactly identical versions.
> 
> In theory, one can also specify a different crate version to use instead
> of basing the versions on Cargo.lock. The update tool (pycargoebuild)
> always builds the list based on Cargo.lock, though.
> 
> Now you have a 90-line package recipe, so have fun with that! :) Of
> course, since any solution for bcachefs-tools is logically consistent
> with handling other rust software, keep in mind that some packages have
> 1,000+ crates. Even more fun!
> 
> 
> Back to Debian. Since each package can only have one source
> *.orig.tar.xz, I would assume that Debian has basically two choices as a
> result of their packaging format:
> 
> - package each crate as a system dependency
> - repack each program source after running `cargo vendor`, and upload
>   *that* as the canonical source code

I already provide 'cargo vendor' tarballs. Yes, it's big (thanks to
bindgen), but it's not _enormous_.

And bindgen is unfortunately the problematic one (and the one that broke
the build) - the semantics of the packed and aligned attributes are
subtle, and bindgen has had difficulty getting them right (due in part
to differences between gcc and microsoft).

So if you want to unbundle bindgen:
- you get to fix it if it breaks, I don't want to see a log of a build
  error months later that you didn't try to debug

- you really should be testing it as well (e.g. create an image with
  bcachefs format --source; verify that it fscks with kernel fsck
  implementation, i.e. that they agree); the stuff in bindgen that's
  been causing issues is absolutely the sort of thing that could cause
  miscompilation and if that ever happens it's going to be a really,
  really bad day.

Don't unbundle the other dependencies. They're tiny, and it's not worth
the pain it could potentially cause.

The cargo dependency model is _really good_ because it means dependency
updates happen in commits in the consuming package, meaning it's
actually possible to bisect for breakage caused by a dependency update -
that's not something we've had before.

Re: PSA: Avoid Debian Stable

[Eli Schwartz]

[-- Attachment #1.1: Type: text/plain, Size: 3695 bytes --]

On 9/3/24 8:04 PM, Kent Overstreet wrote:
>> Back to Debian. Since each package can only have one source
>> *.orig.tar.xz, I would assume that Debian has basically two choices as a
>> result of their packaging format:
>>
>> - package each crate as a system dependency
>> - repack each program source after running `cargo vendor`, and upload
>>   *that* as the canonical source code
> 
> I already provide 'cargo vendor' tarballs. Yes, it's big (thanks to
> bindgen), but it's not _enormous_.


Sure. Debian was probably working with the notion that using vendor
tarballs makes individual packages a "special snowflake", which is a
tempting rationale until you realize you're patching to allow versions
that are way too low.


> And bindgen is unfortunately the problematic one (and the one that broke
> the build) - the semantics of the packed and aligned attributes are
> subtle, and bindgen has had difficulty getting them right (due in part
> to differences between gcc and microsoft).


Heh, no doubt. :)

-bindgen = "0.69.4"
+bindgen = "0.66"

Really makes you think about the wisdom of allowing versions that
upstream specifically marked as incompatible (downgrading, not upgrading!)

Especially since binding between two languages will inherently tend to
violate rust's safety semantics! The entire point of bindgen is to
create a "safe" description of C types...


> Don't unbundle the other dependencies. They're tiny, and it's not worth
> the pain it could potentially cause.


Yeah but again this has nothing to do with unbundling, rather, it's
about modifying version and disrespecting:

- the lockfile
- the Cargo.toml

If the debian packages were simply the same version as all the
Cargo.lock entries, then it would be *unbundling*, but not *modification*.
'
It's the modification that's a problem, and specifically, the
modification of *required lower bounds*. It might be safe to upgrade a
crate, to get a "better" version, but downgrading to get a worse version
generally feels pretty silly.

Again, Simply running `cargo install --git
https://github.com/koverstreet/bcachefs-tools` will totally ignore your
lockfile and feel free to update to "newer == better" versions. Perhaps
bindgen would even fix more attribute issues. :D

But it would always, always, always respect Cargo.toml, which is why
Debian did NOT bother doing anything to the lockfile, but did apply a
patch to Cargo.toml


> The cargo dependency model is _really good_ because it means dependency
> updates happen in commits in the consuming package, meaning it's
> actually possible to bisect for breakage caused by a dependency update -
> that's not something we've had before.


And you can do the same with any C build system that isn't "I like
Makefiles because I think they are simple to write". You do a dependency
update in commits in the consuming package as an update to the version
specification of the url+checksum used to download and build C libraries
as vendored dependencies. Just like Cargo.lock

And like Cargo.toml, you can do the *other* kind of dependency update
(also as commits in the consuming package) by updating the *minimum*
version permissible (e.g. when consuming a system package).

This really is a solved problem in C/C++ and has been for a decade. It
has to be, if for no other reason than that Windows developers ***do not
have a choice*** as there is no such thing as system dependencies. That
is why vcpkg exists (and using it involves adding a manifest to your
Windows project describing the exact pinned versions of your
dependencies to go ahead and build).


-- 
Eli Schwartz


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 236 bytes --]

Re: PSA: Avoid Debian Stable

[Kent Overstreet]

On Tue, Sep 03, 2024 at 08:31:16PM GMT, Eli Schwartz wrote:
> On 9/3/24 8:04 PM, Kent Overstreet wrote:
> >> Back to Debian. Since each package can only have one source
> >> *.orig.tar.xz, I would assume that Debian has basically two choices as a
> >> result of their packaging format:
> >>
> >> - package each crate as a system dependency
> >> - repack each program source after running `cargo vendor`, and upload
> >>   *that* as the canonical source code
> > 
> > I already provide 'cargo vendor' tarballs. Yes, it's big (thanks to
> > bindgen), but it's not _enormous_.
> 
> 
> Sure. Debian was probably working with the notion that using vendor
> tarballs makes individual packages a "special snowflake", which is a
> tempting rationale until you realize you're patching to allow versions
> that are way too low.
> 
> 
> > And bindgen is unfortunately the problematic one (and the one that broke
> > the build) - the semantics of the packed and aligned attributes are
> > subtle, and bindgen has had difficulty getting them right (due in part
> > to differences between gcc and microsoft).
> 
> 
> Heh, no doubt. :)
> 
> -bindgen = "0.69.4"
> +bindgen = "0.66"
> 
> Really makes you think about the wisdom of allowing versions that
> upstream specifically marked as incompatible (downgrading, not upgrading!)
> 
> Especially since binding between two languages will inherently tend to
> violate rust's safety semantics! The entire point of bindgen is to
> create a "safe" description of C types...

Yeah, I think it's been pretty well established by this point that
mistakes were made.

Hopefully we're finally on a track to doing better :)

> > Don't unbundle the other dependencies. They're tiny, and it's not worth
> > the pain it could potentially cause.
> 
> 
> Yeah but again this has nothing to do with unbundling, rather, it's
> about modifying version and disrespecting:
> 
> - the lockfile
> - the Cargo.toml
> 
> If the debian packages were simply the same version as all the
> Cargo.lock entries, then it would be *unbundling*, but not *modification*.
> '
> It's the modification that's a problem, and specifically, the
> modification of *required lower bounds*. It might be safe to upgrade a
> crate, to get a "better" version, but downgrading to get a worse version
> generally feels pretty silly.

But given that different packages probably won't be specifying the exact
same versions of dependencies, so debian isn't going to have the right
one, and most of these dependencies are _tiny_ - why bother with the
enormous hassle?

Not everything tiny bit of code needs to be a distro-level package...

> Again, Simply running `cargo install --git
> https://github.com/koverstreet/bcachefs-tools` will totally ignore your
> lockfile and feel free to update to "newer == better" versions. Perhaps
> bindgen would even fix more attribute issues. :D
> 
> But it would always, always, always respect Cargo.toml, which is why
> Debian did NOT bother doing anything to the lockfile, but did apply a
> patch to Cargo.toml

Yeah, cargo still has some bugs: the modern thinking on how to do this
style of packaging 'right' has been evolving fairly recently.

Cargo.lock should really not be changed by anything but 'cargo update',
and if I was less lazy I'd file a bug about htat.

Re: PSA: Avoid Debian

[Carl E. Thompson]

Whether or not the concept of Debian is a good idea probably isn't a constructive discussion for this list.

The problem here is that what was essentially an _alpha_ piece of software for what at the time was essentially an _alpha_ filesystem was allowed into the _stable_ release of Debian at all. Whoever on the Debian side allowed its inclusion dropped the ball.

> On 2024-08-07 10:29 AM PDT Kent Overstreet <kent.overstreet@linux.dev> wrote:
> 
>  
> On Wed, Aug 07, 2024 at 12:09:53PM GMT, Eli Schwartz wrote:
> > On 8/7/24 12:01 PM, Kent Overstreet wrote:
> > > This is holding up _bugfix releases_.
> > > 
> > > Anyone would run screaming from a distro that didn't ship updates at
> > > all.
> > 
> > 
> > (What if I said that lots of people *do* run screaming from Debian?)
> 
> Heh.
> 
> Personally, in _general_, I feel quite affectionate towards debian; I've
> been running it for 20 years, and there's a lot to like about it and a
> lot of good stuff they've done.
> 
> But lately a _lot_ of the bug reports I'm seeing have "I was running an
> old broken Debian package" as a root cause or additional complicating
> factor.
> 
> And considering that this is due to something we discussed months? a
> year? ago, and they're still insisting on broken policies, I am growing
> _increasingly_ pissed off about it.
> 
> (Personally, this is pushing me to migrate my infrastructure to NixOS
> sooner rather than later...)
> 
> > You have to manually negotiate for those, to avoid the risk of
> > accidentally shipping an updated bugfix release that breaks their
> > spacebar heating: https://xkcd.com/1172/
> 
> Which isn't remotely feasible; I have a lot of distros packaging
> bcachefs, and I don't have time to devote to interactions like this with
> all of them. This is Debian wanting to think they're special, assuming
> that they can dominate with their policies - but that's not a winning
> long term strategy, that's just going to result in them being left
> behind.
> 
> The only honest way of influencing other people, and the only way that
> works long term, is with the quality of your ideas. "But this is our
> policy and you just have to abide" - nope. Even if people don't react
> right away, they see that and take note and start maing other plans.
> 
> > When software cannot be updated by default because it might break
> > someone's workflow, the natural result of sometimes needing an update is
> > that people who want updates are pitted adversarially against people who
> > do not want updates -- you need to plead your case and get permission
> > and, well, fight for your right to receive a bugfix.
> 
> I've put a _lot_ of work into making sure bcachefs updates are as
> painless as they can be, with e.g. seamless upgrade and downgrade paths,
> and ways of dealing with version mismatch between tools/kernel/ondisk
> filesystem.
> 
> Because we _have to be able to ship our work_, and in a timely manner.
> Our systems get steadily more complicated year by year, decade by
> decade, as we build up more processes and tooling around the whole
> business of writing and shipping code. Making progress in our work
> requires shipping code and iterating, so if we can't and we let the
> political process bullshit it's death by a thousand cuts and work slowly
> grinds to a halt.
> 
> > Has anyone volunteered to be the political advocate for bcachefs-tools
> > bugfix releases in Debian?
> 
> No, and nor would I recommend anyone else for that kind of bullshit,
> make-work job.
> 
> The real issue here is just that Debian needs to figure out how to have
> some flexibility, recognize when policies aren't working, and develop a
> better and more practical minded attitude.
> 
> So they can stop wasting my time with this stupid bullshit and I can get
> back to real work.

Re: PSA: Avoid Debian

[Kent Overstreet]

On Wed, Aug 07, 2024 at 10:44:19AM GMT, Carl E. Thompson wrote:
> Whether or not the concept of Debian is a good idea probably isn't a constructive discussion for this list.
> 
> The problem here is that what was essentially an _alpha_ piece of
> software for what at the time was essentially an _alpha_ filesystem
> was allowed into the _stable_ release of Debian at all. Whoever on the
> Debian side allowed its inclusion dropped the ball.

We're primarily talking about the package in Debian _unstable_, although
the ancient -tools package in stable is also causing problems. The
package in unstable is at 1.9.1, and we're just trying to get it to
1.9.4.

And you're blameshifting and making excuses. System critical packages
(and the filesystem is about as critical as it gets) absolutely need
timely updates, no matter what stage of the lifecycle it's at.

"Don't include it until it's perfect" is not an answer, because:
a) there will also be a period of stabilization for the distro rollout
itself where we find bugs that are specific to distro packaging and
distro process
b) software is never perfect

a) is what's going on here, and it's turning into a whole thing because
it's a Debian policy that's causing problems, and the Debian package
maintainer's response to that was "of course we won't change our policy".

So given that, and the number of bugs in my inbox from Debian users for
stuff that's already been fixed, I really have no choice but to tell
users to stay away from Debian until this gets sorted.

Re: PSA: Avoid Debian

[Carl E. Thompson]

Again, my point was (in my opinion) the bcachefs mailing list is probably not the best place to argue about whether Debian's policies / procedures are a good idea.

Of course I wasn't suggesting that bcachefs and the corresponding userspace tools need to be "perfect" before being included in Debian stable. But IMO they should at least be reasonably _stable_ to be included in Debian stable and bcachefs definitely wasn't at the time of its inclusion. Debian isn't Arch.

As far as Debian unstable goes, I have no strong opinions on that.

BTW, I'm not trying to be critical of you or of bcachefs. I have been using it on my personal systems for years now since long before it was in the kernel and there have been relatively few bumps on the road along the way and I have never lost any data. I'm impressed with what you've accomplished and have even donated my own personal money to you. However, as good as bcachefs is, based on the bug reports and rapid changes I've seen on this list I don't think it's truly stable yet. That's just my opinion.

Carl

> On 2024-08-07 11:19 AM PDT Kent Overstreet <kent.overstreet@linux.dev> wrote:
> 
>  
> On Wed, Aug 07, 2024 at 10:44:19AM GMT, Carl E. Thompson wrote:
> > Whether or not the concept of Debian is a good idea probably isn't a constructive discussion for this list.
> > 
> > The problem here is that what was essentially an _alpha_ piece of
> > software for what at the time was essentially an _alpha_ filesystem
> > was allowed into the _stable_ release of Debian at all. Whoever on the
> > Debian side allowed its inclusion dropped the ball.
> 
> We're primarily talking about the package in Debian _unstable_, although
> the ancient -tools package in stable is also causing problems. The
> package in unstable is at 1.9.1, and we're just trying to get it to
> 1.9.4.
> 
> And you're blameshifting and making excuses. System critical packages
> (and the filesystem is about as critical as it gets) absolutely need
> timely updates, no matter what stage of the lifecycle it's at.
> 
> "Don't include it until it's perfect" is not an answer, because:
> a) there will also be a period of stabilization for the distro rollout
> itself where we find bugs that are specific to distro packaging and
> distro process
> b) software is never perfect
> 
> a) is what's going on here, and it's turning into a whole thing because
> it's a Debian policy that's causing problems, and the Debian package
> maintainer's response to that was "of course we won't change our policy".
> 
> So given that, and the number of bugs in my inbox from Debian users for
> stuff that's already been fixed, I really have no choice but to tell
> users to stay away from Debian until this gets sorted.