Change in kernel debian packages between -rc2 and -rc3

Change in kernel debian packages between -rc2 and -rc3

[Theodore Ts'o]

Hi Masahiro,

It appears that there is a backwards-incompatible (and, I believe,
incorrect) change in the package version used when generated Debian
packages between 6.2-rc2 and 6.2-rc3.

I have CONFIG_LOCALVERSION set in my .config:

   CONFIG_LOCALVERSION="-xfstests"

As a result, the uname -a of a kernel that I build contains -xfstests,
e.g:

root@kvm-xfstests:~# uname -r
6.3.0-rc3-xfstests-00043-g463f2e46bf7c

Previously this would be correctly reflected in the Debian package
version:

% dpkg -I /build/linux-image-6.3.0-rc2-xfstests_6.3.0-rc2-xfstests-1016_amd64.deb 
...
 Package: linux-image-6.3.0-rc2-xfstests
 Source: linux-6.3.0-rc2-xfstests
 Version: 6.3.0-rc2-xfstests-1016
...

However, after -rc3, the CONFIG_LOCALVERSION is no longer present in the package version:

% dpkg -I /build/linux-image-6.3.0-rc3-xfstests-00043-g463f2e46bf7c_6.3.0-rc3-00043-g463f2e46bf7c-1017_amd64.deb 
...
 Package: linux-image-6.3.0-rc3-xfstests-00043-g463f2e46bf7c
 Source: linux-6.3.0-rc3-xfstests-00043-g463f2e46bf7c
 Version: 6.3.0-rc3-00043-g463f2e46bf7c-1017
...

Note the missing "-xfstests" in the above Version.  This also shows up
in the name of the .deb file which is generated, and this is breaking
my kernel build script[1].

[1] https://github.com/tytso/xfstests-bld/blob/master/kernel-build/kbuild

I assume the problem is caused by one of these two commits:

36862e14e316 ("kbuild: deb-pkg: use dh_listpackages to know enabled packages")
b611daae5efc ("kbuild: deb-pkg: split image and debug objects staging out into functions")

As these are the two commits which touch the builddeb script.

Could you please take a look?

Many thanks,

						- Ted

Re: Change in kernel debian packages between -rc2 and -rc3

[Masahiro Yamada]

On Sat, Mar 25, 2023 at 11:31 PM Theodore Ts'o <tytso@mit.edu> wrote:
>
> Hi Masahiro,
>
> It appears that there is a backwards-incompatible (and, I believe,
> incorrect) change in the package version used when generated Debian
> packages between 6.2-rc2 and 6.2-rc3.
>
> I have CONFIG_LOCALVERSION set in my .config:
>
>    CONFIG_LOCALVERSION="-xfstests"
>
> As a result, the uname -a of a kernel that I build contains -xfstests,
> e.g:
>
> root@kvm-xfstests:~# uname -r
> 6.3.0-rc3-xfstests-00043-g463f2e46bf7c
>
> Previously this would be correctly reflected in the Debian package
> version:
>
> % dpkg -I /build/linux-image-6.3.0-rc2-xfstests_6.3.0-rc2-xfstests-1016_amd64.deb
> ...
>  Package: linux-image-6.3.0-rc2-xfstests
>  Source: linux-6.3.0-rc2-xfstests
>  Version: 6.3.0-rc2-xfstests-1016
> ...
>
> However, after -rc3, the CONFIG_LOCALVERSION is no longer present in the package version:
>
> % dpkg -I /build/linux-image-6.3.0-rc3-xfstests-00043-g463f2e46bf7c_6.3.0-rc3-00043-g463f2e46bf7c-1017_amd64.deb
> ...
>  Package: linux-image-6.3.0-rc3-xfstests-00043-g463f2e46bf7c
>  Source: linux-6.3.0-rc3-xfstests-00043-g463f2e46bf7c
>  Version: 6.3.0-rc3-00043-g463f2e46bf7c-1017
> ...



I am afraid you are completely misunderstanding the two versions,
the ABI version and the package version.

They do not need to match. Actually, they do not match.

See real Debian (or Ubuntu) systems.


On Debian bullseye.


$ uname -r
5.10.0-21-amd64
$ dpkg-deb -I linux-image-5.10.0-21-amd64_5.10.162-1_amd64.deb
 [snip]
 Package: linux-image-5.10.0-21-amd64
 Source: linux-signed-amd64 (5.10.162+1)
 Version: 5.10.162-1
 Architecture: amd64




`uname -r` returns '5.10.0-21-amd64'.
This is what they call the ABI version, and
this is contained as a part of the package name,
'linux-image-5.10.0-21-amd64'

The package version is '5.10.162-1'

They are different.

This means the Debian kernel team built the kernel package
based on the stable kernel '5.10.162'.
The suffix '-1' indicates the Debian revision.
But, they gave a different uname '5.10.0-21-amd64'.



Get back to your question.

You set CONFIG_LOCALVERSION="-xfstests".
It was reflected in 'uname -r' and in the package name.
This is correct.

It was _not_ reflected in the package version.
This is also correct since the package version
is not meant to reflect such user configuration
as CONFIG_LOCALVERSION.


The upstream version part '6.3.0-rc3-00043-g463f2e46bf7c'
represents the version of the source code.
In other words, it is determined by the git commit hash
(if you are using a git repository) and never affected by
CONFIG options or local files.

You can build multiple debian packages from the same git
commit hash, changing the .config, arch, or whatever.
In this case, the debian revision part, "-1017" is auto-incremented.

If you override the package version, you can use KDEB_PKGVERSION.





> Note the missing "-xfstests" in the above Version.  This also shows up
> in the name of the .deb file which is generated, and this is breaking
> my kernel build script[1].
>
> [1] https://github.com/tytso/xfstests-bld/blob/master/kernel-build/kbuild


This is just because your tool is wrong.
It was just working based on the wrong assumption.

Please fix it.





> I assume the problem is caused by one of these two commits:
>
> 36862e14e316 ("kbuild: deb-pkg: use dh_listpackages to know enabled packages")
> b611daae5efc ("kbuild: deb-pkg: split image and debug objects staging out into functions")
>
> As these are the two commits which touch the builddeb script.
>
> Could you please take a look?
>
> Many thanks,
>
>                                                 - Ted



-- 
Best Regards
Masahiro Yamada

Re: Change in kernel debian packages between -rc2 and -rc3

[Theodore Ts'o]

On Sun, Mar 26, 2023 at 12:39:44PM +0900, Masahiro Yamada wrote:
> 
> I am afraid you are completely misunderstanding the two versions,
> the ABI version and the package version.
> 
> They do not need to match. Actually, they do not match.
> 
> See real Debian (or Ubuntu) systems.
> 
> `uname -r` returns '5.10.0-21-amd64'.
> This is what they call the ABI version, and
> this is contained as a part of the package name,
> 'linux-image-5.10.0-21-amd64'

That may be a convention that Debian uses, but I'll point out that
trying to use this as an ABI version when people are using different
.config's is not going to end well.  That's because many different
Kernel configurations will end up making incompatible changes to the
ABI.  and only works if you are ***super*** careful about not making
any kind of changes (e.g., the number of CPU's, adding or changing
various cgroup controllers, all of which will make incompatible kernel
ABI changes.)

This is "Stable ABI nonsense" is well, nonsense.

> It was _not_ reflected in the package version.
> This is also correct since the package version
> is not meant to reflect such user configuration
> as CONFIG_LOCALVERSION.

Citation, please?  Does the Debian system work in that particular way?

In any case, it's an incompatible regression made during rc2 and rc3,
which I think is at the very least, unfortunate.

      	      	    	     	    - Ted

Re: Change in kernel debian packages between -rc2 and -rc3

[Kalle Valo]

"Theodore Ts'o" <tytso@mit.edu> writes:

> On Sun, Mar 26, 2023 at 12:39:44PM +0900, Masahiro Yamada wrote:
>> 
>> I am afraid you are completely misunderstanding the two versions,
>> the ABI version and the package version.
>> 
>> They do not need to match. Actually, they do not match.
>> 
>> See real Debian (or Ubuntu) systems.
>> 
>> `uname -r` returns '5.10.0-21-amd64'.
>> This is what they call the ABI version, and
>> this is contained as a part of the package name,
>> 'linux-image-5.10.0-21-amd64'
>
> That may be a convention that Debian uses, but I'll point out that
> trying to use this as an ABI version when people are using different
> .config's is not going to end well.  That's because many different
> Kernel configurations will end up making incompatible changes to the
> ABI.  and only works if you are ***super*** careful about not making
> any kind of changes (e.g., the number of CPU's, adding or changing
> various cgroup controllers, all of which will make incompatible kernel
> ABI changes.)
>
> This is "Stable ABI nonsense" is well, nonsense.
>
>> It was _not_ reflected in the package version.
>> This is also correct since the package version
>> is not meant to reflect such user configuration
>> as CONFIG_LOCALVERSION.
>
> Citation, please?  Does the Debian system work in that particular way?
>
> In any case, it's an incompatible regression made during rc2 and rc3,
> which I think is at the very least, unfortunate.

v6.3-rc3 also broke my deb install scripts, so definitely not cool
breaking it like this. Please keep the deb names stable.

-- 
https://patchwork.kernel.org/project/linux-wireless/list/

https://wireless.wiki.kernel.org/en/developers/documentation/submittingpatches

Re: Change in kernel debian packages between -rc2 and -rc3

[Masahiro Yamada]

On Sun, Mar 26, 2023 at 9:11 PM Theodore Ts'o <tytso@mit.edu> wrote:
>
> On Sun, Mar 26, 2023 at 12:39:44PM +0900, Masahiro Yamada wrote:
> >
> > I am afraid you are completely misunderstanding the two versions,
> > the ABI version and the package version.
> >
> > They do not need to match. Actually, they do not match.
> >
> > See real Debian (or Ubuntu) systems.
> >
> > `uname -r` returns '5.10.0-21-amd64'.
> > This is what they call the ABI version, and
> > this is contained as a part of the package name,
> > 'linux-image-5.10.0-21-amd64'
>
> That may be a convention that Debian uses, but I'll point out that
> trying to use this as an ABI version when people are using different
> .config's is not going to end well.  That's because many different
> Kernel configurations will end up making incompatible changes to the
> ABI.  and only works if you are ***super*** careful about not making
> any kind of changes (e.g., the number of CPU's, adding or changing
> various cgroup controllers, all of which will make incompatible kernel
> ABI changes.)
>
> This is "Stable ABI nonsense" is well, nonsense.

Well, that is just the _name_.

The actual effect is whether you can install
multiple kernel packages at the same time or not
because the ABI version is a part of the package name.
The ABI is nothing more than that for upstream kernel packages.



>
> > It was _not_ reflected in the package version.
> > This is also correct since the package version
> > is not meant to reflect such user configuration
> > as CONFIG_LOCALVERSION.
>
> Citation, please?  Does the Debian system work in that particular way?


The Debian policy [1] says that the version consists of:

    <upstream_version>-<debian_revision>

For clarification, I omitted <epoch>.


[1] http://www.fifi.org/doc/debian-policy/policy.html/ch-versions.html


See what the Debian source package looks like.

<upstream_version> determines the basename of *.orig.tar.gz.

*.orig.tar.gz is generated by 'git archive'.
Obviously, the output of 'git archive' is the same
as long as you work on the same git commit hash.

Think about doing 'make deb-pkg' multiple times on
commit 463f2e46bf7c.
(for example one for CONFIG_LOCALVERSION="-foo"
the other for CONFIG_LOCALVERSION="-bar")


First run:
linux-upstream_6.3.0-rc3-00043-g463f2e46bf7c.orig.tar.gz  [1]
linux-upstream_6.3.0-rc3-00043-g463f2e46bf7c-1.dsc
linux-upstream_6.3.0-rc3-00043-g463f2e46bf7c-1.debian.tar.xz

Second run:
linux-upstream_6.3.0-rc3-00043-g463f2e46bf7c.orig.tar.gz  [2]
linux-upstream_6.3.0-rc3-00043-g463f2e46bf7c-2.dsc
linux-upstream_6.3.0-rc3-00043-g463f2e46bf7c-2.debian.tar.xz


[1] and [2] are the same
(since it is the output of "git archive 463f2e46bf7c").
Hence, it is shared by multiple packages, no matter how many
you generate from commit 463f2e46bf7c.


You are requiring the file name to contain "-xfstests", "-foo", or "-bar"
depending on the CONFIG options.
There is no room for CONFIG options to affect <upstream_version>.

So, the rule is quite simple.
If the contents are exactly the same, they have the same version.
If they are different, they have different versions.



> In any case, it's an incompatible regression made during rc2 and rc3,
> which I think is at the very least, unfortunate.


Your tool was working on a wrong assumption, which nobody promised.

Kbuild already provides various hooks.
You can use CONFIG options and KERNELRELEASE to override uname,
and use KDEB_PKGVERSION to override the package version.

They are enough to continue your broken workflow.





>
>                                     - Ted
--
Best Regards
Masahiro Yamada

Re: Change in kernel debian packages between -rc2 and -rc3

[Theodore Ts'o]

On Mon, Mar 27, 2023 at 10:35:09PM +0900, Masahiro Yamada wrote:
> 
> The actual effect is whether you can install
> multiple kernel packages at the same time or not
> because the ABI version is a part of the package name.
> The ABI is nothing more than that for upstream kernel packages.

Well, for me, technically speaking CONFIG_LOCAVERSION *does* specify
an module ABI.  I have a install-kconfig[1] which is **different**
from what I might use in other contexts (for example, when building a
custom kernel for my laptop and desktop).  And in fact, there are two
flavors, with different kernel configs --- and hence, different and
incompatible module ABI's.  "xfstests"[2] and "xfstests-perf"[3]

[1] https://github.com/tytso/xfstests-bld/blob/master/kernel-build/install-kconfig
[2] https://github.com/tytso/xfstests-bld/blob/master/kernel-build/kernel-configs/config-6.1
[3] https://github.com/tytso/xfstests-bld/blob/master/kernel-build/kernel-configs/perf-config-6.1

And since they are different ABI's, and I want to distinguish between
them in the package name, so I can keep them straight --- and in
theory, there would be nothing wrong with wanting to have both an
"xfstests" and "xfstests-perf" installed at the same time, and being
able to select between the two of them using a bootloader.

Ergo, in my case CONFIG_LOCAVERSION is *part* of the version, and it
is a **feature*** that package names should include the
CONFIG_LOCALVERSIONS --- because they are different kernels, with
different properties!


> > In any case, it's an incompatible regression made during rc2 and rc3,
> > which I think is at the very least, unfortunate.
> 
> Your tool was working on a wrong assumption, which nobody promised.

It's still a backwards incompatible change.  And Linus doesn't like
backwards compatible changes which is noticeable, even if it's
something which isn't promised in, say, the POSIX spec.  If we change
kernel behavior, and it's noticeable, it's BAD BAD BAD.  I would claim
the same is true for kbuild infrastructure, unless there's a really
good reason.


> Kbuild already provides various hooks.
> You can use CONFIG options and KERNELRELEASE to override uname,
> and use KDEB_PKGVERSION to override the package version.

KDEB_PKGVERSION is not documented anywhere.  Thank you for telling me
about it, though.

I'd suggest that perhaps KDEB_PKGVERSION is documented, so there is an
explicit promise that this interface will be preserved?  I would hate
for you to later change, and break me, and claim "nobody promised",
since it's not documented.

And if you really want users to be able to replicate a Debian kernel's
"module ABI", then I'd **strongly** suggest that you document how they
are supposed to do that.  After all, Debian and Ubuntu doesn't use the
kbuild infrastructure; they have their own hugely complicated way of
generating files.

And oh by the way, I've just looked up Debian's documentation on that,
and the "ABI name" is not just the kernel version.  So it's not just
(for example) "2.6.18", but "2.6.18-1-686".   See [2].

[2] https://wiki.debian.org/DebianKernelABIChanges

And if you are going to be including the full git commit id in the
version, e.g., "6.3.0-rc3-00043-g463f2e46bf7c", it's not going to be
compatible with the Debian kernel ABI name ***anyway***.  So it's also
equally broken in terms of hwo Debian does things.


> They are enough to continue your broken workflow.

Well, for upstream developers, it's very useful to be able to
distinguish between kernel configs and this very *much* implicates the
"module ABI".  So IMHO think my particular use case is not "broken",
but in fact essential.

There are Very Good Reasons for doing things in this particular way.

      	       	    	    	      - Ted

P.S.  I actually simplified the story a little bit; there are actually
quite a lot more possible "module ABI's", or "kernel configs" than
just the two "xfstests" and "xfstests-perf".  These are all valid
install-kconfig commands:

# This will produce a CONFIG_LOCALVERSION of "xfstests-blktests"
% install-kconfig --blktests

# This will produce a CONFIG_LOCALVERSION of "xfstests-perf-kasan"
% install-kconfig --perf --kasan

# This will produce a CONFIG_LOCALVERSION of "xfstests-dept"
% install-kconfig --dept

# This will produce a CONFIG_LOCALVERSION of "xfstests-debug-lockdep"
% install-kconfig --debug --lockdep

etc.

These are all different kernel configs, producing different "module
ABI"'s, and I want to be able to distinguish between them in the
generated package files!

Re: Change in kernel debian packages between -rc2 and -rc3

[Theodore Ts'o]

I've worked around this in xfstests-bld.

					- Ted

commit e3309f769882397e605f956755dfec30f1f3f196
Author: Theodore Ts'o <tytso@mit.edu>
Date:   Sat Apr 1 23:22:09 2023 -0400

    kernel-build: work around backwards incompatible change in "make bindeb-pkg"
    
    Starting in v6.3-rc3, "make bindeb-pkg" made an incompatible change in
    how the debian package version is constructed, by dropping
    CONFIG_LOCALVERSION from the package version.  This might make sense
    given that Debian and Ubuntu try to adhere to Stable ABI nonsense with
    their Distribution kernels, but the default version chosen by "make
    bindeb-pkg" isn't compatible with the stable ABI nonsense versioning
    scheme in any case, so the change just breaks upstream developers for
    no good reason.
    
    Fix this by overriding the package version using KDEB_PKGVERSION,
    which is not documented, but hopefully it won't also incompatibly
    change in the future.  :-(
    
    Signed-off-by: Theodore Ts'o <tytso@mit.edu>

diff --git a/kernel-build/kbuild b/kernel-build/kbuild
index 16dfb7cd..4ab9ea28 100755
--- a/kernel-build/kbuild
+++ b/kernel-build/kbuild
@@ -153,7 +153,25 @@ rm -f "$BLD_DIR/linux-image.deb" "$BLD_DIR/linux-image-dbg.deb" \
 if test -n "$DO_DPKG" ; then
     $NO_ACTION make "${MAKE_ARGS[@]}" prepare
     REL=$(make "${MAKE_ARGS[@]}" kernelrelease | grep -v ^make)
-    $NO_ACTION time nice make "KDEB_SOURCENAME=linux-${REL}" "${MAKE_ARGS[@]}" \
+    MAJOR=$(echo $REL | awk -F. '{print $1}')
+    MINOR=$(echo $REL | awk -F. '{print $2}')
+    if test -f "$BLD_DIR/.version" ; then
+	NUM=$(cat "$BLD_DIR/.version")
+	# Starting in 6.1, how "make bindeb-pkg" handled the
+	# .version numbering randomly changed; accomodate that
+	# here.   Things are broken starting in 6.1-rc1 until 6.1-rc6
+	# but we won't worry about that here.  See commit
+	# 5db8face97f8 ("kbuild: Restore .version auto-increment behaviour
+	# for Debian packages") which fixed the problem but in a way
+	# which was different compared to how 6.0 and earlier kernels
+	# handled things.
+	if test "$MAJOR" -ge 6 -a "$MINOR" -ge 1 ; then
+	    NUM=$(( $NUM + 1 ))
+	fi
+    else
+	NUM=1
+    fi
+    $NO_ACTION time nice make "KDEB_PKGVERSION=$REL-$NUM" "KDEB_SOURCENAME=linux-${REL}" "${MAKE_ARGS[@]}" \
 	 bindeb-pkg "$@"
     err=$?
     d="$BLD_DIR/.."