[RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model

[RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model

[Artem S. Tashkinov]

Hi all,

The relentless cadence of critical vulnerability disclosures and public 
exploits over the past month—including Copy Fail (CVE-2026-31431), Dirty 
Frag (CVE-2026-43284/500), Fragnesia (CVE-2026-46300), and the ptrace 
exit race (CVE-2026-46333)—has highlighted a severe structural 
bottleneck in how we package and distribute stable backports.

When fatal logic flaws or memory corruptions strike core subsystems, our 
current point-release model fractures. Spinning up whole new point 
releases (7.0.4, 7.0.5, 7.0.7) in a matter of days just to address 
incomplete fixes, subsystem regressions, or independent public 
disclosures (such as the recent GRO managed-frag UAF exploit dropped 
directly to GitHub gists by researchers) creates massive administrative 
fatigue for maintainers and downstream teams alike.

Upstream has long maintained that the stable tree is effectively a 
continuous stream of fixes, and that users should track the tip of the 
stable branch rather than cherry-picking. It is time our release 
infrastructure matches this reality.

### The Proposal

I propose transitioning the stable tree (`linux-x.y.y`) away from 
manual,discrete point-release tarballs (`x.y.z`). Instead, we should 
treat the stable sub-version purely as an append-only, continuous, 
git-native patch stream.

Major releases (e.g., 7.0, 7.1) remain the foundational code boundaries, 
but sub-versions are eliminated as monolithic manual artifacts.

### The Implementation: How It Works

To ensure downstream distributions, enterprise compliance engines, and 
automated testing rings can still securely ingest code, we can replace 
the manual tarball with a decoupled, automated asset pipeline:

1. **The Git-First Stream:** The stable branch (`linux-7.0.y`) remains 
the single source of truth. Commits are pushed as soon as they pass 
stable criteria and automated sanity testing.

2. **The Signed Patch-Stream Archive:** Instead of packaging the entire 
30M+ line source code tree into a new tarball for every quick fix, 
upstream infrastructure maintains a rolling, cumulative patch sequence 
for the major cycle:

linux-7.0-stable.series = \sum (patch_1 + patch_2 + ... + patch_n)

Every time a fix is merged to the stable branch, the patch is appended 
to a publicly accessible, cryptographically signed manifest file
(`linux-7.0-stable-patches.tar.bz2` or a standard `series` file) 
alongside a detached signature.

3. **Automated Snapshot Tags:** If the industry strictly requires an 
immutable archive for compliance, point-release numbers can be replaced 
by automated, time-stamped git tags and machine-generated source 
snapshots cut on a strict, automated interval (e.g., every 48 hours), 
removing human maintainers entirely from the release timing.

### Why This Benefits the Ecosystem

* **Eliminates Churn and Latency:**

When a patch introduces an edge-case regression or requires an immediate 
follow-up (a common reason for rapid point-release sequences), 
maintainers do not need to coordinate a whole new release event. The 
follow-up fix is simply patch $n+1$. Downstream CI pipelines ingest it 
natively via standard git fetches.

* **Maintains Git-Native Debugging:**

Debugging stable regressions via `git bisect` has always been 
patch-based, not release-based. Since point releases are meant strictly 
for backported bug fixes, removing the arbitrary `x.y.z` release tags 
changes nothing about a developer's ability to isolate a regression. If 
anything, it prevents downstream vendors from pulling out-of-order 
patches that complicate bisection across distros.

* **Eases Downstream Automation:**

Modern tracking distributions (Arch, Fedora snapshotting, etc.) can 
switch to trunk-based intake, automatically building from the signed tip.

For enterprise distributions (RHEL, Ubuntu LTS) where constant kernel 
packaging and reboots are untenable, a fluid patch stream allows vendor 
security teams to more rapidly feed live-patching infrastructure 
(`kpatch`, `kgraft`), applying critical CVE fixes directly to runtime 
memory without changing the base package version.

* **Bridges the Compliance Gap:**

Embedded, automotive, or medical compliance pipelines
that legally require a static, verifiable code artifact can validate 
their software against the base major release tarball ($7.0.0$) plus the 
cryptographically signed, append-only stable patch series manifest.

The manual compilation, testing, and cutting of sub-version tarballs is 
an administrative artifact of the late 1990s. Shifting to an explicit, 
signed patch-stream architecture acknowledges the velocity of modern 
vulnerability research, strips away artificial latency, and frees our 
stable maintainers to focus on code quality rather than release 
management overhead.

I would love to hear thoughts, architectural blockers, or feedback from 
the stable maintainers and distribution teams on the feasibility of this 
transition.

Best regards,
Artem S. Tashkinov

Re: [RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model

[Greg Kroah-Hartman]

On Sun, May 24, 2026 at 01:38:55PM +0400, Artem S. Tashkinov wrote:
> Hi all,
> 
> The relentless cadence of critical vulnerability disclosures and public
> exploits over the past month—including Copy Fail (CVE-2026-31431), Dirty
> Frag (CVE-2026-43284/500), Fragnesia (CVE-2026-46300), and the ptrace exit
> race (CVE-2026-46333)—has highlighted a severe structural bottleneck in how
> we package and distribute stable backports.

Who is "we"?

And there's nothing really "new" here, these issues are all normal,
remember, we resolve, on average, 13 CVEs a day, most much more severe
than the ones that happened to get marketing names that you list here
(and how many systems have untrusted users?)

> When fatal logic flaws or memory corruptions strike core subsystems, our
> current point-release model fractures. Spinning up whole new point releases
> (7.0.4, 7.0.5, 7.0.7) in a matter of days just to address incomplete fixes,
> subsystem regressions, or independent public disclosures (such as the recent
> GRO managed-frag UAF exploit dropped directly to GitHub gists by
> researchers) creates massive administrative fatigue for maintainers and
> downstream teams alike.

it takes just a minute to "spin up" a point release, what is difficult
about that?  If needed, just let us know and we can easily do so.

> Upstream has long maintained that the stable tree is effectively a
> continuous stream of fixes, and that users should track the tip of the
> stable branch rather than cherry-picking. It is time our release
> infrastructure matches this reality.
> 
> ### The Proposal
> 
> I propose transitioning the stable tree (`linux-x.y.y`) away from
> manual,discrete point-release tarballs (`x.y.z`). Instead, we should treat
> the stable sub-version purely as an append-only, continuous, git-native
> patch stream.

That's what we do today, we just happen go "jump" on a weekly basis.

> Major releases (e.g., 7.0, 7.1) remain the foundational code boundaries, but
> sub-versions are eliminated as monolithic manual artifacts.
> 
> ### The Implementation: How It Works
> 
> To ensure downstream distributions, enterprise compliance engines, and
> automated testing rings can still securely ingest code, we can replace the
> manual tarball with a decoupled, automated asset pipeline:
> 
> 1. **The Git-First Stream:** The stable branch (`linux-7.0.y`) remains the
> single source of truth. Commits are pushed as soon as they pass stable
> criteria and automated sanity testing.

Again, that's what we do today.

> 2. **The Signed Patch-Stream Archive:** Instead of packaging the entire 30M+
> line source code tree into a new tarball for every quick fix, upstream
> infrastructure maintains a rolling, cumulative patch sequence for the major
> cycle:
> 
> linux-7.0-stable.series = \sum (patch_1 + patch_2 + ... + patch_n)
> 
> Every time a fix is merged to the stable branch, the patch is appended to a
> publicly accessible, cryptographically signed manifest file
> (`linux-7.0-stable-patches.tar.bz2` or a standard `series` file) alongside a
> detached signature.

Who would use/need such a thing?  What's wrong with the 2 systems we
have today that this would somehow help out with?

> 3. **Automated Snapshot Tags:** If the industry strictly requires an
> immutable archive for compliance,

What "compliance"?

> point-release numbers can be replaced by
> automated, time-stamped git tags and machine-generated source snapshots cut
> on a strict, automated interval (e.g., every 48 hours), removing human
> maintainers entirely from the release timing.

That's probably not a good idea anyway.  Are you doing continous testing
of the stable queue?  If so, great, just take from there today.
Everyone adds patches on top of releases anyway, what's a few more if it
happens to resolve specific issues for a day or so before a .y release
can be cut?

> ### Why This Benefits the Ecosystem
> 
> * **Eliminates Churn and Latency:**
> 
> When a patch introduces an edge-case regression or requires an immediate
> follow-up (a common reason for rapid point-release sequences), maintainers
> do not need to coordinate a whole new release event.

No real "coordination" happens here.

> The follow-up fix is simply patch $n+1$. Downstream CI pipelines
> ingest it natively via standard git fetches.

Again, we do that today.

> * **Maintains Git-Native Debugging:**
> 
> Debugging stable regressions via `git bisect` has always been patch-based,
> not release-based. Since point releases are meant strictly for backported
> bug fixes, removing the arbitrary `x.y.z` release tags changes nothing about
> a developer's ability to isolate a regression. If anything, it prevents
> downstream vendors from pulling out-of-order patches that complicate
> bisection across distros.

Who bisects across distros?

> * **Eases Downstream Automation:**
> 
> Modern tracking distributions (Arch, Fedora snapshotting, etc.) can switch
> to trunk-based intake, automatically building from the signed tip.

Have you asked them if they need/want this?

> For enterprise distributions (RHEL, Ubuntu LTS) where constant kernel
> packaging and reboots are untenable,

Why are reboots for these systems untenable?  Why not fix that root
problem instead?

> a fluid patch stream allows vendor
> security teams to more rapidly feed live-patching infrastructure (`kpatch`,
> `kgraft`), applying critical CVE fixes directly to runtime memory without
> changing the base package version.

They can do that today, and do do that today.  So again, what distro
needs this?

> * **Bridges the Compliance Gap:**
> 
> Embedded, automotive, or medical compliance pipelines
> that legally require a static, verifiable code artifact can validate their
> software against the base major release tarball ($7.0.0$) plus the
> cryptographically signed, append-only stable patch series manifest.

Do they really need that?  Again, they can have that today, nothing new
here.

> The manual compilation, testing, and cutting of sub-version tarballs is an
> administrative artifact of the late 1990s.

Weekly releases is not an artivact of the 1990s :)

> Shifting to an explicit, signed
> patch-stream architecture acknowledges the velocity of modern vulnerability
> research, strips away artificial latency, and frees our stable maintainers
> to focus on code quality rather than release management overhead.

Again, we have that today, on a weekly basis.

greg k-h

Re: [RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model

[Theodore Tso]

On Sun, May 24, 2026 at 12:56:54PM +0200, Greg Kroah-Hartman wrote:
> > point-release numbers can be replaced by
> > automated, time-stamped git tags and machine-generated source snapshots cut
> > on a strict, automated interval (e.g., every 48 hours), removing human
> > maintainers entirely from the release timing.
> 
> That's probably not a good idea anyway.  Are you doing continous testing
> of the stable queue?  If so, great, just take from there today.
> Everyone adds patches on top of releases anyway, what's a few more if it
> happens to resolve specific issues for a day or so before a .y release
> can be cut?

It's already the case that not all maintainers have the time to test
the stable queue, and it's not clear that current testing of the
weekly release is all that great.  I have seen stabilty regressions
where an xfstest running against ext4 will cause the kernel to crash
with the 6.1 and 6.6 LTS kernels.  It took me several days to figure
out the 6.1 regression, and I still haven't had time to look into the
6.6 regression, because my day job (which is not ext4, but herding
cats for an AI infrastructure project --- it's amazing how many fellow
developers I met at LSF/MM are actually doing AI infrastructure
projects for $WORK, and not kernel development as their primary job
responsibilities.)

So even the weekly cadence is starting to creek a bit from a quality
perspective.  I can't even *imagine* what a continuous, automated, "it
builds, ship it!" would do to the quality of the stable kernel series.

		     	  	      	  - Ted

P.S.  If someone is interesting in helping to test ext4 and xfs stable
kernel patches, talk to me.  There is partial automation to test
updates to the stable-rc trees, but I've never had time to automate
the rest of the test regression analysis combined with the automated
"which patches need to be backed out to avoid the regression / kernel
crash".  There had been a few companies contributing fractions of
engineers to do XFS stable maintainenace, all of those resources have
been withdrawn by their respective companies in the past year.




> 
> > ### Why This Benefits the Ecosystem
> > 
> > * **Eliminates Churn and Latency:**
> > 
> > When a patch introduces an edge-case regression or requires an immediate
> > follow-up (a common reason for rapid point-release sequences), maintainers
> > do not need to coordinate a whole new release event.
> 
> No real "coordination" happens here.
> 
> > The follow-up fix is simply patch $n+1$. Downstream CI pipelines
> > ingest it natively via standard git fetches.
> 
> Again, we do that today.
> 
> > * **Maintains Git-Native Debugging:**
> > 
> > Debugging stable regressions via `git bisect` has always been patch-based,
> > not release-based. Since point releases are meant strictly for backported
> > bug fixes, removing the arbitrary `x.y.z` release tags changes nothing about
> > a developer's ability to isolate a regression. If anything, it prevents
> > downstream vendors from pulling out-of-order patches that complicate
> > bisection across distros.
> 
> Who bisects across distros?
> 
> > * **Eases Downstream Automation:**
> > 
> > Modern tracking distributions (Arch, Fedora snapshotting, etc.) can switch
> > to trunk-based intake, automatically building from the signed tip.
> 
> Have you asked them if they need/want this?
> 
> > For enterprise distributions (RHEL, Ubuntu LTS) where constant kernel
> > packaging and reboots are untenable,
> 
> Why are reboots for these systems untenable?  Why not fix that root
> problem instead?
> 
> > a fluid patch stream allows vendor
> > security teams to more rapidly feed live-patching infrastructure (`kpatch`,
> > `kgraft`), applying critical CVE fixes directly to runtime memory without
> > changing the base package version.
> 
> They can do that today, and do do that today.  So again, what distro
> needs this?
> 
> > * **Bridges the Compliance Gap:**
> > 
> > Embedded, automotive, or medical compliance pipelines
> > that legally require a static, verifiable code artifact can validate their
> > software against the base major release tarball ($7.0.0$) plus the
> > cryptographically signed, append-only stable patch series manifest.
> 
> Do they really need that?  Again, they can have that today, nothing new
> here.
> 
> > The manual compilation, testing, and cutting of sub-version tarballs is an
> > administrative artifact of the late 1990s.
> 
> Weekly releases is not an artivact of the 1990s :)
> 
> > Shifting to an explicit, signed
> > patch-stream architecture acknowledges the velocity of modern vulnerability
> > research, strips away artificial latency, and frees our stable maintainers
> > to focus on code quality rather than release management overhead.
> 
> Again, we have that today, on a weekly basis.
> 
> greg k-h
> 

Re: [RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model

[Artem S. Tashkinov]

On 5/24/26 10:56 AM, Greg Kroah-Hartman wrote:
> On Sun, May 24, 2026 at 01:38:55PM +0400, Artem S. Tashkinov wrote:
>> Hi all,
>>
>> The relentless cadence of critical vulnerability disclosures and public
>> exploits over the past month—including Copy Fail (CVE-2026-31431), Dirty
>> Frag (CVE-2026-43284/500), Fragnesia (CVE-2026-46300), and the ptrace exit
>> race (CVE-2026-46333)—has highlighted a severe structural bottleneck in how
>> we package and distribute stable backports.
> 
> Who is "we"?
> 
> And there's nothing really "new" here, these issues are all normal,
> remember, we resolve, on average, 13 CVEs a day, most much more severe
> than the ones that happened to get marketing names that you list here
> (and how many systems have untrusted users?)
> 
>> When fatal logic flaws or memory corruptions strike core subsystems, our
>> current point-release model fractures. Spinning up whole new point releases
>> (7.0.4, 7.0.5, 7.0.7) in a matter of days just to address incomplete fixes,
>> subsystem regressions, or independent public disclosures (such as the recent
>> GRO managed-frag UAF exploit dropped directly to GitHub gists by
>> researchers) creates massive administrative fatigue for maintainers and
>> downstream teams alike.
> 
> it takes just a minute to "spin up" a point release, what is difficult
> about that?  If needed, just let us know and we can easily do so.
> 
>> Upstream has long maintained that the stable tree is effectively a
>> continuous stream of fixes, and that users should track the tip of the
>> stable branch rather than cherry-picking. It is time our release
>> infrastructure matches this reality.
>>
>> ### The Proposal
>>
>> I propose transitioning the stable tree (`linux-x.y.y`) away from
>> manual,discrete point-release tarballs (`x.y.z`). Instead, we should treat
>> the stable sub-version purely as an append-only, continuous, git-native
>> patch stream.
> 
> That's what we do today, we just happen go "jump" on a weekly basis.
> 
>> Major releases (e.g., 7.0, 7.1) remain the foundational code boundaries, but
>> sub-versions are eliminated as monolithic manual artifacts.
>>
>> ### The Implementation: How It Works
>>
>> To ensure downstream distributions, enterprise compliance engines, and
>> automated testing rings can still securely ingest code, we can replace the
>> manual tarball with a decoupled, automated asset pipeline:
>>
>> 1. **The Git-First Stream:** The stable branch (`linux-7.0.y`) remains the
>> single source of truth. Commits are pushed as soon as they pass stable
>> criteria and automated sanity testing.
> 
> Again, that's what we do today.
> 
>> 2. **The Signed Patch-Stream Archive:** Instead of packaging the entire 30M+
>> line source code tree into a new tarball for every quick fix, upstream
>> infrastructure maintains a rolling, cumulative patch sequence for the major
>> cycle:
>>
>> linux-7.0-stable.series = \sum (patch_1 + patch_2 + ... + patch_n)
>>
>> Every time a fix is merged to the stable branch, the patch is appended to a
>> publicly accessible, cryptographically signed manifest file
>> (`linux-7.0-stable-patches.tar.bz2` or a standard `series` file) alongside a
>> detached signature.
> 
> Who would use/need such a thing?  What's wrong with the 2 systems we
> have today that this would somehow help out with?
> 
>> 3. **Automated Snapshot Tags:** If the industry strictly requires an
>> immutable archive for compliance,
> 
> What "compliance"?
> 
>> point-release numbers can be replaced by
>> automated, time-stamped git tags and machine-generated source snapshots cut
>> on a strict, automated interval (e.g., every 48 hours), removing human
>> maintainers entirely from the release timing.
> 
> That's probably not a good idea anyway.  Are you doing continous testing
> of the stable queue?  If so, great, just take from there today.
> Everyone adds patches on top of releases anyway, what's a few more if it
> happens to resolve specific issues for a day or so before a .y release
> can be cut?
> 
>> ### Why This Benefits the Ecosystem
>>
>> * **Eliminates Churn and Latency:**
>>
>> When a patch introduces an edge-case regression or requires an immediate
>> follow-up (a common reason for rapid point-release sequences), maintainers
>> do not need to coordinate a whole new release event.
> 
> No real "coordination" happens here.
> 
>> The follow-up fix is simply patch $n+1$. Downstream CI pipelines
>> ingest it natively via standard git fetches.
> 
> Again, we do that today.
> 
>> * **Maintains Git-Native Debugging:**
>>
>> Debugging stable regressions via `git bisect` has always been patch-based,
>> not release-based. Since point releases are meant strictly for backported
>> bug fixes, removing the arbitrary `x.y.z` release tags changes nothing about
>> a developer's ability to isolate a regression. If anything, it prevents
>> downstream vendors from pulling out-of-order patches that complicate
>> bisection across distros.
> 
> Who bisects across distros?
> 
>> * **Eases Downstream Automation:**
>>
>> Modern tracking distributions (Arch, Fedora snapshotting, etc.) can switch
>> to trunk-based intake, automatically building from the signed tip.
> 
> Have you asked them if they need/want this?
> 
>> For enterprise distributions (RHEL, Ubuntu LTS) where constant kernel
>> packaging and reboots are untenable,
> 
> Why are reboots for these systems untenable?  Why not fix that root
> problem instead?
> 
>> a fluid patch stream allows vendor
>> security teams to more rapidly feed live-patching infrastructure (`kpatch`,
>> `kgraft`), applying critical CVE fixes directly to runtime memory without
>> changing the base package version.
> 
> They can do that today, and do do that today.  So again, what distro
> needs this?
> 
>> * **Bridges the Compliance Gap:**
>>
>> Embedded, automotive, or medical compliance pipelines
>> that legally require a static, verifiable code artifact can validate their
>> software against the base major release tarball ($7.0.0$) plus the
>> cryptographically signed, append-only stable patch series manifest.
> 
> Do they really need that?  Again, they can have that today, nothing new
> here.
> 
>> The manual compilation, testing, and cutting of sub-version tarballs is an
>> administrative artifact of the late 1990s.
> 
> Weekly releases is not an artivact of the 1990s :)
> 
>> Shifting to an explicit, signed
>> patch-stream architecture acknowledges the velocity of modern vulnerability
>> research, strips away artificial latency, and frees our stable maintainers
>> to focus on code quality rather than release management overhead.
> 
> Again, we have that today, on a weekly basis.
> 
> greg k-h

Hi Greg,

I understand completely that from an upstream maintainer perspective, 
the current pipeline feels seamless. Running a script to cut a tag takes 
a minute, and git-stable is inherently fluid. But looking at the 
ecosystem from the downstream and infrastructure side, the view is a bit 
different.

Here is where the current model introduces friction that a patch-stream 
or automated snapshot approach could solve:

### 1. The Compounding Downstream Tax

When you spin up a rapid sequence like 7.0.4, 7.0.5, and 7.0.7 to 
address the immediate fallout of complex CVEs, it takes a minute for 
upstream, but it triggers a massive domino effect downstream. Hundreds 
of distribution mirrors sync gigabytes of redundant source data, package 
maintainers rewrite specs, and automated build farms recreate full 
packages. When a fix is incomplete or causes an immediate regression 
requiring a follow-up version 48 hours later, that entire global compute 
and human cycle repeats for what amounts to a few lines of diff.

### 2. The Infrastructure Reality of Reboots

You mentioned that systems should just be fixed so reboots aren't 
untenable. In an ideal architectural world, yes. But in production 
reality, orchestrating reboots across thousands of live cloud nodes 
running high-availability workloads carries immense risk and scheduling 
overhead.

Enterprise distros absolutely do use live-patching (`kpatch`/`kgraft`) 
to mitigate this, but tracking a shifting landscape of discrete 
point-release tarballs complicates their internal backport verification. 
A continuous, signed patch-stream would give vendor security teams a 
clean, linear ledger of upstream-approved deltas to feed directly into 
live-patch compilation engines without the noise of full tree packaging.

### 3. Archive Bloat on the Mirror Network

Every stable point release requires a new ~130MB `.tar.xz` full source 
archive. Multiplying that across multiple active LTS branches, dozens of 
point releases, and hundreds of worldwide mirrors results in massive 
data duplication just to distribute a cumulative handful of text diffs. 
The ecosystem is essentially re-shipping the entire ocean every time we 
need to add a cup of water.

### Why the Proposal Matters

When independent researchers drop working exploit primitives directly to 
public gists (like the recent GRO managed-frag UAF), the turnaround time 
to protected production systems needs to be near-zero.

The proposal wasn't meant to imply that the stable team isn't working 
fast enough—you guys are incredibly fast. It was an observation that the 
*delivery mechanism* (the discrete, human-timed tarball) forces 
downstream consumers to choose between packaging fatigue or artificial 
patch latency. Shifting the sub-version boundary to a machine-managed, 
append-only signed patch stream bridges that gap.

I know you have a workflow that works for **you**, and I respect the 
hell out of the massive volume of fixes you ship daily. I just wanted to 
share the perspective of how that weekly "jump" ripples out to the 
people who have to deploy it.

### Truly Massive Distro Churn Issue

There's also a very important downstream packaging issue that I think 
the current stable-kernel release model underestimates.

Debian and Ubuntu already diverge from the upstream stable versioning 
model in a useful way: they package kernels around their own ABI 
versioning. In practice, this means they can often ship fixes without 
forcing users to install an entirely new `/lib/modules/<kernel-version>` 
tree every time an upstream stable micro-release appears. A new 
ABI/package line is only needed when the kernel ABI they expose to 
packaged or out-of-tree modules changes.

That model is a major downstream advantage. It avoids needless module 
churn, avoids retaining multiple near-identical kernel trees, and makes 
security/regression updates cheaper to ship. ABI-breaking changes do 
happen, but in stable kernels they are rare enough that treating them as 
exceptional events is perfectly reasonable.

My proposal takes this idea further.

Stable kernel releases frequently contain important fixes for 
regressions introduced either in `.0` releases or in earlier stable 
updates. But the current release cadence means that downstream 
distributions often have to react before the next stable point release 
exists. They either wait, leaving users exposed to known regressions, or 
they cherry-pick individual commits from the stable queue. That creates 
avoidable work: maintainers have to select fixes, update packaging 
metadata, rebuild, test, and publish a distro-specific kernel update 
that is supposed to correspond to a “stable” upstream series.

Fedora’s 7.0 cycle is a good example of the downstream churn this 
creates: multiple downstream updates may be needed for what is logically 
the same upstream stable series, simply because important fixes land 
between formal stable releases.

Under the model I am proposing, stable would be treated less like a 
sequence of isolated point releases and more like a continuously 
updated, signed stable branch. Distributions could pin a specific signed 
stable commit and trigger a rebuild from that point, instead of 
maintaining their own ad hoc selection of fixes while waiting for the 
next point release.

This would not remove the need for distro testing or for avoiding 
known-bad commits. But in the common case it would simplify the workflow 
enormously: downstreams would consume the stable branch directly, 
rebuild from a known commit, and get all accepted stable fixes together. 
Only in rare cases would they need to temporarily blacklist or revert a 
problematic stable commit.

That is still much easier than every distribution independently deciding 
which urgent fixes to cherry-pick from the stable queue.

The end result would be less downstream packaging churn, faster delivery 
of regression fixes, fewer distro-specific stable-kernel deltas, and a 
model that better reflects how many downstreams already consume stable 
kernels in practice.

Best regards,
Artem

Re: [RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model

[Artem S. Tashkinov]

Hi Greg,

Let me try to restate the proposal more narrowly, because I think my 
previous wording mixed several related issues together.

I am not arguing that stable.git is not already continuous. It obviously 
is. Nor am I arguing that cutting a point release is expensive for 
upstream. I understand that from the stable maintainer side, tagging a 
release is cheap and well automated.

The problem is that the ecosystem still treats the x.y.z point release 
as the main externally visible consumption boundary, even though it is 
neither the real upstream unit nor the real downstream deployment unit.

The real upstream unit is the stable git branch: a linear sequence of 
accepted backports.

The real downstream deployment unit is a distro-built kernel package: 
base kernel, stable backports through some point in git history, distro 
patches/configuration, compiler/toolchain, signing, modules, CI, and 
user-visible packaging.

The x.y.z point release sits awkwardly in between. It is a useful 
compatibility marker for consumers that want that workflow, but it 
should not have to be the canonical boundary for everyone else.

A better model, in my view, would be:

* stable.git remains the canonical source of truth;
* every downstream-consumable stable state is identified by branch + 
signed commit ID, optionally with a signed machine-readable manifest;
* distros consume `v7.0 + stable commits through <commit>` or 
`linux-7.0.y as of <date/commit>`, according to their own testing and 
release policy;
* point releases may continue to exist for consumers that require them, 
but they become one possible checkpoint format rather than the central 
model.

This decouples two decisions that are currently conflated:

1. upstream’s decision that a commit belongs in the stable branch;
2. a downstream’s decision that a particular aggregate is ready to ship 
to its users.

Those are not the same decision. Fedora, Arch, Debian, Ubuntu, 
enterprise vendors, embedded vendors, and live-patching teams all have 
different risk tolerances, hardware exposure, CI capacity, reboot 
policies, module/signing workflows, and urgency. A single upstream x.y.z 
cadence cannot be the right integration boundary for all of them.

In the recent 7.0 security/regression mess, the useful downstream 
question should not have been “which point release has Greg cut yet?” It 
should have been “which stable commits are required for the complete fix 
set, and has our actual distro kernel artifact built from that stable 
commit range passed enough testing to ship?”

If distros routinely consumed the stable branch as a signed linear 
stream, they could pin a specific stable commit, build from it, test it, 
and declare exactly what they shipped:

```
base: v7.0
stable branch: linux-7.0.y
stable commit: <hash>
included range: v7.0..<hash>
downstream patches/reverts: <list>
```

That is more auditable than chasing point releases or cherry-picking 
individual commits from intermediate states. It also makes clear that 
the distro package, not the upstream tarball, is the object that was 
actually tested and deployed.

There is also a practical artifact problem here. For long-lived stable 
series, publishing a complete source tarball for every x.y.z release is 
very wasteful. By 5.10.258, the 5.10.y series has hundreds of distinct 
tarballs, each over 100 MB compressed, representing tens of gigabytes of 
mostly duplicated source snapshots for one stable line. The real 
information is the base tree plus the incremental stable deltas. Git 
already represents that naturally.

So I am not asking to remove point releases from users that still need 
them. Keep them for conservative workflows, existing scripts, 
announcements, and consumers that require a simple named snapshot.

What I am arguing is that point releases should no longer be the 
privileged stable consumption model. The canonical model should be the 
signed stable git stream, with downstreams free to define their own 
tested integration points on that stream.

That would make the stable process match what it already is technically: 
a continuous sequence of accepted backports, not a sequence of magic 
tarball events.

Regards,
Artem

Re: [RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model

[Willy Tarreau]

On Sun, Jun 14, 2026 at 08:16:06AM +0000, Artem S. Tashkinov wrote:
> A better model, in my view, would be:
> 
> * stable.git remains the canonical source of truth;
> * every downstream-consumable stable state is identified by branch + signed
> commit ID, optionally with a signed machine-readable manifest;

Well, you have stable-queue which is getting closer to that.

> * distros consume `v7.0 + stable commits through <commit>` or `linux-7.0.y
> as of <date/commit>`, according to their own testing and release policy;

what makes you think that distros will want to pick even less tested
or validated commits ? Right now we know that some distros cherry-pick
just what they see fit and they miss important fixes, probably based on
lack of trust of a well-exposed set of patches that compose a version.
If you refine the granularity, there will be even less trust in the
level of test of each commit and they might wait even longer to pick
fixes at all.

> * point releases may continue to exist for consumers that require them, but
> they become one possible checkpoint format rather than the central model.

This model at least gives a good indication of how late your system is
(the last number roughly corresponds to the number of weeks past the
release). It also offers opportunities to maintainers to voice in
saying "not this patch" or "pick that one with it" even if it does
not happen that often. And it increases the exposure on patches
combinations to more users. The finer your granulary, the less tested
each combination is. Here the current model sets discrete points in
time where many users deploy the same version hence the same set of
patches. This is particularly important and it has allowed many times
in the past to quickly spot a regression that would have taken much
longer to be noticed if everyone would deploy at a different point.
> 
> This decouples two decisions that are currently conflated:
> 
> 1. upstream's decision that a commit belongs in the stable branch;
> 2. a downstream's decision that a particular aggregate is ready to ship to
> its users.
> 
> Those are not the same decision. Fedora, Arch, Debian, Ubuntu, enterprise
> vendors, embedded vendors, and live-patching teams all have different risk
> tolerances, hardware exposure, CI capacity, reboot policies, module/signing
> workflows, and urgency. A single upstream x.y.z cadence cannot be the right
> integration boundary for all of them.

I see it the opposite way: those who don't want to follow stable branches
already don't wait for patches to land into -stable to have them, and
those who follow it will have even less reasons to trust aggregates on
which they have no feedback at all.

> In the recent 7.0 security/regression mess, the useful downstream question
> should not have been "which point release has Greg cut yet?" It should have
> been "which stable commits are required for the complete fix set, and has
> our actual distro kernel artifact built from that stable commit range passed
> enough testing to ship?"

So in fact you're just proposing to move the testing more on the distro
side hence to fragment it.

> If distros routinely consumed the stable branch as a signed linear stream,
> they could pin a specific stable commit, build from it, test it, and declare
> exactly what they shipped:

Given that nothing prevents them from doing this already, surely if they
don't do it it's because it's an extra effort that doesn't solve any
problem ?

> ```
> base: v7.0
> stable branch: linux-7.0.y
> stable commit: <hash>
> included range: v7.0..<hash>
> downstream patches/reverts: <list>
> ```
> 
> That is more auditable than chasing point releases or cherry-picking
> individual commits from intermediate states. It also makes clear that the
> distro package, not the upstream tarball, is the object that was actually
> tested and deployed.

That's already the case, usually they have different version numbering.

> There is also a practical artifact problem here. For long-lived stable
> series, publishing a complete source tarball for every x.y.z release is very
> wasteful. By 5.10.258, the 5.10.y series has hundreds of distinct tarballs,
> each over 100 MB compressed, representing tens of gigabytes of mostly
> duplicated source snapshots for one stable line. The real information is the
> base tree plus the incremental stable deltas. Git already represents that
> naturally.

That's an orthogonal aspect. Some tools are happy with the immediate
availability of these tarballs, and if they are ever considered wasteful,
they will probably stop being produced/archived or they might be deleted
after some time. But many users just download incremental patches.

All of this sounds very strange to me personally, more like a solution
looking for a problem.

Willy

Re: [RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model

[Greg Kroah-Hartman]

On Sun, Jun 14, 2026 at 08:16:06AM +0000, Artem S. Tashkinov wrote:
> Hi Greg,
> 
> Let me try to restate the proposal more narrowly, because I think my
> previous wording mixed several related issues together.
> 
> I am not arguing that stable.git is not already continuous. It obviously is.
> Nor am I arguing that cutting a point release is expensive for upstream. I
> understand that from the stable maintainer side, tagging a release is cheap
> and well automated.
> 
> The problem is that the ecosystem still treats the x.y.z point release as
> the main externally visible consumption boundary, even though it is neither
> the real upstream unit nor the real downstream deployment unit.

Not true at all, I would argue that this is the ONLY real deployment
unit as things are NOT tested in the stable releases except on that
boundry.  We do not do full testing on each individual commit, only at
the release points.  So to assume that you can consume a portion of the
stable release feed is not correct at all.

> The real upstream unit is the stable git branch: a linear sequence of
> accepted backports.
> 
> The real downstream deployment unit is a distro-built kernel package: base
> kernel, stable backports through some point in git history, distro
> patches/configuration, compiler/toolchain, signing, modules, CI, and
> user-visible packaging.
> 
> The x.y.z point release sits awkwardly in between. It is a useful
> compatibility marker for consumers that want that workflow, but it should
> not have to be the canonical boundary for everyone else.

Nope, sorry, unless you want to run all tests on each individual point,
this isn't going to change.

Also, you failed to answer my previous questions, like who is "we" that
is having problems with the current release process that is so difficult
to consume as-is?

thanks,

greg k-h