* Re: [RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model
2026-05-24 10:56 ` Greg Kroah-Hartman
@ 2026-05-25 0:33 ` Theodore Tso
2026-06-14 7:49 ` Artem S. Tashkinov
2026-06-14 8:16 ` Artem S. Tashkinov
2 siblings, 0 replies; 7+ messages in thread
From: Theodore Tso @ 2026-05-25 0:33 UTC (permalink / raw)
To: Greg Kroah-Hartman
Cc: Artem S. Tashkinov, Linux Kernel Mailing List, Sasha Levin,
stable
On Sun, May 24, 2026 at 12:56:54PM +0200, Greg Kroah-Hartman wrote:
> > point-release numbers can be replaced by
> > automated, time-stamped git tags and machine-generated source snapshots cut
> > on a strict, automated interval (e.g., every 48 hours), removing human
> > maintainers entirely from the release timing.
>
> That's probably not a good idea anyway. Are you doing continous testing
> of the stable queue? If so, great, just take from there today.
> Everyone adds patches on top of releases anyway, what's a few more if it
> happens to resolve specific issues for a day or so before a .y release
> can be cut?
It's already the case that not all maintainers have the time to test
the stable queue, and it's not clear that current testing of the
weekly release is all that great. I have seen stabilty regressions
where an xfstest running against ext4 will cause the kernel to crash
with the 6.1 and 6.6 LTS kernels. It took me several days to figure
out the 6.1 regression, and I still haven't had time to look into the
6.6 regression, because my day job (which is not ext4, but herding
cats for an AI infrastructure project --- it's amazing how many fellow
developers I met at LSF/MM are actually doing AI infrastructure
projects for $WORK, and not kernel development as their primary job
responsibilities.)
So even the weekly cadence is starting to creek a bit from a quality
perspective. I can't even *imagine* what a continuous, automated, "it
builds, ship it!" would do to the quality of the stable kernel series.
- Ted
P.S. If someone is interesting in helping to test ext4 and xfs stable
kernel patches, talk to me. There is partial automation to test
updates to the stable-rc trees, but I've never had time to automate
the rest of the test regression analysis combined with the automated
"which patches need to be backed out to avoid the regression / kernel
crash". There had been a few companies contributing fractions of
engineers to do XFS stable maintainenace, all of those resources have
been withdrawn by their respective companies in the past year.
>
> > ### Why This Benefits the Ecosystem
> >
> > * **Eliminates Churn and Latency:**
> >
> > When a patch introduces an edge-case regression or requires an immediate
> > follow-up (a common reason for rapid point-release sequences), maintainers
> > do not need to coordinate a whole new release event.
>
> No real "coordination" happens here.
>
> > The follow-up fix is simply patch $n+1$. Downstream CI pipelines
> > ingest it natively via standard git fetches.
>
> Again, we do that today.
>
> > * **Maintains Git-Native Debugging:**
> >
> > Debugging stable regressions via `git bisect` has always been patch-based,
> > not release-based. Since point releases are meant strictly for backported
> > bug fixes, removing the arbitrary `x.y.z` release tags changes nothing about
> > a developer's ability to isolate a regression. If anything, it prevents
> > downstream vendors from pulling out-of-order patches that complicate
> > bisection across distros.
>
> Who bisects across distros?
>
> > * **Eases Downstream Automation:**
> >
> > Modern tracking distributions (Arch, Fedora snapshotting, etc.) can switch
> > to trunk-based intake, automatically building from the signed tip.
>
> Have you asked them if they need/want this?
>
> > For enterprise distributions (RHEL, Ubuntu LTS) where constant kernel
> > packaging and reboots are untenable,
>
> Why are reboots for these systems untenable? Why not fix that root
> problem instead?
>
> > a fluid patch stream allows vendor
> > security teams to more rapidly feed live-patching infrastructure (`kpatch`,
> > `kgraft`), applying critical CVE fixes directly to runtime memory without
> > changing the base package version.
>
> They can do that today, and do do that today. So again, what distro
> needs this?
>
> > * **Bridges the Compliance Gap:**
> >
> > Embedded, automotive, or medical compliance pipelines
> > that legally require a static, verifiable code artifact can validate their
> > software against the base major release tarball ($7.0.0$) plus the
> > cryptographically signed, append-only stable patch series manifest.
>
> Do they really need that? Again, they can have that today, nothing new
> here.
>
> > The manual compilation, testing, and cutting of sub-version tarballs is an
> > administrative artifact of the late 1990s.
>
> Weekly releases is not an artivact of the 1990s :)
>
> > Shifting to an explicit, signed
> > patch-stream architecture acknowledges the velocity of modern vulnerability
> > research, strips away artificial latency, and frees our stable maintainers
> > to focus on code quality rather than release management overhead.
>
> Again, we have that today, on a weekly basis.
>
> greg k-h
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model
2026-05-24 10:56 ` Greg Kroah-Hartman
2026-05-25 0:33 ` Theodore Tso
@ 2026-06-14 7:49 ` Artem S. Tashkinov
2026-06-14 8:16 ` Artem S. Tashkinov
2 siblings, 0 replies; 7+ messages in thread
From: Artem S. Tashkinov @ 2026-06-14 7:49 UTC (permalink / raw)
To: Greg Kroah-Hartman; +Cc: Linux Kernel Mailing List, Sasha Levin, stable
On 5/24/26 10:56 AM, Greg Kroah-Hartman wrote:
> On Sun, May 24, 2026 at 01:38:55PM +0400, Artem S. Tashkinov wrote:
>> Hi all,
>>
>> The relentless cadence of critical vulnerability disclosures and public
>> exploits over the past month—including Copy Fail (CVE-2026-31431), Dirty
>> Frag (CVE-2026-43284/500), Fragnesia (CVE-2026-46300), and the ptrace exit
>> race (CVE-2026-46333)—has highlighted a severe structural bottleneck in how
>> we package and distribute stable backports.
>
> Who is "we"?
>
> And there's nothing really "new" here, these issues are all normal,
> remember, we resolve, on average, 13 CVEs a day, most much more severe
> than the ones that happened to get marketing names that you list here
> (and how many systems have untrusted users?)
>
>> When fatal logic flaws or memory corruptions strike core subsystems, our
>> current point-release model fractures. Spinning up whole new point releases
>> (7.0.4, 7.0.5, 7.0.7) in a matter of days just to address incomplete fixes,
>> subsystem regressions, or independent public disclosures (such as the recent
>> GRO managed-frag UAF exploit dropped directly to GitHub gists by
>> researchers) creates massive administrative fatigue for maintainers and
>> downstream teams alike.
>
> it takes just a minute to "spin up" a point release, what is difficult
> about that? If needed, just let us know and we can easily do so.
>
>> Upstream has long maintained that the stable tree is effectively a
>> continuous stream of fixes, and that users should track the tip of the
>> stable branch rather than cherry-picking. It is time our release
>> infrastructure matches this reality.
>>
>> ### The Proposal
>>
>> I propose transitioning the stable tree (`linux-x.y.y`) away from
>> manual,discrete point-release tarballs (`x.y.z`). Instead, we should treat
>> the stable sub-version purely as an append-only, continuous, git-native
>> patch stream.
>
> That's what we do today, we just happen go "jump" on a weekly basis.
>
>> Major releases (e.g., 7.0, 7.1) remain the foundational code boundaries, but
>> sub-versions are eliminated as monolithic manual artifacts.
>>
>> ### The Implementation: How It Works
>>
>> To ensure downstream distributions, enterprise compliance engines, and
>> automated testing rings can still securely ingest code, we can replace the
>> manual tarball with a decoupled, automated asset pipeline:
>>
>> 1. **The Git-First Stream:** The stable branch (`linux-7.0.y`) remains the
>> single source of truth. Commits are pushed as soon as they pass stable
>> criteria and automated sanity testing.
>
> Again, that's what we do today.
>
>> 2. **The Signed Patch-Stream Archive:** Instead of packaging the entire 30M+
>> line source code tree into a new tarball for every quick fix, upstream
>> infrastructure maintains a rolling, cumulative patch sequence for the major
>> cycle:
>>
>> linux-7.0-stable.series = \sum (patch_1 + patch_2 + ... + patch_n)
>>
>> Every time a fix is merged to the stable branch, the patch is appended to a
>> publicly accessible, cryptographically signed manifest file
>> (`linux-7.0-stable-patches.tar.bz2` or a standard `series` file) alongside a
>> detached signature.
>
> Who would use/need such a thing? What's wrong with the 2 systems we
> have today that this would somehow help out with?
>
>> 3. **Automated Snapshot Tags:** If the industry strictly requires an
>> immutable archive for compliance,
>
> What "compliance"?
>
>> point-release numbers can be replaced by
>> automated, time-stamped git tags and machine-generated source snapshots cut
>> on a strict, automated interval (e.g., every 48 hours), removing human
>> maintainers entirely from the release timing.
>
> That's probably not a good idea anyway. Are you doing continous testing
> of the stable queue? If so, great, just take from there today.
> Everyone adds patches on top of releases anyway, what's a few more if it
> happens to resolve specific issues for a day or so before a .y release
> can be cut?
>
>> ### Why This Benefits the Ecosystem
>>
>> * **Eliminates Churn and Latency:**
>>
>> When a patch introduces an edge-case regression or requires an immediate
>> follow-up (a common reason for rapid point-release sequences), maintainers
>> do not need to coordinate a whole new release event.
>
> No real "coordination" happens here.
>
>> The follow-up fix is simply patch $n+1$. Downstream CI pipelines
>> ingest it natively via standard git fetches.
>
> Again, we do that today.
>
>> * **Maintains Git-Native Debugging:**
>>
>> Debugging stable regressions via `git bisect` has always been patch-based,
>> not release-based. Since point releases are meant strictly for backported
>> bug fixes, removing the arbitrary `x.y.z` release tags changes nothing about
>> a developer's ability to isolate a regression. If anything, it prevents
>> downstream vendors from pulling out-of-order patches that complicate
>> bisection across distros.
>
> Who bisects across distros?
>
>> * **Eases Downstream Automation:**
>>
>> Modern tracking distributions (Arch, Fedora snapshotting, etc.) can switch
>> to trunk-based intake, automatically building from the signed tip.
>
> Have you asked them if they need/want this?
>
>> For enterprise distributions (RHEL, Ubuntu LTS) where constant kernel
>> packaging and reboots are untenable,
>
> Why are reboots for these systems untenable? Why not fix that root
> problem instead?
>
>> a fluid patch stream allows vendor
>> security teams to more rapidly feed live-patching infrastructure (`kpatch`,
>> `kgraft`), applying critical CVE fixes directly to runtime memory without
>> changing the base package version.
>
> They can do that today, and do do that today. So again, what distro
> needs this?
>
>> * **Bridges the Compliance Gap:**
>>
>> Embedded, automotive, or medical compliance pipelines
>> that legally require a static, verifiable code artifact can validate their
>> software against the base major release tarball ($7.0.0$) plus the
>> cryptographically signed, append-only stable patch series manifest.
>
> Do they really need that? Again, they can have that today, nothing new
> here.
>
>> The manual compilation, testing, and cutting of sub-version tarballs is an
>> administrative artifact of the late 1990s.
>
> Weekly releases is not an artivact of the 1990s :)
>
>> Shifting to an explicit, signed
>> patch-stream architecture acknowledges the velocity of modern vulnerability
>> research, strips away artificial latency, and frees our stable maintainers
>> to focus on code quality rather than release management overhead.
>
> Again, we have that today, on a weekly basis.
>
> greg k-h
Hi Greg,
I understand completely that from an upstream maintainer perspective,
the current pipeline feels seamless. Running a script to cut a tag takes
a minute, and git-stable is inherently fluid. But looking at the
ecosystem from the downstream and infrastructure side, the view is a bit
different.
Here is where the current model introduces friction that a patch-stream
or automated snapshot approach could solve:
### 1. The Compounding Downstream Tax
When you spin up a rapid sequence like 7.0.4, 7.0.5, and 7.0.7 to
address the immediate fallout of complex CVEs, it takes a minute for
upstream, but it triggers a massive domino effect downstream. Hundreds
of distribution mirrors sync gigabytes of redundant source data, package
maintainers rewrite specs, and automated build farms recreate full
packages. When a fix is incomplete or causes an immediate regression
requiring a follow-up version 48 hours later, that entire global compute
and human cycle repeats for what amounts to a few lines of diff.
### 2. The Infrastructure Reality of Reboots
You mentioned that systems should just be fixed so reboots aren't
untenable. In an ideal architectural world, yes. But in production
reality, orchestrating reboots across thousands of live cloud nodes
running high-availability workloads carries immense risk and scheduling
overhead.
Enterprise distros absolutely do use live-patching (`kpatch`/`kgraft`)
to mitigate this, but tracking a shifting landscape of discrete
point-release tarballs complicates their internal backport verification.
A continuous, signed patch-stream would give vendor security teams a
clean, linear ledger of upstream-approved deltas to feed directly into
live-patch compilation engines without the noise of full tree packaging.
### 3. Archive Bloat on the Mirror Network
Every stable point release requires a new ~130MB `.tar.xz` full source
archive. Multiplying that across multiple active LTS branches, dozens of
point releases, and hundreds of worldwide mirrors results in massive
data duplication just to distribute a cumulative handful of text diffs.
The ecosystem is essentially re-shipping the entire ocean every time we
need to add a cup of water.
### Why the Proposal Matters
When independent researchers drop working exploit primitives directly to
public gists (like the recent GRO managed-frag UAF), the turnaround time
to protected production systems needs to be near-zero.
The proposal wasn't meant to imply that the stable team isn't working
fast enough—you guys are incredibly fast. It was an observation that the
*delivery mechanism* (the discrete, human-timed tarball) forces
downstream consumers to choose between packaging fatigue or artificial
patch latency. Shifting the sub-version boundary to a machine-managed,
append-only signed patch stream bridges that gap.
I know you have a workflow that works for **you**, and I respect the
hell out of the massive volume of fixes you ship daily. I just wanted to
share the perspective of how that weekly "jump" ripples out to the
people who have to deploy it.
### Truly Massive Distro Churn Issue
There's also a very important downstream packaging issue that I think
the current stable-kernel release model underestimates.
Debian and Ubuntu already diverge from the upstream stable versioning
model in a useful way: they package kernels around their own ABI
versioning. In practice, this means they can often ship fixes without
forcing users to install an entirely new `/lib/modules/<kernel-version>`
tree every time an upstream stable micro-release appears. A new
ABI/package line is only needed when the kernel ABI they expose to
packaged or out-of-tree modules changes.
That model is a major downstream advantage. It avoids needless module
churn, avoids retaining multiple near-identical kernel trees, and makes
security/regression updates cheaper to ship. ABI-breaking changes do
happen, but in stable kernels they are rare enough that treating them as
exceptional events is perfectly reasonable.
My proposal takes this idea further.
Stable kernel releases frequently contain important fixes for
regressions introduced either in `.0` releases or in earlier stable
updates. But the current release cadence means that downstream
distributions often have to react before the next stable point release
exists. They either wait, leaving users exposed to known regressions, or
they cherry-pick individual commits from the stable queue. That creates
avoidable work: maintainers have to select fixes, update packaging
metadata, rebuild, test, and publish a distro-specific kernel update
that is supposed to correspond to a “stable” upstream series.
Fedora’s 7.0 cycle is a good example of the downstream churn this
creates: multiple downstream updates may be needed for what is logically
the same upstream stable series, simply because important fixes land
between formal stable releases.
Under the model I am proposing, stable would be treated less like a
sequence of isolated point releases and more like a continuously
updated, signed stable branch. Distributions could pin a specific signed
stable commit and trigger a rebuild from that point, instead of
maintaining their own ad hoc selection of fixes while waiting for the
next point release.
This would not remove the need for distro testing or for avoiding
known-bad commits. But in the common case it would simplify the workflow
enormously: downstreams would consume the stable branch directly,
rebuild from a known commit, and get all accepted stable fixes together.
Only in rare cases would they need to temporarily blacklist or revert a
problematic stable commit.
That is still much easier than every distribution independently deciding
which urgent fixes to cherry-pick from the stable queue.
The end result would be less downstream packaging churn, faster delivery
of regression fixes, fewer distro-specific stable-kernel deltas, and a
model that better reflects how many downstreams already consume stable
kernels in practice.
Best regards,
Artem
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [RFC/PROPOSAL] Shifting the x.y.z Stable Tree to a Continuous, Signed Patch-Stream Model
2026-05-24 10:56 ` Greg Kroah-Hartman
2026-05-25 0:33 ` Theodore Tso
2026-06-14 7:49 ` Artem S. Tashkinov
@ 2026-06-14 8:16 ` Artem S. Tashkinov
2026-06-14 9:33 ` Willy Tarreau
2026-06-14 11:19 ` Greg Kroah-Hartman
2 siblings, 2 replies; 7+ messages in thread
From: Artem S. Tashkinov @ 2026-06-14 8:16 UTC (permalink / raw)
To: Greg Kroah-Hartman; +Cc: Linux Kernel Mailing List, Sasha Levin, stable
Hi Greg,
Let me try to restate the proposal more narrowly, because I think my
previous wording mixed several related issues together.
I am not arguing that stable.git is not already continuous. It obviously
is. Nor am I arguing that cutting a point release is expensive for
upstream. I understand that from the stable maintainer side, tagging a
release is cheap and well automated.
The problem is that the ecosystem still treats the x.y.z point release
as the main externally visible consumption boundary, even though it is
neither the real upstream unit nor the real downstream deployment unit.
The real upstream unit is the stable git branch: a linear sequence of
accepted backports.
The real downstream deployment unit is a distro-built kernel package:
base kernel, stable backports through some point in git history, distro
patches/configuration, compiler/toolchain, signing, modules, CI, and
user-visible packaging.
The x.y.z point release sits awkwardly in between. It is a useful
compatibility marker for consumers that want that workflow, but it
should not have to be the canonical boundary for everyone else.
A better model, in my view, would be:
* stable.git remains the canonical source of truth;
* every downstream-consumable stable state is identified by branch +
signed commit ID, optionally with a signed machine-readable manifest;
* distros consume `v7.0 + stable commits through <commit>` or
`linux-7.0.y as of <date/commit>`, according to their own testing and
release policy;
* point releases may continue to exist for consumers that require them,
but they become one possible checkpoint format rather than the central
model.
This decouples two decisions that are currently conflated:
1. upstream’s decision that a commit belongs in the stable branch;
2. a downstream’s decision that a particular aggregate is ready to ship
to its users.
Those are not the same decision. Fedora, Arch, Debian, Ubuntu,
enterprise vendors, embedded vendors, and live-patching teams all have
different risk tolerances, hardware exposure, CI capacity, reboot
policies, module/signing workflows, and urgency. A single upstream x.y.z
cadence cannot be the right integration boundary for all of them.
In the recent 7.0 security/regression mess, the useful downstream
question should not have been “which point release has Greg cut yet?” It
should have been “which stable commits are required for the complete fix
set, and has our actual distro kernel artifact built from that stable
commit range passed enough testing to ship?”
If distros routinely consumed the stable branch as a signed linear
stream, they could pin a specific stable commit, build from it, test it,
and declare exactly what they shipped:
```
base: v7.0
stable branch: linux-7.0.y
stable commit: <hash>
included range: v7.0..<hash>
downstream patches/reverts: <list>
```
That is more auditable than chasing point releases or cherry-picking
individual commits from intermediate states. It also makes clear that
the distro package, not the upstream tarball, is the object that was
actually tested and deployed.
There is also a practical artifact problem here. For long-lived stable
series, publishing a complete source tarball for every x.y.z release is
very wasteful. By 5.10.258, the 5.10.y series has hundreds of distinct
tarballs, each over 100 MB compressed, representing tens of gigabytes of
mostly duplicated source snapshots for one stable line. The real
information is the base tree plus the incremental stable deltas. Git
already represents that naturally.
So I am not asking to remove point releases from users that still need
them. Keep them for conservative workflows, existing scripts,
announcements, and consumers that require a simple named snapshot.
What I am arguing is that point releases should no longer be the
privileged stable consumption model. The canonical model should be the
signed stable git stream, with downstreams free to define their own
tested integration points on that stream.
That would make the stable process match what it already is technically:
a continuous sequence of accepted backports, not a sequence of magic
tarball events.
Regards,
Artem
^ permalink raw reply [flat|nested] 7+ messages in thread