Re: [PATCH 5.2 00/20] 5.2.6-stable review

Re: [PATCH 5.2 00/20] 5.2.6-stable review

[Thierry Reding]

From: Thierry Reding <treding@nvidia.com>

On Fri, 02 Aug 2019 11:39:54 +0200, Greg Kroah-Hartman wrote:
> This is the start of the stable review cycle for the 5.2.6 release.
> There are 20 patches in this series, all will be posted as a response
> to this one.  If anyone has any issues with these being applied, please
> let me know.
> 
> Responses should be made by Sun 04 Aug 2019 09:19:34 AM UTC.
> Anything received after that time might be too late.
> 
> The whole patch series can be found in one patch at:
> 	https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.2.6-rc1.gz
> or in the git tree and branch at:
> 	git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.2.y
> and the diffstat can be found below.
> 
> thanks,
> 
> greg k-h

All tests passing for Tegra ...

Test results for stable-v5.2:
    12 builds:	12 pass, 0 fail
    22 boots:	22 pass, 0 fail
    38 tests:	38 pass, 0 fail

Linux version:	5.2.6-rc1-gbe893953fcc2
Boards tested:	tegra124-jetson-tk1, tegra186-p2771-0000,
                tegra194-p2972-0000, tegra20-ventana,
                tegra210-p2371-2180, tegra30-cardhu-a04

Thierry

Re: [PATCH 5.2 00/20] 5.2.6-stable review

[Greg Kroah-Hartman]

On Fri, Aug 02, 2019 at 07:21:05PM +0200, Thierry Reding wrote:
> From: Thierry Reding <treding@nvidia.com>
> 
> On Fri, 02 Aug 2019 11:39:54 +0200, Greg Kroah-Hartman wrote:
> > This is the start of the stable review cycle for the 5.2.6 release.
> > There are 20 patches in this series, all will be posted as a response
> > to this one.  If anyone has any issues with these being applied, please
> > let me know.
> > 
> > Responses should be made by Sun 04 Aug 2019 09:19:34 AM UTC.
> > Anything received after that time might be too late.
> > 
> > The whole patch series can be found in one patch at:
> > 	https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.2.6-rc1.gz
> > or in the git tree and branch at:
> > 	git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.2.y
> > and the diffstat can be found below.
> > 
> > thanks,
> > 
> > greg k-h
> 
> All tests passing for Tegra ...
> 
> Test results for stable-v5.2:
>     12 builds:	12 pass, 0 fail
>     22 boots:	22 pass, 0 fail
>     38 tests:	38 pass, 0 fail
> 
> Linux version:	5.2.6-rc1-gbe893953fcc2
> Boards tested:	tegra124-jetson-tk1, tegra186-p2771-0000,
>                 tegra194-p2972-0000, tegra20-ventana,
>                 tegra210-p2371-2180, tegra30-cardhu-a04

Great!  Thanks for testing all of these and letting me know.

greg k-h

Re: [PATCH 5.2 00/20] 5.2.6-stable review

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 3337 bytes --]

On Sat, Aug 03, 2019 at 09:09:32AM +0200, Greg Kroah-Hartman wrote:
> On Fri, Aug 02, 2019 at 07:21:05PM +0200, Thierry Reding wrote:
> > From: Thierry Reding <treding@nvidia.com>
> > 
> > On Fri, 02 Aug 2019 11:39:54 +0200, Greg Kroah-Hartman wrote:
> > > This is the start of the stable review cycle for the 5.2.6 release.
> > > There are 20 patches in this series, all will be posted as a response
> > > to this one.  If anyone has any issues with these being applied, please
> > > let me know.
> > > 
> > > Responses should be made by Sun 04 Aug 2019 09:19:34 AM UTC.
> > > Anything received after that time might be too late.
> > > 
> > > The whole patch series can be found in one patch at:
> > > 	https://www.kernel.org/pub/linux/kernel/v5.x/stable-review/patch-5.2.6-rc1.gz
> > > or in the git tree and branch at:
> > > 	git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git linux-5.2.y
> > > and the diffstat can be found below.
> > > 
> > > thanks,
> > > 
> > > greg k-h
> > 
> > All tests passing for Tegra ...
> > 
> > Test results for stable-v5.2:
> >     12 builds:	12 pass, 0 fail
> >     22 boots:	22 pass, 0 fail
> >     38 tests:	38 pass, 0 fail
> > 
> > Linux version:	5.2.6-rc1-gbe893953fcc2
> > Boards tested:	tegra124-jetson-tk1, tegra186-p2771-0000,
> >                 tegra194-p2972-0000, tegra20-ventana,
> >                 tegra210-p2371-2180, tegra30-cardhu-a04
> 
> Great!  Thanks for testing all of these and letting me know.

Hi Greg,

I stumbled across something as I was attempting to automate more parts
of our process to generate these reports. The original test results were
from a different version of the tree: 5.2.6-rc1-gdbc7f5c7df28. I suspect
that's the same thing that you were discussing with Pavel regarding the
IP tunnel patch that was added subsequent to the announcement.

Just for my understanding, does this mean that the patch still makes it
into the 5.2.6 release, or was it supposed to go into 5.2.7?

One problem that I ran into was that when I tried to correlate the test
results with your announcement email, there's no indication other than
the branch name and the release candidate name (5.2.6-rc1 in this case),
so there's no way to uniquely identify which test run belongs to the
announcement. Given that there are no tags for the release candidates
means that that's also not an option to uniquely associate with the
builds and tests.

While the differences between the two builds are very minor here, I
wonder if there could perhaps in the future be a problem where I report
successful results for a test, but the same tests would be broken by a
patch added to the stable-rc branch subsequent to the announcement. The
test report would be misleading in that case.

I noticed that you do add a couple of X-KernelTest-* headers to your
announcement emails, so I'm wondering if perhaps it was possible for you
to add another one that contains the exact SHA1 that corresponds to the
snapshot that's the release candidate. That would allow everyone to
uniquely associate test results with a specific release candidate.

That said, perhaps I've just got this all wrong and there's already a
way to connect all the dots that I'm not aware of. Or maybe I'm being
too pedantic here?

Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 5.2 00/20] 5.2.6-stable review

[Greg Kroah-Hartman]

On Mon, Aug 05, 2019 at 01:48:21PM +0200, Thierry Reding wrote:
> Hi Greg,

Sorry for the delay, this got pushed down my queue...

> I stumbled across something as I was attempting to automate more parts
> of our process to generate these reports. The original test results were
> from a different version of the tree: 5.2.6-rc1-gdbc7f5c7df28. I suspect
> that's the same thing that you were discussing with Pavel regarding the
> IP tunnel patch that was added subsequent to the announcement.
> 
> Just for my understanding, does this mean that the patch still makes it
> into the 5.2.6 release, or was it supposed to go into 5.2.7?
> 
> One problem that I ran into was that when I tried to correlate the test
> results with your announcement email, there's no indication other than
> the branch name and the release candidate name (5.2.6-rc1 in this case),
> so there's no way to uniquely identify which test run belongs to the
> announcement. Given that there are no tags for the release candidates
> means that that's also not an option to uniquely associate with the
> builds and tests.
> 
> While the differences between the two builds are very minor here, I
> wonder if there could perhaps in the future be a problem where I report
> successful results for a test, but the same tests would be broken by a
> patch added to the stable-rc branch subsequent to the announcement. The
> test report would be misleading in that case.
> 
> I noticed that you do add a couple of X-KernelTest-* headers to your
> announcement emails, so I'm wondering if perhaps it was possible for you
> to add another one that contains the exact SHA1 that corresponds to the
> snapshot that's the release candidate. That would allow everyone to
> uniquely associate test results with a specific release candidate.
> 
> That said, perhaps I've just got this all wrong and there's already a
> way to connect all the dots that I'm not aware of. Or maybe I'm being
> too pedantic here?

You aren't being pedantic, I think you are missing exactly what the
linux-stable-rc tree is for and how it is created.

Granted, it's not really documented anywhere so it's not your fault :)

The linux-stable-rc tree is there ONLY for people who want to test the
-rc kernels and can not, or do not want to, use the quilt tree of
patches in the stable-queue.git tree on kernel.org.  I generate the
branches there from a script that throws away the current -rc branch and
rebuilds it "from scratch" by applying the patches that are in the
stable-quilt tree and then adds the -rc patch as well.  This tree is
generated multiple times when I am working on the queues and then when I
want to do a "real" -rc release.

The branches are constantly rebased, so you can not rely on 'git pull'
doing the right thing (unless you add --rebase to it), and are there for
testing only.

Yes, you will see different results of a "-rc1" release in the tree
depending on the time of day/week when I created the tree, and sometimes
I generate them multiple times a day just to have some of the
auto-builders give me results quickly (Linaro does pull from it and
sends me results within the hour usually).

So does that help?  Ideally everyone would just use the quilt trees from
stable-queue.git, but no everyone likes that, so the -rc git tree is
there to make automated testing "easier".  If that works with your
workflow, wonderful, feel free to use it.  If not, then go with the
stable-quilt.git tree, or the tarballs on kernel.org.

And as for the SHA1 being in the emails, that doesn't make all that much
sense as that SHA1 doesn't live very long.  When I create the trees
locally, I instantly delete them after pushing them to kernel.org so I
can't regenerate them or do anything with them.

DOes that help explain things better?

thanks,

greg k-h

Re: [PATCH 5.2 00/20] 5.2.6-stable review

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 4690 bytes --]

On Fri, Aug 09, 2019 at 10:52:53AM +0200, Greg Kroah-Hartman wrote:
> On Mon, Aug 05, 2019 at 01:48:21PM +0200, Thierry Reding wrote:
> > Hi Greg,
> 
> Sorry for the delay, this got pushed down my queue...
> 
> > I stumbled across something as I was attempting to automate more parts
> > of our process to generate these reports. The original test results were
> > from a different version of the tree: 5.2.6-rc1-gdbc7f5c7df28. I suspect
> > that's the same thing that you were discussing with Pavel regarding the
> > IP tunnel patch that was added subsequent to the announcement.
> > 
> > Just for my understanding, does this mean that the patch still makes it
> > into the 5.2.6 release, or was it supposed to go into 5.2.7?
> > 
> > One problem that I ran into was that when I tried to correlate the test
> > results with your announcement email, there's no indication other than
> > the branch name and the release candidate name (5.2.6-rc1 in this case),
> > so there's no way to uniquely identify which test run belongs to the
> > announcement. Given that there are no tags for the release candidates
> > means that that's also not an option to uniquely associate with the
> > builds and tests.
> > 
> > While the differences between the two builds are very minor here, I
> > wonder if there could perhaps in the future be a problem where I report
> > successful results for a test, but the same tests would be broken by a
> > patch added to the stable-rc branch subsequent to the announcement. The
> > test report would be misleading in that case.
> > 
> > I noticed that you do add a couple of X-KernelTest-* headers to your
> > announcement emails, so I'm wondering if perhaps it was possible for you
> > to add another one that contains the exact SHA1 that corresponds to the
> > snapshot that's the release candidate. That would allow everyone to
> > uniquely associate test results with a specific release candidate.
> > 
> > That said, perhaps I've just got this all wrong and there's already a
> > way to connect all the dots that I'm not aware of. Or maybe I'm being
> > too pedantic here?
> 
> You aren't being pedantic, I think you are missing exactly what the
> linux-stable-rc tree is for and how it is created.
> 
> Granted, it's not really documented anywhere so it's not your fault :)
> 
> The linux-stable-rc tree is there ONLY for people who want to test the
> -rc kernels and can not, or do not want to, use the quilt tree of
> patches in the stable-queue.git tree on kernel.org.  I generate the
> branches there from a script that throws away the current -rc branch and
> rebuilds it "from scratch" by applying the patches that are in the
> stable-quilt tree and then adds the -rc patch as well.  This tree is
> generated multiple times when I am working on the queues and then when I
> want to do a "real" -rc release.
> 
> The branches are constantly rebased, so you can not rely on 'git pull'
> doing the right thing (unless you add --rebase to it), and are there for
> testing only.
> 
> Yes, you will see different results of a "-rc1" release in the tree
> depending on the time of day/week when I created the tree, and sometimes
> I generate them multiple times a day just to have some of the
> auto-builders give me results quickly (Linaro does pull from it and
> sends me results within the hour usually).
> 
> So does that help?  Ideally everyone would just use the quilt trees from
> stable-queue.git, but no everyone likes that, so the -rc git tree is
> there to make automated testing "easier".  If that works with your
> workflow, wonderful, feel free to use it.  If not, then go with the
> stable-quilt.git tree, or the tarballs on kernel.org.

I'll have to look into that, to see if that'd work. I have to admit that
having a git tree to point scripts at is rather convenient for
automation.

> And as for the SHA1 being in the emails, that doesn't make all that much
> sense as that SHA1 doesn't live very long.  When I create the trees
> locally, I instantly delete them after pushing them to kernel.org so I
> can't regenerate them or do anything with them.

Fair enough. I suppose the worst thing that could happen is that we may
fail to test a couple of commits occasionally. In the rare case where
this actually matters we'll likely end up reporting the failure for the
stable release, in which case it can be fixed in the next one.

> DOes that help explain things better?

Yes, makes a lot more sense now. Thanks for taking the time to explain
it. Do you want me to snapshot this and submit it as documentation
somewhere for later reference?

Thierry

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 5.2 00/20] 5.2.6-stable review

[Greg Kroah-Hartman]

On Fri, Aug 09, 2019 at 03:04:49PM +0200, Thierry Reding wrote:
> On Fri, Aug 09, 2019 at 10:52:53AM +0200, Greg Kroah-Hartman wrote:
> > On Mon, Aug 05, 2019 at 01:48:21PM +0200, Thierry Reding wrote:
> > > Hi Greg,
> > 
> > Sorry for the delay, this got pushed down my queue...
> > 
> > > I stumbled across something as I was attempting to automate more parts
> > > of our process to generate these reports. The original test results were
> > > from a different version of the tree: 5.2.6-rc1-gdbc7f5c7df28. I suspect
> > > that's the same thing that you were discussing with Pavel regarding the
> > > IP tunnel patch that was added subsequent to the announcement.
> > > 
> > > Just for my understanding, does this mean that the patch still makes it
> > > into the 5.2.6 release, or was it supposed to go into 5.2.7?
> > > 
> > > One problem that I ran into was that when I tried to correlate the test
> > > results with your announcement email, there's no indication other than
> > > the branch name and the release candidate name (5.2.6-rc1 in this case),
> > > so there's no way to uniquely identify which test run belongs to the
> > > announcement. Given that there are no tags for the release candidates
> > > means that that's also not an option to uniquely associate with the
> > > builds and tests.
> > > 
> > > While the differences between the two builds are very minor here, I
> > > wonder if there could perhaps in the future be a problem where I report
> > > successful results for a test, but the same tests would be broken by a
> > > patch added to the stable-rc branch subsequent to the announcement. The
> > > test report would be misleading in that case.
> > > 
> > > I noticed that you do add a couple of X-KernelTest-* headers to your
> > > announcement emails, so I'm wondering if perhaps it was possible for you
> > > to add another one that contains the exact SHA1 that corresponds to the
> > > snapshot that's the release candidate. That would allow everyone to
> > > uniquely associate test results with a specific release candidate.
> > > 
> > > That said, perhaps I've just got this all wrong and there's already a
> > > way to connect all the dots that I'm not aware of. Or maybe I'm being
> > > too pedantic here?
> > 
> > You aren't being pedantic, I think you are missing exactly what the
> > linux-stable-rc tree is for and how it is created.
> > 
> > Granted, it's not really documented anywhere so it's not your fault :)
> > 
> > The linux-stable-rc tree is there ONLY for people who want to test the
> > -rc kernels and can not, or do not want to, use the quilt tree of
> > patches in the stable-queue.git tree on kernel.org.  I generate the
> > branches there from a script that throws away the current -rc branch and
> > rebuilds it "from scratch" by applying the patches that are in the
> > stable-quilt tree and then adds the -rc patch as well.  This tree is
> > generated multiple times when I am working on the queues and then when I
> > want to do a "real" -rc release.
> > 
> > The branches are constantly rebased, so you can not rely on 'git pull'
> > doing the right thing (unless you add --rebase to it), and are there for
> > testing only.
> > 
> > Yes, you will see different results of a "-rc1" release in the tree
> > depending on the time of day/week when I created the tree, and sometimes
> > I generate them multiple times a day just to have some of the
> > auto-builders give me results quickly (Linaro does pull from it and
> > sends me results within the hour usually).
> > 
> > So does that help?  Ideally everyone would just use the quilt trees from
> > stable-queue.git, but no everyone likes that, so the -rc git tree is
> > there to make automated testing "easier".  If that works with your
> > workflow, wonderful, feel free to use it.  If not, then go with the
> > stable-quilt.git tree, or the tarballs on kernel.org.
> 
> I'll have to look into that, to see if that'd work. I have to admit that
> having a git tree to point scripts at is rather convenient for
> automation.
> 
> > And as for the SHA1 being in the emails, that doesn't make all that much
> > sense as that SHA1 doesn't live very long.  When I create the trees
> > locally, I instantly delete them after pushing them to kernel.org so I
> > can't regenerate them or do anything with them.
> 
> Fair enough. I suppose the worst thing that could happen is that we may
> fail to test a couple of commits occasionally. In the rare case where
> this actually matters we'll likely end up reporting the failure for the
> stable release, in which case it can be fixed in the next one.
> 
> > DOes that help explain things better?
> 
> Yes, makes a lot more sense now. Thanks for taking the time to explain
> it. Do you want me to snapshot this and submit it as documentation
> somewhere for later reference?

It probably should go somewhere, but as the number of people that do
"test stable kernels in an automated way" is very low, so far I've been
doing ok with a one-by-one explaination.  I guess if it's more obvious,
more people would test, so sure, this should be cleaned up...

thanks,

greg k-h

Re: [PATCH 5.2 00/20] 5.2.6-stable review

[Thierry Reding]

[-- Attachment #1: Type: text/plain, Size: 7880 bytes --]

On Fri, Aug 09, 2019 at 05:49:59PM +0200, Greg Kroah-Hartman wrote:
> On Fri, Aug 09, 2019 at 03:04:49PM +0200, Thierry Reding wrote:
> > On Fri, Aug 09, 2019 at 10:52:53AM +0200, Greg Kroah-Hartman wrote:
> > > On Mon, Aug 05, 2019 at 01:48:21PM +0200, Thierry Reding wrote:
> > > > Hi Greg,
> > > 
> > > Sorry for the delay, this got pushed down my queue...
> > > 
> > > > I stumbled across something as I was attempting to automate more parts
> > > > of our process to generate these reports. The original test results were
> > > > from a different version of the tree: 5.2.6-rc1-gdbc7f5c7df28. I suspect
> > > > that's the same thing that you were discussing with Pavel regarding the
> > > > IP tunnel patch that was added subsequent to the announcement.
> > > > 
> > > > Just for my understanding, does this mean that the patch still makes it
> > > > into the 5.2.6 release, or was it supposed to go into 5.2.7?
> > > > 
> > > > One problem that I ran into was that when I tried to correlate the test
> > > > results with your announcement email, there's no indication other than
> > > > the branch name and the release candidate name (5.2.6-rc1 in this case),
> > > > so there's no way to uniquely identify which test run belongs to the
> > > > announcement. Given that there are no tags for the release candidates
> > > > means that that's also not an option to uniquely associate with the
> > > > builds and tests.
> > > > 
> > > > While the differences between the two builds are very minor here, I
> > > > wonder if there could perhaps in the future be a problem where I report
> > > > successful results for a test, but the same tests would be broken by a
> > > > patch added to the stable-rc branch subsequent to the announcement. The
> > > > test report would be misleading in that case.
> > > > 
> > > > I noticed that you do add a couple of X-KernelTest-* headers to your
> > > > announcement emails, so I'm wondering if perhaps it was possible for you
> > > > to add another one that contains the exact SHA1 that corresponds to the
> > > > snapshot that's the release candidate. That would allow everyone to
> > > > uniquely associate test results with a specific release candidate.
> > > > 
> > > > That said, perhaps I've just got this all wrong and there's already a
> > > > way to connect all the dots that I'm not aware of. Or maybe I'm being
> > > > too pedantic here?
> > > 
> > > You aren't being pedantic, I think you are missing exactly what the
> > > linux-stable-rc tree is for and how it is created.
> > > 
> > > Granted, it's not really documented anywhere so it's not your fault :)
> > > 
> > > The linux-stable-rc tree is there ONLY for people who want to test the
> > > -rc kernels and can not, or do not want to, use the quilt tree of
> > > patches in the stable-queue.git tree on kernel.org.  I generate the
> > > branches there from a script that throws away the current -rc branch and
> > > rebuilds it "from scratch" by applying the patches that are in the
> > > stable-quilt tree and then adds the -rc patch as well.  This tree is
> > > generated multiple times when I am working on the queues and then when I
> > > want to do a "real" -rc release.
> > > 
> > > The branches are constantly rebased, so you can not rely on 'git pull'
> > > doing the right thing (unless you add --rebase to it), and are there for
> > > testing only.
> > > 
> > > Yes, you will see different results of a "-rc1" release in the tree
> > > depending on the time of day/week when I created the tree, and sometimes
> > > I generate them multiple times a day just to have some of the
> > > auto-builders give me results quickly (Linaro does pull from it and
> > > sends me results within the hour usually).
> > > 
> > > So does that help?  Ideally everyone would just use the quilt trees from
> > > stable-queue.git, but no everyone likes that, so the -rc git tree is
> > > there to make automated testing "easier".  If that works with your
> > > workflow, wonderful, feel free to use it.  If not, then go with the
> > > stable-quilt.git tree, or the tarballs on kernel.org.
> > 
> > I'll have to look into that, to see if that'd work. I have to admit that
> > having a git tree to point scripts at is rather convenient for
> > automation.
> > 
> > > And as for the SHA1 being in the emails, that doesn't make all that much
> > > sense as that SHA1 doesn't live very long.  When I create the trees
> > > locally, I instantly delete them after pushing them to kernel.org so I
> > > can't regenerate them or do anything with them.
> > 
> > Fair enough. I suppose the worst thing that could happen is that we may
> > fail to test a couple of commits occasionally. In the rare case where
> > this actually matters we'll likely end up reporting the failure for the
> > stable release, in which case it can be fixed in the next one.
> > 
> > > DOes that help explain things better?
> > 
> > Yes, makes a lot more sense now. Thanks for taking the time to explain
> > it. Do you want me to snapshot this and submit it as documentation
> > somewhere for later reference?
> 
> It probably should go somewhere, but as the number of people that do
> "test stable kernels in an automated way" is very low, so far I've been
> doing ok with a one-by-one explaination.  I guess if it's more obvious,
> more people would test, so sure, this should be cleaned up...

How about something like the below. It applies to the stable-queue.git
repository.

Thierry

--- >8 ---
From 4083add6ccb4a1c23edeba650170470bcc5d5205 Mon Sep 17 00:00:00 2001
From: Thierry Reding <treding@nvidia.com>
Date: Mon, 12 Aug 2019 10:58:35 +0200
Subject: [PATCH] Describe the stable-queue release process

Add a README file that describes how release candidates for stable
kernels are created and how users are expected to use them. This is
reworded from Greg's reply here:

	https://lore.kernel.org/stable/20190809085253.GA25046@kroah.com/
---
 README | 31 +++++++++++++++++++++++++++++++
 1 file changed, 31 insertions(+)
 create mode 100644 README

diff --git a/README b/README
new file mode 100644
index 000000000000..868469a73f68
--- /dev/null
+++ b/README
@@ -0,0 +1,31 @@
+This repository is the canonical source for patches that make up the stable
+kernel releases. It consists of a set of directories for each of the stable
+kernels, as well as a directory that contains a snapshot of the patches for
+each stable release.
+
+The patches for each release can be found along with a complete tarball of
+a release in the following location:
+
+	https://kernel.org/pub/linux/kernel/vX.Y/
+
+For each stable release candidate, a patch representing the diff of all the
+patches in the stable queue is uploaded here:
+
+	https://kernel.org/pub/linux/kernel/vX.Y/stable-review/
+
+As a convenience for people that want to test release candidates of stable
+releases, a branch of the kernel git tree is created containing all of the
+patches in the given stable queue. These branches are available in the
+following repository:
+
+	git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
+
+A branch exists for each of the stable releases. Note, though, that these
+branches are recreated from scratch by applying the queued stable patches
+on top of the prior release. As a consequence, the branches are not fast-
+forward and can change after a release candidate has been announced. The
+contents of the branch may therefore not match exactly what was released
+as the release candidate, depending on when you fetch it. No tags are
+created to track individual release candidates. If you're interested in
+exact reproducibility of a stable release candidate, please use the patches
+from the location mentioned above.
-- 
2.22.0


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 5.2 00/20] 5.2.6-stable review

[Greg Kroah-Hartman]

On Mon, Aug 12, 2019 at 11:05:53AM +0200, Thierry Reding wrote:
> On Fri, Aug 09, 2019 at 05:49:59PM +0200, Greg Kroah-Hartman wrote:
> > On Fri, Aug 09, 2019 at 03:04:49PM +0200, Thierry Reding wrote:
> > > On Fri, Aug 09, 2019 at 10:52:53AM +0200, Greg Kroah-Hartman wrote:
> > > > On Mon, Aug 05, 2019 at 01:48:21PM +0200, Thierry Reding wrote:
> > > > > Hi Greg,
> > > > 
> > > > Sorry for the delay, this got pushed down my queue...
> > > > 
> > > > > I stumbled across something as I was attempting to automate more parts
> > > > > of our process to generate these reports. The original test results were
> > > > > from a different version of the tree: 5.2.6-rc1-gdbc7f5c7df28. I suspect
> > > > > that's the same thing that you were discussing with Pavel regarding the
> > > > > IP tunnel patch that was added subsequent to the announcement.
> > > > > 
> > > > > Just for my understanding, does this mean that the patch still makes it
> > > > > into the 5.2.6 release, or was it supposed to go into 5.2.7?
> > > > > 
> > > > > One problem that I ran into was that when I tried to correlate the test
> > > > > results with your announcement email, there's no indication other than
> > > > > the branch name and the release candidate name (5.2.6-rc1 in this case),
> > > > > so there's no way to uniquely identify which test run belongs to the
> > > > > announcement. Given that there are no tags for the release candidates
> > > > > means that that's also not an option to uniquely associate with the
> > > > > builds and tests.
> > > > > 
> > > > > While the differences between the two builds are very minor here, I
> > > > > wonder if there could perhaps in the future be a problem where I report
> > > > > successful results for a test, but the same tests would be broken by a
> > > > > patch added to the stable-rc branch subsequent to the announcement. The
> > > > > test report would be misleading in that case.
> > > > > 
> > > > > I noticed that you do add a couple of X-KernelTest-* headers to your
> > > > > announcement emails, so I'm wondering if perhaps it was possible for you
> > > > > to add another one that contains the exact SHA1 that corresponds to the
> > > > > snapshot that's the release candidate. That would allow everyone to
> > > > > uniquely associate test results with a specific release candidate.
> > > > > 
> > > > > That said, perhaps I've just got this all wrong and there's already a
> > > > > way to connect all the dots that I'm not aware of. Or maybe I'm being
> > > > > too pedantic here?
> > > > 
> > > > You aren't being pedantic, I think you are missing exactly what the
> > > > linux-stable-rc tree is for and how it is created.
> > > > 
> > > > Granted, it's not really documented anywhere so it's not your fault :)
> > > > 
> > > > The linux-stable-rc tree is there ONLY for people who want to test the
> > > > -rc kernels and can not, or do not want to, use the quilt tree of
> > > > patches in the stable-queue.git tree on kernel.org.  I generate the
> > > > branches there from a script that throws away the current -rc branch and
> > > > rebuilds it "from scratch" by applying the patches that are in the
> > > > stable-quilt tree and then adds the -rc patch as well.  This tree is
> > > > generated multiple times when I am working on the queues and then when I
> > > > want to do a "real" -rc release.
> > > > 
> > > > The branches are constantly rebased, so you can not rely on 'git pull'
> > > > doing the right thing (unless you add --rebase to it), and are there for
> > > > testing only.
> > > > 
> > > > Yes, you will see different results of a "-rc1" release in the tree
> > > > depending on the time of day/week when I created the tree, and sometimes
> > > > I generate them multiple times a day just to have some of the
> > > > auto-builders give me results quickly (Linaro does pull from it and
> > > > sends me results within the hour usually).
> > > > 
> > > > So does that help?  Ideally everyone would just use the quilt trees from
> > > > stable-queue.git, but no everyone likes that, so the -rc git tree is
> > > > there to make automated testing "easier".  If that works with your
> > > > workflow, wonderful, feel free to use it.  If not, then go with the
> > > > stable-quilt.git tree, or the tarballs on kernel.org.
> > > 
> > > I'll have to look into that, to see if that'd work. I have to admit that
> > > having a git tree to point scripts at is rather convenient for
> > > automation.
> > > 
> > > > And as for the SHA1 being in the emails, that doesn't make all that much
> > > > sense as that SHA1 doesn't live very long.  When I create the trees
> > > > locally, I instantly delete them after pushing them to kernel.org so I
> > > > can't regenerate them or do anything with them.
> > > 
> > > Fair enough. I suppose the worst thing that could happen is that we may
> > > fail to test a couple of commits occasionally. In the rare case where
> > > this actually matters we'll likely end up reporting the failure for the
> > > stable release, in which case it can be fixed in the next one.
> > > 
> > > > DOes that help explain things better?
> > > 
> > > Yes, makes a lot more sense now. Thanks for taking the time to explain
> > > it. Do you want me to snapshot this and submit it as documentation
> > > somewhere for later reference?
> > 
> > It probably should go somewhere, but as the number of people that do
> > "test stable kernels in an automated way" is very low, so far I've been
> > doing ok with a one-by-one explaination.  I guess if it's more obvious,
> > more people would test, so sure, this should be cleaned up...
> 
> How about something like the below. It applies to the stable-queue.git
> repository.
> 
> Thierry
> 
> --- >8 ---
> From 4083add6ccb4a1c23edeba650170470bcc5d5205 Mon Sep 17 00:00:00 2001
> From: Thierry Reding <treding@nvidia.com>
> Date: Mon, 12 Aug 2019 10:58:35 +0200
> Subject: [PATCH] Describe the stable-queue release process
> 
> Add a README file that describes how release candidates for stable
> kernels are created and how users are expected to use them. This is
> reworded from Greg's reply here:
> 
> 	https://lore.kernel.org/stable/20190809085253.GA25046@kroah.com/
> ---
>  README | 31 +++++++++++++++++++++++++++++++
>  1 file changed, 31 insertions(+)
>  create mode 100644 README
> 
> diff --git a/README b/README
> new file mode 100644
> index 000000000000..868469a73f68
> --- /dev/null
> +++ b/README
> @@ -0,0 +1,31 @@
> +This repository is the canonical source for patches that make up the stable
> +kernel releases. It consists of a set of directories for each of the stable
> +kernels, as well as a directory that contains a snapshot of the patches for
> +each stable release.
> +
> +The patches for each release can be found along with a complete tarball of
> +a release in the following location:
> +
> +	https://kernel.org/pub/linux/kernel/vX.Y/
> +
> +For each stable release candidate, a patch representing the diff of all the
> +patches in the stable queue is uploaded here:
> +
> +	https://kernel.org/pub/linux/kernel/vX.Y/stable-review/
> +
> +As a convenience for people that want to test release candidates of stable
> +releases, a branch of the kernel git tree is created containing all of the
> +patches in the given stable queue. These branches are available in the
> +following repository:
> +
> +	git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git
> +
> +A branch exists for each of the stable releases. Note, though, that these
> +branches are recreated from scratch by applying the queued stable patches
> +on top of the prior release. As a consequence, the branches are not fast-
> +forward and can change after a release candidate has been announced. The
> +contents of the branch may therefore not match exactly what was released
> +as the release candidate, depending on when you fetch it. No tags are
> +created to track individual release candidates. If you're interested in
> +exact reproducibility of a stable release candidate, please use the patches
> +from the location mentioned above.
> -- 

Sorry for the very long delay, cleaning out old emails...

This looks really good, thanks for writing it up, I've now applied it to
the stable-queue tree.

greg k-h