Changing checksums of RT patches

Changing checksums of RT patches

[Dominic Sacré]

Hi,

I noticed that recently the compressed RT patches (.patch.gz) at
https://www.kernel.org/pub/linux/kernel/projects/rt started changing.
The actual content of the patches seems to remain unchanged, but it
looks like the gzipped files are being regenerated, resulting in
different checksums.

This is a problem for example for OpenEmbedded-based projects, which use
MD5 and SHA256 checksums (basically hardcoded into Bitbake recipes) to
verify the integrity of files being downloaded.

Does someone know why this is happening?


Thanks,

Dominic

Re: Changing checksums of RT patches

[Sebastian Andrzej Siewior]

On 2017-02-09 10:09:16 [+0100], Dominic Sacré wrote:
> Hi,
Hi,

> I noticed that recently the compressed RT patches (.patch.gz) at
> https://www.kernel.org/pub/linux/kernel/projects/rt started changing.
> The actual content of the patches seems to remain unchanged, but it
> looks like the gzipped files are being regenerated, resulting in
> different checksums.

Can you tell me which file in which folder changed?
Plus I suggest to use
 https://cdn.kernel.org/pub/linux/kernel/projects/rt
which is mostly the same content but should lower the load on the
"origin" servers.

> This is a problem for example for OpenEmbedded-based projects, which use
> MD5 and SHA256 checksums (basically hardcoded into Bitbake recipes) to
> verify the integrity of files being downloaded.

I see. You should be able to verify the uncompressed patch against a gpg
key. However I guess this is not how OpenEmbedded works.

> Does someone know why this is happening?

So if you tell me which file changed I will ask around and let you know.

> Thanks,
> 
> Dominic

Sebastian

Re: Changing checksums of RT patches

[Dominic Sacré]

On 2017-02-10 19:54, Sebastian Andrzej Siewior wrote:
> On 2017-02-09 10:09:16 [+0100], Dominic Sacré wrote:
>> I noticed that recently the compressed RT patches (.patch.gz) at
>> https://www.kernel.org/pub/linux/kernel/projects/rt started changing.
>> The actual content of the patches seems to remain unchanged, but it
>> looks like the gzipped files are being regenerated, resulting in
>> different checksums.
> 
> Can you tell me which file in which folder changed?

The files I'm currently concerned with are in the rt/4.1/older directory.

For example, the Bitbake recipe I'm working on
(https://github.com/Freescale/meta-freescale/blob/master/recipes-kernel/linux/linux-fslc-imx-rt_4.1-2.0.x.bb)
used the file rt/4.1/older/patch-4.1.35-rt41.patch.gz. At some point
that file changed, so the recipe would fail to build from scratch (I
don't know when exactly or how often the file changed).

I then updated the recipe to a later kernel version, using
patch-4.1.37-rt43.patch.gz instead. Two days later I noticed builds
starting to fail again, because that file too had changed, *after* I had
updated the recipe.

>> This is a problem for example for OpenEmbedded-based projects, which use
>> MD5 and SHA256 checksums (basically hardcoded into Bitbake recipes) to
>> verify the integrity of files being downloaded.
> 
> I see. You should be able to verify the uncompressed patch against a gpg
> key. However I guess this is not how OpenEmbedded works.

Bitbake/OpenEmbedded uses only SHA256 and MD5 checksums, and I don't
think I can get it to check the hashes after uncompressing the patches.


Thanks,

Dominic

Re: Changing checksums of RT patches

[Dominic Sacré]

[-- Attachment #1: Type: text/plain, Size: 491 bytes --]

On 2017-02-13 23:56, Dominic Sacré wrote:
> On 2017-02-10 19:54, Sebastian Andrzej Siewior wrote:
>> Can you tell me which file in which folder changed?
> 
> The files I'm currently concerned with are in the rt/4.1/older directory.

While I was writing my previous mail, 4.1.38-rt45 was released. Attached
is a diff between rt/4.1/older/sha256sums.asc before (4 days ago) and
after -rt45 was added.
Note how the gzip-compressed files for 4.1.35-rt40 and 4.1.38-rt44
changed in the meantime.

[-- Attachment #2: sha256sums.asc.diff --]
[-- Type: text/x-patch, Size: 4019 bytes --]

--- sha256sums.asc.orig	2017-02-14 00:07:29.727603816 +0100
+++ sha256sums.asc	2017-02-14 00:00:20.757619931 +0100
@@ -89,9 +89,9 @@
 840e9d8f6d310b33d0878eb9680e824a7eb0c52ac279c2eb501e30fc6e0c62d8  patch-4.1.34-rt39.patch.xz
 a4ba7d5df224d743e1f8b1422172a81dc0ff2e222809cf8820ccb934b6a7fc80  patches-4.1.34-rt39.tar.gz
 80d6ed60aeb166827c40a83f6f556def2dd0768142793947f3ea63e749863d5d  patches-4.1.34-rt39.tar.xz
-0393fcda51fc76ffa5eeb60344a0b60c96eeefe965dee1b94cdd03a0919bfc4f  patch-4.1.35-rt40.patch.gz
+cac1fc90758d942caa89cce1ba7e9e4ac6eab61bc5006d4e7641451ace345173  patch-4.1.35-rt40.patch.gz
 109032535d63aeac66cc2a76a24c98647bcb93a367a731350f6897397a55a703  patch-4.1.35-rt40.patch.xz
-b102fa127ff6650454ccf4a18544855f790959f967ad9fb56a172ab465cfabdd  patches-4.1.35-rt40.tar.gz
+a509fc7b09d2f1e196c347126be6f03ee66530baf00ae89f6a80a5f62dfa564e  patches-4.1.35-rt40.tar.gz
 9200687e7d104b48ec9326833b904708a30add9d8a8e2ade047c41f10a43a04c  patches-4.1.35-rt40.tar.xz
 284a1bc0094df0a61e6dcb9996eceea6a3791ccba1e5763e36f251d0dfeecd32  patch-4.1.35-rt41.patch.gz
 af1d65e1a9560b06599a0188a5c971ff116919473453565e2ab53dc7b1f80945  patch-4.1.35-rt41.patch.xz
@@ -105,24 +105,28 @@
 8c1a5064bcbcdd6aa0e613cf3cd0b2e3558854da5808d66b32950ebf900f034b  patch-4.1.37-rt43.patch.xz
 cf51d66c0532a002ae8d55d32bc39bc2bacea51593f32f3b4bd4830f75069b05  patches-4.1.37-rt43.tar.gz
 badde46bb4fbbeeee13a6457726a25a083da2d3b390fd8f5bcdbfd4034d87b92  patches-4.1.37-rt43.tar.xz
-646df5300a5180f7201284ab5ca20fb6ac868b64b3f780a4950ea3b5b8595a09  patch-4.1.38-rt44.patch.gz
+98730eeee49a07e555a83ae92f7eee7bb7596eb5967b121f3a9d7bf119b95c5a  patch-4.1.38-rt44.patch.gz
 14fb957835abd1ef9106561e7866ef2be94e1b02d83dd78a02d5c0d4204fed31  patch-4.1.38-rt44.patch.xz
-9f0442a699ea52b9c3364e1237181f0a746cfa56d0d761a1212036c272f069ed  patches-4.1.38-rt44.tar.gz
+fb67487ad230716a7b75d5beb8449efe0bcd9d5edbb4e59aca842ad6d38c886e  patches-4.1.38-rt44.tar.gz
 66a74b1792367806d33a7081f723cf1234c595eb8de57fadfe1456a1f238ab53  patches-4.1.38-rt44.tar.xz
+7b2e9f5ec78f508fda1f439fef08e1ee2d81e137b82cf40e62c40def781c7d41  patch-4.1.38-rt45.patch.gz
+fb2cc71641e7078509cda637f00a5964c6bd57061639919b4f1f18a07db24e4a  patch-4.1.38-rt45.patch.xz
+418268844d353bb5c7194d664440f2e89175c629bba376cda1892c9e727fae7b  patches-4.1.38-rt45.tar.gz
+67aead5194a0b1276111a47d4c76a5f6502e8cb4c852eded9d2d21876560bb85  patches-4.1.38-rt45.tar.xz
 -----BEGIN PGP SIGNATURE-----
 Version: GnuPG v1.4.12 (GNU/Linux)
 
-iQIcBAEBCAAGBQJYmzHoAAoJEGMtOgZYnaaxub0QAIUPnG0xxDuhQ/HpFwkDcLq3
-tLFO933qa6TyuR0OeVQHhLpWvtaQensZcqQ9nrrPINlmU8a4N3vBSNblaxG62GBu
-KaVI9KqPskkEw9D9pxyKDlid9wYbi57Y5bYn6s5PAFCFv3kVCfV+65jTwH2MZWMp
-oAC8Y8Jb/HWmFmy9i/6g4dS/qaJ3BdSOYRsX8QKIb1Orbne7UE8jLl2lBF/DUXgz
-tbZPUjp798MTfcg4Z1owCrSaXmUWQp0tFare87WHPTEn4TEbO7qJkbS5WHl+JpQK
-zprqeEN26kXj+MChK23RMLudbYgggdLSuf29MfufW/ESzrb/me1SDqRLmRz2jpna
-c2MkmMzT3SMf8hDujzQiiZ5oJ6Vexj1e+lkGdC3gsPSfhwwTsvJrvJxQUF7mDA1v
-CeEFNMqR3ygzvSTgz9N/CB56nL/HMK9ymZLB8Dcyhv2LyFfPezbZFZXF8jarQLV6
-Z2Rfikg+/94YACM78LKONajv+JB6mjsnIOXJ/3GV/07QrK5hBwLRXsKe843b72+x
-YyzPTsbI1gyiSHJFUolieNxnv1Bs+O+JaKTGG4MxKlDMz3/Dhxt5TMX2WjrZUtR7
-XwdZQtJo7mM34okKz8GyBib6vAgMtZyZaDg63aYs0fJGs6y+ByQnrhBK2XHoRAfZ
-003O5TS9BelNYrpL4lr6
-=iqvs
+iQIcBAEBCAAGBQJYojCAAAoJEGMtOgZYnaax+HoQAIxcyvuFvaW8ZY8EJT35CUWr
+8dtPA+Z7d/JHT4E9UIJc877tDt53G0rM6pzgPzlRy5djgyT6tfsTF6PkDNEt8g8l
+ZyxNQQ6THUYOjlrgheFYN5hu55jhYmUJ8N+/4YoXLF7Oh5ZIt3rruXDz1xPwUhX8
+IyzR+rWX3UQfjNEHf3a/tqNX59MQ3kkTrqn+gzEZionVYFfY0woe91LjNVh4z3nI
+PyY4WYVObV1HNFsmmzsyJEULX96SADuU0zTrKuRx0EJY+r9hdkW7pgI4xR8tGH0p
+2jJFaWelH2PgVsvkxPQchpS+VEh7jA5B9qXovH3ClZ4UfTqy/+Baen1qup9+tnA9
+rs4mKbL+em3A0D3P4DkAR5CTNIc0HDK6O16xuF/+NvhKqJq5mNiTZc0DiwroHEOR
+SxKo5Rz7hcJb90q3BwxJXN1yDpLctyphjxAdLnUE1y98ZvHyGLsMt9rtrYqrytI0
+5y/56Ujc2hqOxjXl872tZwRaWNnTCPvSnd+thfkvadzqGZ0+/FaN5izaR3v2ipWV
+uc+Gn8ZvP2eioV9LKN+8or8pb9mWW+JCloC/Wu+ATsKd2F6lYgrNuXVkPcZxoWD/
+LeowYbDXhowHV6BgyKP2LBaYGhE1LVMeuVKYSkJsWtWXlC7j0HBceE5jXY4zoPEp
+eN4S/hWgHpECnp71p4au
+=29uN
 -----END PGP SIGNATURE-----

Re: Changing checksums of RT patches

[Steven Rostedt]

On Tue, 14 Feb 2017 00:15:33 +0100
Dominic Sacré <dominic.sacre@gmx.de> wrote:

> On 2017-02-13 23:56, Dominic Sacré wrote:
> > On 2017-02-10 19:54, Sebastian Andrzej Siewior wrote:  
> >> Can you tell me which file in which folder changed?  
> > 
> > The files I'm currently concerned with are in the rt/4.1/older directory.  
> 
> While I was writing my previous mail, 4.1.38-rt45 was released. Attached
> is a diff between rt/4.1/older/sha256sums.asc before (4 days ago) and
> after -rt45 was added.
> Note how the gzip-compressed files for 4.1.35-rt40 and 4.1.38-rt44
> changed in the meantime.

I wonder if my process caused this. I use to just have a copy of the
current patches in the main directory, and someone asked me to place a
copy in the older directory, which I started doing. Now when I release
a newer kernel, I move the previous version from the main directory
into the older directory deleting the copy that was there.

Could this be the cause of your issue?

-- Steve

Re: Changing checksums of RT patches

[Dominic Sacré]

On 2017-02-14 15:18, Steven Rostedt wrote:
> On Tue, 14 Feb 2017 00:15:33 +0100
> Dominic Sacré <dominic.sacre@gmx.de> wrote:
>> While I was writing my previous mail, 4.1.38-rt45 was released. Attached
>> is a diff between rt/4.1/older/sha256sums.asc before (4 days ago) and
>> after -rt45 was added.
>> Note how the gzip-compressed files for 4.1.35-rt40 and 4.1.38-rt44
>> changed in the meantime.
> 
> I wonder if my process caused this. I use to just have a copy of the
> current patches in the main directory, and someone asked me to place a
> copy in the older directory, which I started doing.

Thanks! To me it's very useful to have a permanent URL for all patches,
including the latest one.

> Now when I release a newer kernel, I move the previous version from
> the main directory into the older directory deleting the copy that
> was there.
> 
> Could this be the cause of your issue?

I think so. Looking at the current contents of rt/4.1, I see that
patch-4.1.38-rt45.patch.gz and older/patch-4.1.38-rt45.patch.gz have
different sha256 hashes. I would have expected them to be copies of the
exact same file, so I guess the question is why they're different in the
first place.


Dominic

Re: Changing checksums of RT patches

[Steven Rostedt]

On Tue, 14 Feb 2017 19:42:35 +0100
Dominic Sacré <dominic.sacre@gmx.de> wrote:

> Could this be the cause of your issue?  
> 
> I think so. Looking at the current contents of rt/4.1, I see that
> patch-4.1.38-rt45.patch.gz and older/patch-4.1.38-rt45.patch.gz have
> different sha256 hashes. I would have expected them to be copies of the
> exact same file, so I guess the question is why they're different in the
> first place.

It's the way kernel.org does the updates. I upload a .xz file and a
signed gpg file of the uncompressed image. kernel.org creates the gz
file from the uncompressed version. As I upload it twice, kernel.org
does two gzips of the uncompressed file. I'm guessing that there's a
timestamp that gets used as well, making both gzipped files different,
even though what they contain are the same.

I may need to change my workflow to simply delete the file in the
current directory than to move it. Although, I had better make sure
that there's a copy in the older directory first. Maybe I'll change my
tool to download the older and current versions, uncompress them, make
sure they are the same, and if they are, remove the current version,
else, move the current version on the older one.

-- Steve

Re: Changing checksums of RT patches

[Dominic Sacré]

On 2017-02-14 19:57, Steven Rostedt wrote:
> It's the way kernel.org does the updates. I upload a .xz file and a
> signed gpg file of the uncompressed image. kernel.org creates the gz
> file from the uncompressed version. As I upload it twice, kernel.org
> does two gzips of the uncompressed file. I'm guessing that there's a
> timestamp that gets used as well, making both gzipped files different,
> even though what they contain are the same.
> 
> I may need to change my workflow to simply delete the file in the
> current directory than to move it. Although, I had better make sure
> that there's a copy in the older directory first. Maybe I'll change my
> tool to download the older and current versions, uncompress them, make
> sure they are the same, and if they are, remove the current version,
> else, move the current version on the older one.

Sounds good to me!

I'll upgrade my Bitbake recipe to use older/patch-4.1.38-rt45.patch.gz
then. I've checked that the uncompressed contents are identical, so
everything should be fine with the process you describe.

(I've also just submitted a patch to OpenEmbedded to support
xz-compressed patches, which AFAICT are not affected by the whole issue.)


Thanks,

Dominic

Re: Changing checksums of RT patches

[Sebastian Andrzej Siewior]

On 2017-02-15 14:38:38 [+0100], Dominic Sacré wrote:
> (I've also just submitted a patch to OpenEmbedded to support
> xz-compressed patches, which AFAICT are not affected by the whole issue.)

please also consider www.kernel.org => cdn.kernel.org

> Thanks,
> 
> Dominic

Sebastian

Re: Changing checksums of RT patches

[Dominic Sacré]

On 2017-02-15 14:52, Sebastian Andrzej Siewior wrote:
> please also consider www.kernel.org => cdn.kernel.org

Actually, OpenEmbedded globally defines a placeholder
${KERNELORG_MIRROR}, so I should probably use that, and not hardcode the
server in the RT-kernel recipe at all.

However, as far as I can see, the only mirror that's actually defined in
OpenEmbedded is www.kernel.org. I could try to add cdn.kernel.org there,
but I don't really know how exactly OpenEmbedded handles mirror servers,
and what the policies are.


Dominic

Re: Changing checksums of RT patches

[Sebastian Andrzej Siewior]

On 2017-02-15 15:50:48 [+0100], Dominic Sacré wrote:
> However, as far as I can see, the only mirror that's actually defined in
> OpenEmbedded is www.kernel.org. I could try to add cdn.kernel.org there,
> but I don't really know how exactly OpenEmbedded handles mirror servers,
> and what the policies are.

please replace instead of add
	https://kernel.org/introducing-fastly-cdn.html

so unless you (OpenEmbedded) wants to avoid fastly/cdn I don't see a
reason why there should be a policy against it :)
 
> Dominic

Sebastian

Re: Changing checksums of RT patches

[Thomas Gleixner]

[-- Attachment #1: Type: text/plain, Size: 2114 bytes --]

On Tue, 14 Feb 2017, Steven Rostedt wrote:
> On Tue, 14 Feb 2017 19:42:35 +0100
> Dominic Sacré <dominic.sacre@gmx.de> wrote:
> 
> > Could this be the cause of your issue?  
> > 
> > I think so. Looking at the current contents of rt/4.1, I see that
> > patch-4.1.38-rt45.patch.gz and older/patch-4.1.38-rt45.patch.gz have
> > different sha256 hashes. I would have expected them to be copies of the
> > exact same file, so I guess the question is why they're different in the
> > first place.
> 
> It's the way kernel.org does the updates. I upload a .xz file and a
> signed gpg file of the uncompressed image. kernel.org creates the gz
> file from the uncompressed version. As I upload it twice, kernel.org
> does two gzips of the uncompressed file. I'm guessing that there's a
> timestamp that gets used as well, making both gzipped files different,
> even though what they contain are the same.

Right. And it does not matter at all.

The sha256 tells you that the file you downloaded is the one which
kernel.org advertised to you. Not more, not less.

The content is protected by the *.patch.sign and *.tar.sign files.

> I may need to change my workflow to simply delete the file in the
> current directory than to move it.

And the point is?

> Although, I had better make sure that there's a copy in the older
> directory first. Maybe I'll change my tool to download the older and
> current versions, uncompress them, make sure they are the same, and if
> they are, remove the current version, else, move the current version on
> the older one.

That does not make any sense.

The proper thing to do is move file from rt/4.x/ to rt/4.x/older.

If kernel.org decides to redo the archives, then you can't do anything
about it. And if you upload it another time there is no guarantee either
that the resulting gz/xz are the same.

What matters is the content and not the compressed thingy. Again:

 - The sha256 only tells you that the download was not corrupted.

 - The PGP signature is what protects the content and that does not change
   whether you move it or upload the same thing again.

Thanks,

	tglx

Re: Changing checksums of RT patches

[Steven Rostedt]

On Tue, 28 Feb 2017 19:03:29 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> On Tue, 14 Feb 2017, Steven Rostedt wrote:
> > On Tue, 14 Feb 2017 19:42:35 +0100
> > Dominic Sacré <dominic.sacre@gmx.de> wrote:
> >   
> > > Could this be the cause of your issue?  
> > > 
> > > I think so. Looking at the current contents of rt/4.1, I see that
> > > patch-4.1.38-rt45.patch.gz and older/patch-4.1.38-rt45.patch.gz have
> > > different sha256 hashes. I would have expected them to be copies of the
> > > exact same file, so I guess the question is why they're different in the
> > > first place.  
> > 
> > It's the way kernel.org does the updates. I upload a .xz file and a
> > signed gpg file of the uncompressed image. kernel.org creates the gz
> > file from the uncompressed version. As I upload it twice, kernel.org
> > does two gzips of the uncompressed file. I'm guessing that there's a
> > timestamp that gets used as well, making both gzipped files different,
> > even though what they contain are the same.  
> 
> Right. And it does not matter at all.
> 
> The sha256 tells you that the file you downloaded is the one which
> kernel.org advertised to you. Not more, not less.
> 
> The content is protected by the *.patch.sign and *.tar.sign files.
> 
> > I may need to change my workflow to simply delete the file in the
> > current directory than to move it.  
> 
> And the point is?

The issue I believe is that there's a third party downloading the
compressed files and doing a checksum on them.

> 
> > Although, I had better make sure that there's a copy in the older
> > directory first. Maybe I'll change my tool to download the older and
> > current versions, uncompress them, make sure they are the same, and if
> > they are, remove the current version, else, move the current version on
> > the older one.  
> 
> That does not make any sense.
> 
> The proper thing to do is move file from rt/4.x/ to rt/4.x/older.

This is a fallout of the change in work flow I have already made for
multiple people that have asked.

The issue is that we have a "current" and "older" directory. When we
do a new release, the current files get moved to the older directory.
This caused issues with people that supply scripts to down load the
tarballs. The problem is that they would be looking for a specific
version and suddenly it would move. I was asked to place the current
version in the "older" directory too. What I was told was that Sebastian
would upload the same file to both the current and older directories.
Which is what I changed to do as well.



> 
> If kernel.org decides to redo the archives, then you can't do anything
> about it. And if you upload it another time there is no guarantee either
> that the resulting gz/xz are the same.
> 
> What matters is the content and not the compressed thingy. Again:
> 
>  - The sha256 only tells you that the download was not corrupted.
> 
>  - The PGP signature is what protects the content and that does not change
>    whether you move it or upload the same thing again.

Actually, there's no "upload again" step. I believe Sebastian does the
same thing that I'm changing my workflow into. That is, when I upload
for the first time, I upload the same file into both "current" and
"older". This was added to my old workflow. But when I do a new
release, I didn't change my workflow to just remove what was in
current, but instead I moved current into older *overwriting* what was
already in older. That is the problem people are having.

They had a checksum of the .gz file in "older", and when I did the move
from current to older, I overwrote that file, and the checksum changed,
as it was compressed twice.

If I just do what I believe Sebastian does, and simply remove the files
in "current" as they already exist in "older", then everyone should be
happy.

-- Steve

Re: Changing checksums of RT patches

[Dominic Sacré]

Hi all,

I'm sorry to bring this up again, but it looks like
4.1/older/patch-4.1.38-rt45.patch.gz changed when -rt46 was released, so
the Bitbake recipes that use this patch are now broken again.

On 2017-02-28 19:03, Thomas Gleixner wrote:
> What matters is the content and not the compressed thingy. Again:
> 
>  - The sha256 only tells you that the download was not corrupted.
> 
>  - The PGP signature is what protects the content and that does not change
>    whether you move it or upload the same thing again.

I don't disagree, but there's currently no support for verifying
downloads by PGP signature in Bitbake; sha256 is the best we've got.

Besides, we're talking about a system for automated builds here.
I would argue that verifying the authenticity of files (e.g. using PGP
signatures) is the responsibility of the person writing the Bitbake recipe.
For those who then use that recipe to build the software, all that
matters is that the download is not corrupted, and that the file being
downloaded is actually the same one that was used by the author of the
recipe.

Of course, it really is the content that matters. But working with
sha256 checksums of the compressed files simplifies both the build
system, and, perhaps more importantly, the process of writing recipes.

IMHO it's bad practice for files to change once they've been released
(with a version number and everything), even if the contents remain the
same. The sha256-based approach seems to be working for thousands of
projects in OpenEmbedded; I don't see any particular reason for patches
on kernel.org to be any different.

By the way, the 4.1.38-rt46 release is currently still missing from the
"older" directory.


Cheers,

Dominic

Re: Changing checksums of RT patches

[Steven Rostedt]

On Sat, 11 Mar 2017 00:13:25 +0100
Dominic Sacré <dominic.sacre@gmx.de> wrote:

> Hi all,
> 
> I'm sorry to bring this up again, but it looks like
> 4.1/older/patch-4.1.38-rt45.patch.gz changed when -rt46 was released, so
> the Bitbake recipes that use this patch are now broken again.

Note, Julia just took over for 4.1-rt maintainership, so there is going
to be some expected hiccups along the way until she is fully up to
speed. Please have patience during this time.

> 
> On 2017-02-28 19:03, Thomas Gleixner wrote:
> > What matters is the content and not the compressed thingy. Again:
> > 
> >  - The sha256 only tells you that the download was not corrupted.
> > 
> >  - The PGP signature is what protects the content and that does not change
> >    whether you move it or upload the same thing again.  
> 
> I don't disagree, but there's currently no support for verifying
> downloads by PGP signature in Bitbake; sha256 is the best we've got.
> 
> Besides, we're talking about a system for automated builds here.
> I would argue that verifying the authenticity of files (e.g. using PGP
> signatures) is the responsibility of the person writing the Bitbake recipe.
> For those who then use that recipe to build the software, all that
> matters is that the download is not corrupted, and that the file being
> downloaded is actually the same one that was used by the author of the
> recipe.
> 
> Of course, it really is the content that matters. But working with
> sha256 checksums of the compressed files simplifies both the build
> system, and, perhaps more importantly, the process of writing recipes.
> 
> IMHO it's bad practice for files to change once they've been released
> (with a version number and everything), even if the contents remain the
> same. The sha256-based approach seems to be working for thousands of
> projects in OpenEmbedded; I don't see any particular reason for patches
> on kernel.org to be any different.
> 
> By the way, the 4.1.38-rt46 release is currently still missing from the
> "older" directory.

Julia,

Just an FYI, especially if you are using my older scripts (I haven't
updated the ones I posted). I was asked when I do my upload of a new
version to copy to both the main and older directory. But my scripts
still moved the main to the older one when a new release went out. This
caused the checksums of the created .gz to change in the older
directory.

What I do now is to still upload to both main and older, but when I
have a new release, I simply delete the main one, as the older one
already exists.

Thanks!

-- Steve

Re: Changing checksums of RT patches

[Julia Cartwright]

On Fri, Mar 10, 2017 at 06:19:47PM -0500, Steven Rostedt wrote:
> On Sat, 11 Mar 2017 00:13:25 +0100
> Dominic Sacré <dominic.sacre@gmx.de> wrote:
> 
> > Hi all,
> > 
> > I'm sorry to bring this up again, but it looks like
> > 4.1/older/patch-4.1.38-rt45.patch.gz changed when -rt46 was released, so
> > the Bitbake recipes that use this patch are now broken again.
> 
> Note, Julia just took over for 4.1-rt maintainership, so there is going
> to be some expected hiccups along the way until she is fully up to
> speed. Please have patience during this time.

*sigh*.  I was setup to fail here, wasn't I? :).  I even recalled this
thread, but thought "surely if I do the most obvious thing, I won't run
into problems!".

[..]
> > IMHO it's bad practice for files to change once they've been released
> > (with a version number and everything), even if the contents remain the
> > same. The sha256-based approach seems to be working for thousands of
> > projects in OpenEmbedded; I don't see any particular reason for patches
> > on kernel.org to be any different.
> > 
> > By the way, the 4.1.38-rt46 release is currently still missing from the
> > "older" directory.
> 
> Julia,
> 
> Just an FYI, especially if you are using my older scripts (I haven't
> updated the ones I posted). I was asked when I do my upload of a new
> version to copy to both the main and older directory. But my scripts
> still moved the main to the older one when a new release went out. This
> caused the checksums of the created .gz to change in the older
> directory.
>
> What I do now is to still upload to both main and older, but when I
> have a new release, I simply delete the main one, as the older one
> already exists.

Okay, sounds simple enough.  I've uploaded both the patch and series
tarball to older/, which should now remain unchanged.

I'd still like to understand why kernel.org decides to re-compress when
the simply moving patches around...

Thanks,

   Julia

Re: Changing checksums of RT patches

[Steven Rostedt]

On Fri, 10 Mar 2017 20:27:51 -0600
Julia Cartwright <julia@ni.com> wrote:

> I'd still like to understand why kernel.org decides to re-compress when
> the simply moving patches around...

It doesn't but I'm guessing that each upload has a different
compression result. The change happens when we remove the one from the
older directory and replace it with the one from the top directory. The
moved files still have the same checksum, but it's not the same as what
it replaced. Then it's reported that the one that got replaced had its
checksum changed.

-- Steve

Re: Changing checksums of RT patches

[Dominic Sacré]

On 2017-03-11 03:27, Julia Cartwright wrote:
> On Fri, Mar 10, 2017 at 06:19:47PM -0500, Steven Rostedt wrote:
>> What I do now is to still upload to both main and older, but when I
>> have a new release, I simply delete the main one, as the older one
>> already exists.
> 
> Okay, sounds simple enough.  I've uploaded both the patch and series
> tarball to older/, which should now remain unchanged.

Thanks, Julia.
Would it also be possible for you to restore patch-4.1.38-rt45.patch.gz
to the version that was previously in older/? That way we could minimize
the breakage. I still have the original file, if that helps.


Dominic

Re: Changing checksums of RT patches

[Steven Rostedt]

On Sat, 11 Mar 2017 12:14:00 +0100
Dominic Sacré <dominic.sacre@gmx.de> wrote:

> Thanks, Julia.
> Would it also be possible for you to restore patch-4.1.38-rt45.patch.gz
> to the version that was previously in older/? That way we could minimize
> the breakage. I still have the original file, if that helps.

No it's not possible. The .gz files are created by kernel.org when we
upload them. That's the reason they are different in the first place.
Because we upload the same file twice, and each time kernel.org creates
the .gz file which creates the differences.

-- Steve

Re: Changing checksums of RT patches

[Thomas Gleixner]

On Fri, 10 Mar 2017, Steven Rostedt wrote:
> Just an FYI, especially if you are using my older scripts (I haven't
> updated the ones I posted). I was asked when I do my upload of a new
> version to copy to both the main and older directory. But my scripts
> still moved the main to the older one when a new release went out. This
> caused the checksums of the created .gz to change in the older
> directory.
> 
> What I do now is to still upload to both main and older, but when I
> have a new release, I simply delete the main one, as the older one
> already exists.

That double upload does not make sense. Upload it once to the older folder
and make a link. kup supports that.

Thanks,

	tglx

Re: Changing checksums of RT patches

[Steven Rostedt]

On Sat, 11 Mar 2017 10:45:32 +0100 (CET)
Thomas Gleixner <tglx@linutronix.de> wrote:

> On Fri, 10 Mar 2017, Steven Rostedt wrote:
> > Just an FYI, especially if you are using my older scripts (I haven't
> > updated the ones I posted). I was asked when I do my upload of a new
> > version to copy to both the main and older directory. But my scripts
> > still moved the main to the older one when a new release went out. This
> > caused the checksums of the created .gz to change in the older
> > directory.
> > 
> > What I do now is to still upload to both main and older, but when I
> > have a new release, I simply delete the main one, as the older one
> > already exists.  
> 
> That double upload does not make sense. Upload it once to the older folder
> and make a link. kup supports that.
> 

OK, I'll have to update my scripts to do this.

Thanks,

-- Steve