[Buildroot] [PATCH] ubifs: Fix regression in ubifs_readdir()

[Buildroot] [PATCH] ubifs: Fix regression in ubifs_readdir()

[Thomas Petazzoni]

Hello J?rg,

[E-mail thread hijacked from the linux-mtd mailing list, since there is
a Buildroot related problem reported.]

On Sat, 29 Oct 2016 00:23:15 +0200, J?rg Krause wrote:

> > > I'm not sure if it's related to the issue reported by Peter Rosin
> > > and
> > > Ralph Sennhauser, but I am still getting a kernel panic using UBIFS
> > > with OverlayFS on Linux v4.9.0-rc2 with this patch applied:  
> > 
> > Does reverting c83ed4c9dbb35 help?
> > And are you 100% sure you applied the fix?  
> 
> I double double checked. The fix was applied on the git tree, but the
> compiler cache (I am using Buildroot with this option enabled) fooled
> me by using an old copy. After disabling the compiler cache I got a
> fixed build of the kernel. The panic is gone! Thanks!

This is *really* bad. Which Buildroot version are you using? Are you
able to reproduce the bad ccache behavior here? A modified source code
should definitely lead to a different hash of the preprocessed code,
and therefore there shouldn't be such a confusion between two cache
results.

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH] ubifs: Fix regression in ubifs_readdir()

[Jörg Krause]

On Tue, 2016-11-01 at 23:22 +0100, Thomas Petazzoni wrote:
> Hello J?rg,
> 
> [E-mail thread hijacked from the linux-mtd mailing list, since there
> is
> a Buildroot related problem reported.]
> 
> On Sat, 29 Oct 2016 00:23:15 +0200, J?rg Krause wrote:
> 
> > > > I'm not sure if it's related to the issue reported by Peter
> > > > Rosin
> > > > and
> > > > Ralph Sennhauser, but I am still getting a kernel panic using
> > > > UBIFS
> > > > with OverlayFS on Linux v4.9.0-rc2 with this patch applied:??
> > > 
> > > Does reverting c83ed4c9dbb35 help?
> > > And are you 100% sure you applied the fix???
> > 
> > I double double checked. The fix was applied on the git tree, but
> > the
> > compiler cache (I am using Buildroot with this option enabled)
> > fooled
> > me by using an old copy. After disabling the compiler cache I got a
> > fixed build of the kernel. The panic is gone! Thanks!
> 
> This is *really* bad. Which Buildroot version are you using?

2016.11 #

Note that I am using Buildroot as a submodule [1] and I needed to port
the br2-external tree. Maybe I messed something up?

> Are you able to reproduce the bad ccache behavior here?

Yes, I am. Linux kernel source directory is located locally and the
path is set using LINUX_OVERRIDE_SRCDIR in local.mk.

1/ Checkout Linux kernel version 4.7.10
2/ make linux-dirclean all
3/ Booted Linux version is 4.7.10

4/ Checkout Linux kernel version 4.8.5
5/ make linux-dirclean all
6/ Booted Linux version is still 4.7.10

> A modified source code should definitely lead to a different hash of
> the preprocessed code, and therefore there shouldn't be such a
> confusion between two cache results.

I checked the Linux source files in output/build/linux-custom/ and they
are correct - after step 2 the directory contains the Linux sources of
version 4.7.10 and after step 5 it contains version 4.8.5.

[1] https://github.com/Openwide-Ingenierie/buildroot-submodule

Best regards,
J?rg Krause

[Buildroot] [PATCH] ubifs: Fix regression in ubifs_readdir()

[Thomas Petazzoni]

Hello,

On Wed, 02 Nov 2016 20:56:59 +0100, J?rg Krause wrote:

> > This is *really* bad. Which Buildroot version are you using?  
> 
> 2016.11 #
> 
> Note that I am using Buildroot as a submodule [1] and I needed to port
> the br2-external tree. Maybe I messed something up?
> 
> > Are you able to reproduce the bad ccache behavior here?  
> 
> Yes, I am. Linux kernel source directory is located locally and the
> path is set using LINUX_OVERRIDE_SRCDIR in local.mk.
> 
> 1/ Checkout Linux kernel version 4.7.10
> 2/ make linux-dirclean all
> 3/ Booted Linux version is 4.7.10
> 
> 4/ Checkout Linux kernel version 4.8.5
> 5/ make linux-dirclean all
> 6/ Booted Linux version is still 4.7.10

And this scenario works fine with ccache *disabled*, but breaks when
you enable ccache support in Buildroot?

Thanks,

Thomas
-- 
Thomas Petazzoni, CTO, Free Electrons
Embedded Linux and Kernel engineering
http://free-electrons.com

[Buildroot] [PATCH] ubifs: Fix regression in ubifs_readdir()

[Jörg Krause]

On Wed, 2016-11-02 at 21:49 +0100, Thomas Petazzoni wrote:
> Hello,
> 
> On Wed, 02 Nov 2016 20:56:59 +0100, J?rg Krause wrote:
> 
> > > This is *really* bad. Which Buildroot version are you using???
> > 
> > 2016.11 #
> > 
> > Note that I am using Buildroot as a submodule [1] and I needed to
> > port
> > the br2-external tree. Maybe I messed something up?
> > 
> > > Are you able to reproduce the bad ccache behavior here???
> > 
> > Yes, I am. Linux kernel source directory is located locally and the
> > path is set using LINUX_OVERRIDE_SRCDIR in local.mk.
> > 
> > 1/ Checkout Linux kernel version 4.7.10
> > 2/ make linux-dirclean all
> > 3/ Booted Linux version is 4.7.10
> > 
> > 4/ Checkout Linux kernel version 4.8.5
> > 5/ make linux-dirclean all
> > 6/ Booted Linux version is still 4.7.10
> 
> And this scenario works fine with ccache *disabled*, but breaks when
> you enable ccache support in Buildroot?

Yes.

[Buildroot] [PATCH] ubifs: Fix regression in ubifs_readdir()

[Arnout Vandecappelle]

On 02-11-16 23:49, J?rg Krause wrote:
> On Wed, 2016-11-02 at 21:49 +0100, Thomas Petazzoni wrote:
>> Hello,
>>
>> On Wed, 02 Nov 2016 20:56:59 +0100, J?rg Krause wrote:
>>
>>>> This is *really* bad. Which Buildroot version are you using?  
>>>
>>> 2016.11 #
>>>
>>> Note that I am using Buildroot as a submodule [1] and I needed to
>>> port
>>> the br2-external tree. Maybe I messed something up?
>>>
>>>> Are you able to reproduce the bad ccache behavior here?  
>>>
>>> Yes, I am. Linux kernel source directory is located locally and the
>>> path is set using LINUX_OVERRIDE_SRCDIR in local.mk.
>>>
>>> 1/ Checkout Linux kernel version 4.7.10
>>> 2/ make linux-dirclean all
>>> 3/ Booted Linux version is 4.7.10
>>>
>>> 4/ Checkout Linux kernel version 4.8.5
>>> 5/ make linux-dirclean all
>>> 6/ Booted Linux version is still 4.7.10
>>
>> And this scenario works fine with ccache *disabled*, but breaks when
>> you enable ccache support in Buildroot?
> 
> Yes.

 I can't reproduce this. I've got a nice mix of cache misses and cache hits when
building a different version, and version.o is definitely a cache miss.

 Can you double-check the version in output/build/linux-custom/init/version.o?
And the one in output/build/linux-custom/include/generated/utsrelease.h?

 Hm, hang on, is there maybe a utsrelease.h that ends up in your staging dir and
that is used by ccache?

 Regards,
 Arnout


-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] ubifs: Fix regression in ubifs_readdir()

[Jörg Krause]

On Thu, 2016-11-03 at 03:46 +0100, Arnout Vandecappelle wrote:
> 
> On 02-11-16 23:49, J?rg Krause wrote:
> > On Wed, 2016-11-02 at 21:49 +0100, Thomas Petazzoni wrote:
> > > Hello,
> > > 
> > > On Wed, 02 Nov 2016 20:56:59 +0100, J?rg Krause wrote:
> > > 
> > > > > This is *really* bad. Which Buildroot version are you
> > > > > using???
> > > > 
> > > > 2016.11 #
> > > > 
> > > > Note that I am using Buildroot as a submodule [1] and I needed
> > > > to
> > > > port
> > > > the br2-external tree. Maybe I messed something up?
> > > > 
> > > > > Are you able to reproduce the bad ccache behavior here???
> > > > 
> > > > Yes, I am. Linux kernel source directory is located locally and
> > > > the
> > > > path is set using LINUX_OVERRIDE_SRCDIR in local.mk.
> > > > 
> > > > 1/ Checkout Linux kernel version 4.7.10
> > > > 2/ make linux-dirclean all
> > > > 3/ Booted Linux version is 4.7.10
> > > > 
> > > > 4/ Checkout Linux kernel version 4.8.5
> > > > 5/ make linux-dirclean all
> > > > 6/ Booted Linux version is still 4.7.10
> > > 
> > > And this scenario works fine with ccache *disabled*, but breaks
> > > when
> > > you enable ccache support in Buildroot?
> > 
> > Yes.
> 
> ?I can't reproduce this. I've got a nice mix of cache misses and
> cache hits when
> building a different version, and version.o is definitely a cache
> miss.
> 
> ?Can you double-check the version in output/build/linux-
> custom/init/version.o?

I checked the version string in version.o with hexdump -C and they
differ.

> And the one in output/build/linux-
> custom/include/generated/utsrelease.h?

This as well.

> ?Hm, hang on, is there maybe a utsrelease.h that ends up in your
> staging dir and that is used by ccache?

No, there is only the linux generated file.

---

I am not experienced with ccache, but I got some statistics:

I cleared the cache by removing ~/.buildroot-ccache, than run `make
linux-dirclean linux`:

cache hit (direct)?????????????????????0
cache hit (preprocessed)???????????????0
cache miss??????????????????????????1088
cache hit rate??????????????????????0.00 %
called for link???????????????????????10
called for preprocessing???????????????5
no input file????????????????????????394
cleanups performed?????????????????????0
files in cache??????????????????????3250

Then I re-run `make linux-dirclean linux` (without altering the linux
sources):

cache hit (direct)??????????????????1076
cache hit (preprocessed)??????????????10
cache miss??????????????????????????1090
cache hit rate?????????????????????49.91 %
called for link???????????????????????20
called for preprocessing??????????????10
no input file????????????????????????788
cleanups performed?????????????????????0
files in cache??????????????????????3264

Again, clear the cache and run `make linux-dirclean linux`, but alter
the linux sources by checking out a different branch:

cache hit (direct)????????????????????10
cache hit (preprocessed)??????????????62
cache miss??????????????????????????2102
cache hit rate??????????????????????3.31 %
called for link???????????????????????20
called for preprocessing??????????????10
no input file????????????????????????773
cleanups performed?????????????????????0
files in cache??????????????????????6386

---

Furthermore, I did the following for the different linux builds:

After building branch-4.7.y:
# cp output/build/linux-custom/arch/arm/boot/zImage zImage-4.7.10

After building branch-4.8.y:
# cp output/build/linux-custom/arch/arm/boot/zImage zImage-4.8.5

Afterwards, diff shows no differences:
# diff zImage-4.7.10 zImage-4.8.5

Best regards,
J?rg Krause

[Buildroot] [PATCH] ubifs: Fix regression in ubifs_readdir()

[Arnout Vandecappelle]

On 03-11-16 08:23, J?rg Krause wrote:
> On Thu, 2016-11-03 at 03:46 +0100, Arnout Vandecappelle wrote:
>>
>> On 02-11-16 23:49, J?rg Krause wrote:
[snip]
>>>>> Yes, I am. Linux kernel source directory is located locally and
>>>>> the
>>>>> path is set using LINUX_OVERRIDE_SRCDIR in local.mk.
>>>>>
>>>>> 1/ Checkout Linux kernel version 4.7.10
>>>>> 2/ make linux-dirclean all
>>>>> 3/ Booted Linux version is 4.7.10
>>>>>
>>>>> 4/ Checkout Linux kernel version 4.8.5
>>>>> 5/ make linux-dirclean all
>>>>> 6/ Booted Linux version is still 4.7.10
>>>>
>>>> And this scenario works fine with ccache *disabled*, but breaks
>>>> when
>>>> you enable ccache support in Buildroot?
>>>
>>> Yes.
>>
>>  I can't reproduce this. I've got a nice mix of cache misses and
>> cache hits when
>> building a different version, and version.o is definitely a cache
>> miss.
>>
>>  Can you double-check the version in output/build/linux-
>> custom/init/version.o?
> 
> I checked the version string in version.o with hexdump -C and they
> differ.

 And the version that is in there is correct, right? (Use strings to get it, the
first string it finds should be the version string as in uname -a).

 This is very weird, because version.o gets linked into the vmlinux.elf and
eventually the zImage, and ccache doesn't get activated anymore in that path
because it's all linking and no compiling...

> 
>> And the one in output/build/linux-
>> custom/include/generated/utsrelease.h?
> 
> This as well.
> 
>>  Hm, hang on, is there maybe a utsrelease.h that ends up in your
>> staging dir and that is used by ccache?
> 
> No, there is only the linux generated file.
> 
> ---
> 
> I am not experienced with ccache, but I got some statistics:
> 
> I cleared the cache by removing ~/.buildroot-ccache, than run `make
> linux-dirclean linux`:
> 
> cache hit (direct)                     0
> cache hit (preprocessed)               0
> cache miss                          1088
> cache hit rate                      0.00 %
> called for link                       10
> called for preprocessing               5
> no input file                        394
> cleanups performed                     0
> files in cache                      3250
> 
> Then I re-run `make linux-dirclean linux` (without altering the linux
> sources):
> 
> cache hit (direct)                  1076
> cache hit (preprocessed)              10
> cache miss                          1090
> cache hit rate                     49.91 %
> called for link                       20
> called for preprocessing              10
> no input file                        788
> cleanups performed                     0
> files in cache                      3264
> 
> Again, clear the cache and run `make linux-dirclean linux`, but alter
> the linux sources by checking out a different branch:
> 
> cache hit (direct)                    10

 This is weird, it shouldn't go down... Did you clear the cache or the
statistics somewhere in between?

> cache hit (preprocessed)              62
> cache miss                          2102
> cache hit rate                      3.31 %
> called for link                       20
> called for preprocessing              10
> no input file                        773
> cleanups performed                     0
> files in cache                      6386
> 
> ---
> 
> Furthermore, I did the following for the different linux builds:
> 
> After building branch-4.7.y:
> # cp output/build/linux-custom/arch/arm/boot/zImage zImage-4.7.10
> 
> After building branch-4.8.y:
> # cp output/build/linux-custom/arch/arm/boot/zImage zImage-4.8.5
> 
> Afterwards, diff shows no differences:
> # diff zImage-4.7.10 zImage-4.8.5

 Since version.o differs, it means the version.o doesn't end up in the zImage...
Again, can you check for the intermediate files what happens to them? E.g.
init/built-in.o, .tmp_vmlinux1, arch/arm/boot/Image. Maybe just do "mv
output/build/linux-custom output/build/linux-custom.orig" before rebuilding and
do a full tree compare to find which files are identical.

 Regards,
 Arnout

> 
> Best regards,
> J?rg Krause
> 

-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7493 020B C7E3 8618 8DEC 222C 82EB F404 F9AC 0DDF

[Buildroot] [PATCH] ubifs: Fix regression in ubifs_readdir()

[Jörg Krause]

On Thu, 2016-11-03 at 11:22 +0100, Arnout Vandecappelle wrote:
> 
> On 03-11-16 08:23, J?rg Krause wrote:
> > On Thu, 2016-11-03 at 03:46 +0100, Arnout Vandecappelle wrote:
> > > 
> > > On 02-11-16 23:49, J?rg Krause wrote:
> 
> [snip]
> > > > > > Yes, I am. Linux kernel source directory is located locally
> > > > > > and
> > > > > > the
> > > > > > path is set using LINUX_OVERRIDE_SRCDIR in local.mk.
> > > > > > 
> > > > > > 1/ Checkout Linux kernel version 4.7.10
> > > > > > 2/ make linux-dirclean all
> > > > > > 3/ Booted Linux version is 4.7.10
> > > > > > 
> > > > > > 4/ Checkout Linux kernel version 4.8.5
> > > > > > 5/ make linux-dirclean all
> > > > > > 6/ Booted Linux version is still 4.7.10
> > > > > 
> > > > > And this scenario works fine with ccache *disabled*, but
> > > > > breaks
> > > > > when
> > > > > you enable ccache support in Buildroot?
> > > > 
> > > > Yes.
> > > 
> > > ?I can't reproduce this. I've got a nice mix of cache misses and
> > > cache hits when
> > > building a different version, and version.o is definitely a cache
> > > miss.
> > > 
> > > ?Can you double-check the version in output/build/linux-
> > > custom/init/version.o?
> > 
> > I checked the version string in version.o with hexdump -C and they
> > differ.
> 
> ?And the version that is in there is correct, right? (Use strings to
> get it, the
> first string it finds should be the version string as in uname -a).
> 
> ?This is very weird, because version.o gets linked into the
> vmlinux.elf and
> eventually the zImage, and ccache doesn't get activated anymore in
> that path
> because it's all linking and no compiling...
> 
> > 
> > > And the one in output/build/linux-
> > > custom/include/generated/utsrelease.h?
> > 
> > This as well.
> > 
> > > ?Hm, hang on, is there maybe a utsrelease.h that ends up in your
> > > staging dir and that is used by ccache?
> > 
> > No, there is only the linux generated file.
> > 
> > ---
> > 
> > I am not experienced with ccache, but I got some statistics:
> > 
> > I cleared the cache by removing ~/.buildroot-ccache, than run `make
> > linux-dirclean linux`:
> > 
> > cache hit (direct)?????????????????????0
> > cache hit (preprocessed)???????????????0
> > cache miss??????????????????????????1088
> > cache hit rate??????????????????????0.00 %
> > called for link???????????????????????10
> > called for preprocessing???????????????5
> > no input file????????????????????????394
> > cleanups performed?????????????????????0
> > files in cache??????????????????????3250
> > 
> > Then I re-run `make linux-dirclean linux` (without altering the
> > linux
> > sources):
> > 
> > cache hit (direct)??????????????????1076
> > cache hit (preprocessed)??????????????10
> > cache miss??????????????????????????1090
> > cache hit rate?????????????????????49.91 %
> > called for link???????????????????????20
> > called for preprocessing??????????????10
> > no input file????????????????????????788
> > cleanups performed?????????????????????0
> > files in cache??????????????????????3264
> > 
> > Again, clear the cache and run `make linux-dirclean linux`, but
> > alter
> > the linux sources by checking out a different branch:
> > 
> > cache hit (direct)????????????????????10
> 
> ?This is weird, it shouldn't go down... Did you clear the cache or
> the
> statistics somewhere in between?

Yes. Maybe you missed it, but I wrote "Again, clear the cache...".

> > cache hit (preprocessed)??????????????62
> > cache miss??????????????????????????2102
> > cache hit rate??????????????????????3.31 %
> > called for link???????????????????????20
> > called for preprocessing??????????????10
> > no input file????????????????????????773
> > cleanups performed?????????????????????0
> > files in cache??????????????????????6386
> > 
> > ---
> > 
> > Furthermore, I did the following for the different linux builds:
> > 
> > After building branch-4.7.y:
> > # cp output/build/linux-custom/arch/arm/boot/zImage zImage-4.7.10
> > 
> > After building branch-4.8.y:
> > # cp output/build/linux-custom/arch/arm/boot/zImage zImage-4.8.5
> > 
> > Afterwards, diff shows no differences:
> > # diff zImage-4.7.10 zImage-4.8.5
> 
> ?Since version.o differs, it means the version.o doesn't end up in
> the zImage...
> Again, can you check for the intermediate files what happens to them?
> E.g.
> init/built-in.o, .tmp_vmlinux1, arch/arm/boot/Image. Maybe just do
> "mv
> output/build/linux-custom output/build/linux-custom.orig" before
> rebuilding and
> do a full tree compare to find which files are identical.

I did the following:

1/ checkout linux-4.8 branch
2/ make linux-dirclean linux
3/ move output/build/linux-custom to output/build/linux-custom.orig
4/ checkout linux-4.7 branch
5/ make linux-dirclean linux
6/ diff:

# diff output/build/linux-custom/arch/arm/boot/Image
output/build/linux-custom.orig/arch/arm/boot/Image
Binary files output/build/linux-custom/arch/arm/boot/Image and
output/build/linux-custom.orig/arch/arm/boot/Image differ
# diff output/build/linux-custom/arch/arm/boot/zImage
output/build/linux-custom.orig/arch/arm/boot/zImage

Weird, the Image binaries differ, but the zImage don't. The zImage in
output/build/linux-custom/arch/arm/boot/ is exactly the same as in
output/build/linux-custom.orig/arch/arm/boot/

J?rg

[Buildroot] [PATCH] ubifs: Fix regression in ubifs_readdir()

[Jörg Krause]

On Mon, 2016-11-07 at 23:08 +0100, J?rg Krause wrote:

[snip]

> > 
> > ?Since version.o differs, it means the version.o doesn't end up in
> > the zImage...
> > Again, can you check for the intermediate files what happens to
> > them?
> > E.g.
> > init/built-in.o, .tmp_vmlinux1, arch/arm/boot/Image. Maybe just do
> > "mv
> > output/build/linux-custom output/build/linux-custom.orig" before
> > rebuilding and
> > do a full tree compare to find which files are identical.
> 
> I did the following:
> 
> 1/ checkout linux-4.8 branch
> 2/ make linux-dirclean linux
> 3/ move output/build/linux-custom to output/build/linux-custom.orig
> 4/ checkout linux-4.7 branch
> 5/ make linux-dirclean linux
> 6/ diff:
> 
> # diff output/build/linux-custom/arch/arm/boot/Image
> output/build/linux-custom.orig/arch/arm/boot/Image
> Binary files output/build/linux-custom/arch/arm/boot/Image and
> output/build/linux-custom.orig/arch/arm/boot/Image differ
> # diff output/build/linux-custom/arch/arm/boot/zImage
> output/build/linux-custom.orig/arch/arm/boot/zImage
> 
> Weird, the Image binaries differ, but the zImage don't. The zImage in
> output/build/linux-custom/arch/arm/boot/ is exactly the same as in
> output/build/linux-custom.orig/arch/arm/boot/

After running the above steps I did the following:

7/ disable compiler cache option in Buildroot
8/ make linux-dirclean linux
9/ diff:

diff output/build/linux-custom/arch/arm/boot/zImage output/build/linux-
custom.orig/arch/arm/boot/zImage
Binary files hbm10/output/build/linux-custom/arch/arm/boot/zImage and
hbm10/output/build/linux-custom.orig/arch/arm/boot/zImage differ

10/ re-enable compiler cache in Buildroot
11/ Binary files hbm10/output/build/linux-custom/arch/arm/boot/zImage
and hbm10/output/build/linux-custom.orig/arch/arm/boot/zImage differ
12/ diff:

diff output/build/linux-custom/arch/arm/boot/zImage output/build/linux-
custom.orig/arch/arm/boot/zImage
(No message)

No idea!

J?rg

[Buildroot] [PATCH] ubifs: Fix regression in ubifs_readdir()

[Jörg Krause]

On Thu, 2016-11-03 at 11:22 +0100, Arnout Vandecappelle wrote:
> 
> On 03-11-16 08:23, J?rg Krause wrote:
> > On Thu, 2016-11-03 at 03:46 +0100, Arnout Vandecappelle wrote:
> > > 
> > > On 02-11-16 23:49, J?rg Krause wrote:

[snip]

> ?Since version.o differs, it means the version.o doesn't end up in
> the zImage...
> Again, can you check for the intermediate files what happens to them?
> E.g.
> init/built-in.o, .tmp_vmlinux1, arch/arm/boot/Image. Maybe just do
> "mv
> output/build/linux-custom output/build/linux-custom.orig" before
> rebuilding and
> do a full tree compare to find which files are identical.

Got it! ccache <3.3.3 is affected by bug concerning rebuilding the
kernel [1].

So, I guess we should update ccache.

[1] https://github.com/ccache/ccache/issues/136

Many thanks for your help!