Not as much ccache win as I expected

Not as much ccache win as I expected

[Tim Bird]

I'm running an automated test which does numerous compiles
of the Linux kernel.  One of the things I do is create a localversion
file at the root of the kernel source tree with a unique identifier
that I use later on in testing.

I started using ccache to improve the performance of my builds,
but found that the hit rate on the cache was not very good.
 $ ccache -s
cache directory                     /home/tbird/.ccache
cache hit                          74416
cache miss                         59400
called for link                    87252
compile failed                        21
not a C/C++ file                  143449
no input file                      49336
files in cache                     42844
cache size                           1.8 Gbytes
max cache size                       2.0 Gbytes

Thinking that the problem might be having a unique version for
every build (and that this change flowed to every file via the
version.h file), I tried building without this change.  I saw
an improvement, but not much.

Is there anything else obvious which is prevents ccache from
working well with a kernel build (that is, anything else that
would, for otherwise identical C files with a similar build,
cause a difference?)

Any tips would be appreciated.
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Not as much ccache win as I expected

[Oleg Verych]

Tim Bird @ Fri, 13 Jun 2008 12:06:05 -0700:

> I'm running an automated test which does numerous compiles
> of the Linux kernel.  One of the things I do is create a localversion
> file at the root of the kernel source tree with a unique identifier
> that I use later on in testing.

And what kinds of source/kconfig changes are made for every build?
(any versions, e.g. localversion, .version, aren't important, they are for
modules ko and vmlinux, afaik)

[...]
> Is there anything else obvious which is prevents ccache from
> working well with a kernel build (that is, anything else that
> would, for otherwise identical C files with a similar build,
> cause a difference?)

kbuild is `ccache` on itself. Every *.o.cmd is kind of info `ccache`
hashes (except things like stderr, gcc version) to check repeated
rebuilds. Also kconfig<->kbuild link via header magic may confuse
general-purpose `ccache`.

For rebuilds of the same codebase, it's better to use separate
kbuild object output directories.

> Any tips would be appreciated.

Just my handwaving, but test with couple core config symbols
toggling, shows only one `ccache` hit.

olecom@flower:/mnt/zdev0/blinux$ CCACHE_DIR=_ccache/ ccache -s
cache directory                     _ccache/
cache hit                           1133
cache miss                          1141
called for link                       28
not a C/C++ file                      64
no input file                        282
files in cache                      2282
cache size                          15.7 Mbytes
max cache size                     976.6 Mbytes
olecom@flower:/mnt/zdev0/blinux$ make menuconfig # toggle
olecom@flower:/mnt/zdev0/blinux$ diff -u1 .config.old .config
--- .config.old 2008-06-13 22:52:37.000000000 +0200
+++ .config     2008-06-13 23:01:07.000000000 +0200
@@ -3,3 +3,3 @@
 # Linux kernel version: 2.6.24
-# Fri Jun 13 22:52:37 2008
+# Fri Jun 13 23:01:07 2008
 #
@@ -67,3 +67,4 @@
 CONFIG_POSIX_MQUEUE=y
-# CONFIG_BSD_PROCESS_ACCT is not set
+CONFIG_BSD_PROCESS_ACCT=y
+CONFIG_BSD_PROCESS_ACCT_V3=y
 # CONFIG_TASKSTATS is not set
olecom@flower:/mnt/zdev0/blinux$
olecom@flower:/mnt/zdev0/blinux$ CCACHE_DIR=_ccache/ ccache -s
cache directory                     _ccache/
cache hit                           1134
cache miss                          1879
called for link                       33
not a C/C++ file                      79
no input file                        401
files in cache                      3758
cache size                          28.1 Mbytes
max cache size                     976.6 Mbytes
olecom@flower:/mnt/zdev0/blinux$

-- 
sed 'sed && sh + olecom = love'  <<  ''
-o--=O`C
 #oo'L O
<___=E M

Re: Not as much ccache win as I expected

[Tim Bird]

Oleg Verych wrote:
> And what kinds of source/kconfig changes are made for every build?

I start with a baseline config for an embedded board, then
alter, one at a time, individual config items related to kernel size.
No source changes are made.

I do full removal of the kernel source tree and build area
before the start of each test.

> (any versions, e.g. localversion, .version, aren't important, they are for
> modules ko and vmlinux, afaik)
Ok - this is helpful.

> kbuild is `ccache` on itself. Every *.o.cmd is kind of info `ccache`
> hashes (except things like stderr, gcc version) to check repeated
> rebuilds.
Yeah, I'm pretty impressed with how well kbuild avoids rebuilding
stuff in the first place.

Maybe I should just be grateful for any ccache hits I get.

Thanks,
 -- Tim

=============================
Tim Bird
Architecture Group Chair, CE Linux Forum
Senior Staff Engineer, Sony Corporation of America
=============================

Re: Not as much ccache win as I expected

[Jörn Engel]

On Fri, 13 June 2008 14:10:29 -0700, Tim Bird wrote:
> 
> Maybe I should just be grateful for any ccache hits I get.

ccache's usefulness depends on your workload.  If you make a change to
include/linux/fs.h, close to 100% of the kernel is rebuilt, with or
without ccache.  But when you revert that change, the build time differs
dramatically.  Without ccache, fs.h was simply changed again and
everything is rebuild.  With ccache, there are hits for the old version
and all is pulled from the cache - provided you have allotted enough
disk for it.

If you never revert to an old version or do some equivalent operation,
ccache can even be a net loss.  On a fast machine, the additional disk
accesses are easily more expensive than the minimal cpu gains.

Jörn

-- 
Public Domain  - Free as in Beer
General Public - Free as in Speech
BSD License    - Free as in Enterprise
Shared Source  - Free as in "Work will make you..."
--
To unsubscribe from this list: send the line "unsubscribe linux-kbuild" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html