[Buildroot] ccache directory

[Buildroot] ccache directory

[Thomas De Schampheleire]

Thomas, Peter,

A while ago, there were some changes in the ccache handling, with
which the cache location also changed to $(HOME)/.buildroot-ccache.
Before that, the cache directory was configurable in .config. What is
the reason of changing the cache location?

In my (corporate) environment, the home directories are actually on
the network (NFS) which is not ideal for performance. I would like to
configure another cache directory.

Would you accept a patch that re-introduces BR2_CCACHE_DIR ?

Thanks,
Thomas

[Buildroot] ccache directory

[Thomas Petazzoni]

Le Wed, 20 Jul 2011 17:42:52 +0200,
Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> a ?crit :

> A while ago, there were some changes in the ccache handling, with
> which the cache location also changed to $(HOME)/.buildroot-ccache.
> Before that, the cache directory was configurable in .config. What is
> the reason of changing the cache location?

Simplifying the ccache support, mainly.

> In my (corporate) environment, the home directories are actually on
> the network (NFS) which is not ideal for performance. I would like to
> configure another cache directory.
> 
> Would you accept a patch that re-introduces BR2_CCACHE_DIR ?

I guess your use case is valid, so I would personally accept such a
patch. I don't like cluttering the "Build options" menu with too many
options, but I guess this option is a valid thing to have when ccache
support is enabled.

Regards,

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] ccache directory

[Peter Korsgaard]

>>>>> "Thomas" == Thomas Petazzoni <thomas.petazzoni@free-electrons.com> writes:

Hi,

 >> Would you accept a patch that re-introduces BR2_CCACHE_DIR ?

 Thomas> I guess your use case is valid, so I would personally accept
 Thomas> such a patch. I don't like cluttering the "Build options" menu
 Thomas> with too many options, but I guess this option is a valid thing
 Thomas> to have when ccache support is enabled.

Or simply drop the custom ccache location and let people set CCACHE_DIR
if they don't like the default (and hence expected) ~/.ccache location.

-- 
Bye, Peter Korsgaard

[Buildroot] ccache directory

[Thomas Petazzoni]

Le Wed, 20 Jul 2011 22:49:19 +0200,
Peter Korsgaard <jacmet@uclibc.org> a ?crit :

> Or simply drop the custom ccache location and let people set CCACHE_DIR
> if they don't like the default (and hence expected) ~/.ccache location.

That's another possibility, but I find it quite confusing to have some
parameters set by environment variables, and some other by
configuration options.

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] ccache directory

[Michael S. Zick]

On Thu July 21 2011, Thomas Petazzoni wrote:
> Le Wed, 20 Jul 2011 22:49:19 +0200,
> Peter Korsgaard <jacmet@uclibc.org> a ?crit :
> 
> > Or simply drop the custom ccache location and let people set CCACHE_DIR
> > if they don't like the default (and hence expected) ~/.ccache location.
> 
> That's another possibility, but I find it quite confusing to have some
> parameters set by environment variables, and some other by
> configuration options.
> 

And if one of the purposes of having a programmed build system is to
provide a reproducible build at some future time (or place or by someone else);
setting an environment variable leaves no record in the stored build system.

Mike
> Thomas

[Buildroot] ccache directory

[Peter Korsgaard]

>>>>> "Michael" == Michael S Zick <minimod@morethan.org> writes:

 Michael> And if one of the purposes of having a programmed build system
 Michael> is to provide a reproducible build at some future time (or
 Michael> place or by someone else); setting an environment variable
 Michael> leaves no record in the stored build system.

True, but the ccache directory (like BUILDROOT_DL_DIR) don't affect the
output, they are only local build time optimizations.

-- 
Bye, Peter Korsgaard

[Buildroot] ccache directory

[Thomas De Schampheleire]

On Thu, Jul 21, 2011 at 7:58 AM, Thomas Petazzoni
<thomas.petazzoni@free-electrons.com> wrote:
> Le Wed, 20 Jul 2011 22:49:19 +0200,
> Peter Korsgaard <jacmet@uclibc.org> a ?crit :
>
>> Or simply drop the custom ccache location and let people set CCACHE_DIR
>> if they don't like the default (and hence expected) ~/.ccache location.
>
> That's another possibility, but I find it quite confusing to have some
> parameters set by environment variables, and some other by
> configuration options.

The disadvantage with this particular configuration variable, is that
it requires a forced recompilation of host-ccache in order for changes
to this variable to take effect.

Another solution is to remove the hardcoded cache directory, use the
CCACHE_DIR environment variable, but let buildroot set that variable
based on BR2_CCACHE_DIR (a configuration option). This addresses
Thomas' point regarding mixing environment variables and configuration
options, and my request of being able to change the cache directory
and making sure these changes are automatically in effect.

What do you think?

Thanks,
Thomas

[Buildroot] ccache directory

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> writes:


Hi,

 >> That's another possibility, but I find it quite confusing to have some
 >> parameters set by environment variables, and some other by
 >> configuration options.

 Thomas> The disadvantage with this particular configuration variable, is that
 Thomas> it requires a forced recompilation of host-ccache in order for changes
 Thomas> to this variable to take effect.

 Thomas> Another solution is to remove the hardcoded cache directory, use the
 Thomas> CCACHE_DIR environment variable, but let buildroot set that variable
 Thomas> based on BR2_CCACHE_DIR (a configuration option). This addresses
 Thomas> Thomas' point regarding mixing environment variables and configuration
 Thomas> options, and my request of being able to change the cache directory
 Thomas> and making sure these changes are automatically in effect.

 Thomas> What do you think?

Yes, something like:

# use BR setting if not set in environment
ifndef CCACHE_DIR
CCACHE_DIR:=$(call qstrip,BR2_CCACHE_DIR)
endif

export CCACHE_DIR

in the toplevel Makefile.

-- 
Bye, Peter Korsgaard

[Buildroot] ccache directory

[Peter Korsgaard]

>>>>> "Peter" == Peter Korsgaard <jacmet@uclibc.org> writes:

Hi,

 Thomas> Another solution is to remove the hardcoded cache directory,
 Thomas> use the CCACHE_DIR environment variable, but let buildroot set
 Thomas> that variable based on BR2_CCACHE_DIR (a configuration
 Thomas> option). This addresses Thomas' point regarding mixing
 Thomas> environment variables and configuration options, and my request
 Thomas> of being able to change the cache directory and making sure
 Thomas> these changes are automatically in effect.

 Thomas> What do you think?

 Peter> Yes, something like:

 Peter> # use BR setting if not set in environment
 Peter> ifndef CCACHE_DIR
 Peter> CCACHE_DIR:=$(call qstrip,BR2_CCACHE_DIR)
 Peter> endif

 Peter> export CCACHE_DIR

 Peter> in the toplevel Makefile.

Except that doesn't work. The GENTARGETS stuff for ccache already
creates a CCACHE_DIR variable for the ccache build directory, and the
export line only gets evaluated just before the targets are executed :/

As variables are automatically reexported when they were set in the
environment, this also means that we would always clutter the CCACHE_DIR
environment setting.

It's furthermore not easy to handle a BR_CCACHE_DIR variable containing
shell variables (like $HOME) that should get expanded before make
handles it.

So that leaves either:

- Keep things like they are

- Add a BR2_CCACHE_DIR but tell people that they cannot use $VAR in it

- Patch ccache to look for another environment variable

- Rework GENTARGETS to not use <pkg>_DIR, but several packages would
  need to be updated, and users might rely on it for local packages

- Rename ccache package to something else, breaking existing .configs

I think option two (defaulting to ~/.buildroot-ccache or perhaps
~/.ccache) is probably the best option, followed by 1.

-- 
Bye, Peter Korsgaard

[Buildroot] ccache directory

[Thomas De Schampheleire]

Hi Peter,

On Sun, Jul 24, 2011 at 6:06 PM, Peter Korsgaard <jacmet@sunsite.dk> wrote:
>>>>>> "Peter" == Peter Korsgaard <jacmet@uclibc.org> writes:
>
> Hi,
>
> ?Thomas> Another solution is to remove the hardcoded cache directory,
> ?Thomas> use the CCACHE_DIR environment variable, but let buildroot set
> ?Thomas> that variable based on BR2_CCACHE_DIR (a configuration
> ?Thomas> option). This addresses Thomas' point regarding mixing
> ?Thomas> environment variables and configuration options, and my request
> ?Thomas> of being able to change the cache directory and making sure
> ?Thomas> these changes are automatically in effect.
>
> ?Thomas> What do you think?
>
> ?Peter> Yes, something like:
>
> ?Peter> # use BR setting if not set in environment
> ?Peter> ifndef CCACHE_DIR
> ?Peter> CCACHE_DIR:=$(call qstrip,BR2_CCACHE_DIR)
> ?Peter> endif
>
> ?Peter> export CCACHE_DIR
>
> ?Peter> in the toplevel Makefile.
>
> Except that doesn't work. The GENTARGETS stuff for ccache already
> creates a CCACHE_DIR variable for the ccache build directory, and the
> export line only gets evaluated just before the targets are executed :/
>
> As variables are automatically reexported when they were set in the
> environment, this also means that we would always clutter the CCACHE_DIR
> environment setting.
>
> It's furthermore not easy to handle a BR_CCACHE_DIR variable containing
> shell variables (like $HOME) that should get expanded before make
> handles it.

What exactly is the problem here? Other configuration options can use
shell variables like $(USER) without problem. The only thing to keep
in mind is to write these variables in make syntax with parentheses,
rather than in shell syntax with either curly braces or nothing.

>
> So that leaves either:
>
> - Keep things like they are
>
> - Add a BR2_CCACHE_DIR but tell people that they cannot use $VAR in it

How would you pass it to ccache then, given the restriction you
mentioned above? On each compiler command-line?

Or were you thinking of compiling this variable hardcoded in ccache
using the existing patch?

>
> - Patch ccache to look for another environment variable
>
> - Rework GENTARGETS to not use <pkg>_DIR, but several packages would
> ?need to be updated, and users might rely on it for local packages

In itself I'm not sure whether this is a problem. With
buildroot-2011.05, the board support also changed significantly, which
required some rework in non-mainstream projects. Moreover, the change
would be nothing more than a variable rename.

>
> - Rename ccache package to something else, breaking existing .configs
>
> I think option two (defaulting to ~/.buildroot-ccache or perhaps
> ~/.ccache) is probably the best option, followed by 1.
>
> --
> Bye, Peter Korsgaard

[Buildroot] ccache directory

[Peter Korsgaard]

>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> writes:

Hi,

 >> It's furthermore not easy to handle a BR_CCACHE_DIR variable
 >> containing shell variables (like $HOME) that should get expanded
 >> before make handles it.

 Thomas> What exactly is the problem here? Other configuration options
 Thomas> can use shell variables like $(USER) without problem. The only
 Thomas> thing to keep in mind is to write these variables in make
 Thomas> syntax with parentheses, rather than in shell syntax with
 Thomas> either curly braces or nothing.

Indeed, that's the (potential) problem. People need to write the patch
in make syntax rather than shell, which may or may not be a problem if
they expect something similar to the CCACHE_DIR shell environment
variable.

 >> - Add a BR2_CCACHE_DIR but tell people that they cannot use $VAR in it

 Thomas> How would you pass it to ccache then, given the restriction you
 Thomas> mentioned above? On each compiler command-line?

 Thomas> Or were you thinking of compiling this variable hardcoded in ccache
 Thomas> using the existing patch?

Hardcode it like it is now would be simplest option.


 >> - Rework GENTARGETS to not use <pkg>_DIR, but several packages would
 >> ?need to be updated, and users might rely on it for local packages

 Thomas> In itself I'm not sure whether this is a problem. With
 Thomas> buildroot-2011.05, the board support also changed
 Thomas> significantly, which required some rework in non-mainstream
 Thomas> projects. Moreover, the change would be nothing more than a
 Thomas> variable rename.

True - It would still mean that we would have to fix up a number of
packages though, E.G.:

git grep '([A-Z0-9]*_DIR)' package/**/*.mk | \
    egrep -v 'STAGING_|BUILD_|TARGET_|HOST_|DL_' | \
    cut -f1 -d: | sort -u | wc -l
34

-- 
Bye, Peter Korsgaard

[Buildroot] ccache directory

[Yann E. MORIN]

On Sunday 24 July 2011 22:07:52 Peter Korsgaard wrote:
> >>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> writes:
>  >> It's furthermore not easy to handle a BR_CCACHE_DIR variable
>  >> containing shell variables (like $HOME) that should get expanded
>  >> before make handles it.
> 
>  Thomas> What exactly is the problem here? Other configuration options
>  Thomas> can use shell variables like $(USER) without problem. The only
>  Thomas> thing to keep in mind is to write these variables in make
>  Thomas> syntax with parentheses, rather than in shell syntax with
>  Thomas> either curly braces or nothing.
> 
> Indeed, that's the (potential) problem. People need to write the patch
> in make syntax rather than shell,

Well, parenthesis or braces, make does not mind:
  http://www.gnu.org/software/make/manual/make.html#Reference

Excerpt:
  To substitute a variable's value, write a dollar sign followed by the name
  of the variable in parentheses or braces: either ?$(foo)? or ?${foo}? is a
  valid reference to the variable foo.

Although parenthesis are being traditionally used in Makefile.

Regards,
Yann E. MORIN.

-- 
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 223 225 172 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'

[Buildroot] ccache directory

[Thomas De Schampheleire]

Hi,

On Sun, Jul 24, 2011 at 10:07 PM, Peter Korsgaard <jacmet@sunsite.dk> wrote:
>>>>>> "Thomas" == Thomas De Schampheleire <patrickdepinguin+buildroot@gmail.com> writes:
>
> Hi,
>
> ?>> It's furthermore not easy to handle a BR_CCACHE_DIR variable
> ?>> containing shell variables (like $HOME) that should get expanded
> ?>> before make handles it.
>
> ?Thomas> What exactly is the problem here? Other configuration options
> ?Thomas> can use shell variables like $(USER) without problem. The only
> ?Thomas> thing to keep in mind is to write these variables in make
> ?Thomas> syntax with parentheses, rather than in shell syntax with
> ?Thomas> either curly braces or nothing.
>
> Indeed, that's the (potential) problem. People need to write the patch
> in make syntax rather than shell, which may or may not be a problem if
> they expect something similar to the CCACHE_DIR shell environment
> variable.
>
> ?>> - Add a BR2_CCACHE_DIR but tell people that they cannot use $VAR in it
>
> ?Thomas> How would you pass it to ccache then, given the restriction you
> ?Thomas> mentioned above? On each compiler command-line?
>
> ?Thomas> Or were you thinking of compiling this variable hardcoded in ccache
> ?Thomas> using the existing patch?
>
> Hardcode it like it is now would be simplest option.

But has the disadvantage of not picking up changes in BR2_CCACHE_DIR
automatically; you have to force a rebuild of host-ccache.
Do you have any objection against passing the CCACHE_DIR on the CC /
CCX command-line? This does not interfere with gentargets, but does
allow for easy changing of the config option.

>
>
> ?>> - Rework GENTARGETS to not use <pkg>_DIR, but several packages would
> ?>> ?need to be updated, and users might rely on it for local packages
>
> ?Thomas> In itself I'm not sure whether this is a problem. With
> ?Thomas> buildroot-2011.05, the board support also changed
> ?Thomas> significantly, which required some rework in non-mainstream
> ?Thomas> projects. Moreover, the change would be nothing more than a
> ?Thomas> variable rename.
>
> True - It would still mean that we would have to fix up a number of
> packages though, E.G.:
>
> git grep '([A-Z0-9]*_DIR)' package/**/*.mk | \
> ? ?egrep -v 'STAGING_|BUILD_|TARGET_|HOST_|DL_' | \
> ? ?cut -f1 -d: | sort -u | wc -l
> 34
>
> --
> Bye, Peter Korsgaard
>