[Buildroot] SVN process started at make

[Buildroot] SVN process started at make

[Ruud Commandeur]

Hi Everyone,

Since today, all my 'make' commands in the buildroot dir take a very
long time before they actually start doing something. It appears to be
that on each make command, an svn process is started, that takes 5-10%
CPU time for a couple of minutes. Once this process stops, the 'normal'
make actions start.

My sources have been under svn control for a long time by using
SmartSVN, but since today I installed the commandline tool of
subversion. This must have triggered this behaviour, but can anyone
explain to me how I can prevent this? Is this someting in the
Makefile(s)?

Regards,

Ruud

[Buildroot] SVN process started at make

[Peter Korsgaard]

>>>>> "Ruud" == Ruud Commandeur <RCommandeur@clb.nl> writes:

 Ruud> Hi Everyone,
 Ruud> Since today, all my 'make' commands in the buildroot dir take a very
 Ruud> long time before they actually start doing something. It appears to be
 Ruud> that on each make command, an svn process is started, that takes 5-10%
 Ruud> CPU time for a couple of minutes. Once this process stops, the 'normal'
 Ruud> make actions start.

 Ruud> My sources have been under svn control for a long time by using
 Ruud> SmartSVN, but since today I installed the commandline tool of
 Ruud> subversion. This must have triggered this behaviour, but can anyone
 Ruud> explain to me how I can prevent this? Is this someting in the
 Ruud> Makefile(s)?

Odd. Would it be because of ./support/scripts/setlocalversion?

-- 
Bye, Peter Korsgaard

[Buildroot] SVN process started at make

[Arnout Vandecappelle]

On 09/03/13 16:50, Peter Korsgaard wrote:
>>>>>> "Ruud" == Ruud Commandeur <RCommandeur@clb.nl> writes:
>
>   Ruud> Hi Everyone,
>   Ruud> Since today, all my 'make' commands in the buildroot dir take a very
>   Ruud> long time before they actually start doing something. It appears to be
>   Ruud> that on each make command, an svn process is started, that takes 5-10%
>   Ruud> CPU time for a couple of minutes. Once this process stops, the 'normal'
>   Ruud> make actions start.
>
>   Ruud> My sources have been under svn control for a long time by using
>   Ruud> SmartSVN, but since today I installed the commandline tool of
>   Ruud> subversion. This must have triggered this behaviour, but can anyone
>   Ruud> explain to me how I can prevent this? Is this someting in the
>   Ruud> Makefile(s)?
>
> Odd. Would it be because of ./support/scripts/setlocalversion?

  I think it is. I experienced that as well (not personally since I use 
git-svn when the customer stores his buildroot tree in svn, but my 
colleagues complained about it).

  Maybe we should put something like

LOCALVERSION ?= $(shell $(TOPDIR)/support/scripts/setlocalversion)
BR2_VERSION_FULL := $(BR2_VERSION)$(LOCALVERSION)

in the top-level Makefile.


  Regards,
  Arnout
-- 
Arnout Vandecappelle                          arnout at mind be
Senior Embedded Software Architect            +32-16-286500
Essensium/Mind                                http://www.mind.be
G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 RPR Leuven
LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F

[Buildroot] SVN process started at make

[Ruud Commandeur]

Peter, Arnout,

Thanks for your reply. This does indeed seem to be the cause. If I make
this change to the top-level Makefile:

# Compute the full local version string so packages can use it as-is
# Need to export it, so it can be got from environment in children (eg.
mconf)
-export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell
$(TOPDIR)/support/scripts/setlocalversion)
+export BR2_VERSION_FULL:=$(BR2_VERSION)

things are back to normal.

Am I correct, that what this setlocalversion does, is add a revision
mark to the version string? Does it run that for each file to be
processed (since it can take that long)? For my build it reports
"2013.02-svn9-dirty". So it seems to check only a number of files and/or
folders, since the trunk is at revision 84 now, but some folders are
indeed at 9.

Anyway: for now I will use my changed Makefile. Please let me know if I
should try some other fix to be of help.

Regards,

Ruud

> -----Oorspronkelijk bericht-----
> Van: Arnout Vandecappelle [mailto:arnout at mind.be] 
> Verzonden: dinsdag 3 september 2013 18:28
> Aan: Peter Korsgaard
> CC: Ruud Commandeur; buildroot at uclibc.org
> Onderwerp: Re: [Buildroot] SVN process started at make
> 
> On 09/03/13 16:50, Peter Korsgaard wrote:
> >>>>>> "Ruud" == Ruud Commandeur <RCommandeur@clb.nl> writes:
> >
> >   Ruud> Hi Everyone,
> >   Ruud> Since today, all my 'make' commands in the 
> buildroot dir take a very
> >   Ruud> long time before they actually start doing 
> something. It appears to be
> >   Ruud> that on each make command, an svn process is 
> started, that takes 5-10%
> >   Ruud> CPU time for a couple of minutes. Once this process 
> stops, the 'normal'
> >   Ruud> make actions start.
> >
> >   Ruud> My sources have been under svn control for a long 
> time by using
> >   Ruud> SmartSVN, but since today I installed the 
> commandline tool of
> >   Ruud> subversion. This must have triggered this 
> behaviour, but can anyone
> >   Ruud> explain to me how I can prevent this? Is this 
> someting in the
> >   Ruud> Makefile(s)?
> >
> > Odd. Would it be because of ./support/scripts/setlocalversion?
> 
>   I think it is. I experienced that as well (not personally 
> since I use 
> git-svn when the customer stores his buildroot tree in svn, but my 
> colleagues complained about it).
> 
>   Maybe we should put something like
> 
> LOCALVERSION ?= $(shell $(TOPDIR)/support/scripts/setlocalversion)
> BR2_VERSION_FULL := $(BR2_VERSION)$(LOCALVERSION)
> 
> in the top-level Makefile.
> 
> 
>   Regards,
>   Arnout
> -- 
> Arnout Vandecappelle                          arnout at mind be
> Senior Embedded Software Architect            +32-16-286500
> Essensium/Mind                                http://www.mind.be
> G.Geenslaan 9, 3001 Leuven, Belgium           BE 872 984 063 
> RPR Leuven
> LinkedIn profile: http://www.linkedin.com/in/arnoutvandecappelle
> GPG fingerprint:  7CB5 E4CC 6C2E EFD4 6E3D A754 F963 ECAB 2450 2F1F
> 

[Buildroot] SVN process started at make

[Thomas Petazzoni]

Dear Ruud Commandeur,

On Wed, 4 Sep 2013 09:55:50 +0200, Ruud Commandeur wrote:

> Thanks for your reply. This does indeed seem to be the cause. If I make
> this change to the top-level Makefile:
> 
> # Compute the full local version string so packages can use it as-is
> # Need to export it, so it can be got from environment in children (eg.
> mconf)
> -export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell
> $(TOPDIR)/support/scripts/setlocalversion)
> +export BR2_VERSION_FULL:=$(BR2_VERSION)
> 
> things are back to normal.
> 
> Am I correct, that what this setlocalversion does, is add a revision
> mark to the version string? Does it run that for each file to be
> processed (since it can take that long)? For my build it reports
> "2013.02-svn9-dirty". So it seems to check only a number of files and/or
> folders, since the trunk is at revision 84 now, but some folders are
> indeed at 9.

You can check what this script is doing by looking at it in
support/scripts/setlocalversion. For SVN, it does:

# Check for svn and a svn repo.
if rev=`LC_ALL=C svn info 2>/dev/null | grep '^Last Changed Rev'`; then
        rev=`echo $rev | awk '{print $NF}'`
        changes=`LC_ALL=C svn status 2>/dev/null | grep '^[AMD]' | wc -l`

        # Are there uncommitted changes?
        if [ $changes != 0 ]; then
                printf -- '-svn%s%s' "$rev" -dirty
        else
                printf -- '-svn%s' "$rev"
        fi

        # All done with svn
        exit
fi

So there are two svn commands, and it normally does them only *once*
per build (i.e per invocation of make).

Thomas
-- 
Thomas Petazzoni, Free Electrons
Kernel, drivers, real-time and embedded Linux
development, consulting, training and support.
http://free-electrons.com

[Buildroot] SVN process started at make

[Ruud Commandeur]

Dear Thomas,

Well, I did take a quick look at this file, but could have seen this
myself indeed :-)

So it runs an "svn info" and "svn status". This last command can take
quite some time for a large repository, so this explains the delay.

Regards,

Ruud


> -----Oorspronkelijk bericht-----
> Van: Thomas Petazzoni [mailto:thomas.petazzoni at free-electrons.com] 
> Verzonden: woensdag 4 september 2013 10:08
> Aan: Ruud Commandeur
> CC: Arnout Vandecappelle; Peter Korsgaard; buildroot at uclibc.org
> Onderwerp: Re: [Buildroot] SVN process started at make
> 
> Dear Ruud Commandeur,
> 
> On Wed, 4 Sep 2013 09:55:50 +0200, Ruud Commandeur wrote:
> 
> > Thanks for your reply. This does indeed seem to be the 
> cause. If I make
> > this change to the top-level Makefile:
> > 
> > # Compute the full local version string so packages can use it as-is
> > # Need to export it, so it can be got from environment in 
> children (eg.
> > mconf)
> > -export BR2_VERSION_FULL:=$(BR2_VERSION)$(shell
> > $(TOPDIR)/support/scripts/setlocalversion)
> > +export BR2_VERSION_FULL:=$(BR2_VERSION)
> > 
> > things are back to normal.
> > 
> > Am I correct, that what this setlocalversion does, is add a revision
> > mark to the version string? Does it run that for each file to be
> > processed (since it can take that long)? For my build it reports
> > "2013.02-svn9-dirty". So it seems to check only a number of 
> files and/or
> > folders, since the trunk is at revision 84 now, but some folders are
> > indeed at 9.
> 
> You can check what this script is doing by looking at it in
> support/scripts/setlocalversion. For SVN, it does:
> 
> # Check for svn and a svn repo.
> if rev=`LC_ALL=C svn info 2>/dev/null | grep '^Last Changed 
> Rev'`; then
>         rev=`echo $rev | awk '{print $NF}'`
>         changes=`LC_ALL=C svn status 2>/dev/null | grep 
> '^[AMD]' | wc -l`
> 
>         # Are there uncommitted changes?
>         if [ $changes != 0 ]; then
>                 printf -- '-svn%s%s' "$rev" -dirty
>         else
>                 printf -- '-svn%s' "$rev"
>         fi
> 
>         # All done with svn
>         exit
> fi
> 
> So there are two svn commands, and it normally does them only *once*
> per build (i.e per invocation of make).
> 
> Thomas
> -- 
> Thomas Petazzoni, Free Electrons
> Kernel, drivers, real-time and embedded Linux
> development, consulting, training and support.
> http://free-electrons.com
> 

[Buildroot] SVN process started at make

[Peter Korsgaard]

>>>>> "Ruud" == Ruud Commandeur <RCommandeur@clb.nl> writes:

 Ruud> Dear Thomas,
 Ruud> Well, I did take a quick look at this file, but could have seen this
 Ruud> myself indeed :-)

 Ruud> So it runs an "svn info" and "svn status". This last command can take
 Ruud> quite some time for a large repository, so this explains the delay.

Ok, I'm afraid there's not much to do about svn being crap :/

How come you are only seeing this now? We've been calling
setlocalversion since 2010. Did you recently upgrade your svn version?

Looking at the kernel history (where setlocalversion comes from), I see
a commit from Mike Frysinger where the svn status step is skipped as it
was deemed too slow, so perhaps we should just resync our
setlocalversion script with the kernel:

Author: Mike Frysinger <vapier.adi@gmail.com>
Date:   Thu Feb 5 16:13:32 2009 +0800

    kbuild,setlocalversion: shorten the make time when using svn
    
    Don't bother doing `svn st` as it takes a retarded amount of time when
    the source is cold
    
    Signed-off-by: Mike Frysinger <vapier.adi@gmail.com>
    Signed-off-by: Bryan Wu <cooloney@kernel.org>
    Signed-off-by: Sam Ravnborg <sam@ravnborg.org>

diff --git a/scripts/setlocalversion b/scripts/setlocalversion
index f6946cf..f1c4b35 100755
--- a/scripts/setlocalversion
+++ b/scripts/setlocalversion
@@ -58,14 +58,7 @@ fi
 # Check for svn and a svn repo.
 if rev=`svn info 2>/dev/null | grep '^Last Changed Rev'`; then
        rev=`echo $rev | awk '{print $NF}'`
-       changes=`svn status 2>/dev/null | grep '^[AMD]' | wc -l`
-
-       # Are there uncommitted changes?
-       if [ $changes != 0 ]; then
-               printf -- '-svn%s%s' "$rev" -dirty
-       else
-               printf -- '-svn%s' "$rev"
-       fi
+       printf -- '-svn%s' "$rev"
 
        # All done with svn
        exit

I'll do that now.

-- 
Bye, Peter Korsgaard

[Buildroot] SVN process started at make

[Peter Korsgaard]

>>>>> "Ruud" == Ruud Commandeur <RCommandeur@clb.nl> writes:

 Peter> Ok, I'm afraid there's not much to do about svn being crap :/

 Peter> How come you are only seeing this now? We've been calling
 Peter> setlocalversion since 2010. Did you recently upgrade your svn version?

 Peter> Looking at the kernel history (where setlocalversion comes from), I see
 Peter> a commit from Mike Frysinger where the svn status step is skipped as it
 Peter> was deemed too slow, so perhaps we should just resync our
 Peter> setlocalversion script with the kernel:

 Peter> I'll do that now.

It is now in git. Could you give it a test please?

-- 
Bye, Peter Korsgaard

[Buildroot] SVN process started at make

[Ruud Commandeur]

Hi Peter,

I did upgrade the setlocalversion from
http://git.buildroot.net/buildroot/tree/support/scripts

Tested:

user at ubuntu-ruud:~/1122-ipcan/buildroot$ make print-version
2013.02-svn84
user at ubuntu-ruud:~/1122-ipcan/buildroot$ 

So the result is OK as might be expected, with a normal delay between
the 1st and 3rd line.

Thanks,

Ruud

> -----Oorspronkelijk bericht-----
> Van: Peter Korsgaard [mailto:jacmet at gmail.com] Namens Peter Korsgaard
> Verzonden: woensdag 4 september 2013 12:03
> Aan: Ruud Commandeur
> CC: Thomas Petazzoni; Arnout Vandecappelle; buildroot at uclibc.org
> Onderwerp: Re: SVN process started at make
> 
> >>>>> "Ruud" == Ruud Commandeur <RCommandeur@clb.nl> writes:
> 
>  Peter> Ok, I'm afraid there's not much to do about svn being crap :/
> 
>  Peter> How come you are only seeing this now? We've been calling
>  Peter> setlocalversion since 2010. Did you recently upgrade 
> your svn version?
> 
>  Peter> Looking at the kernel history (where setlocalversion 
> comes from), I see
>  Peter> a commit from Mike Frysinger where the svn status 
> step is skipped as it
>  Peter> was deemed too slow, so perhaps we should just resync our
>  Peter> setlocalversion script with the kernel:
> 
>  Peter> I'll do that now.
> 
> It is now in git. Could you give it a test please?
> 
> -- 
> Bye, Peter Korsgaard
> 

[Buildroot] SVN process started at make

[Ruud Commandeur]

Hi Peter,

I'm only seeing this now, because I installed the subversion package
yesterday. The project has been under svn control from the beginning
with SmartSvn, which has it's own svn built-in. But now the
setlocalversion script can finally do it's intended job.

The change from Mike Frysinger will certainly solve this problem, since
the "svn status" part is skipped. 

Regards,

Ruud
 

> -----Oorspronkelijk bericht-----
> Van: Peter Korsgaard [mailto:jacmet at gmail.com] Namens Peter Korsgaard
> Verzonden: woensdag 4 september 2013 11:15
> Aan: Ruud Commandeur
> CC: Thomas Petazzoni; Arnout Vandecappelle; Peter Korsgaard; 
> buildroot at uclibc.org
> Onderwerp: Re: SVN process started at make
> 
> >>>>> "Ruud" == Ruud Commandeur <RCommandeur@clb.nl> writes:
> 
>  Ruud> Dear Thomas,
>  Ruud> Well, I did take a quick look at this file, but could 
> have seen this
>  Ruud> myself indeed :-)
> 
>  Ruud> So it runs an "svn info" and "svn status". This last 
> command can take
>  Ruud> quite some time for a large repository, so this 
> explains the delay.
> 
> Ok, I'm afraid there's not much to do about svn being crap :/
> 
> How come you are only seeing this now? We've been calling
> setlocalversion since 2010. Did you recently upgrade your svn version?
> 
> Looking at the kernel history (where setlocalversion comes 
> from), I see
> a commit from Mike Frysinger where the svn status step is 
> skipped as it
> was deemed too slow, so perhaps we should just resync our
> setlocalversion script with the kernel:
> 
> Author: Mike Frysinger <vapier.adi@gmail.com>
> Date:   Thu Feb 5 16:13:32 2009 +0800
> 
>     kbuild,setlocalversion: shorten the make time when using svn
>     
>     Don't bother doing `svn st` as it takes a retarded amount 
> of time when
>     the source is cold
>     
>     Signed-off-by: Mike Frysinger <vapier.adi@gmail.com>
>     Signed-off-by: Bryan Wu <cooloney@kernel.org>
>     Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
> 
> diff --git a/scripts/setlocalversion b/scripts/setlocalversion
> index f6946cf..f1c4b35 100755
> --- a/scripts/setlocalversion
> +++ b/scripts/setlocalversion
> @@ -58,14 +58,7 @@ fi
>  # Check for svn and a svn repo.
>  if rev=`svn info 2>/dev/null | grep '^Last Changed Rev'`; then
>         rev=`echo $rev | awk '{print $NF}'`
> -       changes=`svn status 2>/dev/null | grep '^[AMD]' | wc -l`
> -
> -       # Are there uncommitted changes?
> -       if [ $changes != 0 ]; then
> -               printf -- '-svn%s%s' "$rev" -dirty
> -       else
> -               printf -- '-svn%s' "$rev"
> -       fi
> +       printf -- '-svn%s' "$rev"
>  
>         # All done with svn
>         exit
> 
> I'll do that now.
> 
> -- 
> Bye, Peter Korsgaard
>