script to remove orphaned files

script to remove orphaned files

[Frans Meulenbroeks]

Dear all,

As I indicated before there are quite some orphaned patch files that
are not used in any recipe.

To resolve this I've written a small bash script to remove unused
patches/files from a directory.
Below is a copy of the script.
This is posted to get feedback on the script (with the review comments
added this can probably be put in the contrib dir).
If the script is considered to be good, we can run it on the
directories that we have to clean them up.
My preference is that recipe owners do this, but as a lot of dirs are
orphaned, I am planning to do this myself in due time (after the
review and after giving recipe owners a chance to do it themselves).

Attached is the script. It takes a recipe dir as argument (e.g. recipes/gcc)
It searches for all .patch and .diff files in that dir, and checks if
they are mentioned in a .bb file or .inc file
if the file is there, nothing is done.
If the fileis not there and there is a -d argument before the dir
name, the files are actually removed and the change committed.
Otherwise this filename is just listed.

Proposal is to start with a 1 week script review period.

Frans.

The script:

#!/bin/bash
# clean-recipe: a small shell script to clean unneeded patch/diff
files from a recipe folder
if [ $# -eq 0 ]
then
    echo "usage " $0 "[-d] recipe-dir-name"
    exit
fi
delete=0
if [ $1 = "-d" ]
then
    delete=1
    shift;
fi
dir=$1
if [ ! -d $dir ]
then
    echo $dir " is not a directory"
    exit
fi
cd $dir
removed=0
find -name "*.diff" -o -name "*.patch" | (while  read name
    do
        bname=`basename $name`
        grep -q $bname *.bb *.inc || \
        if [ $delete -eq 0 ]
        then
        echo $name " is unused"
        else
        git rm $name
        removed=1
        fi
    done
    if [ $removed -eq 1 ]
    then
        echo $dir ": removed unneeded files" | git commit -s -F -
    fi )

Re: script to remove orphaned files

[Khem Raj]

On Mon, Aug 23, 2010 at 1:19 PM, Frans Meulenbroeks
<fransmeulenbroeks@gmail.com> wrote:
> Dear all,
>
> As I indicated before there are quite some orphaned patch files that
> are not used in any recipe.
>
> To resolve this I've written a small bash script to remove unused
> patches/files from a directory.
> Below is a copy of the script.
> This is posted to get feedback on the script (with the review comments
> added this can probably be put in the contrib dir).
> If the script is considered to be good, we can run it on the
> directories that we have to clean them up.
> My preference is that recipe owners do this, but as a lot of dirs are
> orphaned, I am planning to do this myself in due time (after the
> review and after giving recipe owners a chance to do it themselves).
>
> Attached is the script. It takes a recipe dir as argument (e.g. recipes/gcc)
> It searches for all .patch and .diff files in that dir, and checks if
> they are mentioned in a .bb file or .inc file
> if the file is there, nothing is done.
> If the fileis not there and there is a -d argument before the dir
> name, the files are actually removed and the change committed.
> Otherwise this filename is just listed.
>
> Proposal is to start with a 1 week script review period.
>
> Frans.
>
> The script:
>
> #!/bin/bash
> # clean-recipe: a small shell script to clean unneeded patch/diff
> files from a recipe folder
> if [ $# -eq 0 ]
> then
>    echo "usage " $0 "[-d] recipe-dir-name"
>    exit
> fi
> delete=0
> if [ $1 = "-d" ]
> then
>    delete=1
>    shift;
> fi
> dir=$1
> if [ ! -d $dir ]
> then
>    echo $dir " is not a directory"
>    exit
> fi
> cd $dir
> removed=0
> find -name "*.diff" -o -name "*.patch" | (while  read name
>    do
>        bname=`basename $name`
>        grep -q $bname *.bb *.inc || \
>        if [ $delete -eq 0 ]
>        then
>        echo $name " is unused"
>        else
>        git rm $name
>        removed=1
>        fi
>    done
>    if [ $removed -eq 1 ]
>    then
>        echo $dir ": removed unneeded files" | git commit -s -F -
>    fi )
>

Sometimes I run into a issue and then I look into the existing patches
and apply it or port it forward/backward
that solves it. This will be not easily possible once these patches
are removed. but thats not such a big issue I do
not have a strong opinion on this either way.



> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>

Re: script to remove orphaned files

[Chris Larson]

On Mon, Aug 23, 2010 at 11:32 PM, Frans Meulenbroeks <
fransmeulenbroeks@gmail.com> wrote:

> 2010/8/23 Khem Raj <raj.khem@gmail.com>:
> > On Mon, Aug 23, 2010 at 1:19 PM, Frans Meulenbroeks
>
> >
> > Sometimes I run into a issue and then I look into the existing patches
> > and apply it or port it forward/backward
> > that solves it. This will be not easily possible once these patches
> > are removed. but thats not such a big issue I do
> > not have a strong opinion on this either way.
> >
>
> I've been thinking about that as well. Onthe other hand some of these
> files seem to be relics of the past that apply to older versions and
> are not relevant at all.
> Anyway, if there is a maintainer who feels some files are useful, we
> can just keep them afaik. Just want to get rid of the orphaned stuff.
>
> BTW; there is one change needed for the script. Right after turning
> off my computer last evening I realised that some patches use ${PV}.
> Just did a quick check: there are 78 or so of them (and most are
> expanded only once). The script fails for these.
>
> Solution is to add the following line before the find:
> grep -q "file://.*\\$" *.bb *.inc && echo "cannot handle recipes with
> metavariables in the name" && exit
>
> Actually it would probably be nicer to have a parser that could parse
> the recipes and based upon SRC_URI could decide on what is needed and
> not.
> (and maybe also reorder things, detect for things that are syntaxwise
> ok, but that are an indication of an error (e.g. a double assignment
> of a variable).
> It could also tell about unused variables etc.
>
> Currently the parser is in bitbake, I have no idea on how to hook
> those kind of checks in the parser, or how to craft a python parser
> that would be able to do the above.
> It would be nice if someone with more python knowledge could give that a
> stab.
> BTW: I have been thinking about crafting a parser with lex/yacc, but
> especially the override mechanism does not allow to make a simple .y
> file for it (e.g. SRC_URI cannot be made a token because we also can
> have SRC_URI_xyz etc)


SRC_URI shouldn't be a token.  It's not special to the file format, it's
just a variable :)
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Re: script to remove orphaned files

[Frans Meulenbroeks]

2010/8/24 Chris Larson <clarson@kergoth.com>:
> On Mon, Aug 23, 2010 at 11:32 PM, Frans Meulenbroeks <
> fransmeulenbroeks@gmail.com> wrote:
>

>> Currently the parser is in bitbake, I have no idea on how to hook
>> those kind of checks in the parser, or how to craft a python parser
>> that would be able to do the above.
>> It would be nice if someone with more python knowledge could give that a
>> stab.
>> BTW: I have been thinking about crafting a parser with lex/yacc, but
>> especially the override mechanism does not allow to make a simple .y
>> file for it (e.g. SRC_URI cannot be made a token because we also can
>> have SRC_URI_xyz etc)
>
>
> SRC_URI shouldn't be a token.  It's not special to the file format, it's
> just a variable :)

True.
Language it is not different from another variable, but semantically
it has a specific meaning, that's why I also see it as a token
although technically it isn't.

Frans.

Re: script to remove orphaned files

[Frans Meulenbroeks]

As I don't really see progress on this, I am planning to take action
in a few days.
Since Khem expressed a preference to keep the files accessible, I'm
planning not to remove the unneeded patches, but instead move the to
the obsolete dir.
(alternate could be to move them to a dir called unused in the recipe dir).
Removal will be tested by verifying that bitbake -c patch remains
working on all dirs where unused files are removed.

Frans.

Re: script to remove orphaned files

[Khem Raj]

On Mon, Sep 13, 2010 at 1:20 PM, Frans Meulenbroeks
<fransmeulenbroeks@gmail.com> wrote:
> As I don't really see progress on this, I am planning to take action
> in a few days.
> Since Khem expressed a preference to keep the files accessible, I'm
> planning not to remove the unneeded patches, but instead move the to
> the obsolete dir.
> (alternate could be to move them to a dir called unused in the recipe dir).
> Removal will be tested by verifying that bitbake -c patch remains
> working on all dirs where unused files are removed.
>

sometimes same names patches exist in different dirs and if you remove
one then it will start picking the other one
silently so make sure such cases are not there. In general I am wary
of mass changes which could creep in some bugs like that
Ideally I would suggest that you look at each and every change before its final.

> Frans.
>
> _______________________________________________
> Openembedded-devel mailing list
> Openembedded-devel@lists.openembedded.org
> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>



-- 
-Khem

Re: script to remove orphaned files

[Frans Meulenbroeks]

2010/9/13 Khem Raj <raj.khem@gmail.com>:
> On Mon, Sep 13, 2010 at 1:20 PM, Frans Meulenbroeks
> <fransmeulenbroeks@gmail.com> wrote:
>> As I don't really see progress on this, I am planning to take action
>> in a few days.
>> Since Khem expressed a preference to keep the files accessible, I'm
>> planning not to remove the unneeded patches, but instead move the to
>> the obsolete dir.
>> (alternate could be to move them to a dir called unused in the recipe dir).
>> Removal will be tested by verifying that bitbake -c patch remains
>> working on all dirs where unused files are removed.
>>
>
> sometimes same names patches exist in different dirs and if you remove
> one then it will start picking the other one
> silently so make sure such cases are not there. In general I am wary
> of mass changes which could creep in some bugs like that
> Ideally I would suggest that you look at each and every change before its final.
>

I am aware of thi (and the trickery behind FILESDIR and FILESPATHPKG
(if I remembered the names right).
The starting point of this is the patch file.
If the file occurs in any recipe (*.bb or *.inc file) in that dir the
file will not be moved.

The script goes like:
find -name "*.diff" -o -name "*.patch" | (while  read name
   do
       bname=`basename $name`
       grep -q $bname *.bb *.inc || \
       if [ $delete -eq 0 ]
       then
       echo $name " is unused"
       else
# cmd to create dir before the move has to be added here
       git mv $name unused/$name
       removed=1
       fi
   done

Also recipes which have a file:// construct with ${PV} in it will be
untouched (or manually processed).
I can make and post patches for this but the amount will probably not
make anyone happy. (don't have the exact number of files handy, it is
listed before in an earlier thread, but I vaguley recall some 200+
files, not sure about the # of recipes affected.

Frans

Re: script to remove orphaned files

[Philip Balister]

On 09/13/2010 04:42 PM, Khem Raj wrote:
> On Mon, Sep 13, 2010 at 1:20 PM, Frans Meulenbroeks
> <fransmeulenbroeks@gmail.com>  wrote:
>> As I don't really see progress on this, I am planning to take action
>> in a few days.
>> Since Khem expressed a preference to keep the files accessible, I'm
>> planning not to remove the unneeded patches, but instead move the to
>> the obsolete dir.
>> (alternate could be to move them to a dir called unused in the recipe dir).
>> Removal will be tested by verifying that bitbake -c patch remains
>> working on all dirs where unused files are removed.
>>
>
> sometimes same names patches exist in different dirs and if you remove
> one then it will start picking the other one
> silently so make sure such cases are not there. In general I am wary
> of mass changes which could creep in some bugs like that
> Ideally I would suggest that you look at each and every change before its final.

I agree with Khem. please do this a directory of recipes at a time.

Philip

>
>> Frans.
>>
>> _______________________________________________
>> Openembedded-devel mailing list
>> Openembedded-devel@lists.openembedded.org
>> http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-devel
>>
>
>
>

Re: script to remove orphaned files

[Frans Meulenbroeks]

2010/9/13 Philip Balister <philip@balister.org>:
> On 09/13/2010 04:42 PM, Khem Raj wrote:
>>
>> On Mon, Sep 13, 2010 at 1:20 PM, Frans Meulenbroeks
>> <fransmeulenbroeks@gmail.com>  wrote:
>>>
>>> As I don't really see progress on this, I am planning to take action
>>> in a few days.
>>> Since Khem expressed a preference to keep the files accessible, I'm
>>> planning not to remove the unneeded patches, but instead move the to
>>> the obsolete dir.
>>> (alternate could be to move them to a dir called unused in the recipe
>>> dir).
>>> Removal will be tested by verifying that bitbake -c patch remains
>>> working on all dirs where unused files are removed.
>>>
>>
>> sometimes same names patches exist in different dirs and if you remove
>> one then it will start picking the other one
>> silently so make sure such cases are not there. In general I am wary
>> of mass changes which could creep in some bugs like that
>> Ideally I would suggest that you look at each and every change before its
>> final.
>
> I agree with Khem. please do this a directory of recipes at a time.
>
> Philip
>
Dear all,

I've just pushed the first few recipes. One at a time would stretch it
very long (there are some 160 recipe dirs with unused files), so was
planning to take the recipes starting with the same letter at the same
time (or if there only one or two recipes in it I might combine a
little bit, e.g. as a starter I've taken the recipes starting with z
(1 recipe folder with dups) and a (5 recipe folders with dups).
For letters that have many recipes with dups I'll probably split in two.
Btw I am not going to do this in strict alphabetical order. b and g
(which contain quite some dups for binutils and gcc) will probably be
done last.

And if a recipe maintainer wants to do the work him/herself: greatly
appreciated. Just go ahead.
I've pushed the script I use to do this. It is in contrib and called
clean-recipe.

The patches I just pushed are tested using bitbake -c patch for all
recipes as far as possible).
(in the recipes I pushed I could not test alsa-scenario_git.bb as it
didn't fetch for me and alsa-driver_0.9.6-hh4c.bb as it was for h3600
and h3900 only),
Results are both visually reviewed as well as a grep on failed. The
two failing recipes got an additional verification by checking that
the patches for it were not deleted.

Enjoy! Frans.

PS: wrt the remark of Khem: if a patch file is used in any recipe the
file is never removed. I'm staying at the safe side here (and the
downside is that there might still be a few unused files left, but at
least the bulk of the work is done, doing a full-fledged cleanup would
require parsing the bb files fully and emitting a list of all patches
that are applied (including directory), then remove the files in the
list. That can also deal with things like ${PV} in a patch. Probably
it is not overly complicated to write something like this, but it is
beyond my current capabilities, guess it would require instrumenting
do_patch to echo the name, then do a -c patchall or so). Anyway, no
plans to do that for now. FM.

Re: script to remove orphaned files

[Tom Rini]

Frans Meulenbroeks wrote:

> The patches I just pushed are tested using bitbake -c patch for all
> recipes as far as possible).
> (in the recipes I pushed I could not test alsa-scenario_git.bb as it
> didn't fetch for me and alsa-driver_0.9.6-hh4c.bb as it was for h3600
> and h3900 only),

Not to stop the work here, but is there a reason you couldn't do 
MACHINE=h3600 bitbake -c patch -b 
openembedded/recipes/alsa/alsa-driver_0.9.6-hh4c.bb ?  Yes, the initial 
parsing sucks but I think it's important for things like this to go out 
and find the case where a given recipe is supposed to work and make sure 
it still works.  "It didn't fetch" is a legitimate stopper (and I think 
candidate for moving to nonworking) of testing something.

-- 
Tom Rini
Mentor Graphics Corporation

Re: script to remove orphaned files

[Frans Meulenbroeks]

2010/9/16 Tom Rini <tom_rini@mentor.com>:
> Frans Meulenbroeks wrote:
>
>> The patches I just pushed are tested using bitbake -c patch for all
>> recipes as far as possible).
>> (in the recipes I pushed I could not test alsa-scenario_git.bb as it
>> didn't fetch for me and alsa-driver_0.9.6-hh4c.bb as it was for h3600
>> and h3900 only),
>
> Not to stop the work here, but is there a reason you couldn't do
> MACHINE=h3600 bitbake -c patch -b
> openembedded/recipes/alsa/alsa-driver_0.9.6-hh4c.bb ?  Yes, the initial
> parsing sucks but I think it's important for things like this to go out and
> find the case where a given recipe is supposed to work and make sure it
> still works.  "It didn't fetch" is a legitimate stopper (and I think
> candidate for moving to nonworking) of testing something.

I'll try the MACHINE = thingie tonight or tomorrow. Actually
originally I skipped this because I thought it will build me a whole
toolchain (which is quite time consuming) but thinking of it -c patch
will probably only require a few native recipes (like patch-native)
that already exists anyway, so it is limited to the parsing.

Wrt alsa-scenario: If it was not a git recipe I would have proposed it
for deletion, but maybe this is just a srcrev that does not exist any
more or so (after a merge).
Actually didn't check yesterday, but did so today, the git does not
clone for me.

frans@frans-desktop:/tmp$ git clone git://slimlogic.co.uk/alsa-scenario
Initialized empty Git repository in /tmp/alsa-scenario/.git/
fatal: The remote end hung up unexpectedly

Will post a separate message about it

Frans

Re: script to remove orphaned files

[Tom Rini]

Frans Meulenbroeks wrote:
> 2010/9/16 Tom Rini <tom_rini@mentor.com>:
>> Frans Meulenbroeks wrote:
>>
>>> The patches I just pushed are tested using bitbake -c patch for all
>>> recipes as far as possible).
>>> (in the recipes I pushed I could not test alsa-scenario_git.bb as it
>>> didn't fetch for me and alsa-driver_0.9.6-hh4c.bb as it was for h3600
>>> and h3900 only),
>> Not to stop the work here, but is there a reason you couldn't do
>> MACHINE=h3600 bitbake -c patch -b
>> openembedded/recipes/alsa/alsa-driver_0.9.6-hh4c.bb ?  Yes, the initial
>> parsing sucks but I think it's important for things like this to go out and
>> find the case where a given recipe is supposed to work and make sure it
>> still works.  "It didn't fetch" is a legitimate stopper (and I think
>> candidate for moving to nonworking) of testing something.
> 
> I'll try the MACHINE = thingie tonight or tomorrow. Actually

OK.  Note that doing it on the commandline means having it in 
BB_ENV_EXTRAWHITE.  Otherwise it's just short-hand for editing your 
local.conf :)  Thanks!

-- 
Tom Rini
Mentor Graphics Corporation

Re: script to remove orphaned files

[Frans Meulenbroeks]

2010/9/17 Tom Rini <tom_rini@mentor.com>:
> Frans Meulenbroeks wrote:
>>
>> 2010/9/16 Tom Rini <tom_rini@mentor.com>:
>>>
>>> Frans Meulenbroeks wrote:
>>>
>>>> The patches I just pushed are tested using bitbake -c patch for all
>>>> recipes as far as possible).
>>>> (in the recipes I pushed I could not test alsa-scenario_git.bb as it
>>>> didn't fetch for me and alsa-driver_0.9.6-hh4c.bb as it was for h3600
>>>> and h3900 only),
>>>
>>> Not to stop the work here, but is there a reason you couldn't do
>>> MACHINE=h3600 bitbake -c patch -b
>>> openembedded/recipes/alsa/alsa-driver_0.9.6-hh4c.bb ?  Yes, the initial
>>> parsing sucks but I think it's important for things like this to go out
>>> and
>>> find the case where a given recipe is supposed to work and make sure it
>>> still works.  "It didn't fetch" is a legitimate stopper (and I think
>>> candidate for moving to nonworking) of testing something.
>>
>> I'll try the MACHINE = thingie tonight or tomorrow. Actually
>
> OK.  Note that doing it on the commandline means having it in
> BB_ENV_EXTRAWHITE.  Otherwise it's just short-hand for editing your
> local.conf :)  Thanks!

I know.
Just tried it for openembedded/recipes/alsa/alsa-driver_0.9.6-hh4c.bb
This recipe fetches from ftp.handhelds.org which resolves to
h2.handhelds.org, but does not give data.

If I ftp to there from the cmd line I get connected but after a while I get
421 Service not available, remote server timed out. Connection closed

So this one seems dead too.

Frans

Re: script to remove orphaned files

[Frans Meulenbroeks]

2010/9/17 Frans Meulenbroeks <fransmeulenbroeks@gmail.com>:
> 2010/9/17 Tom Rini <tom_rini@mentor.com>:
>> Frans Meulenbroeks wrote:
>>>
>>> 2010/9/16 Tom Rini <tom_rini@mentor.com>:
>>>>
>>>> Frans Meulenbroeks wrote:
>>>>
>>>>> The patches I just pushed are tested using bitbake -c patch for all
>>>>> recipes as far as possible).
>>>>> (in the recipes I pushed I could not test alsa-scenario_git.bb as it
>>>>> didn't fetch for me and alsa-driver_0.9.6-hh4c.bb as it was for h3600
>>>>> and h3900 only),
>>>>
>>>> Not to stop the work here, but is there a reason you couldn't do
>>>> MACHINE=h3600 bitbake -c patch -b
>>>> openembedded/recipes/alsa/alsa-driver_0.9.6-hh4c.bb ?  Yes, the initial
>>>> parsing sucks but I think it's important for things like this to go out
>>>> and
>>>> find the case where a given recipe is supposed to work and make sure it
>>>> still works.  "It didn't fetch" is a legitimate stopper (and I think
>>>> candidate for moving to nonworking) of testing something.
>>>
>>> I'll try the MACHINE = thingie tonight or tomorrow. Actually
>>
>> OK.  Note that doing it on the commandline means having it in
>> BB_ENV_EXTRAWHITE.  Otherwise it's just short-hand for editing your
>> local.conf :)  Thanks!
>
> I know.
> Just tried it for openembedded/recipes/alsa/alsa-driver_0.9.6-hh4c.bb
> This recipe fetches from ftp.handhelds.org which resolves to
> h2.handhelds.org, but does not give data.
>
> If I ftp to there from the cmd line I get connected but after a while I get
> 421 Service not available, remote server timed out. Connection closed
>
> So this one seems dead too.
>
> Frans
>
Forgot to mention: no one has alsa-driver in its dependencies.
Perhaps this is from the stone age when alsa was not yet in the kernel.

Frans

Re: script to remove orphaned files

[Frans Meulenbroeks]

Dear all,

I just pushed the last batch of changes.
All recipes dirs have been cleaned except for boost (someone
explicitly asked not to clean it; if I recall correctly because he was
working on that recipe and that patch).
Should something have been moved that should not be moved, feel free
to revert the patch or restore the missing file from obsolete.

I would appreciate it if people would check every once-in-a-while if
they do not leave stale files behind.
A maintainer of a recipe is in a much better position to judge this than I am.

Also please note that I am not saying that there are no unused files left.
A few recipes used things like ${PV} in the patches. I did not process
these dirs.

Furthermore I did not analyse file paths or so. I've taken a cautious
approach, so if a patch file was referenced in any bb or inc file from
that recipe it was kept as it was too hard to reliably judge if *that*
copy was actually used.
So e.g. if you have v1/patch and v2/patch both files are still there
even though maybe only v2/patch was used.
Doing that kind of cleanup would require a recipe parser that marks
all files that it uses.
(and if someone can create such a thing, that would be nice).

Best regards, Frans.