[RFC] do we really need all OVERRIDES in FILESPATH?

[RFC] do we really need all OVERRIDES in FILESPATH?

[Martin Jansa]

Currently
bitbake -e -b xserver-xorg-conf_0.1.bb | grep ^FILESPATH= | sed "s/:/\n/g" | wc -l
shows 65 directories where file:// from SRC_URI can be found.

base_do_unpack is looking for first directory where requested file
exists. Most files are IMHO found in FILESPATHPKG PN, files or P
(without an override used)

Number of directories tried before
PN:    38
files: 51
P:     25

I see many recipes really using that MACHINE or DISTRO is in FILESPATH,
few users of TARGET_ARCH and quick find/grep doesn't show any users of 
other OVERRIDES in FILESPATH.

BTW: ie initscripts have initscripts/files/arm/alignment.sh but that's only
alignment.sh and SRC_URI_append_arm = " file://alignment.sh" would work
ok even without arm in FILESPATH.

What about using only ${TARGET_ARCH}:${DISTRO}:${MACHINE} in FILESPATH
instead all OVERRIDES?

And if there is need to use some special OVERRIDE then recipe can use
SRC_URI_append_override.

Number of directories tried before
PN:    11
files: 15
P:      7
end:   20

If I'm overlooking other usages of FILESPATH or if this I/O saving is
not worth it, let me know.

Regards,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

Re: [RFC] do we really need all OVERRIDES in FILESPATH?

[Koen Kooi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 17-10-10 23:15, Martin Jansa wrote:
> Currently
> bitbake -e -b xserver-xorg-conf_0.1.bb | grep ^FILESPATH= | sed "s/:/\n/g" | wc -l
> shows 65 directories where file:// from SRC_URI can be found.
> 
> base_do_unpack is looking for first directory where requested file
> exists. Most files are IMHO found in FILESPATHPKG PN, files or P
> (without an override used)
> 
> Number of directories tried before
> PN:    38
> files: 51
> P:     25
> 
> I see many recipes really using that MACHINE or DISTRO is in FILESPATH,
> few users of TARGET_ARCH and quick find/grep doesn't show any users of 
> other OVERRIDES in FILESPATH.
> 
> BTW: ie initscripts have initscripts/files/arm/alignment.sh but that's only
> alignment.sh and SRC_URI_append_arm = " file://alignment.sh" would work
> ok even without arm in FILESPATH.
> 
> What about using only ${TARGET_ARCH}:${DISTRO}:${MACHINE} in FILESPATH
> instead all OVERRIDES?

At least I additionally need BASE_PACKAGE_ARCH and SOC_FAMILY in it and
sometimes libc-$LIBC.

regards,

Koen
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Darwin)

iD8DBQFMu/V+MkyGM64RGpERAk8TAKCNwmBmVhDmFC0jzhrxWVo3WxI9iwCfRrGd
03R+AyA6hh/3vtl/OBdnbkc=
=iUyw
-----END PGP SIGNATURE-----

Re: [RFC] do we really need all OVERRIDES in FILESPATH?

[Frans Meulenbroeks]

2010/10/17 Martin Jansa <martin.jansa@gmail.com>:
> Currently
> bitbake -e -b xserver-xorg-conf_0.1.bb | grep ^FILESPATH= | sed "s/:/\n/g" | wc -l
> shows 65 directories where file:// from SRC_URI can be found.

Actually I got 84. This is git head from last friday or so. Seems a
good plan to reduce this somewhat.
BTW this was for:
DISTRO = "angstrom-2008.1"
MACHINE = "ixp4xxbe"

Frans

frans@frans-desktop:~/workspace/openembedded.git/recipes/xserver-common$
bitbake -e -b xserver-xorg-conf_0.1.bb | grep ^FILESPATH= | sed
"s/:/\n/g" | wc -l
84
frans@frans-desktop:~/workspace/openembedded.git/recipes/xserver-common$
bitbake -e -b xserver-xorg-conf_0.1.bb | grep ^FILESPATH= | sed
"s/:/\n/g"
FILESPATH="/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34/local
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34/ixp4xxbe
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34/angstrom
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34/linux-gnueabi
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34/armeb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34/build-linux
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34/fail-fast
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34/pn-xserver-xorg-conf
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34/armv5teb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34/Unknown
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34/libc-glibc
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1-r34
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/local
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/ixp4xxbe
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/angstrom
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/linux-gnueabi
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/armeb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/build-linux
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/fail-fast
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/pn-xserver-xorg-conf
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/armv5teb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/Unknown
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/libc-glibc
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/local
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/ixp4xxbe
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/angstrom
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/linux-gnueabi
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/armeb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/build-linux
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/fail-fast
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/pn-xserver-xorg-conf
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/armv5teb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/Unknown
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/libc-glibc
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/local
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/ixp4xxbe
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/angstrom
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/linux-gnueabi
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/armeb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/build-linux
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/fail-fast
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/pn-xserver-xorg-conf
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/armv5teb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/Unknown
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1/libc-glibc
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf-0.1
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/local
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/ixp4xxbe
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/angstrom
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/linux-gnueabi
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/armeb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/build-linux
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/fail-fast
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/pn-xserver-xorg-conf
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/armv5teb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/Unknown
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf/libc-glibc
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/xserver-xorg-conf
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files/local
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files/ixp4xxbe
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files/angstrom
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files/linux-gnueabi
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files/armeb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files/build-linux
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files/fail-fast
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files/pn-xserver-xorg-conf
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files/armv5teb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files/Unknown
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files/libc-glibc
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/files
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/local
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/ixp4xxbe
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/angstrom
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/linux-gnueabi
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/armeb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/build-linux
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/fail-fast
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/pn-xserver-xorg-conf
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/armv5teb
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/Unknown
/home/frans/workspace/openembedded.git/recipes/xorg-xserver/libc-glibc
/home/frans/workspace/openembedded.git/recipes/xorg-xserver"

Re: [RFC] do we really need all OVERRIDES in FILESPATH?

[Martin Jansa]

On Mon, Oct 18, 2010 at 09:33:28AM +0200, Frans Meulenbroeks wrote:
> 2010/10/17 Martin Jansa <martin.jansa@gmail.com>:
> > Currently
> > bitbake -e -b xserver-xorg-conf_0.1.bb | grep ^FILESPATH= | sed "s/:/\n/g" | wc -l
> > shows 65 directories where file:// from SRC_URI can be found.
> 
> Actually I got 84. This is git head from last friday or so. Seems a
> good plan to reduce this somewhat.
> BTW this was for:
> DISTRO = "angstrom-2008.1"
> MACHINE = "ixp4xxbe"

Why do you have P and PN twice in FILESPATHPKG? something in local.conf?
I don't see any in-tree conf/class changing FILESPATHPKG that way

Regards,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

Re: [RFC] do we really need all OVERRIDES in FILESPATH?

[Richard Purdie]

On Mon, 2010-10-18 at 09:21 +0200, Koen Kooi wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 17-10-10 23:15, Martin Jansa wrote:
> > Currently
> > bitbake -e -b xserver-xorg-conf_0.1.bb | grep ^FILESPATH= | sed "s/:/\n/g" | wc -l
> > shows 65 directories where file:// from SRC_URI can be found.
> > 
> > base_do_unpack is looking for first directory where requested file
> > exists. Most files are IMHO found in FILESPATHPKG PN, files or P
> > (without an override used)
> > 
> > Number of directories tried before
> > PN:    38
> > files: 51
> > P:     25
> > 
> > I see many recipes really using that MACHINE or DISTRO is in FILESPATH,
> > few users of TARGET_ARCH and quick find/grep doesn't show any users of 
> > other OVERRIDES in FILESPATH.
> > 
> > BTW: ie initscripts have initscripts/files/arm/alignment.sh but that's only
> > alignment.sh and SRC_URI_append_arm = " file://alignment.sh" would work
> > ok even without arm in FILESPATH.
> > 
> > What about using only ${TARGET_ARCH}:${DISTRO}:${MACHINE} in FILESPATH
> > instead all OVERRIDES?
> 
> At least I additionally need BASE_PACKAGE_ARCH and SOC_FAMILY in it and
> sometimes libc-$LIBC.

I have to admit, the structure of FILESPATH is a little unwieldy to me.
I'd propose that we should have a small set of defaults like ${BPN} and
${BPN}-${PV} and if any recipe wants more than this, it should have a
variable it can append a list to.

I'd also suggest "files" should probably be deprecated in favour of BPN.

It becomes obvious how horrible the current structure is when you try
and use .bbappend and need to add some extra directory to the search
path.

Cheers,

Richard

Re: [RFC] do we really need all OVERRIDES in FILESPATH?

[Frans Meulenbroeks]

2010/10/18 Richard Purdie <rpurdie@rpsys.net>:
> On Mon, 2010-10-18 at 09:21 +0200, Koen Kooi wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 17-10-10 23:15, Martin Jansa wrote:
>> > Currently
>> > bitbake -e -b xserver-xorg-conf_0.1.bb | grep ^FILESPATH= | sed "s/:/\n/g" | wc -l
>> > shows 65 directories where file:// from SRC_URI can be found.
>> >
>> > base_do_unpack is looking for first directory where requested file
>> > exists. Most files are IMHO found in FILESPATHPKG PN, files or P
>> > (without an override used)
>> >
>> > Number of directories tried before
>> > PN:    38
>> > files: 51
>> > P:     25
>> >
>> > I see many recipes really using that MACHINE or DISTRO is in FILESPATH,
>> > few users of TARGET_ARCH and quick find/grep doesn't show any users of
>> > other OVERRIDES in FILESPATH.
>> >
>> > BTW: ie initscripts have initscripts/files/arm/alignment.sh but that's only
>> > alignment.sh and SRC_URI_append_arm = " file://alignment.sh" would work
>> > ok even without arm in FILESPATH.
>> >
>> > What about using only ${TARGET_ARCH}:${DISTRO}:${MACHINE} in FILESPATH
>> > instead all OVERRIDES?
>>
>> At least I additionally need BASE_PACKAGE_ARCH and SOC_FAMILY in it and
>> sometimes libc-$LIBC.
>
> I have to admit, the structure of FILESPATH is a little unwieldy to me.
> I'd propose that we should have a small set of defaults like ${BPN} and
> ${BPN}-${PV} and if any recipe wants more than this, it should have a
> variable it can append a list to.
>
> I'd also suggest "files" should probably be deprecated in favour of BPN.

I'm all in favour of that but....
- we currently have 724 files directories (so this is some work)
- I'm not fully sure this will work for packages that have recipes
with different names (e.g. alsa), ofc we could rework these and add a
FILESPATHPKG
- not sure if removing files completely will not cause any problems
for the top level files dir (which contains some static device tables
(then again I didn't really found that to be an appropriate place
either)

BTW in my overlay in the top level files folders I also keep my own
busybox config. Not sure how others deal with that, but I can image
that also becomes broken
>
> It becomes obvious how horrible the current structure is when you try
> and use .bbappend and need to add some extra directory to the search
> path.
>

Agree. Frans.