* Sharing sstate
@ 2011-09-27 14:22 Gary Thomas
2011-09-27 16:17 ` Richard Purdie
0 siblings, 1 reply; 6+ messages in thread
From: Gary Thomas @ 2011-09-27 14:22 UTC (permalink / raw)
To: Poky Project
I'd really like to be able to share sstate effectively. One problem I have with
it is the size - for a build of my system (something similar to core-image-sato
on BeagleBoard), the sstate cache for a build-from-scratch is nearly 5GB! Not
exactly simple to share across sites.
It seems to me that maybe I don't need to share everything. For example, the
webkit-gtk package:
$ ls -l sstate-cache/*webkit-gtk*
-rw-rw-r-- 1 gary gary 331221793 Sep 23 16:21
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz
-rw-rw-r-- 1 gary gary 12740 Sep 23 16:21
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz.siginfo
-rw-r--r-- 1 gary gary 326376172 Sep 23 16:15
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz
-rw-r--r-- 1 gary gary 21652 Sep 23 16:15
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz.siginfo
-rw-r--r-- 1 gary gary 665832323 Sep 23 16:19
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz
-rw-r--r-- 1 gary gary 98746 Sep 23 16:19
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz.siginfo
-rw-rw-r-- 1 gary gary 3117 Sep 23 15:38
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz
-rw-rw-r-- 1 gary gary 8359 Sep 23 15:38
sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz.siginfo
Would it be possible to only share part of this, maybe the "package" file?
It seems to me that this could make the needed (shared) cache much smaller...
--
------------------------------------------------------------
Gary Thomas | Consulting for the
MLB Associates | Embedded world
------------------------------------------------------------
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Sharing sstate
2011-09-27 14:22 Sharing sstate Gary Thomas
@ 2011-09-27 16:17 ` Richard Purdie
2011-09-28 2:25 ` Gary Thomas
0 siblings, 1 reply; 6+ messages in thread
From: Richard Purdie @ 2011-09-27 16:17 UTC (permalink / raw)
To: Gary Thomas; +Cc: Poky Project
On Tue, 2011-09-27 at 08:22 -0600, Gary Thomas wrote:
> I'd really like to be able to share sstate effectively. One problem I have with
> it is the size - for a build of my system (something similar to core-image-sato
> on BeagleBoard), the sstate cache for a build-from-scratch is nearly 5GB! Not
> exactly simple to share across sites.
>
> It seems to me that maybe I don't need to share everything. For example, the
> webkit-gtk package:
>
> $ ls -l sstate-cache/*webkit-gtk*
> -rw-rw-r-- 1 gary gary 331221793 Sep 23 16:21
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz
> -rw-rw-r-- 1 gary gary 12740 Sep 23 16:21
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz.siginfo
> -rw-r--r-- 1 gary gary 326376172 Sep 23 16:15
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz
> -rw-r--r-- 1 gary gary 21652 Sep 23 16:15
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz.siginfo
> -rw-r--r-- 1 gary gary 665832323 Sep 23 16:19
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz
> -rw-r--r-- 1 gary gary 98746 Sep 23 16:19
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz.siginfo
> -rw-rw-r-- 1 gary gary 3117 Sep 23 15:38
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz
> -rw-rw-r-- 1 gary gary 8359 Sep 23 15:38
> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz.siginfo
>
> Would it be possible to only share part of this, maybe the "package" file?
> It seems to me that this could make the needed (shared) cache much smaller...
The build will only use the pieces it needs. If your build only needs
the deploy-ipk pieces and doesn't need to write new packages files it
can survive without the package part. If it doesn't need to compile
something against webkit, it won't use the populate-sysroot file.
So what I'm saying is you likely could get away with not sharing the
*_package files at least. The only downside is if you turn on rpm
packaging it would have to rebuild.
Cheers,
Richard
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Sharing sstate
2011-09-27 16:17 ` Richard Purdie
@ 2011-09-28 2:25 ` Gary Thomas
0 siblings, 0 replies; 6+ messages in thread
From: Gary Thomas @ 2011-09-28 2:25 UTC (permalink / raw)
To: Richard Purdie; +Cc: Poky Project
On 2011-09-27 10:17, Richard Purdie wrote:
> On Tue, 2011-09-27 at 08:22 -0600, Gary Thomas wrote:
>> I'd really like to be able to share sstate effectively. One problem I have with
>> it is the size - for a build of my system (something similar to core-image-sato
>> on BeagleBoard), the sstate cache for a build-from-scratch is nearly 5GB! Not
>> exactly simple to share across sites.
>>
>> It seems to me that maybe I don't need to share everything. For example, the
>> webkit-gtk package:
>>
>> $ ls -l sstate-cache/*webkit-gtk*
>> -rw-rw-r-- 1 gary gary 331221793 Sep 23 16:21
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz
>> -rw-rw-r-- 1 gary gary 12740 Sep 23 16:21
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-4a04fc210c0a565767a2ad9bb11d8322_deploy-ipk.tgz.siginfo
>> -rw-r--r-- 1 gary gary 326376172 Sep 23 16:15
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz
>> -rw-r--r-- 1 gary gary 21652 Sep 23 16:15
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-67b4ac238844551dd48eed3255ea33d0_populate-sysroot.tgz.siginfo
>> -rw-r--r-- 1 gary gary 665832323 Sep 23 16:19
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz
>> -rw-r--r-- 1 gary gary 98746 Sep 23 16:19
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-726c31c08826bf24e6db77fdf5f4408b_package.tgz.siginfo
>> -rw-rw-r-- 1 gary gary 3117 Sep 23 15:38
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz
>> -rw-rw-r-- 1 gary gary 8359 Sep 23 15:38
>> sstate-cache/sstate-webkit-gtk-armv7a-vfp-neon-amltd-linux-gnueabi-1.5.1+svnr90727-r0-armv7a-vfp-neon-2-f9267d59a61b218bd6ab76ed55f12e8e_populate-lic.tgz.siginfo
>>
>> Would it be possible to only share part of this, maybe the "package" file?
>> It seems to me that this could make the needed (shared) cache much smaller...
>
> The build will only use the pieces it needs. If your build only needs
> the deploy-ipk pieces and doesn't need to write new packages files it
> can survive without the package part. If it doesn't need to compile
> something against webkit, it won't use the populate-sysroot file.
>
> So what I'm saying is you likely could get away with not sharing the
> *_package files at least. The only downside is if you turn on rpm
> packaging it would have to rebuild.
I tried removing the *package* files and the build ended up rebuilding
most things, in particular the toolchain bits. That's the part I really
want to be able to package the sstate for :-(
So then I tried having just a small subset of packages in the sstate
cache, but including all the pieces of each, like this:
rsync -auv sstate-cache/*eglibc-locale* \
sstate-cache/*autoconf-native* \
sstate-cache/*automake-native* \
sstate-cache/*bzip2-native* \
sstate-cache/*gettext-native* \
sstate-cache/*gnu-config-native* \
sstate-cache/*help2man-native* \
sstate-cache/*libtool-native* \
sstate-cache/*libxml2-native* \
sstate-cache/*m4-native* \
sstate-cache/*ncurses-native* \
sstate-cache/*openssl-native* \
sstate-cache/*pkgconfig-native* \
sstate-cache/*pseudo-native* \
sstate-cache/*python-native* \
sstate-cache/*quilt-native* \
sstate-cache/*readline-native* \
sstate-cache/*sqlite3-native* \
sstate-cache/*tar-replacement-native* \
sstate-cache/*zlib-native* \
sstate-cache/*gcc* \
sstate-cache/*binutils* \
sstate-cache/*eglibc* \
/tmp/sstate-test
This also fails, but differently:
ERROR: Task 756 (/home/local/poky-multi/meta/recipes-devtools/libtool/libtool-cross_2.4.bb, do_configure) failed with exit code '1'
looking at the config.log:
/home/local/p60_sstate_test/tmp/sysroots/i686-linux/usr/bin/armv7a-vfp-neon-amltd-linux-gnueabi/../../libexec/armv7a-vfp
-neon-amltd-linux-gnueabi/gcc/arm-amltd-linux-gnueabi/4.6.1/cc1: error while loading shared libraries: libmpc.so.2: cann
ot open shared object file: No such file or directory
Filed as http://bugzilla.pokylinux.org/show_bug.cgi?id=1536
--
------------------------------------------------------------
Gary Thomas | Consulting for the
MLB Associates | Embedded world
------------------------------------------------------------
^ permalink raw reply [flat|nested] 6+ messages in thread
* Sharing sstate
@ 2015-03-26 13:44 Gary Thomas
2015-03-26 23:13 ` Gary Thomas
2015-03-27 7:46 ` Mike Looijmans
0 siblings, 2 replies; 6+ messages in thread
From: Gary Thomas @ 2015-03-26 13:44 UTC (permalink / raw)
To: Yocto Project
I have two development servers, one running an older version
of Fedora (x86 *), the other runs a recent Ubuntu (x86_64).
Both are the same CPU core (Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz)
I've just tried building the same target, using the same version
of Poky/Yocto, sharing the sstate cache (fully populated) from
the x86_64 machine with the x86 machine. This was done using
HTTP. Sadly, there was not a single file found in the cache
that could be used.
Should this not be expected to work? Does sstate cache only
work across like systems (32 vs 64 bit hosts)?
Also, Mark Hatle mentioned a few days ago that there are ways
to "lock" versions of the sstate cache. I still haven't found
any info/documentation on this. Can someone please explain more?
Thanks
(*) I can't update my x86 machine to a more recent (x86_64)
distribution as I have many older "projects" that only run
on that hardware and no time/opportunity to upgrade, hence
my desire to share in this manner.
--
------------------------------------------------------------
Gary Thomas | Consulting for the
MLB Associates | Embedded world
------------------------------------------------------------
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Sharing sstate
2015-03-26 13:44 Gary Thomas
@ 2015-03-26 23:13 ` Gary Thomas
2015-03-27 7:46 ` Mike Looijmans
1 sibling, 0 replies; 6+ messages in thread
From: Gary Thomas @ 2015-03-26 23:13 UTC (permalink / raw)
To: yocto
On 2015-03-26 07:44, Gary Thomas wrote:
> I have two development servers, one running an older version
> of Fedora (x86 *), the other runs a recent Ubuntu (x86_64).
> Both are the same CPU core (Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz)
>
> I've just tried building the same target, using the same version
> of Poky/Yocto, sharing the sstate cache (fully populated) from
> the x86_64 machine with the x86 machine. This was done using
> HTTP. Sadly, there was not a single file found in the cache
> that could be used.
>
> Should this not be expected to work? Does sstate cache only
> work across like systems (32 vs 64 bit hosts)?
>
> Also, Mark Hatle mentioned a few days ago that there are ways
> to "lock" versions of the sstate cache. I still haven't found
> any info/documentation on this. Can someone please explain more?
>
> Thanks
>
> (*) I can't update my x86 machine to a more recent (x86_64)
> distribution as I have many older "projects" that only run
> on that hardware and no time/opportunity to upgrade, hence
> my desire to share in this manner.
>
I've just verified that if I move my build tree from the x86
machine to the x86_64 (really just build/conf), then sstate
works as expected.
--
------------------------------------------------------------
Gary Thomas | Consulting for the
MLB Associates | Embedded world
------------------------------------------------------------
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: Sharing sstate
2015-03-26 13:44 Gary Thomas
2015-03-26 23:13 ` Gary Thomas
@ 2015-03-27 7:46 ` Mike Looijmans
1 sibling, 0 replies; 6+ messages in thread
From: Mike Looijmans @ 2015-03-27 7:46 UTC (permalink / raw)
To: yocto
On 26-03-15 14:44, Gary Thomas wrote:
> I have two development servers, one running an older version
> of Fedora (x86 *), the other runs a recent Ubuntu (x86_64).
> Both are the same CPU core (Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz)
>
> I've just tried building the same target, using the same version
> of Poky/Yocto, sharing the sstate cache (fully populated) from
> the x86_64 machine with the x86 machine. This was done using
> HTTP. Sadly, there was not a single file found in the cache
> that could be used.
>
> Should this not be expected to work? Does sstate cache only
> work across like systems (32 vs 64 bit hosts)?
The hosts need to be similar for the build tools to be recycled. If they have
different OS versions, or even different CFLAGS you'll see a rebuild of
"native" packages.
It should however be able to recycle things for the target, because these
should not depend on the build host's configuration.
We currently have a "server" with ubuntu 12 on it, while most "clients" run
14. Typically the clients build their own tools, but they do re-use all target
packages (like FPGA bitstream and ARM software packages).
> Also, Mark Hatle mentioned a few days ago that there are ways
> to "lock" versions of the sstate cache. I still haven't found
> any info/documentation on this. Can someone please explain more?
>
> Thanks
>
> (*) I can't update my x86 machine to a more recent (x86_64)
> distribution as I have many older "projects" that only run
> on that hardware and no time/opportunity to upgrade, hence
> my desire to share in this manner.
>
Kind regards,
Mike Looijmans
System Expert
TOPIC Embedded Products
Eindhovenseweg 32-C, NL-5683 KH Best
Postbus 440, NL-5680 AK Best
Telefoon: +31 (0) 499 33 69 79
Telefax: +31 (0) 499 33 69 70
E-mail: mike.looijmans@topicproducts.com
Website: www.topicproducts.com
Please consider the environment before printing this e-mail
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2015-03-27 7:47 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2011-09-27 14:22 Sharing sstate Gary Thomas
2011-09-27 16:17 ` Richard Purdie
2011-09-28 2:25 ` Gary Thomas
-- strict thread matches above, loose matches on Subject: below --
2015-03-26 13:44 Gary Thomas
2015-03-26 23:13 ` Gary Thomas
2015-03-27 7:46 ` Mike Looijmans
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.