* Re: [OE-core] About the sstate cache directory hierarchy
[not found] <16CE5D64E2C30C30.17092@lists.openembedded.org>
@ 2022-01-28 7:18 ` ChenQi
0 siblings, 0 replies; 5+ messages in thread
From: ChenQi @ 2022-01-28 7:18 UTC (permalink / raw)
To: openembedded-core@lists.openembedded.org, Richard Purdie
[-- Attachment #1: Type: text/plain, Size: 6291 bytes --]
On 1/28/22 3:06 PM, Chen Qi wrote:
> Hi All,
>
> I'm sending out this email because I'm wondering if we can change the
> sstate cache directory to use ${PN} and taskname as subditories. Hope
> to hear your opinions.
>
> Below is the long story.
>
> Recently I noticed that running `bitbake xxx -c cleansstate' usually
> takes more than 10 minutes.
>
> After some investigation, I can see that most of the time is spent on
> file searching. This is because we have:
>
> SSTATE_PATHSPEC =
> "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
>
Sorry. The above one is the patched one. Below is our current setting.
SSTATE_PATHSPEC =
"${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}*/*/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
> And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as
> sub-directories.
>
> This essentially means that all sub-directories are searched. This
> would take a long time, especially when run for the first time. I made
> some changes to output the time and the logs are as below.
>
> $ bitbake glibc -c cleansstate
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds
>
> I figured that unlike git, we do have knowledge on our sstate objects.
> It does not seem necessary to use hash value as sub directory. So I
> changed the sstate directory hierarchy to use ${PN}/taskname/ as sub
> directories, and here's the result.
>
> $ bitbake libgcc -c cleansstate
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.020630598068237305
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0011608600616455078
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007557868957519531
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0013995170593261719
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007488727569580078
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0005896091461181641
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125
> seconds
>
> It's much faster.
>
> In addition, the sub dirs now give more info, which should potentially
> make sstate cache easier to manage.
>
> Attached is the patch to quickly try things out. Hope to hear your
> opinions.
>
> Best Regards,
>
> Chen Qi
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161067): https://lists.openembedded.org/g/openembedded-core/message/161067
> Mute This Topic: https://lists.openembedded.org/mt/88740516/3618072
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [Qi.Chen@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
[-- Attachment #2: Type: text/html, Size: 8027 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core] About the sstate cache directory hierarchy
2022-01-28 7:06 ChenQi
@ 2022-01-28 10:29 ` Alexander Kanavin
2022-02-07 7:37 ` ChenQi
2022-05-16 5:30 ` Jacob Kroon
1 sibling, 1 reply; 5+ messages in thread
From: Alexander Kanavin @ 2022-01-28 10:29 UTC (permalink / raw)
To: Chen Qi; +Cc: openembedded-core@lists.openembedded.org, Richard Purdie
[-- Attachment #1: Type: text/plain, Size: 6388 bytes --]
I do like the idea; not everyone has a pcie gen4/5 ssd for builds, or
rigorously trims sstate on a schedule. But there may be consequences or
regressions, maybe RP will immediately shoot it down :)
I would however still place a single level of hash[:2] *under* the
pn/task/, to avoid too many files piling up in a single directory.
Can you send this as an RFC patch?
Alex
On Fri, 28 Jan 2022 at 08:06, Chen Qi <Qi.Chen@windriver.com> wrote:
> Hi All,
>
> I'm sending out this email because I'm wondering if we can change the
> sstate cache directory to use ${PN} and taskname as subditories. Hope to
> hear your opinions.
>
> Below is the long story.
>
> Recently I noticed that running `bitbake xxx -c cleansstate' usually
> takes more than 10 minutes.
>
> After some investigation, I can see that most of the time is spent on
> file searching. This is because we have:
>
> SSTATE_PATHSPEC =
>
> "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
>
> And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as
> sub-directories.
>
> This essentially means that all sub-directories are searched. This would
> take a long time, especially when run for the first time. I made some
> changes to output the time and the logs are as below.
>
> $ bitbake glibc -c cleansstate
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds
>
> I figured that unlike git, we do have knowledge on our sstate objects.
> It does not seem necessary to use hash value as sub directory. So I
> changed the sstate directory hierarchy to use ${PN}/taskname/ as sub
> directories, and here's the result.
>
> $ bitbake libgcc -c cleansstate
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.020630598068237305 seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0011608600616455078
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007557868957519531
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0013995170593261719
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007488727569580078
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0005896091461181641
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
>
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125 seconds
>
> It's much faster.
>
> In addition, the sub dirs now give more info, which should potentially
> make sstate cache easier to manage.
>
> Attached is the patch to quickly try things out. Hope to hear your
> opinions.
>
> Best Regards,
>
> Chen Qi
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161067):
> https://lists.openembedded.org/g/openembedded-core/message/161067
> Mute This Topic: https://lists.openembedded.org/mt/88740516/1686489
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
> alex.kanavin@gmail.com]
> -=-=-=-=-=-=-=-=-=-=-=-
>
>
[-- Attachment #2: Type: text/html, Size: 7554 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core] About the sstate cache directory hierarchy
2022-01-28 10:29 ` [OE-core] " Alexander Kanavin
@ 2022-02-07 7:37 ` ChenQi
0 siblings, 0 replies; 5+ messages in thread
From: ChenQi @ 2022-02-07 7:37 UTC (permalink / raw)
To: Alexander Kanavin
Cc: openembedded-core@lists.openembedded.org, Richard Purdie
[-- Attachment #1: Type: text/plain, Size: 7729 bytes --]
On 1/28/22 6:29 PM, Alexander Kanavin wrote:
> I do like the idea; not everyone has a pcie gen4/5 ssd for builds, or
> rigorously trims sstate on a schedule. But there may be consequences
> or regressions, maybe RP will immediately shoot it down :)
>
> I would however still place a single level of hash[:2] *under* the
> pn/task/, to avoid too many files piling up in a single directory.
>
> Can you send this as an RFC patch?
>
> Alex
Yes. I will ensure that all oe selftests pass before sending out patch.
Regards,
Qi
>
> On Fri, 28 Jan 2022 at 08:06, Chen Qi <Qi.Chen@windriver.com
> <mailto:Qi.Chen@windriver.com>> wrote:
>
> Hi All,
>
> I'm sending out this email because I'm wondering if we can change the
> sstate cache directory to use ${PN} and taskname as subditories.
> Hope to
> hear your opinions.
>
> Below is the long story.
>
> Recently I noticed that running `bitbake xxx -c cleansstate' usually
> takes more than 10 minutes.
>
> After some investigation, I can see that most of the time is spent on
> file searching. This is because we have:
>
> SSTATE_PATHSPEC =
> "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
>
> And our sstate cache directory's hierarchy uses
> hash[:2]/hash[2:4]/ as
> sub-directories.
>
> This essentially means that all sub-directories are searched. This
> would
> take a long time, especially when run for the first time. I made some
> changes to output the time and the logs are as below.
>
> $ bitbake glibc -c cleansstate
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst*
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds
>
> I figured that unlike git, we do have knowledge on our sstate
> objects.
> It does not seem necessary to use hash value as sub directory. So I
> changed the sstate directory hierarchy to use ${PN}/taskname/ as sub
> directories, and here's the result.
>
> $ bitbake libgcc -c cleansstate
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
> 0.020630598068237305 seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
> 0.0011608600616455078 seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
> 0.0007557868957519531 seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
> 0.0013995170593261719 seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
> 0.0007488727569580078 seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took
> 0.0005896091461181641 seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst*
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125
> seconds
>
> It's much faster.
>
> In addition, the sub dirs now give more info, which should
> potentially
> make sstate cache easier to manage.
>
> Attached is the patch to quickly try things out. Hope to hear your
> opinions.
>
> Best Regards,
>
> Chen Qi
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#161067):
> https://lists.openembedded.org/g/openembedded-core/message/161067
> <https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/message/161067__;!!AjveYdw8EvQ!N5Dyvu4eenTGK1TY1BrreaZ7uMlcahVqvjTbg01fXC5Sptlu1wS-nKbk5PGtw6ODQA$>
> Mute This Topic:
> https://lists.openembedded.org/mt/88740516/1686489
> <https://urldefense.com/v3/__https://lists.openembedded.org/mt/88740516/1686489__;!!AjveYdw8EvQ!N5Dyvu4eenTGK1TY1BrreaZ7uMlcahVqvjTbg01fXC5Sptlu1wS-nKbk5PGWaXA87A$>
> Group Owner: openembedded-core+owner@lists.openembedded.org
> <mailto:openembedded-core%2Bowner@lists.openembedded.org>
> Unsubscribe:
> https://lists.openembedded.org/g/openembedded-core/unsub
> <https://urldefense.com/v3/__https://lists.openembedded.org/g/openembedded-core/unsub__;!!AjveYdw8EvQ!N5Dyvu4eenTGK1TY1BrreaZ7uMlcahVqvjTbg01fXC5Sptlu1wS-nKbk5PEkTurZng$>
> [alex.kanavin@gmail.com <mailto:alex.kanavin@gmail.com>]
> -=-=-=-=-=-=-=-=-=-=-=-
>
[-- Attachment #2: Type: text/html, Size: 9923 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [OE-core] About the sstate cache directory hierarchy
2022-01-28 7:06 ChenQi
2022-01-28 10:29 ` [OE-core] " Alexander Kanavin
@ 2022-05-16 5:30 ` Jacob Kroon
2022-05-16 5:35 ` Chen, Qi
1 sibling, 1 reply; 5+ messages in thread
From: Jacob Kroon @ 2022-05-16 5:30 UTC (permalink / raw)
To: Chen Qi, openembedded-core@lists.openembedded.org, Richard Purdie
On 1/28/22 08:06, Chen Qi wrote:
> Hi All,
>
> I'm sending out this email because I'm wondering if we can change the
> sstate cache directory to use ${PN} and taskname as subditories. Hope to
> hear your opinions.
>
> Below is the long story.
>
> Recently I noticed that running `bitbake xxx -c cleansstate' usually
> takes more than 10 minutes.
>
> After some investigation, I can see that most of the time is spent on
> file searching. This is because we have:
>
> SSTATE_PATHSPEC =
> "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
>
>
> And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as
> sub-directories.
>
> This essentially means that all sub-directories are searched. This would
> take a long time, especially when run for the first time. I made some
> changes to output the time and the logs are as below.
>
> $ bitbake glibc -c cleansstate
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.34:r0::7:*_populate_lic.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:core2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds
>
> I figured that unlike git, we do have knowledge on our sstate objects.
> It does not seem necessary to use hash value as sub directory. So I
> changed the sstate directory hierarchy to use ${PN}/taskname/ as sub
> directories, and here's the result.
>
> $ bitbake libgcc -c cleansstate
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source_date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.020630598068237305 seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0011608600616455078
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007557868957519531
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write_rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_write_rpm.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0013995170593261719
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007488727569580078
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0005896091461181641
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysroot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_sysroot.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125 seconds
>
> It's much faster.
>
> In addition, the sub dirs now give more info, which should potentially
> make sstate cache easier to manage.
>
> Attached is the patch to quickly try things out. Hope to hear your
> opinions.
>
> Best Regards,
>
> Chen Qi
>
I thought this sounded interesting. Has anything come out of this that I
missed ?
Jacob
^ permalink raw reply [flat|nested] 5+ messages in thread
* RE: [OE-core] About the sstate cache directory hierarchy
2022-05-16 5:30 ` Jacob Kroon
@ 2022-05-16 5:35 ` Chen, Qi
0 siblings, 0 replies; 5+ messages in thread
From: Chen, Qi @ 2022-05-16 5:35 UTC (permalink / raw)
To: Jacob Kroon, openembedded-core@lists.openembedded.org,
Richard Purdie
Hi Jacob,
No. I haven't done anything since then, because I've been involved in something else.
Regards,
Qi
-----Original Message-----
From: Jacob Kroon <jacob.kroon@gmail.com>
Sent: Monday, May 16, 2022 1:30 PM
To: Chen, Qi <Qi.Chen@windriver.com>; openembedded-core@lists.openembedded.org; Richard Purdie <richard.purdie@linuxfoundation.org>
Subject: Re: [OE-core] About the sstate cache directory hierarchy
On 1/28/22 08:06, Chen Qi wrote:
> Hi All,
>
> I'm sending out this email because I'm wondering if we can change the
> sstate cache directory to use ${PN} and taskname as subditories. Hope
> to hear your opinions.
>
> Below is the long story.
>
> Recently I noticed that running `bitbake xxx -c cleansstate' usually
> takes more than 10 minutes.
>
> After some investigation, I can see that most of the time is spent on
> file searching. This is because we have:
>
> SSTATE_PATHSPEC =
> "${SSTATE_DIR}/${SSTATE_EXTRAPATHWILDCARD}${PN}/${SSTATE_PATH_CURRTASK}/${SSTATE_PKGSPEC}*_${SSTATE_PATH_CURRTASK}.tar.zst*"
>
>
> And our sstate cache directory's hierarchy uses hash[:2]/hash[2:4]/ as
> sub-directories.
>
> This essentially means that all sub-directories are searched. This
> would take a long time, especially when run for the first time. I made
> some changes to output the time and the logs are as below.
>
> $ bitbake glibc -c cleansstate
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor
> e2-64-poky-linux:2.34:r0:core2-64:7:*_deploy_source_date_epoch.tar.zst
> *
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 611.8865714073181 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor
> e2-64-poky-linux:2.34:r0:core2-64:7:*_package.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.3219327926635742 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor
> e2-64-poky-linux:2.34:r0:core2-64:7:*_package_qa.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.470815658569336 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor
> e2-64-poky-linux:2.34:r0:core2-64:7:*_package_write_rpm.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.251939058303833 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor
> e2-64-poky-linux:2.34:r0:core2-64:7:*_packagedata.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.2369801998138428 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc::2.
> 34:r0::7:*_populate_lic.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.1668426990509033 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor
> e2-64-poky-linux:2.34:r0:core2-64:7:*_populate_sysroot.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.385568380355835 seconds
> WARNING: glibc-2.34-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/*/*/sstate:glibc:cor
> e2-64-poky-linux:2.34:r0:core2-64:7:*_stash_locale.tar.zst*
>
> WARNING: glibc-2.34-r0 do_cleansstate: Took 1.4884181022644043 seconds
>
> I figured that unlike git, we do have knowledge on our sstate objects.
> It does not seem necessary to use hash value as sub directory. So I
> changed the sstate directory hierarchy to use ${PN}/taskname/ as sub
> directories, and here's the result.
>
> $ bitbake libgcc -c cleansstate
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/deploy_source
> _date_epoch/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_d
> eploy_source_date_epoch.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.020630598068237305
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package/sstat
> e:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0011608600616455078
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_qa/ss
> tate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_qa.tar.
> zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007557868957519531
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/package_write
> _rpm/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_package_
> write_rpm.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0013995170593261719
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/packagedata/s
> state:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_packagedata.ta
> r.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0007488727569580078
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_lic/
> sstate:libgcc::11.2.0:r0::7:*_populate_lic.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.0005896091461181641
> seconds
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Removing
> /ala-lpggp72/qichen/LAT/builds/share/sstate-cache/libgcc/populate_sysr
> oot/sstate:libgcc:core2-64-poky-linux:11.2.0:r0:core2-64:7:*_populate_
> sysroot.tar.zst*
>
> WARNING: libgcc-11.2.0-r0 do_cleansstate: Took 0.00080108642578125
> seconds
>
> It's much faster.
>
> In addition, the sub dirs now give more info, which should potentially
> make sstate cache easier to manage.
>
> Attached is the patch to quickly try things out. Hope to hear your
> opinions.
>
> Best Regards,
>
> Chen Qi
>
I thought this sounded interesting. Has anything come out of this that I missed ?
Jacob
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2022-05-16 5:35 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
[not found] <16CE5D64E2C30C30.17092@lists.openembedded.org>
2022-01-28 7:18 ` [OE-core] About the sstate cache directory hierarchy ChenQi
2022-01-28 7:06 ChenQi
2022-01-28 10:29 ` [OE-core] " Alexander Kanavin
2022-02-07 7:37 ` ChenQi
2022-05-16 5:30 ` Jacob Kroon
2022-05-16 5:35 ` Chen, Qi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox