[PATCH 0/1] [RFC/RFT] Shorten file names in sstate-cache

[PATCH 0/1] [RFC/RFT] Shorten file names in sstate-cache

[Paul Barker]

As discussed briefly on IRC earlier in the week, here's a patch which attempts
to shorten the file name lengths seen in the sstate cache whilst retaining as
much useful human-readable info as possible.

This has been tested locally by building, wiping out tmp and then re-building to
confirm that sstate was still re-used correctly. I'm not an expert on sstate
internals though so please feed back with any comments or further testing that
should be done before this is merged to master.

I'd also like to know if there's a simpler method of limiting the length of a
variable expansion so that the python function limit_sstate_frag() can be
dropped.

Paul Barker (1):
  sstate: Shorten file names in sstate-cache

 meta/classes/sstate.bbclass | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

-- 
2.17.1

[PATCH 1/1] [RFC/RFT] sstate: Shorten file names in sstate-cache

[Paul Barker]

With the recent move to using sha256 checksums in sstate, the lengths of
some file names can now exceed the 255 char limit seen in most Linux
filesystems. When this occurs, bitbake crashes as it is unable to create
the sstate archive or sigfile.

To avoid this issue we need to shorten the length of the file names used
in the sstate cache. We can remove the 'sstate:' prefix as these files
already reside in a directory which only contains sstate files. We can
also remove the target triplet (set by the expansion of
${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}) as we also have the
package arch later via the expansion of ${SSTATE_PKGARCH}. And for
simplicity we can replace the expansion of ${PN}, ${PV} and ${PR} with
${PF}.

With the above changes we may still get bitten by packages with
exceptionally long names and/or version strings. To avoid this we limit
the length of the expansions of ${PF} to 100 chars and ${SSTATE_PKGARCH}
to 32 chars. Allowing a further 64 chars for the sha256 hash this leaves
about 50 chars for the task name and extensions (in the worst case)
which should be sufficient.

Signed-off-by: Paul Barker <paul@betafive.co.uk>
---
 meta/classes/sstate.bbclass | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/meta/classes/sstate.bbclass b/meta/classes/sstate.bbclass
index da0807d6e9..987218afaf 100644
--- a/meta/classes/sstate.bbclass
+++ b/meta/classes/sstate.bbclass
@@ -8,9 +8,12 @@ def generate_sstatefn(spec, hash, d):
         hash = "INVALID"
     return hash[:2] + "/" + spec + hash
 
+def limit_sstate_frag(frag, limit, d):
+    return frag[:limit]
+
 SSTATE_PKGARCH    = "${PACKAGE_ARCH}"
-SSTATE_PKGSPEC    = "sstate:${PN}:${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}:${PV}:${PR}:${SSTATE_PKGARCH}:${SSTATE_VERSION}:"
-SSTATE_SWSPEC     = "sstate:${PN}::${PV}:${PR}::${SSTATE_VERSION}:"
+SSTATE_PKGSPEC    = "${@limit_sstate_frag(d.getVar('PF'), 100, d)}:${@limit_sstate_frag(d.getVar('SSTATE_PKGARCH'), 32, d)}:${SSTATE_VERSION}:"
+SSTATE_SWSPEC     = "${@limit_sstate_frag(d.getVar('PF'), 100, d)}::${SSTATE_VERSION}:"
 SSTATE_PKGNAME    = "${SSTATE_EXTRAPATH}${@generate_sstatefn(d.getVar('SSTATE_PKGSPEC'), d.getVar('BB_UNIHASH'), d)}"
 SSTATE_PKG        = "${SSTATE_DIR}/${SSTATE_PKGNAME}"
 SSTATE_EXTRAPATH   = ""
-- 
2.17.1

Re: [PATCH 1/1] [RFC/RFT] sstate: Shorten file names in sstate-cache

[Richard Purdie]

Hi Paul,

On Fri, 2019-01-11 at 16:44 +0000, Paul Barker wrote:
> With the recent move to using sha256 checksums in sstate, the lengths
> of some file names can now exceed the 255 char limit seen in most
> Linux filesystems. When this occurs, bitbake crashes as it is unable
> to create the sstate archive or sigfile. 

> To avoid this issue we need to shorten the length of the file names
> used in the sstate cache. We can remove the 'sstate:' prefix as these
> files already reside in a directory which only contains sstate files.
> We can also remove the target triplet (set by the expansion of
> ${PACKAGE_ARCH}${TARGET_VENDOR}-${TARGET_OS}) as we also have the
> package arch later via the expansion of ${SSTATE_PKGARCH}. And for
> simplicity we can replace the expansion of ${PN}, ${PV} and ${PR}
> with ${PF}.

Thanks for looking at this. Firstly, could you split this patch up?
You're making multiple changes in one patch and I'd really like to make
it clear what we're changing here.

You've removed the sstate prefix. I'm torn on that since it does tell
you what the file is when looked at in isolation so I've mixed
feelings. I guess it was insurance in case we ever place other files in
the sstate directory.

In SSTATE_SWSPEC the expression is: "${PN}::${PV}:${PR}" and we need
to be very clear this is not equal to ${PF} (which is ${PN}-
${EXTENDPE}${PV}-${PR}). Its a different set of fields and a different
set of delimiters.

The delimiter was specifically chosen to be ":" so that we could stand
a chance of separating the data back out by machine. PN/PV/PR can
contain "-" which makes it a bad choice of delimiter and makes it hard
to delete " all the sstate for perl-native" or similar.

I agree that the duplication of PACKAGE_ARCH in there is probably
unneeded and can likely remove it, the ${TARGET_VENDOR}-${TARGET_OS}
piece may be useful to people with multiple distro configs. These may
be better captured in SSTATE_PKGARCH, not sure but it is a loss of
information.

I think we need to make a conscious decision on each of these changes
as well as length limitation changes.

Cheers,

Richard

✗ patchtest: failure for Shorten file names in sstate-cache

[Patchwork]

== Series Details ==

Series: Shorten file names in sstate-cache
Revision: 1
URL   : https://patchwork.openembedded.org/series/15617/
State : failure

== Summary ==


Thank you for submitting this patch series to OpenEmbedded Core. This is
an automated response. Several tests have been executed on the proposed
series by patchtest resulting in the following failures:



* Issue             Series does not apply on top of target branch [test_series_merge_on_head] 
  Suggested fix    Rebase your series on top of targeted branch
  Targeted branch  master (currently at 65c419b8c4)



If you believe any of these test results are incorrect, please reply to the
mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
Otherwise we would appreciate you correcting the issues and submitting a new
version of the patchset if applicable. Please ensure you add/increment the
version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
[PATCH v3] -> ...).

---
Guidelines:     https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe

Re: ✗ patchtest: failure for Shorten file names in sstate-cache

[Paul Barker]

On Fri, 11 Jan 2019, at 17:05, Patchwork wrote:
> == Series Details ==
> 
> Series: Shorten file names in sstate-cache
> Revision: 1
> URL   : https://patchwork.openembedded.org/series/15617/
> State : failure

That's weird... the patch applies fine onto master here, I've just tested applying it to a clean clone of openembedded-core.

> 
> == Summary ==
> 
> 
> Thank you for submitting this patch series to OpenEmbedded Core. This is
> an automated response. Several tests have been executed on the proposed
> series by patchtest resulting in the following failures:
> 
> 
> 
> * Issue             Series does not apply on top of target branch 
> [test_series_merge_on_head] 
>   Suggested fix    Rebase your series on top of targeted branch
>   Targeted branch  master (currently at 65c419b8c4)

Git can't find a commit hash starting with 65c419b8c4 in poky or oe-core.

> 
> 
> 
> If you believe any of these test results are incorrect, please reply to the
> mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
> Otherwise we would appreciate you correcting the issues and submitting a new
> version of the patchset if applicable. Please ensure you add/increment the
> version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
> [PATCH v3] -> ...).
> 
> ---
> Guidelines:     
> https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
> Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
> Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
> 


-- 
Paul Barker
Managing Director & Principal Engineer
Beta Five Ltd

Re: ✗ patchtest: failure for Shorten file names in sstate-cache

[Michael Halstead]

On 1/11/19 9:16 AM, Paul Barker wrote:
> On Fri, 11 Jan 2019, at 17:05, Patchwork wrote:
>> == Series Details ==
>>
>> Series: Shorten file names in sstate-cache
>> Revision: 1
>> URL   : https://patchwork.openembedded.org/series/15617/
>> State : failure
> That's weird... the patch applies fine onto master here, I've just tested applying it to a clean clone of openembedded-core.

The patchtest server was stuck resetting its working repository to a 
local merge. I've repaired the issue on the patchtest server and 
submitted 
https://lists.yoctoproject.org/pipermail/yocto/2019-January/043790.html 
to prevent it from reoccurring.

>> == Summary ==
>>
>>
>> Thank you for submitting this patch series to OpenEmbedded Core. This is
>> an automated response. Several tests have been executed on the proposed
>> series by patchtest resulting in the following failures:
>>
>>
>>
>> * Issue             Series does not apply on top of target branch
>> [test_series_merge_on_head]
>>    Suggested fix    Rebase your series on top of targeted branch
>>    Targeted branch  master (currently at 65c419b8c4)
> Git can't find a commit hash starting with 65c419b8c4 in poky or oe-core.
>
>>
>>
>> If you believe any of these test results are incorrect, please reply to the
>> mailing list (openembedded-core@lists.openembedded.org) raising your concerns.
>> Otherwise we would appreciate you correcting the issues and submitting a new
>> version of the patchset if applicable. Please ensure you add/increment the
>> version number when sending the new version (i.e. [PATCH] -> [PATCH v2] ->
>> [PATCH v3] -> ...).
>>
>> ---
>> Guidelines:
>> https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines
>> Test framework: http://git.yoctoproject.org/cgit/cgit.cgi/patchtest
>> Test suite:     http://git.yoctoproject.org/cgit/cgit.cgi/patchtest-oe
>>
>
-- 
Michael Halstead
Linux Foundation / SysAdmin