[PATCH] bitbake: fetch2/svn: Add tarsnapshots parameter

[PATCH] bitbake: fetch2/svn: Add tarsnapshots parameter

[Niko Mauno]

By adding 'tarsnapshots=currentonly' option to svn-specific component in
SRC_URI, associated subversion repository module's existing tarball
snapshots are removed from DL_DIR, before creating a new one.

This provides a means to avoid downloads directory becoming congested
with a subversion repository module's tarball snapshots, which may occur
for example during circumstances where source code residing in a
subversion repository is subject to long-term development work, and
recipe used in continuous integration builds has SRCREV set to AUTOREV.

Signed-off-by: Niko Mauno <niko.mauno@iki.fi>
---
 bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml | 6 ++++++
 bitbake/lib/bb/fetch2/svn.py                                     | 6 ++++++
 2 files changed, 12 insertions(+)

diff --git a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
index 29ae486a7c..b4f1f673ec 100644
--- a/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
+++ b/bitbake/doc/bitbake-user-manual/bitbake-user-manual-fetching.xml
@@ -484,6 +484,12 @@
                         You can use this parameter to specify the ssh
                         program used by svn.
                         </para></listitem>
+                    <listitem><para><emphasis>"tarsnapshots":</emphasis>
+                        If this optional parameter is set to "currentonly" then
+                        any previously created tarball snapshots of associated
+                        repository module are removed from DL_DIR, before
+                        creating a new one.
+                        </para></listitem>
                     <listitem><para><emphasis>"transportuser":</emphasis>
                         When required, sets the username for the transport.
                         By default, this parameter is empty.
diff --git a/bitbake/lib/bb/fetch2/svn.py b/bitbake/lib/bb/fetch2/svn.py
index 3f172eec9b..3b58492f8f 100644
--- a/bitbake/lib/bb/fetch2/svn.py
+++ b/bitbake/lib/bb/fetch2/svn.py
@@ -148,6 +148,12 @@ class Svn(FetchMethod):
         else:
             tar_flags = "--exclude='.svn'"
 
+        tarsnapshots = ud.parm.get("tarsnapshots", "")
+        if tarsnapshots == "currentonly":
+            rm_path = ud.localpath.replace('_%s_.tar.' % ud.revision, '_[0-9]*_.tar.')
+            bb.utils.remove(rm_path)
+            bb.utils.remove(rm_path + '.done')
+
         # tar them up to a defined filename
         runfetchcmd("tar %s -czf %s %s" % (tar_flags, ud.localpath, ud.path_spec), d,
                     cleanup=[ud.localpath], workdir=ud.pkgdir)
-- 
2.16.2

Re: [PATCH] bitbake: fetch2/svn: Add tarsnapshots parameter

[Richard Purdie]

On Fri, 2018-03-23 at 11:28 +0200, Niko Mauno wrote:
> By adding 'tarsnapshots=currentonly' option to svn-specific component
> in
> SRC_URI, associated subversion repository module's existing tarball
> snapshots are removed from DL_DIR, before creating a new one.
> 
> This provides a means to avoid downloads directory becoming congested
> with a subversion repository module's tarball snapshots, which may
> occur
> for example during circumstances where source code residing in a
> subversion repository is subject to long-term development work, and
> recipe used in continuous integration builds has SRCREV set to
> AUTOREV.
> 
> Signed-off-by: Niko Mauno <niko.mauno@iki.fi>

This model does't really work as if multiple things share the downloads
directory there is a race window between the old tarballs being removed
and the new one being created. The downloads and sstate directories are
commonly shared between builds so we need to be mindful of this.

Cheers,

Richard

Re: [PATCH] bitbake: fetch2/svn: Add tarsnapshots parameter

[Niko Mauno]

On 04/04/2018 10:56 AM, Richard Purdie wrote:
> On Fri, 2018-03-23 at 11:28 +0200, Niko Mauno wrote:
>> By adding 'tarsnapshots=currentonly' option to svn-specific component
>> in
>> SRC_URI, associated subversion repository module's existing tarball
>> snapshots are removed from DL_DIR, before creating a new one.
>>
>> This provides a means to avoid downloads directory becoming congested
>> with a subversion repository module's tarball snapshots, which may
>> occur
>> for example during circumstances where source code residing in a
>> subversion repository is subject to long-term development work, and
>> recipe used in continuous integration builds has SRCREV set to
>> AUTOREV.
>>
>> Signed-off-by: Niko Mauno <niko.mauno@iki.fi>
> 
> This model does't really work as if multiple things share the downloads
> directory there is a race window between the old tarballs being removed
> and the new one being created. The downloads and sstate directories are
> commonly shared between builds so we need to be mindful of this.
> 
> Cheers,
> 
> Richard
> 

Thank You for pointing this out.

As an observation related to using a shared downloads directory, it currently seems to already be subject to races when subversion is used - for example in a situation where two (or more) builds perform a temporally overlapping do_fetch for same recipe but with differing SRCREVs, and tarballs for respective SRCREVs have not been created yet.

In such cases error seems to occur because several distinct actors are trying to 'svn update' or 'svn checkout' in same directory at same time, resulting in a bitbake error for all build instances (typically involving a "working copy locked" error message).