From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Greylist: delayed 457 seconds by postgrey-1.34 at layers.openembedded.org; Wed, 13 Apr 2016 15:34:44 UTC Received: from avasout05.plus.net (avasout05.plus.net [84.93.230.250]) by mail.openembedded.org (Postfix) with ESMTP id 8529173169 for ; Wed, 13 Apr 2016 15:34:44 +0000 (UTC) Received: from deneb ([80.229.24.9]) by avasout05 with smtp id hrT51s0080BmcFC01rT6tn; Wed, 13 Apr 2016 16:27:06 +0100 X-CM-Score: 0.00 X-CNFS-Analysis: v=2.1 cv=Iat6Ijea c=1 sm=1 tr=0 a=E/9URZZQ5L3bK/voZ0g0HQ==:117 a=E/9URZZQ5L3bK/voZ0g0HQ==:17 a=L9H7d07YOLsA:10 a=9cW_t1CCXrUA:10 a=s5jvgZ67dGcA:10 a=kziv93cY1bsA:10 a=-An2I_7KAAAA:8 a=n8jgqLjJPfYFeFoXIKAA:9 a=CjuIK1q_8ugA:10 a=Sc8CNStod5HAc5t90EwA:9 a=5uX4msJQyvSBJEi7:21 a=buVCt8jQGOgHBdqW:21 Received: from mac by deneb with local (Exim 4.84_2) (envelope-from ) id 1aqMh6-000599-L1; Wed, 13 Apr 2016 16:27:04 +0100 Date: Wed, 13 Apr 2016 16:27:04 +0100 From: Mike Crowe To: Otavio Salvador Message-ID: <20160413152704.GA19112@mcrowe.com> References: <20160329125626.GA11712@mcrowe.com> <1459260670.21672.25.camel@linuxfoundation.org> <20160412185126.GA14627@mcrowe.com> <1460494253.9308.70.camel@linuxfoundation.org> <20160413140103.GA14803@mcrowe.com> MIME-Version: 1.0 In-Reply-To: User-Agent: Mutt/1.5.23 (2014-03-12) Cc: Mike Crowe , Patches and discussions about the oe-core layer Subject: Re: Over-pruning the sstate cache X-BeenThere: openembedded-core@lists.openembedded.org X-Mailman-Version: 2.1.12 Precedence: list List-Id: Patches and discussions about the oe-core layer List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Apr 2016 15:34:45 -0000 X-Groupsio-MsgNum: 80838 Content-Type: multipart/mixed; boundary="wRRV7LY7NUeQGEoC" Content-Disposition: inline --wRRV7LY7NUeQGEoC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wednesday 13 April 2016 at 11:11:11 -0300, Otavio Salvador wrote: > On Wed, Apr 13, 2016 at 11:01 AM, Mike Crowe wrote: > > On Wednesday 13 April 2016 at 10:47:13 -0300, Otavio Salvador wrote: > >> Couldn't this to be done, similar to the fetchall task? > > > > That's the sort of thing I was thinking of with my "all_populate_sysroot" > > task in my original question. > > > > But, I've a working script using Richard's method now. I'll share it once > > I've tested it a bit more. > > Having it as a task allows it to avoid races and like, when running > inside of the BitBake. It also can reuse the metadata database > information and ought to be faster. Once you share the script I will > see if I can rework it to a task and see how it looks. I'm not really sure how much my (not very pretty) script will have in common with a task-based solution. I agree that integrating it into Bitbake would be better though. At the moment generating the locked-sigs.inc file is by far the slowest part of the operation though. Since you've requested it, here's the not-really-tested-properly script. Mike. --wRRV7LY7NUeQGEoC Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=touch-sstate #!/usr/bin/python2 # # touch-sstate # # Update atime on sstate-cache files for locked-sigs.inc file. # # Mike Crowe # # If nothing needs recompiling when building an image then bitbake # won't access the sstate files that are required to populate the # sysroot. This means that they might be expired as being unused. To # avoid the problem this script processes the "locked-sigs.inc" file # generated by running "bitbake XXX --dump-signatures=none". import glob import os import re import sys import time # These variables may need tweaking when the build environment changes native_lsb_string = "Debian-8" target_vendor = "" target_os = "oe-linux-gnueabi" host_os = "linux" sstate_version = "3" native_arches = [ 'x86-64' ] sstate_cache_dir = "sstate-cache/" # The start of a variable declaration that tells us the arch variable_re = re.compile('^SIGGEN_LOCKEDSIGS_t-([^ ]*) = \"\\\\$') # A single hash for a task sig_re = re.compile('^ *([^:]+):([^:]+):([^:]+) \\\\$') # Ignore the trailing quote, the final line and blank lines ignore_re = re.compile('^( *\")|(SIGGEN_LOCKEDSIGS_TYPES_)|$') arch1 = "noarch" arch2 = "noarch" def TouchMatches(pattern, deadline): matches = glob.glob(pattern) for match in matches: # It's likely that we won't need to touch the file so it's # cheaper to do the stat to check first than to always open # the file for reading. The inode might be in cache but the # file contents not. if os.stat(match).st_atime < deadline: print match # We can't use os.utime for this since we may not own the # file. f = open(match, 'r') # We actually have to read something from the file in # order to update atime. f.read(1) f.close() # In order to avoid unnecessarily thrashing the disk we only want to # modify the atime of the file if the atime is more than a day ago. # This matches the behaviour of the "relatime" mount option. We allow # a bit of leeway in case we take a long time to run and the expiry # script wants to expire stuff that's only more than a day old though. deadline = time.time() - 23 * 3600 line_number = 0 for line in open("locked-sigs.inc", "r"): line_number += 1 line = line.rstrip() variable_match = variable_re.match(line) if variable_match: arch = variable_match.group(1) arch_underscore = arch.replace("-", "_") if arch in native_arches: dir = sstate_cache_dir + native_lsb_string + "/" arch1 = arch_underscore + "-" + host_os arch2 = arch_underscore else: dir = sstate_cache_dir arch1 = arch_underscore + target_vendor + "-" + target_os arch2 = arch_underscore else: sig_match = sig_re.match(line) if sig_match: pn = sig_match.group(1) task = sig_match.group(2)[3:] sig = sig_match.group(3) arch_file = dir + sig[0:2] + "/sstate:" + pn + ":" + arch1 + ":*:*:" + arch2 + ":" + sstate_version + ":" + sig + "_" + task + ".tgz" noarch_file = dir + sig[0:2] + "/sstate:" + pn + "::*:*::" + sstate_version + ":" + sig + "_" + task + ".tgz" TouchMatches(arch_file, deadline) TouchMatches(noarch_file, deadline) elif ignore_re.match(line): pass else: sys.stderr.write('touch-sstate: Unparsed line %d\n' % line_number) sys.exit(1) --wRRV7LY7NUeQGEoC--