[PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Richard Purdie]

Its useful to understand where the delta starts against an existing sstate cache
for a given target. Adding this to the output of the -S option seems like a 
natural fit.

We use the hashvalidate function to figure this out and assume it can find siginfo
files for more than just the setscene tasks.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
---
diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
index 428cff1..b7a602b 100644
--- a/bitbake/lib/bb/runqueue.py
+++ b/bitbake/lib/bb/runqueue.py
@@ -1039,6 +1039,7 @@ class RunQueue:
 
         if self.state is runQueueSceneInit:
             if self.cooker.configuration.dump_signatures:
+                self.print_diffscenetasks()
                 self.dump_signatures()
             else:
                 self.start_worker()
@@ -1123,6 +1124,82 @@ class RunQueue:
 
         return
 
+    def print_diffscenetasks(self):
+
+        valid = []
+        sq_hash = []
+        sq_hashfn = []
+        sq_fn = []
+        sq_taskname = []
+        sq_task = []
+        noexec = []
+        stamppresent = []
+        valid_new = set()
+
+        for task in xrange(len(self.rqdata.runq_fnid)):
+            fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]]
+            taskname = self.rqdata.runq_task[task]
+            taskdep = self.rqdata.dataCache.task_deps[fn]
+
+            if 'noexec' in taskdep and taskname in taskdep['noexec']:
+                noexec.append(task)
+                continue
+
+            sq_fn.append(fn)
+            sq_hashfn.append(self.rqdata.dataCache.hashfn[fn])
+            sq_hash.append(self.rqdata.runq_hash[task])
+            sq_taskname.append(taskname)
+            sq_task.append(task)
+        call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
+        locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
+        valid = bb.utils.better_eval(call, locs)
+        for v in valid:
+            valid_new.add(sq_task[v])
+
+        # Tasks which are both setscene and noexec never care about dependencies
+        # We therefore find tasks which are setscene and noexec and mark their
+        # unique dependencies as valid.
+        for task in noexec:
+            if task not in self.rqdata.runq_setscene:
+                continue
+            for dep in self.rqdata.runq_depends[task]:
+                hasnoexecparents = True
+                for dep2 in self.rqdata.runq_revdeps[dep]:
+                    if dep2 in self.rqdata.runq_setscene and dep2 in noexec:
+                        continue
+                    hasnoexecparents = False
+                    break
+                if hasnoexecparents:
+                    valid_new.add(dep)
+
+        invalidtasks = set()
+        for task in xrange(len(self.rqdata.runq_fnid)):
+            if task not in valid_new and task not in noexec:
+                invalidtasks.add(task)
+
+        found = set()
+        processed = set()
+        for task in invalidtasks:
+            toprocess = set([task])
+            while toprocess:
+                next = set()
+                for t in toprocess:
+                    for dep in self.rqdata.runq_depends[t]:
+                        if dep in invalidtasks:
+                            found.add(task)
+                        if dep not in processed:
+                            processed.add(dep)
+                            next.add(dep)
+                toprocess = next
+                if task in found:
+                    toprocess = set()
+
+        tasklist = []
+        for task in invalidtasks.difference(found):
+            tasklist.append(self.rqdata.get_user_idstring(task))
+
+        if tasklist:
+            bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist))
 
 class RunQueueExecute:
 

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 12519 bytes --]

On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote:
> Its useful to understand where the delta starts against an existing sstate cache
> for a given target. Adding this to the output of the -S option seems like a 
> natural fit.
> 
> We use the hashvalidate function to figure this out and assume it can find siginfo
> files for more than just the setscene tasks.

I haven't tried to find smaller reproducer or exact part of code where
it happends, but in bigger builds I have it fails with

I'll try to reproduce and narrow it a bit, but if you have tip where
to look first..

IndexError: list index out of range:

NOTE: Reparsing files to collect dependency data

    >                self.write_diffscenetasks(invalidtasks)
                     self.state = runQueueComplete
  File "/OE/webos/owub/build-starfish-master/bitbake/lib/bb/runqueue.py", line 1227, in RunQueue.write_diffscenetasks(invalidtasks=set([16694, 11264, 5803, 4100, 2054, 16391, 2060, 16, 14354, 4115, 22, 14360, 6148, 8218, 2075, 10268, 13658, 8224, 4130, 18467, 16421, 38, 2087, 10283, 6188, 18687, 4145, 6194, 12686, 2105, 10298, 12347, 62, 14399, 4160, 12353, 16452, 14406, 7231, 6218, 13312, 77, 6224, 2130, 83, 14421, 4184, 15756, 4190, 6239, 2145, 98, 10939, 14436, 10598, 2151, 8296, 4202, 8302, 11665, 6257, 6163, 16763, 14949, 123, 4220, 14461, 2175, 10368, 129, 14468, 2181, 8326, 6849, 12507, 16748, 4235, 6286, 10383, 14480, 13336, 146, 8341, 10390, 10391, 4250, 6302, 161, 14498, 2211, 8357, 11735, 16552, 4265, 8363, 6316, 8562, 12462, 10415, 14513, 2226, 10421, 16567, 8378, 4283, 4286, 8565, 5152, 10436, 14710, 12486, 4295, 200, 8393, 2250, 16588, 12322, 206, 14543, 10452, 6357, 4310, 8408, 6363, 221, 14558, 2272, 7888, 10619, 6375, 17623, 12522, 14375, 10476, 4334, 2287, 8432, 16784, 16627, 18677, 8438, 16633, 4349, 2302, 17409, 10497, 6402, 4355, 7851, 13697, 6408, 12554, 268, 2317, 8462, 13357, 13818, 8468, 16376, 14650, 16663, 6424, 283, 2332, 15823, 10528, 8483, 16648, 6439, 2347, 16192, 4401, 8498, 14644, 6454, 10552, 313, 14023, 7903, 9285, 8513, 2371, 6469, 13718, 14741, 2377, 16446, 14668, 10573, 4431, 337, 6484, 12629, 343, 11741, 16733, 4446, 8544, 14689, 6499, 2404, 358, 1425, 10604, 12656, 6514, 2419, 8254, 374, 6889, 12692, 14665, 5183, 8574, 12671, 6529, 2434, 16107, 389, 4487, 2440, 4493, 8590, 14735, 6331, 10644, 405, 406, 2455, 6553, 4508, 8605, 3141, 420, 12709, 2470, 10667, 17938, 8621, 6574, 12153, 4532, 10313, 4379, 16825, 4538, 6559, 444, 6589, 13381, 7584, 450, 14787, 2500, 8645, 17814, 10697, 8651, 6604, 8269, 4562, 2515, 16856, 10713, 474, 6620, 477, 13733, 12291, 14817, 4578, 12710, 4584, 2539, 12370, 12382, 14832, 2545, 498, 12371, 501, 4599, 8696, 15444, 8814, 6653, 510, 14847, 6656, 2561, 12033, 6665, 8714, 8717, 2575, 10328, 4629, 535, 8728, 7599, 18567, 541, 2590, 12831, 6689, 9307, 4645, 8742, 6695, 14888, 7943, 556, 2605, 17584, 7262, 6710, 12855, 12724, 571, 10813, 14910, 15991, 10821, 2630, 16823, 2636, 4685, 17273, 15459, 596, 8789, 10838, 10839, 15460, 12891, 4700, 10682, 16300, 13111, 14955, 613, 11068, 2664, 619, 6764, 16146, 2670, 6770, 8820, 5566, 14970, 9663, 2685, 13251, 3861, 4736, 6786, 643, 12932, 10006, 14985, 7523, 10893, 4751, 658, 12947, 10900, 14038, 15000, 12740, 8859, 2716, 6815, 8865, 674, 10915, 12093, 10353, 14108, 13426, 689, 12978, 2740, 15030, 695, 8889, 2746, 4795, 8895, 4801, 710, 2761, 10954, 13004, 13005, 8910, 8529, 4816, 13971, 15060, 6873, 13019, 735, 10976, 2785, 16507, 10822, 741, 8934, 2791, 4841, 16528, 13035, 15084, 8943, 8946, 12755, 9777, 5588, 4858, 6271, 8956, 15146, 11007, 2816, 12416, 15106, 772, 2822, 16513, 4873, 6922, 9687, 11022, 4879, 6928, 15121, 787, 2837, 14620, 13080, 17348, 8666, 4894, 12779, 802, 2852, 11047, 6952, 7644, 9002, 9003, 15090, 11053, 6621, 13104, 817, 2868, 4918, 6967, 13112, 5940, 17690, 4924, 14772, 832, 9026, 15499, 9029, 15176, 17483, 2891, 10262, 15182, 847, 13136, 6997, 15161, 4954, 12431, 15197, 862, 2912, 13153, 9062, 4969, 877, 10767, 10728, 7029, 11128, 4985, 7038, 2943, 13184, 7041, 6635, 9093, 902, 13191, 5000, 15243, 908, 11153, 17300, 11159, 5016, 5017, 15258, 9117, 13215, 2976, 7075, 933, 5031, 15273, 9132, 8690, 2991, 7090, 15435, 948, 11189, 16603, 5047, 7096, 3003, 957, 16467, 4939, 11204, 5062, 7111, 9162, 971, 3021, 15312, 12785, 5077, 7126, 17368, 9177, 986, 3036, 7674, 9183, 11234, 5092, 7141, 1001, 16799, 3051, 9199, 11249, 5107, 7157, 1016, 3066, 12117, 9216, 10752, 7171, 15364, 17765, 3081, 5131, 9228, 12446, 1040, 5137, 7186, 11696, 15380, 3096, 11289, 9243, 2906, 13342, 1056, 15392, 1062, 3111, 7210, 9261, 11311, 7216, 15410, 13402, 1078, 17615, 5177, 14799, 16709, 11326, 3135, 16483, 9056, 11098, 1093, 9291, 7246, 10083, 13396, 1109, 11350, 5208, 9991, 4623, 11356, 3165, 5214, 16229, 3171, 1124, 16406, 15474, 9322, 8039, 5229, 13501, 3186, 1139, 8835, 16006, 13432, 9337, 15551, 5244, 10773, 1156, 1157, 7302, 13447, 3210, 5259, 11478, 1171, 7317, 13462, 9367, 7320, 3225, 5274, 15515, 3231, 15521, 11426, 16582, 9383, 5289, 7340, 7341, 13486, 6041, 7973, 1202, 9398, 3255, 5304, 15545, 7355, 3261, 18557, 15221, 1218, 12815, 13510, 11463, 13513, 7372, 7373, 12316, 9423, 11596, 3285, 5334, 2937, 1242, 3291, 11083, 1248, 13538, 9445, 5350, 17639, 7402, 13863, 15596, 3309, 11502, 14704, 9460, 6064, 1273, 13562, 15611, 5374, 12501, 11520, 13569, 9475, 9430, 11526, 5383, 3336, 5386, 15575, 1294, 10797, 5395, 11541, 7447, 3352, 3354, 9499, 1309, 13598, 15237, 5410, 7387, 11556, 7477, 7462, 7729, 3368, 9514, 16022, 11571, 1333, 13622, 15672, 9529, 1339, 13628, 11296, 9535, 3392, 5441, 7493, 3398, 15687, 13643, 8774, 9550, 8024, 11602, 1363, 7508, 3413, 15702, 13881, 5466, 5472, 11617, 7820, 1379, 3428, 15590, 9575, 13673, 17770, 13276, 5487, 1394, 3443, 17784, 14247, 11643, 7548, 9597, 3648, 1410, 10208, 13703, 17644, 11658, 3467, 9612, 5518, 3473, 15762, 11667, 14871, 14574, 12800, 1431, 9627, 3488, 16283, 7750, 1447, 11690, 15787, 5550, 3503, 5552, 15793, 9651, 9660, 13757, 3518, 1471, 15808, 13763, 7623, 9672, 12876, 17868, 7629, 1486, 14575, 14017, 5585, 13778, 1492, 11711, 3542, 7417, 3548, 5597, 13221, 17888, 13793, 10960, 9702, 15848, 7659, 5612, 1517, 253, 11371, 3572, 9717, 11766, 6740, 3578, 5627, 11772, 1533, 15870, 13824, 1539, 9732, 3593, 5642, 11787, 15885, 13839, 1554, 9747, 5654, 15961, 3608, 7705, 11802, 11183, 17948, 1569, 9762, 17958, 3623, 5672, 11441, 14599, 15916, 9138, 7554, 1288, 11829, 7735, 13236, 15931, 15285, 1600, 5697, 11844, 14262, 3654, 13896, 12215, 13902, 9807, 5712, 11859, 7765, 5718, 1624, 11865, 17679, 16177, 13917, 1630, 10853, 955, 5733, 9831, 16316, 13932, 7789, 1646, 11889, 15635, 5748, 3701, 9846, 11895, 13947, 7804, 4715, 3716, 1670, 11911, 9864, 18028, 3722, 1676, 18061, 9870, 12568, 7826, 10558, 4718, 11926, 15741, 13977, 13050, 5787, 9885, 16034, 3747, 1701, 3753, 1707, 9901, 11950, 5809, 14002, 14605, 16052, 3765, 12916, 1722, 9916, 17354, 15717, 7873, 15947, 11972, 18038, 3783, 1737, 9931, 16076, 15650, 5840, 3798, 9946, 16091, 11996, 14925, 5855, 13592, 1761, 12002, 18644, 3814, 1767, 9961, 12583, 5870, 10877, 15656, 12018, 3828, 14069, 7928, 16122, 1792, 9985, 3843, 14084, 5895, 16140, 7949, 1807, 12048, 7698, 5763, 1813, 5910, 3864, 17028, 5916, 12063, 3873, 16162, 1829, 11228, 7979, 12078, 3888, 14129, 1844, 5944, 15327, 5949, 10046, 3903, 14144, 7277, 8003, 1860, 12598, 14756, 5961, 12269, 13282, 3918, 14159, 17720, 8018, 10068, 1877, 5976, 16217, 3933, 14174, 15333, 16226, 5435, 12132, 1893, 5991, 13065, 16362, 10098, 12147, 1908, 3957, 8054, 14199, 10160, 3963, 16253, 6015, 1923, 8069, 14216, 12169, 10123, 16268, 14222, 6031, 1938, 3987, 8084, 6039, 12184, 3993, 10139, 17591, 12613, 1953, 9792, 13297, 4008, 6812, 10154, 8108, 1968, 12209, 8114, 6070, 4425, 14268, 16317, 12861, 1983, 4033, 8130, 6085, 12230, 4039, 8727, 10185, 14283, 1998, 10193, 10194, 12963, 16340, 12245, 4055, 16346, 6112, 8163, 14308, 4069, 6118, 2023, 12547, 14314, 15015, 18071, 2029, 10223, 7569, 8178, 6133, 14327, 4088, 4091, 16844, 12285, 10238])):
                 matches = {k : v for k, v in matches.iteritems() if h not in k}
    >            latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1]
                 prevh = __find_md5__.search(latestmatch).group(0)
IndexError: list index out of range

> 
> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
> ---
> diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py
> index 428cff1..b7a602b 100644
> --- a/bitbake/lib/bb/runqueue.py
> +++ b/bitbake/lib/bb/runqueue.py
> @@ -1039,6 +1039,7 @@ class RunQueue:
>  
>          if self.state is runQueueSceneInit:
>              if self.cooker.configuration.dump_signatures:
> +                self.print_diffscenetasks()
>                  self.dump_signatures()
>              else:
>                  self.start_worker()
> @@ -1123,6 +1124,82 @@ class RunQueue:
>  
>          return
>  
> +    def print_diffscenetasks(self):
> +
> +        valid = []
> +        sq_hash = []
> +        sq_hashfn = []
> +        sq_fn = []
> +        sq_taskname = []
> +        sq_task = []
> +        noexec = []
> +        stamppresent = []
> +        valid_new = set()
> +
> +        for task in xrange(len(self.rqdata.runq_fnid)):
> +            fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]]
> +            taskname = self.rqdata.runq_task[task]
> +            taskdep = self.rqdata.dataCache.task_deps[fn]
> +
> +            if 'noexec' in taskdep and taskname in taskdep['noexec']:
> +                noexec.append(task)
> +                continue
> +
> +            sq_fn.append(fn)
> +            sq_hashfn.append(self.rqdata.dataCache.hashfn[fn])
> +            sq_hash.append(self.rqdata.runq_hash[task])
> +            sq_taskname.append(taskname)
> +            sq_task.append(task)
> +        call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)"
> +        locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data }
> +        valid = bb.utils.better_eval(call, locs)
> +        for v in valid:
> +            valid_new.add(sq_task[v])
> +
> +        # Tasks which are both setscene and noexec never care about dependencies
> +        # We therefore find tasks which are setscene and noexec and mark their
> +        # unique dependencies as valid.
> +        for task in noexec:
> +            if task not in self.rqdata.runq_setscene:
> +                continue
> +            for dep in self.rqdata.runq_depends[task]:
> +                hasnoexecparents = True
> +                for dep2 in self.rqdata.runq_revdeps[dep]:
> +                    if dep2 in self.rqdata.runq_setscene and dep2 in noexec:
> +                        continue
> +                    hasnoexecparents = False
> +                    break
> +                if hasnoexecparents:
> +                    valid_new.add(dep)
> +
> +        invalidtasks = set()
> +        for task in xrange(len(self.rqdata.runq_fnid)):
> +            if task not in valid_new and task not in noexec:
> +                invalidtasks.add(task)
> +
> +        found = set()
> +        processed = set()
> +        for task in invalidtasks:
> +            toprocess = set([task])
> +            while toprocess:
> +                next = set()
> +                for t in toprocess:
> +                    for dep in self.rqdata.runq_depends[t]:
> +                        if dep in invalidtasks:
> +                            found.add(task)
> +                        if dep not in processed:
> +                            processed.add(dep)
> +                            next.add(dep)
> +                toprocess = next
> +                if task in found:
> +                    toprocess = set()
> +
> +        tasklist = []
> +        for task in invalidtasks.difference(found):
> +            tasklist.append(self.rqdata.get_user_idstring(task))
> +
> +        if tasklist:
> +            bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist))
>  
>  class RunQueueExecute:
>  
> 
> 
> _______________________________________________
> bitbake-devel mailing list
> bitbake-devel@lists.openembedded.org
> http://lists.openembedded.org/mailman/listinfo/bitbake-devel

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 954 bytes --]

On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <martin.jansa@gmail.com>wrote:

> On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote:
> > Its useful to understand where the delta starts against an existing
> sstate cache
> > for a given target. Adding this to the output of the -S option seems
> like a
> > natural fit.
> >
> > We use the hashvalidate function to figure this out and assume it can
> find siginfo
> > files for more than just the setscene tasks.
>
> I haven't tried to find smaller reproducer or exact part of code where
> it happends, but in bigger builds I have it fails with
>
> I'll try to reproduce and narrow it a bit, but if you have tip where
> to look first..
>
> IndexError: list index out of range:


I’m hitting this now as well.
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 1474 bytes --]

[RFC][WIP][PATCH] runqueue: Don't fail without match found

[Martin Jansa]

* WIP, because I haven't checked the reasons why this happens
  I'm using this now only to see which recipes/tasks are failing
  like this

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
 lib/bb/runqueue.py | 12 ++++++++----
 1 file changed, 8 insertions(+), 4 deletions(-)

diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 4577236..413b95e 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -1224,10 +1224,14 @@ class RunQueue:
             if match is None:
                 bb.fatal("Can't find a task we're supposed to have written out? (hash: %s)?" % h)
             matches = {k : v for k, v in matches.iteritems() if h not in k}
-            latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1]
-            prevh = __find_md5__.search(latestmatch).group(0)
-            output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb)
-            bb.plain("\nTask %s:%s couldn't be used from the cache because:\n  We need hash %s, closest matching task was %s\n  " % (pn, taskname, h, prevh) + '\n  '.join(output))
+            sortedmatches=sorted(matches.keys(), key=lambda f: matches[f])
+            if sortedmatches:
+                latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1]
+                prevh = __find_md5__.search(latestmatch).group(0)
+                output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb)
+                bb.plain("\nTask %s:%s couldn't be used from the cache because:\n  We need hash %s, closest matching task was %s\n  " % (pn, taskname, h, prevh) + '\n  '.join(output))
+            else:
+                bb.plain("\nTask %s:%s couldn't find any matches for hash %s" % (pn, taskname, h))
              
 
 
-- 
1.8.4.3

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 1659 bytes --]

On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <clarson@kergoth.com> wrote:

> On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <martin.jansa@gmail.com>wrote:
>
>> On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote:
>> > Its useful to understand where the delta starts against an existing
>> sstate cache
>> > for a given target. Adding this to the output of the -S option seems
>> like a
>> > natural fit.
>> >
>> > We use the hashvalidate function to figure this out and assume it can
>> find siginfo
>> > files for more than just the setscene tasks.
>>
>> I haven't tried to find smaller reproducer or exact part of code where
>> it happends, but in bigger builds I have it fails with
>>
>> I'll try to reproduce and narrow it a bit, but if you have tip where
>> to look first..
>>
>> IndexError: list index out of range:
>
>
> I’m hitting this now as well.


To clarify further, I’m hitting this with stock poky master, qemux86,
bitbake -S core-image-base.

Actually, this is getting in my way trying to do something I don’t want it
doing anyway. I want to dump the signature data from one config, and use
bitbake-whatchanged from another build. If I wanted to compare something
right here, I’d have run bitbake-whatchanged. This change confuses me a
bit. -S emits signature data, according to the bitbake help and the
semantics of the command. Are we trying to kill bitbake-whatchanged in
favor of making -S do both jobs, emit and compare?
-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 2546 bytes --]

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2190 bytes --]

On Mon, Dec 30, 2013 at 09:29:38AM -0700, Chris Larson wrote:
> On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <clarson@kergoth.com> wrote:
> 
> > On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <martin.jansa@gmail.com>wrote:
> >
> >> On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote:
> >> > Its useful to understand where the delta starts against an existing
> >> sstate cache
> >> > for a given target. Adding this to the output of the -S option seems
> >> like a
> >> > natural fit.
> >> >
> >> > We use the hashvalidate function to figure this out and assume it can
> >> find siginfo
> >> > files for more than just the setscene tasks.
> >>
> >> I haven't tried to find smaller reproducer or exact part of code where
> >> it happends, but in bigger builds I have it fails with
> >>
> >> I'll try to reproduce and narrow it a bit, but if you have tip where
> >> to look first..
> >>
> >> IndexError: list index out of range:
> >
> >
> > I’m hitting this now as well.
> 
> 
> To clarify further, I’m hitting this with stock poky master, qemux86,
> bitbake -S core-image-base.
> 
> Actually, this is getting in my way trying to do something I don’t want it
> doing anyway. I want to dump the signature data from one config, and use
> bitbake-whatchanged from another build. If I wanted to compare something
> right here, I’d have run bitbake-whatchanged. This change confuses me a
> bit. -S emits signature data, according to the bitbake help and the
> semantics of the command. Are we trying to kill bitbake-whatchanged in
> favor of making -S do both jobs, emit and compare?

Same here, I'm using bitbake -S from
oe-core/scripts/sstate-diff-machines.sh

and this added analysis and output is IMHO very useful, but it would be
nice to control it with some new option (and keep default -S behavior
fast and simple).

It looks like the error is triggered for every task which wasn't built
before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or
bitbake -S world after bitbake some-image).

I see it for many do_fetch and do_rm_work_all tasks.

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Richard Purdie]

On Mon, 2013-12-30 at 17:35 +0100, Martin Jansa wrote:
> On Mon, Dec 30, 2013 at 09:29:38AM -0700, Chris Larson wrote:
> > On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <clarson@kergoth.com> wrote:
> > 
> > > On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <martin.jansa@gmail.com>wrote:
> > >
> > >> On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote:
> > >> > Its useful to understand where the delta starts against an existing
> > >> sstate cache
> > >> > for a given target. Adding this to the output of the -S option seems
> > >> like a
> > >> > natural fit.
> > >> >
> > >> > We use the hashvalidate function to figure this out and assume it can
> > >> find siginfo
> > >> > files for more than just the setscene tasks.
> > >>
> > >> I haven't tried to find smaller reproducer or exact part of code where
> > >> it happends, but in bigger builds I have it fails with
> > >>
> > >> I'll try to reproduce and narrow it a bit, but if you have tip where
> > >> to look first..
> > >>
> > >> IndexError: list index out of range:
> > >
> > >
> > > I’m hitting this now as well.
> > 
> > 
> > To clarify further, I’m hitting this with stock poky master, qemux86,
> > bitbake -S core-image-base.
> > 
> > Actually, this is getting in my way trying to do something I don’t want it
> > doing anyway. I want to dump the signature data from one config, and use
> > bitbake-whatchanged from another build. If I wanted to compare something
> > right here, I’d have run bitbake-whatchanged. This change confuses me a
> > bit. -S emits signature data, according to the bitbake help and the
> > semantics of the command. Are we trying to kill bitbake-whatchanged in
> > favor of making -S do both jobs, emit and compare?
> 
> Same here, I'm using bitbake -S from
> oe-core/scripts/sstate-diff-machines.sh
> 
> and this added analysis and output is IMHO very useful, but it would be
> nice to control it with some new option (and keep default -S behavior
> fast and simple).
> 
> It looks like the error is triggered for every task which wasn't built
> before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or
> bitbake -S world after bitbake some-image).
> 
> I see it for many do_fetch and do_rm_work_all tasks.

To put things in context, I was trying to work with the existing
"whatchanged" type tools or more specifically I wanted to know "why
isn't my sstate cache being used". Having to do a two step process for
any of these enquires seems a little odd. There are also two specific
and as it turns our quite different work flows:

a) What changed since my last build?
b) What is the least delta between my current build and whatever I have
cached?

Those experiments revealed some big holes in the sstate metadata and
some good improvements there. It also highlights that there are two
different questions which have different answers.

In the process, I found bitbake does have some state in runqueue which
can help pinpoint the "whatchanged" part of the problem much more
accurately/quickly. Obviously that state is lost after bitbake -S
finishes so I started printing out some extra data.

Clearly bitbake shouldn't crash, that is a bug and I think Martin is
heading in the right direction with the other patch.

As for bitbake -S being kept fast, I think (but I need to double check
with measurements) that the parsing part of bitbake -S is the slow part
and this extra data is effectively "for free".

So basically consider the current state an experiment to see if we can
make -S more useful/interesting (without much cost) and to try to better
understand where/what functionality we need going forward.

One possible outcome might be we generate an extra file (or files) (like
we do with task-depends.dot) which contains this extra tree data which
some other command can then parse and display to the user. I'm still
struggling to get a complete grip on the exact data and format that file
needs to be. Part of the reason for having -S print something was to see
if it printed what the user wanted to know or if not, we could then try
and figure out what the right thing was so we could improve it.

So sorry if this is causing pain, the code is easy to disable by
commenting out the call, or reverting the patch adding it. Equally, I
think we need to make sstate hashes and reasons for rebuilds more
accessible to users and I think this does move us closer to that
assuming we fix the crash.

I'm trying to take a bit of a break over the holiday period (been riding
a motorcycle in driving snow/rain today and am rather damp/cold) but
I'll get the crash fixed sometime next week at the latest.

Cheers,

Richard

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Chris Larson]

[-- Attachment #1: Type: text/plain, Size: 5209 bytes --]

On Monday, December 30, 2013, Richard Purdie wrote:

> On Mon, 2013-12-30 at 17:35 +0100, Martin Jansa wrote:
> > On Mon, Dec 30, 2013 at 09:29:38AM -0700, Chris Larson wrote:
> > > On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <clarson@kergoth.com<javascript:;>>
> wrote:
> > >
> > > > On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <
> martin.jansa@gmail.com <javascript:;>>wrote:
> > > >
> > > >> On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote:
> > > >> > Its useful to understand where the delta starts against an
> existing
> > > >> sstate cache
> > > >> > for a given target. Adding this to the output of the -S option
> seems
> > > >> like a
> > > >> > natural fit.
> > > >> >
> > > >> > We use the hashvalidate function to figure this out and assume it
> can
> > > >> find siginfo
> > > >> > files for more than just the setscene tasks.
> > > >>
> > > >> I haven't tried to find smaller reproducer or exact part of code
> where
> > > >> it happends, but in bigger builds I have it fails with
> > > >>
> > > >> I'll try to reproduce and narrow it a bit, but if you have tip where
> > > >> to look first..
> > > >>
> > > >> IndexError: list index out of range:
> > > >
> > > >
> > > > I’m hitting this now as well.
> > >
> > >
> > > To clarify further, I’m hitting this with stock poky master, qemux86,
> > > bitbake -S core-image-base.
> > >
> > > Actually, this is getting in my way trying to do something I don’t
> want it
> > > doing anyway. I want to dump the signature data from one config, and
> use
> > > bitbake-whatchanged from another build. If I wanted to compare
> something
> > > right here, I’d have run bitbake-whatchanged. This change confuses me a
> > > bit. -S emits signature data, according to the bitbake help and the
> > > semantics of the command. Are we trying to kill bitbake-whatchanged in
> > > favor of making -S do both jobs, emit and compare?
> >
> > Same here, I'm using bitbake -S from
> > oe-core/scripts/sstate-diff-machines.sh
> >
> > and this added analysis and output is IMHO very useful, but it would be
> > nice to control it with some new option (and keep default -S behavior
> > fast and simple).
> >
> > It looks like the error is triggered for every task which wasn't built
> > before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or
> > bitbake -S world after bitbake some-image).
> >
> > I see it for many do_fetch and do_rm_work_all tasks.
>
> To put things in context, I was trying to work with the existing
> "whatchanged" type tools or more specifically I wanted to know "why
> isn't my sstate cache being used". Having to do a two step process for
> any of these enquires seems a little odd. There are also two specific
> and as it turns our quite different work flows:
>
> a) What changed since my last build?
> b) What is the least delta between my current build and whatever I have
> cached?


> Those experiments revealed some big holes in the sstate metadata and
> some good improvements there. It also highlights that there are two
> different questions which have different answers.
>
> In the process, I found bitbake does have some state in runqueue which
> can help pinpoint the "whatchanged" part of the problem much more
> accurately/quickly. Obviously that state is lost after bitbake -S
> finishes so I started printing out some extra data.
>
> Clearly bitbake shouldn't crash, that is a bug and I think Martin is
> heading in the right direction with the other patch.
>
> As for bitbake -S being kept fast, I think (but I need to double check
> with measurements) that the parsing part of bitbake -S is the slow part
> and this extra data is effectively "for free".
>
> So basically consider the current state an experiment to see if we can
> make -S more useful/interesting (without much cost) and to try to better
> understand where/what functionality we need going forward.
>

I can see that, but I think you're making the semantics of the command-line
argument confusing by trying to combine the two operations into one. I also
think that trying to find the "least delta" is extremely weird, and not
always likely to produce anything worthwhile. My SSTATE_DIR has weeks if
not months of output from hundreds of builds of different combinations of
machine, distro, layer selection, upstream vs mel, as well as between tons
of build environments with slight local alterations. Unless I'm
misunderstanding something, how exactly is it going to know what I want it
to compare against, and how it is going to keep finding the "least delta"
fast, amongst all that data? With the current tools, I can explicitly use
-S in one env and whatchanged in another (with appropriate STAMPS_DIR) to
compare between two exact build environments, rather than letting bitbake
try to automagically find what I meant. Now that the two are combined into
one, I'm not seeing how to do what I'm currently doing. Can you explain
this?

-Chris


-- 
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

[-- Attachment #2: Type: text/html, Size: 6312 bytes --]

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Richard Purdie]

On Mon, 2013-12-30 at 15:34 -0700, Chris Larson wrote:
> I can see that, but I think you're making the semantics of the
> command-line argument confusing by trying to combine the two
> operations into one. I also think that trying to find the "least
> delta" is extremely weird, and not always likely to produce anything
> worthwhile. My SSTATE_DIR has weeks if not months of output from
> hundreds of builds of different combinations of machine, distro, layer
> selection, upstream vs mel, as well as between tons of build
> environments with slight local alterations. Unless I'm
> misunderstanding something, how exactly is it going to know what I
> want it to compare against, and how it is going to keep finding the
> "least delta" fast, amongst all that data? With the current tools, I
> can explicitly use -S in one env and whatchanged in another (with
> appropriate STAMPS_DIR) to compare between two exact build
> environments, rather than letting bitbake try to automagically find
> what I meant. Now that the two are combined into one, I'm not seeing
> how to do what I'm currently doing. Can you explain this?

Nothing changed in the data -S writes out. 

Yes, it does currently crash in some cases which is obviously a bug
however assuming we fix that you can ignore what -S prints on the screen
or writes to a file or whatever we end up doing and just do what you're
doing now.

On the other hand we have a huge hole in the current model when you
don't have the two environments to compare. You have the current build
and you want to know why the cache isn't being used. That is a valid
user question which shouldn't need another build to compare against.

So can we find the "least delta" fast? Its not actually that hard
computationally or on resources, at least in the experiments I've made.
We know the hashes of the current target's tasks and we can quickly tell
which are in the cache and which are not (using the same function sstate
uses for that purpose).

That gives you a set of delta points with the sstate cache but you also
know the recipe name and other details about the target of interest and
that they should have common parent tasks. Based on that information you
can come up with a list of possibilities of least difference and sort
those by date (which bitbake-diffsigs -t uses) or whatever other
mechanism we end up choosing to display the data.

I agree its not always going to give useful information but its a start
in the right direction and my local tests suggest its perhaps good
enough to give the user the hints they want about cache reuse (or lack
of).

I agree we're not there yet, equally, to be quite honest the current
debug of sstate sucks and we need to do something about it. Whilst you
and I may understand it and are able to debug issues, many users are
struggling with it :(.

Cheers,

Richard

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2166 bytes --]

On Wed, Jan 01, 2014 at 11:38:01PM +0000, Richard Purdie wrote:
> So can we find the "least delta" fast? Its not actually that hard
> computationally or on resources, at least in the experiments I've made.
> We know the hashes of the current target's tasks and we can quickly tell
> which are in the cache and which are not (using the same function sstate
> uses for that purpose).

I've finally got some numbers to support my claim that it "feels" slower
with this added functionality.

My script is calling
openembedded-core/scripts/sstate-diff-machines.sh --machines="qemuarm qemux86 qemux86copy" --targets=world --tmpdir=tmp-eglibc/;

which in turn runs
bitbake -S world
three times (once for each machine).

The script on jenkins doing world builds is completely crazy (I've
killed it in last run assuming it was stuck, but it wasn't just terribly
slow).
http://jenkins.nas-admin.org/job/oe_world_compare_signatures/11/
Running 21 hours already and doing 2nd machine from 3 (maybe because
longer SSTATE_MIRROR?). There is one python process using 100% cpu.

In local builds without SSTATE_MIRROR it's not so bad, but still
significantly slower with each release (dylan-dora-master).

Master builds have that WIP patch included in order to finish.

It's not very accurate, because it was running on my desktop and each one has
different metadata, but at least the layers included are similar (only
systems which got separated in dora and removal
of meta-webos-backports which isn't needed for dora and newer).

At least I've executed each test twice.

dylan:
real   9m37.271s
user   6m37.104s
sys    3m12.625s

real   10m5.787s
user   6m37.832s
sys    3m23.614s

dora:
real   26m11.374s
user   9m7.325s
sys    3m0.048s

real   28m10.606s
user   8m27.895s
sys    2m57.497s

master:
real   38m17.641s
user   17m46.905s
sys    5m12.061s

real    36m12.748s
user    16m32.961s
sys     4m43.103s

master-without-S-changes:
real    27m34.228s
user    8m6.811s
sys     2m53.724s

real    27m49.752s
user    8m25.026s
sys     2m50.921s

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Richard Purdie]

On Wed, 2014-01-01 at 23:38 +0000, Richard Purdie wrote:
> Nothing changed in the data -S writes out. 
> 
> Yes, it does currently crash in some cases which is obviously a bug
> however assuming we fix that you can ignore what -S prints on the screen
> or writes to a file or whatever we end up doing and just do what you're
> doing now.
> 
> On the other hand we have a huge hole in the current model when you
> don't have the two environments to compare. You have the current build
> and you want to know why the cache isn't being used. That is a valid
> user question which shouldn't need another build to compare against.
> 
> So can we find the "least delta" fast? Its not actually that hard
> computationally or on resources, at least in the experiments I've made.
> We know the hashes of the current target's tasks and we can quickly tell
> which are in the cache and which are not (using the same function sstate
> uses for that purpose).
> 
> That gives you a set of delta points with the sstate cache but you also
> know the recipe name and other details about the target of interest and
> that they should have common parent tasks. Based on that information you
> can come up with a list of possibilities of least difference and sort
> those by date (which bitbake-diffsigs -t uses) or whatever other
> mechanism we end up choosing to display the data.
> 
> I agree its not always going to give useful information but its a start
> in the right direction and my local tests suggest its perhaps good
> enough to give the user the hints they want about cache reuse (or lack
> of).
> 
> I agree we're not there yet, equally, to be quite honest the current
> debug of sstate sucks and we need to do something about it. Whilst you
> and I may understand it and are able to debug issues, many users are
> struggling with it :(.

I appreciate its taken me a while to loop back around on this but I do
now have an idea. What I'm proposing we do is add an optional parameter
to -S. Without any parameter it would behave as it did classically.

With a parameter, it could trigger the current "debug sstate" behaviour
to stdout and it also allows it to trigger an sstate siggen specific
behaviour as appripriate. I have some prototype code with writes out a
"locked-sigs.inc" file for example, triggered from this same code block.

I think this should let us do more creative things with sstate (even
from the OE siggen class) yet also let it remain useful for different
people.

I don't have any code to do this yet but at least we have a plan
assuming no strong objections.

Cheers,

Richard

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2904 bytes --]

On Tue, Jan 21, 2014 at 10:37:33PM +0000, Richard Purdie wrote:
> On Wed, 2014-01-01 at 23:38 +0000, Richard Purdie wrote:
> > Nothing changed in the data -S writes out. 
> > 
> > Yes, it does currently crash in some cases which is obviously a bug
> > however assuming we fix that you can ignore what -S prints on the screen
> > or writes to a file or whatever we end up doing and just do what you're
> > doing now.
> > 
> > On the other hand we have a huge hole in the current model when you
> > don't have the two environments to compare. You have the current build
> > and you want to know why the cache isn't being used. That is a valid
> > user question which shouldn't need another build to compare against.
> > 
> > So can we find the "least delta" fast? Its not actually that hard
> > computationally or on resources, at least in the experiments I've made.
> > We know the hashes of the current target's tasks and we can quickly tell
> > which are in the cache and which are not (using the same function sstate
> > uses for that purpose).
> > 
> > That gives you a set of delta points with the sstate cache but you also
> > know the recipe name and other details about the target of interest and
> > that they should have common parent tasks. Based on that information you
> > can come up with a list of possibilities of least difference and sort
> > those by date (which bitbake-diffsigs -t uses) or whatever other
> > mechanism we end up choosing to display the data.
> > 
> > I agree its not always going to give useful information but its a start
> > in the right direction and my local tests suggest its perhaps good
> > enough to give the user the hints they want about cache reuse (or lack
> > of).
> > 
> > I agree we're not there yet, equally, to be quite honest the current
> > debug of sstate sucks and we need to do something about it. Whilst you
> > and I may understand it and are able to debug issues, many users are
> > struggling with it :(.
> 
> I appreciate its taken me a while to loop back around on this but I do
> now have an idea. What I'm proposing we do is add an optional parameter
> to -S. Without any parameter it would behave as it did classically.
> 
> With a parameter, it could trigger the current "debug sstate" behaviour
> to stdout and it also allows it to trigger an sstate siggen specific
> behaviour as appripriate. I have some prototype code with writes out a
> "locked-sigs.inc" file for example, triggered from this same code block.
> 
> I think this should let us do more creative things with sstate (even
> from the OE siggen class) yet also let it remain useful for different
> people.
> 
> I don't have any code to do this yet but at least we have a plan
> assuming no strong objections.

I strongly support this plan :).

Cheers,

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 2829 bytes --]

On Fri, Jan 10, 2014 at 03:14:17PM +0100, Martin Jansa wrote:
> On Wed, Jan 01, 2014 at 11:38:01PM +0000, Richard Purdie wrote:
> > So can we find the "least delta" fast? Its not actually that hard
> > computationally or on resources, at least in the experiments I've made.
> > We know the hashes of the current target's tasks and we can quickly tell
> > which are in the cache and which are not (using the same function sstate
> > uses for that purpose).
> 
> I've finally got some numbers to support my claim that it "feels" slower
> with this added functionality.
> 
> My script is calling
> openembedded-core/scripts/sstate-diff-machines.sh --machines="qemuarm qemux86 qemux86copy" --targets=world --tmpdir=tmp-eglibc/;
> 
> which in turn runs
> bitbake -S world
> three times (once for each machine).
> 
> The script on jenkins doing world builds is completely crazy (I've
> killed it in last run assuming it was stuck, but it wasn't just terribly
> slow).
> http://jenkins.nas-admin.org/job/oe_world_compare_signatures/11/
> Running 21 hours already and doing 2nd machine from 3 (maybe because
> longer SSTATE_MIRROR?). There is one python process using 100% cpu.

FWIW: this jenkins job was completed yesterday after 16 days

small part of strace:
http://logs.nslu2-linux.org/buildlogs/oe/oe-shr-core-branches/log.world.20140109_144047.log/log.strace

complete log:
http://logs.nslu2-linux.org/buildlogs/oe/oe-shr-core-branches/log.world.20140109_144047.log/signatures.sstate-diff-machines

Previous runs of the same job took 25-40 minutes

> In local builds without SSTATE_MIRROR it's not so bad, but still
> significantly slower with each release (dylan-dora-master).
> 
> Master builds have that WIP patch included in order to finish.
> 
> It's not very accurate, because it was running on my desktop and each one has
> different metadata, but at least the layers included are similar (only
> systems which got separated in dora and removal
> of meta-webos-backports which isn't needed for dora and newer).
> 
> At least I've executed each test twice.
> 
> dylan:
> real   9m37.271s
> user   6m37.104s
> sys    3m12.625s
> 
> real   10m5.787s
> user   6m37.832s
> sys    3m23.614s
> 
> dora:
> real   26m11.374s
> user   9m7.325s
> sys    3m0.048s
> 
> real   28m10.606s
> user   8m27.895s
> sys    2m57.497s
> 
> master:
> real   38m17.641s
> user   17m46.905s
> sys    5m12.061s
> 
> real    36m12.748s
> user    16m32.961s
> sys     4m43.103s
> 
> master-without-S-changes:
> real    27m34.228s
> user    8m6.811s
> sys     2m53.724s
> 
> real    27m49.752s
> user    8m25.026s
> sys     2m50.921s
> 
> -- 
> Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com



-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Richard Purdie]

On Tue, 2014-01-21 at 22:37 +0000, Richard Purdie wrote:
> I appreciate its taken me a while to loop back around on this but I do
> now have an idea. What I'm proposing we do is add an optional parameter
> to -S. Without any parameter it would behave as it did classically.
> 
> With a parameter, it could trigger the current "debug sstate" behaviour
> to stdout and it also allows it to trigger an sstate siggen specific
> behaviour as appripriate. I have some prototype code with writes out a
> "locked-sigs.inc" file for example, triggered from this same code block.
> 
> I think this should let us do more creative things with sstate (even
> from the OE siggen class) yet also let it remain useful for different
> people.

So we have a small potential issue here. We use python's optparse which
doesn't allow optional arguments. This means we can make -S take options
but the options must always be specified.

It has good reason for this as its near impossible for optparse to tell
the difference between:

bitbake image -S tracesigs
and
bitbake image -S image2

So our options are:

a) Require a new parameter to -S always
b) Hack it to use the option after -S as a parameter (meaning -S should
always be last on the commandline). I do have a proof of concept but it
makes me uneasy
c) Add a new option with a different name

In the interests of not doing something which binds us into a problem in
future, I'm thinking -S should start to always take a parameter. The
only issue is I'd like that parameter to be metadata (siggen) definable
so we can't validate the option passed.

Any opinions?

Cheers,

Richard

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Martin Jansa]

[-- Attachment #1: Type: text/plain, Size: 1969 bytes --]

On Mon, Jan 27, 2014 at 01:23:25PM +0000, Richard Purdie wrote:
> On Tue, 2014-01-21 at 22:37 +0000, Richard Purdie wrote:
> > I appreciate its taken me a while to loop back around on this but I do
> > now have an idea. What I'm proposing we do is add an optional parameter
> > to -S. Without any parameter it would behave as it did classically.
> > 
> > With a parameter, it could trigger the current "debug sstate" behaviour
> > to stdout and it also allows it to trigger an sstate siggen specific
> > behaviour as appripriate. I have some prototype code with writes out a
> > "locked-sigs.inc" file for example, triggered from this same code block.
> > 
> > I think this should let us do more creative things with sstate (even
> > from the OE siggen class) yet also let it remain useful for different
> > people.
> 
> So we have a small potential issue here. We use python's optparse which
> doesn't allow optional arguments. This means we can make -S take options
> but the options must always be specified.
> 
> It has good reason for this as its near impossible for optparse to tell
> the difference between:
> 
> bitbake image -S tracesigs
> and
> bitbake image -S image2
> 
> So our options are:
> 
> a) Require a new parameter to -S always
> b) Hack it to use the option after -S as a parameter (meaning -S should
> always be last on the commandline). I do have a proof of concept but it
> makes me uneasy
> c) Add a new option with a different name
> 
> In the interests of not doing something which binds us into a problem in
> future, I'm thinking -S should start to always take a parameter. The
> only issue is I'd like that parameter to be metadata (siggen) definable
> so we can't validate the option passed.
> 
> Any opinions?

What about using some existing parameter in combination with -S?

-S + -v shows tracesigs
-S without -v old behavior

-- 
Martin 'JaMa' Jansa     jabber: Martin.Jansa@gmail.com

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 205 bytes --]

Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache

[Richard Purdie]

On Mon, 2014-01-27 at 14:39 +0100, Martin Jansa wrote:
> On Mon, Jan 27, 2014 at 01:23:25PM +0000, Richard Purdie wrote:
> > On Tue, 2014-01-21 at 22:37 +0000, Richard Purdie wrote:
> > > I appreciate its taken me a while to loop back around on this but I do
> > > now have an idea. What I'm proposing we do is add an optional parameter
> > > to -S. Without any parameter it would behave as it did classically.
> > > 
> > > With a parameter, it could trigger the current "debug sstate" behaviour
> > > to stdout and it also allows it to trigger an sstate siggen specific
> > > behaviour as appripriate. I have some prototype code with writes out a
> > > "locked-sigs.inc" file for example, triggered from this same code block.
> > > 
> > > I think this should let us do more creative things with sstate (even
> > > from the OE siggen class) yet also let it remain useful for different
> > > people.
> > 
> > So we have a small potential issue here. We use python's optparse which
> > doesn't allow optional arguments. This means we can make -S take options
> > but the options must always be specified.
> > 
> > It has good reason for this as its near impossible for optparse to tell
> > the difference between:
> > 
> > bitbake image -S tracesigs
> > and
> > bitbake image -S image2
> > 
> > So our options are:
> > 
> > a) Require a new parameter to -S always
> > b) Hack it to use the option after -S as a parameter (meaning -S should
> > always be last on the commandline). I do have a proof of concept but it
> > makes me uneasy
> > c) Add a new option with a different name
> > 
> > In the interests of not doing something which binds us into a problem in
> > future, I'm thinking -S should start to always take a parameter. The
> > only issue is I'd like that parameter to be metadata (siggen) definable
> > so we can't validate the option passed.
> > 
> > Any opinions?
> 
> What about using some existing parameter in combination with -S?
> 
> -S + -v shows tracesigs
> -S without -v old behavior

I have some use cases locally (such as locked sigs) which could really
benefit from a string parameter which gets passed into the siggen code.
I can see us needing more than two "behaviours" of -S...

I also think using something like -v will lead to more confusion and the
code won't be particularly nice either :/.

Cheers,

Richard