* [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache @ 2013-12-18 16:21 Richard Purdie 2013-12-28 13:24 ` Martin Jansa 0 siblings, 1 reply; 16+ messages in thread From: Richard Purdie @ 2013-12-18 16:21 UTC (permalink / raw) To: bitbake-devel Its useful to understand where the delta starts against an existing sstate cache for a given target. Adding this to the output of the -S option seems like a natural fit. We use the hashvalidate function to figure this out and assume it can find siginfo files for more than just the setscene tasks. Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> --- diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py index 428cff1..b7a602b 100644 --- a/bitbake/lib/bb/runqueue.py +++ b/bitbake/lib/bb/runqueue.py @@ -1039,6 +1039,7 @@ class RunQueue: if self.state is runQueueSceneInit: if self.cooker.configuration.dump_signatures: + self.print_diffscenetasks() self.dump_signatures() else: self.start_worker() @@ -1123,6 +1124,82 @@ class RunQueue: return + def print_diffscenetasks(self): + + valid = [] + sq_hash = [] + sq_hashfn = [] + sq_fn = [] + sq_taskname = [] + sq_task = [] + noexec = [] + stamppresent = [] + valid_new = set() + + for task in xrange(len(self.rqdata.runq_fnid)): + fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]] + taskname = self.rqdata.runq_task[task] + taskdep = self.rqdata.dataCache.task_deps[fn] + + if 'noexec' in taskdep and taskname in taskdep['noexec']: + noexec.append(task) + continue + + sq_fn.append(fn) + sq_hashfn.append(self.rqdata.dataCache.hashfn[fn]) + sq_hash.append(self.rqdata.runq_hash[task]) + sq_taskname.append(taskname) + sq_task.append(task) + call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)" + locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data } + valid = bb.utils.better_eval(call, locs) + for v in valid: + valid_new.add(sq_task[v]) + + # Tasks which are both setscene and noexec never care about dependencies + # We therefore find tasks which are setscene and noexec and mark their + # unique dependencies as valid. + for task in noexec: + if task not in self.rqdata.runq_setscene: + continue + for dep in self.rqdata.runq_depends[task]: + hasnoexecparents = True + for dep2 in self.rqdata.runq_revdeps[dep]: + if dep2 in self.rqdata.runq_setscene and dep2 in noexec: + continue + hasnoexecparents = False + break + if hasnoexecparents: + valid_new.add(dep) + + invalidtasks = set() + for task in xrange(len(self.rqdata.runq_fnid)): + if task not in valid_new and task not in noexec: + invalidtasks.add(task) + + found = set() + processed = set() + for task in invalidtasks: + toprocess = set([task]) + while toprocess: + next = set() + for t in toprocess: + for dep in self.rqdata.runq_depends[t]: + if dep in invalidtasks: + found.add(task) + if dep not in processed: + processed.add(dep) + next.add(dep) + toprocess = next + if task in found: + toprocess = set() + + tasklist = [] + for task in invalidtasks.difference(found): + tasklist.append(self.rqdata.get_user_idstring(task)) + + if tasklist: + bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist)) class RunQueueExecute: ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2013-12-18 16:21 [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache Richard Purdie @ 2013-12-28 13:24 ` Martin Jansa 2013-12-30 16:18 ` Chris Larson 0 siblings, 1 reply; 16+ messages in thread From: Martin Jansa @ 2013-12-28 13:24 UTC (permalink / raw) To: Richard Purdie; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 12519 bytes --] On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote: > Its useful to understand where the delta starts against an existing sstate cache > for a given target. Adding this to the output of the -S option seems like a > natural fit. > > We use the hashvalidate function to figure this out and assume it can find siginfo > files for more than just the setscene tasks. I haven't tried to find smaller reproducer or exact part of code where it happends, but in bigger builds I have it fails with I'll try to reproduce and narrow it a bit, but if you have tip where to look first.. IndexError: list index out of range: NOTE: Reparsing files to collect dependency data > self.write_diffscenetasks(invalidtasks) self.state = runQueueComplete File "/OE/webos/owub/build-starfish-master/bitbake/lib/bb/runqueue.py", line 1227, in RunQueue.write_diffscenetasks(invalidtasks=set([16694, 11264, 5803, 4100, 2054, 16391, 2060, 16, 14354, 4115, 22, 14360, 6148, 8218, 2075, 10268, 13658, 8224, 4130, 18467, 16421, 38, 2087, 10283, 6188, 18687, 4145, 6194, 12686, 2105, 10298, 12347, 62, 14399, 4160, 12353, 16452, 14406, 7231, 6218, 13312, 77, 6224, 2130, 83, 14421, 4184, 15756, 4190, 6239, 2145, 98, 10939, 14436, 10598, 2151, 8296, 4202, 8302, 11665, 6257, 6163, 16763, 14949, 123, 4220, 14461, 2175, 10368, 129, 14468, 2181, 8326, 6849, 12507, 16748, 4235, 6286, 10383, 14480, 13336, 146, 8341, 10390, 10391, 4250, 6302, 161, 14498, 2211, 8357, 11735, 16552, 4265, 8363, 6316, 8562, 12462, 10415, 14513, 2226, 10421, 16567, 8378, 4283, 4286, 8565, 5152, 10436, 14710, 12486, 4295, 200, 8393, 2250, 16588, 12322, 206, 14543, 10452, 6357, 4310, 8408, 6363, 221, 14558, 2272, 7888, 10619, 6375, 17623, 12522, 14375, 10476, 4334, 2287, 8432, 16784, 16627, 18677, 8438, 16633, 4349, 2302, 17409, 10497, 6402, 4355, 7851, 13697, 6408, 12554, 268, 2317, 8462, 13357, 13818, 8468, 16376, 14650, 16663, 6424, 283, 2332, 15823, 10528, 8483, 16648, 6439, 2347, 16192, 4401, 8498, 14644, 6454, 10552, 313, 14023, 7903, 9285, 8513, 2371, 6469, 13718, 14741, 2377, 16446, 14668, 10573, 4431, 337, 6484, 12629, 343, 11741, 16733, 4446, 8544, 14689, 6499, 2404, 358, 1425, 10604, 12656, 6514, 2419, 8254, 374, 6889, 12692, 14665, 5183, 8574, 12671, 6529, 2434, 16107, 389, 4487, 2440, 4493, 8590, 14735, 6331, 10644, 405, 406, 2455, 6553, 4508, 8605, 3141, 420, 12709, 2470, 10667, 17938, 8621, 6574, 12153, 4532, 10313, 4379, 16825, 4538, 6559, 444, 6589, 13381, 7584, 450, 14787, 2500, 8645, 17814, 10697, 8651, 6604, 8269, 4562, 2515, 16856, 10713, 474, 6620, 477, 13733, 12291, 14817, 4578, 12710, 4584, 2539, 12370, 12382, 14832, 2545, 498, 12371, 501, 4599, 8696, 15444, 8814, 6653, 510, 14847, 6656, 2561, 12033, 6665, 8714, 8717, 2575, 10328, 4629, 535, 8728, 7599, 18567, 541, 2590, 12831, 6689, 9307, 4645, 8742, 6695, 14888, 7943, 556, 2605, 17584, 7262, 6710, 12855, 12724, 571, 10813, 14910, 15991, 10821, 2630, 16823, 2636, 4685, 17273, 15459, 596, 8789, 10838, 10839, 15460, 12891, 4700, 10682, 16300, 13111, 14955, 613, 11068, 2664, 619, 6764, 16146, 2670, 6770, 8820, 5566, 14970, 9663, 2685, 13251, 3861, 4736, 6786, 643, 12932, 10006, 14985, 7523, 10893, 4751, 658, 12947, 10900, 14038, 15000, 12740, 8859, 2716, 6815, 8865, 674, 10915, 12093, 10353, 14108, 13426, 689, 12978, 2740, 15030, 695, 8889, 2746, 4795, 8895, 4801, 710, 2761, 10954, 13004, 13005, 8910, 8529, 4816, 13971, 15060, 6873, 13019, 735, 10976, 2785, 16507, 10822, 741, 8934, 2791, 4841, 16528, 13035, 15084, 8943, 8946, 12755, 9777, 5588, 4858, 6271, 8956, 15146, 11007, 2816, 12416, 15106, 772, 2822, 16513, 4873, 6922, 9687, 11022, 4879, 6928, 15121, 787, 2837, 14620, 13080, 17348, 8666, 4894, 12779, 802, 2852, 11047, 6952, 7644, 9002, 9003, 15090, 11053, 6621, 13104, 817, 2868, 4918, 6967, 13112, 5940, 17690, 4924, 14772, 832, 9026, 15499, 9029, 15176, 17483, 2891, 10262, 15182, 847, 13136, 6997, 15161, 4954, 12431, 15197, 862, 2912, 13153, 9062, 4969, 877, 10767, 10728, 7029, 11128, 4985, 7038, 2943, 13184, 7041, 6635, 9093, 902, 13191, 5000, 15243, 908, 11153, 17300, 11159, 5016, 5017, 15258, 9117, 13215, 2976, 7075, 933, 5031, 15273, 9132, 8690, 2991, 7090, 15435, 948, 11189, 16603, 5047, 7096, 3003, 957, 16467, 4939, 11204, 5062, 7111, 9162, 971, 3021, 15312, 12785, 5077, 7126, 17368, 9177, 986, 3036, 7674, 9183, 11234, 5092, 7141, 1001, 16799, 3051, 9199, 11249, 5107, 7157, 1016, 3066, 12117, 9216, 10752, 7171, 15364, 17765, 3081, 5131, 9228, 12446, 1040, 5137, 7186, 11696, 15380, 3096, 11289, 9243, 2906, 13342, 1056, 15392, 1062, 3111, 7210, 9261, 11311, 7216, 15410, 13402, 1078, 17615, 5177, 14799, 16709, 11326, 3135, 16483, 9056, 11098, 1093, 9291, 7246, 10083, 13396, 1109, 11350, 5208, 9991, 4623, 11356, 3165, 5214, 16229, 3171, 1124, 16406, 15474, 9322, 8039, 5229, 13501, 3186, 1139, 8835, 16006, 13432, 9337, 15551, 5244, 10773, 1156, 1157, 7302, 13447, 3210, 5259, 11478, 1171, 7317, 13462, 9367, 7320, 3225, 5274, 15515, 3231, 15521, 11426, 16582, 9383, 5289, 7340, 7341, 13486, 6041, 7973, 1202, 9398, 3255, 5304, 15545, 7355, 3261, 18557, 15221, 1218, 12815, 13510, 11463, 13513, 7372, 7373, 12316, 9423, 11596, 3285, 5334, 2937, 1242, 3291, 11083, 1248, 13538, 9445, 5350, 17639, 7402, 13863, 15596, 3309, 11502, 14704, 9460, 6064, 1273, 13562, 15611, 5374, 12501, 11520, 13569, 9475, 9430, 11526, 5383, 3336, 5386, 15575, 1294, 10797, 5395, 11541, 7447, 3352, 3354, 9499, 1309, 13598, 15237, 5410, 7387, 11556, 7477, 7462, 7729, 3368, 9514, 16022, 11571, 1333, 13622, 15672, 9529, 1339, 13628, 11296, 9535, 3392, 5441, 7493, 3398, 15687, 13643, 8774, 9550, 8024, 11602, 1363, 7508, 3413, 15702, 13881, 5466, 5472, 11617, 7820, 1379, 3428, 15590, 9575, 13673, 17770, 13276, 5487, 1394, 3443, 17784, 14247, 11643, 7548, 9597, 3648, 1410, 10208, 13703, 17644, 11658, 3467, 9612, 5518, 3473, 15762, 11667, 14871, 14574, 12800, 1431, 9627, 3488, 16283, 7750, 1447, 11690, 15787, 5550, 3503, 5552, 15793, 9651, 9660, 13757, 3518, 1471, 15808, 13763, 7623, 9672, 12876, 17868, 7629, 1486, 14575, 14017, 5585, 13778, 1492, 11711, 3542, 7417, 3548, 5597, 13221, 17888, 13793, 10960, 9702, 15848, 7659, 5612, 1517, 253, 11371, 3572, 9717, 11766, 6740, 3578, 5627, 11772, 1533, 15870, 13824, 1539, 9732, 3593, 5642, 11787, 15885, 13839, 1554, 9747, 5654, 15961, 3608, 7705, 11802, 11183, 17948, 1569, 9762, 17958, 3623, 5672, 11441, 14599, 15916, 9138, 7554, 1288, 11829, 7735, 13236, 15931, 15285, 1600, 5697, 11844, 14262, 3654, 13896, 12215, 13902, 9807, 5712, 11859, 7765, 5718, 1624, 11865, 17679, 16177, 13917, 1630, 10853, 955, 5733, 9831, 16316, 13932, 7789, 1646, 11889, 15635, 5748, 3701, 9846, 11895, 13947, 7804, 4715, 3716, 1670, 11911, 9864, 18028, 3722, 1676, 18061, 9870, 12568, 7826, 10558, 4718, 11926, 15741, 13977, 13050, 5787, 9885, 16034, 3747, 1701, 3753, 1707, 9901, 11950, 5809, 14002, 14605, 16052, 3765, 12916, 1722, 9916, 17354, 15717, 7873, 15947, 11972, 18038, 3783, 1737, 9931, 16076, 15650, 5840, 3798, 9946, 16091, 11996, 14925, 5855, 13592, 1761, 12002, 18644, 3814, 1767, 9961, 12583, 5870, 10877, 15656, 12018, 3828, 14069, 7928, 16122, 1792, 9985, 3843, 14084, 5895, 16140, 7949, 1807, 12048, 7698, 5763, 1813, 5910, 3864, 17028, 5916, 12063, 3873, 16162, 1829, 11228, 7979, 12078, 3888, 14129, 1844, 5944, 15327, 5949, 10046, 3903, 14144, 7277, 8003, 1860, 12598, 14756, 5961, 12269, 13282, 3918, 14159, 17720, 8018, 10068, 1877, 5976, 16217, 3933, 14174, 15333, 16226, 5435, 12132, 1893, 5991, 13065, 16362, 10098, 12147, 1908, 3957, 8054, 14199, 10160, 3963, 16253, 6015, 1923, 8069, 14216, 12169, 10123, 16268, 14222, 6031, 1938, 3987, 8084, 6039, 12184, 3993, 10139, 17591, 12613, 1953, 9792, 13297, 4008, 6812, 10154, 8108, 1968, 12209, 8114, 6070, 4425, 14268, 16317, 12861, 1983, 4033, 8130, 6085, 12230, 4039, 8727, 10185, 14283, 1998, 10193, 10194, 12963, 16340, 12245, 4055, 16346, 6112, 8163, 14308, 4069, 6118, 2023, 12547, 14314, 15015, 18071, 2029, 10223, 7569, 8178, 6133, 14327, 4088, 4091, 16844, 12285, 10238])): matches = {k : v for k, v in matches.iteritems() if h not in k} > latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1] prevh = __find_md5__.search(latestmatch).group(0) IndexError: list index out of range > > Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org> > --- > diff --git a/bitbake/lib/bb/runqueue.py b/bitbake/lib/bb/runqueue.py > index 428cff1..b7a602b 100644 > --- a/bitbake/lib/bb/runqueue.py > +++ b/bitbake/lib/bb/runqueue.py > @@ -1039,6 +1039,7 @@ class RunQueue: > > if self.state is runQueueSceneInit: > if self.cooker.configuration.dump_signatures: > + self.print_diffscenetasks() > self.dump_signatures() > else: > self.start_worker() > @@ -1123,6 +1124,82 @@ class RunQueue: > > return > > + def print_diffscenetasks(self): > + > + valid = [] > + sq_hash = [] > + sq_hashfn = [] > + sq_fn = [] > + sq_taskname = [] > + sq_task = [] > + noexec = [] > + stamppresent = [] > + valid_new = set() > + > + for task in xrange(len(self.rqdata.runq_fnid)): > + fn = self.rqdata.taskData.fn_index[self.rqdata.runq_fnid[task]] > + taskname = self.rqdata.runq_task[task] > + taskdep = self.rqdata.dataCache.task_deps[fn] > + > + if 'noexec' in taskdep and taskname in taskdep['noexec']: > + noexec.append(task) > + continue > + > + sq_fn.append(fn) > + sq_hashfn.append(self.rqdata.dataCache.hashfn[fn]) > + sq_hash.append(self.rqdata.runq_hash[task]) > + sq_taskname.append(taskname) > + sq_task.append(task) > + call = self.hashvalidate + "(sq_fn, sq_task, sq_hash, sq_hashfn, d)" > + locs = { "sq_fn" : sq_fn, "sq_task" : sq_taskname, "sq_hash" : sq_hash, "sq_hashfn" : sq_hashfn, "d" : self.cooker.data } > + valid = bb.utils.better_eval(call, locs) > + for v in valid: > + valid_new.add(sq_task[v]) > + > + # Tasks which are both setscene and noexec never care about dependencies > + # We therefore find tasks which are setscene and noexec and mark their > + # unique dependencies as valid. > + for task in noexec: > + if task not in self.rqdata.runq_setscene: > + continue > + for dep in self.rqdata.runq_depends[task]: > + hasnoexecparents = True > + for dep2 in self.rqdata.runq_revdeps[dep]: > + if dep2 in self.rqdata.runq_setscene and dep2 in noexec: > + continue > + hasnoexecparents = False > + break > + if hasnoexecparents: > + valid_new.add(dep) > + > + invalidtasks = set() > + for task in xrange(len(self.rqdata.runq_fnid)): > + if task not in valid_new and task not in noexec: > + invalidtasks.add(task) > + > + found = set() > + processed = set() > + for task in invalidtasks: > + toprocess = set([task]) > + while toprocess: > + next = set() > + for t in toprocess: > + for dep in self.rqdata.runq_depends[t]: > + if dep in invalidtasks: > + found.add(task) > + if dep not in processed: > + processed.add(dep) > + next.add(dep) > + toprocess = next > + if task in found: > + toprocess = set() > + > + tasklist = [] > + for task in invalidtasks.difference(found): > + tasklist.append(self.rqdata.get_user_idstring(task)) > + > + if tasklist: > + bb.plain("The differences between the current build and any cached tasks start at the following tasks:\n" + "\n".join(tasklist)) > > class RunQueueExecute: > > > > _______________________________________________ > bitbake-devel mailing list > bitbake-devel@lists.openembedded.org > http://lists.openembedded.org/mailman/listinfo/bitbake-devel -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2013-12-28 13:24 ` Martin Jansa @ 2013-12-30 16:18 ` Chris Larson 2013-12-30 16:25 ` [RFC][WIP][PATCH] runqueue: Don't fail without match found Martin Jansa 2013-12-30 16:29 ` [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache Chris Larson 0 siblings, 2 replies; 16+ messages in thread From: Chris Larson @ 2013-12-30 16:18 UTC (permalink / raw) To: Martin Jansa; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 954 bytes --] On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <martin.jansa@gmail.com>wrote: > On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote: > > Its useful to understand where the delta starts against an existing > sstate cache > > for a given target. Adding this to the output of the -S option seems > like a > > natural fit. > > > > We use the hashvalidate function to figure this out and assume it can > find siginfo > > files for more than just the setscene tasks. > > I haven't tried to find smaller reproducer or exact part of code where > it happends, but in bigger builds I have it fails with > > I'll try to reproduce and narrow it a bit, but if you have tip where > to look first.. > > IndexError: list index out of range: I’m hitting this now as well. -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics [-- Attachment #2: Type: text/html, Size: 1474 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* [RFC][WIP][PATCH] runqueue: Don't fail without match found 2013-12-30 16:18 ` Chris Larson @ 2013-12-30 16:25 ` Martin Jansa 2013-12-30 16:29 ` [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache Chris Larson 1 sibling, 0 replies; 16+ messages in thread From: Martin Jansa @ 2013-12-30 16:25 UTC (permalink / raw) To: bitbake-devel * WIP, because I haven't checked the reasons why this happens I'm using this now only to see which recipes/tasks are failing like this Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com> --- lib/bb/runqueue.py | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py index 4577236..413b95e 100644 --- a/lib/bb/runqueue.py +++ b/lib/bb/runqueue.py @@ -1224,10 +1224,14 @@ class RunQueue: if match is None: bb.fatal("Can't find a task we're supposed to have written out? (hash: %s)?" % h) matches = {k : v for k, v in matches.iteritems() if h not in k} - latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1] - prevh = __find_md5__.search(latestmatch).group(0) - output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb) - bb.plain("\nTask %s:%s couldn't be used from the cache because:\n We need hash %s, closest matching task was %s\n " % (pn, taskname, h, prevh) + '\n '.join(output)) + sortedmatches=sorted(matches.keys(), key=lambda f: matches[f]) + if sortedmatches: + latestmatch = sorted(matches.keys(), key=lambda f: matches[f])[-1] + prevh = __find_md5__.search(latestmatch).group(0) + output = bb.siggen.compare_sigfiles(latestmatch, match, recursecb) + bb.plain("\nTask %s:%s couldn't be used from the cache because:\n We need hash %s, closest matching task was %s\n " % (pn, taskname, h, prevh) + '\n '.join(output)) + else: + bb.plain("\nTask %s:%s couldn't find any matches for hash %s" % (pn, taskname, h)) -- 1.8.4.3 ^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2013-12-30 16:18 ` Chris Larson 2013-12-30 16:25 ` [RFC][WIP][PATCH] runqueue: Don't fail without match found Martin Jansa @ 2013-12-30 16:29 ` Chris Larson 2013-12-30 16:35 ` Martin Jansa 1 sibling, 1 reply; 16+ messages in thread From: Chris Larson @ 2013-12-30 16:29 UTC (permalink / raw) To: Martin Jansa; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 1659 bytes --] On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <clarson@kergoth.com> wrote: > On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <martin.jansa@gmail.com>wrote: > >> On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote: >> > Its useful to understand where the delta starts against an existing >> sstate cache >> > for a given target. Adding this to the output of the -S option seems >> like a >> > natural fit. >> > >> > We use the hashvalidate function to figure this out and assume it can >> find siginfo >> > files for more than just the setscene tasks. >> >> I haven't tried to find smaller reproducer or exact part of code where >> it happends, but in bigger builds I have it fails with >> >> I'll try to reproduce and narrow it a bit, but if you have tip where >> to look first.. >> >> IndexError: list index out of range: > > > I’m hitting this now as well. To clarify further, I’m hitting this with stock poky master, qemux86, bitbake -S core-image-base. Actually, this is getting in my way trying to do something I don’t want it doing anyway. I want to dump the signature data from one config, and use bitbake-whatchanged from another build. If I wanted to compare something right here, I’d have run bitbake-whatchanged. This change confuses me a bit. -S emits signature data, according to the bitbake help and the semantics of the command. Are we trying to kill bitbake-whatchanged in favor of making -S do both jobs, emit and compare? -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics [-- Attachment #2: Type: text/html, Size: 2546 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2013-12-30 16:29 ` [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache Chris Larson @ 2013-12-30 16:35 ` Martin Jansa 2013-12-30 22:10 ` Richard Purdie 0 siblings, 1 reply; 16+ messages in thread From: Martin Jansa @ 2013-12-30 16:35 UTC (permalink / raw) To: Chris Larson; +Cc: bitbake-devel [-- Attachment #1: Type: text/plain, Size: 2190 bytes --] On Mon, Dec 30, 2013 at 09:29:38AM -0700, Chris Larson wrote: > On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <clarson@kergoth.com> wrote: > > > On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <martin.jansa@gmail.com>wrote: > > > >> On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote: > >> > Its useful to understand where the delta starts against an existing > >> sstate cache > >> > for a given target. Adding this to the output of the -S option seems > >> like a > >> > natural fit. > >> > > >> > We use the hashvalidate function to figure this out and assume it can > >> find siginfo > >> > files for more than just the setscene tasks. > >> > >> I haven't tried to find smaller reproducer or exact part of code where > >> it happends, but in bigger builds I have it fails with > >> > >> I'll try to reproduce and narrow it a bit, but if you have tip where > >> to look first.. > >> > >> IndexError: list index out of range: > > > > > > I’m hitting this now as well. > > > To clarify further, I’m hitting this with stock poky master, qemux86, > bitbake -S core-image-base. > > Actually, this is getting in my way trying to do something I don’t want it > doing anyway. I want to dump the signature data from one config, and use > bitbake-whatchanged from another build. If I wanted to compare something > right here, I’d have run bitbake-whatchanged. This change confuses me a > bit. -S emits signature data, according to the bitbake help and the > semantics of the command. Are we trying to kill bitbake-whatchanged in > favor of making -S do both jobs, emit and compare? Same here, I'm using bitbake -S from oe-core/scripts/sstate-diff-machines.sh and this added analysis and output is IMHO very useful, but it would be nice to control it with some new option (and keep default -S behavior fast and simple). It looks like the error is triggered for every task which wasn't built before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or bitbake -S world after bitbake some-image). I see it for many do_fetch and do_rm_work_all tasks. -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2013-12-30 16:35 ` Martin Jansa @ 2013-12-30 22:10 ` Richard Purdie 2013-12-30 22:34 ` Chris Larson 0 siblings, 1 reply; 16+ messages in thread From: Richard Purdie @ 2013-12-30 22:10 UTC (permalink / raw) To: Martin Jansa; +Cc: Chris Larson, bitbake-devel On Mon, 2013-12-30 at 17:35 +0100, Martin Jansa wrote: > On Mon, Dec 30, 2013 at 09:29:38AM -0700, Chris Larson wrote: > > On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <clarson@kergoth.com> wrote: > > > > > On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa <martin.jansa@gmail.com>wrote: > > > > > >> On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote: > > >> > Its useful to understand where the delta starts against an existing > > >> sstate cache > > >> > for a given target. Adding this to the output of the -S option seems > > >> like a > > >> > natural fit. > > >> > > > >> > We use the hashvalidate function to figure this out and assume it can > > >> find siginfo > > >> > files for more than just the setscene tasks. > > >> > > >> I haven't tried to find smaller reproducer or exact part of code where > > >> it happends, but in bigger builds I have it fails with > > >> > > >> I'll try to reproduce and narrow it a bit, but if you have tip where > > >> to look first.. > > >> > > >> IndexError: list index out of range: > > > > > > > > > I’m hitting this now as well. > > > > > > To clarify further, I’m hitting this with stock poky master, qemux86, > > bitbake -S core-image-base. > > > > Actually, this is getting in my way trying to do something I don’t want it > > doing anyway. I want to dump the signature data from one config, and use > > bitbake-whatchanged from another build. If I wanted to compare something > > right here, I’d have run bitbake-whatchanged. This change confuses me a > > bit. -S emits signature data, according to the bitbake help and the > > semantics of the command. Are we trying to kill bitbake-whatchanged in > > favor of making -S do both jobs, emit and compare? > > Same here, I'm using bitbake -S from > oe-core/scripts/sstate-diff-machines.sh > > and this added analysis and output is IMHO very useful, but it would be > nice to control it with some new option (and keep default -S behavior > fast and simple). > > It looks like the error is triggered for every task which wasn't built > before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or > bitbake -S world after bitbake some-image). > > I see it for many do_fetch and do_rm_work_all tasks. To put things in context, I was trying to work with the existing "whatchanged" type tools or more specifically I wanted to know "why isn't my sstate cache being used". Having to do a two step process for any of these enquires seems a little odd. There are also two specific and as it turns our quite different work flows: a) What changed since my last build? b) What is the least delta between my current build and whatever I have cached? Those experiments revealed some big holes in the sstate metadata and some good improvements there. It also highlights that there are two different questions which have different answers. In the process, I found bitbake does have some state in runqueue which can help pinpoint the "whatchanged" part of the problem much more accurately/quickly. Obviously that state is lost after bitbake -S finishes so I started printing out some extra data. Clearly bitbake shouldn't crash, that is a bug and I think Martin is heading in the right direction with the other patch. As for bitbake -S being kept fast, I think (but I need to double check with measurements) that the parsing part of bitbake -S is the slow part and this extra data is effectively "for free". So basically consider the current state an experiment to see if we can make -S more useful/interesting (without much cost) and to try to better understand where/what functionality we need going forward. One possible outcome might be we generate an extra file (or files) (like we do with task-depends.dot) which contains this extra tree data which some other command can then parse and display to the user. I'm still struggling to get a complete grip on the exact data and format that file needs to be. Part of the reason for having -S print something was to see if it printed what the user wanted to know or if not, we could then try and figure out what the right thing was so we could improve it. So sorry if this is causing pain, the code is easy to disable by commenting out the call, or reverting the patch adding it. Equally, I think we need to make sstate hashes and reasons for rebuilds more accessible to users and I think this does move us closer to that assuming we fix the crash. I'm trying to take a bit of a break over the holiday period (been riding a motorcycle in driving snow/rain today and am rather damp/cold) but I'll get the crash fixed sometime next week at the latest. Cheers, Richard ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2013-12-30 22:10 ` Richard Purdie @ 2013-12-30 22:34 ` Chris Larson 2014-01-01 23:38 ` Richard Purdie 0 siblings, 1 reply; 16+ messages in thread From: Chris Larson @ 2013-12-30 22:34 UTC (permalink / raw) To: Richard Purdie; +Cc: Chris Larson, bitbake-devel [-- Attachment #1: Type: text/plain, Size: 5209 bytes --] On Monday, December 30, 2013, Richard Purdie wrote: > On Mon, 2013-12-30 at 17:35 +0100, Martin Jansa wrote: > > On Mon, Dec 30, 2013 at 09:29:38AM -0700, Chris Larson wrote: > > > On Mon, Dec 30, 2013 at 9:18 AM, Chris Larson <clarson@kergoth.com<javascript:;>> > wrote: > > > > > > > On Sat, Dec 28, 2013 at 6:24 AM, Martin Jansa < > martin.jansa@gmail.com <javascript:;>>wrote: > > > > > > > >> On Wed, Dec 18, 2013 at 04:21:27PM +0000, Richard Purdie wrote: > > > >> > Its useful to understand where the delta starts against an > existing > > > >> sstate cache > > > >> > for a given target. Adding this to the output of the -S option > seems > > > >> like a > > > >> > natural fit. > > > >> > > > > >> > We use the hashvalidate function to figure this out and assume it > can > > > >> find siginfo > > > >> > files for more than just the setscene tasks. > > > >> > > > >> I haven't tried to find smaller reproducer or exact part of code > where > > > >> it happends, but in bigger builds I have it fails with > > > >> > > > >> I'll try to reproduce and narrow it a bit, but if you have tip where > > > >> to look first.. > > > >> > > > >> IndexError: list index out of range: > > > > > > > > > > > > I’m hitting this now as well. > > > > > > > > > To clarify further, I’m hitting this with stock poky master, qemux86, > > > bitbake -S core-image-base. > > > > > > Actually, this is getting in my way trying to do something I don’t > want it > > > doing anyway. I want to dump the signature data from one config, and > use > > > bitbake-whatchanged from another build. If I wanted to compare > something > > > right here, I’d have run bitbake-whatchanged. This change confuses me a > > > bit. -S emits signature data, according to the bitbake help and the > > > semantics of the command. Are we trying to kill bitbake-whatchanged in > > > favor of making -S do both jobs, emit and compare? > > > > Same here, I'm using bitbake -S from > > oe-core/scripts/sstate-diff-machines.sh > > > > and this added analysis and output is IMHO very useful, but it would be > > nice to control it with some new option (and keep default -S behavior > > fast and simple). > > > > It looks like the error is triggered for every task which wasn't built > > before (e.g. running bitbake -S with empty sstate-cache/tmp-eglibc or > > bitbake -S world after bitbake some-image). > > > > I see it for many do_fetch and do_rm_work_all tasks. > > To put things in context, I was trying to work with the existing > "whatchanged" type tools or more specifically I wanted to know "why > isn't my sstate cache being used". Having to do a two step process for > any of these enquires seems a little odd. There are also two specific > and as it turns our quite different work flows: > > a) What changed since my last build? > b) What is the least delta between my current build and whatever I have > cached? > Those experiments revealed some big holes in the sstate metadata and > some good improvements there. It also highlights that there are two > different questions which have different answers. > > In the process, I found bitbake does have some state in runqueue which > can help pinpoint the "whatchanged" part of the problem much more > accurately/quickly. Obviously that state is lost after bitbake -S > finishes so I started printing out some extra data. > > Clearly bitbake shouldn't crash, that is a bug and I think Martin is > heading in the right direction with the other patch. > > As for bitbake -S being kept fast, I think (but I need to double check > with measurements) that the parsing part of bitbake -S is the slow part > and this extra data is effectively "for free". > > So basically consider the current state an experiment to see if we can > make -S more useful/interesting (without much cost) and to try to better > understand where/what functionality we need going forward. > I can see that, but I think you're making the semantics of the command-line argument confusing by trying to combine the two operations into one. I also think that trying to find the "least delta" is extremely weird, and not always likely to produce anything worthwhile. My SSTATE_DIR has weeks if not months of output from hundreds of builds of different combinations of machine, distro, layer selection, upstream vs mel, as well as between tons of build environments with slight local alterations. Unless I'm misunderstanding something, how exactly is it going to know what I want it to compare against, and how it is going to keep finding the "least delta" fast, amongst all that data? With the current tools, I can explicitly use -S in one env and whatchanged in another (with appropriate STAMPS_DIR) to compare between two exact build environments, rather than letting bitbake try to automagically find what I meant. Now that the two are combined into one, I'm not seeing how to do what I'm currently doing. Can you explain this? -Chris -- Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics [-- Attachment #2: Type: text/html, Size: 6312 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2013-12-30 22:34 ` Chris Larson @ 2014-01-01 23:38 ` Richard Purdie 2014-01-10 14:14 ` Martin Jansa 2014-01-21 22:37 ` Richard Purdie 0 siblings, 2 replies; 16+ messages in thread From: Richard Purdie @ 2014-01-01 23:38 UTC (permalink / raw) To: Chris Larson; +Cc: bitbake-devel On Mon, 2013-12-30 at 15:34 -0700, Chris Larson wrote: > I can see that, but I think you're making the semantics of the > command-line argument confusing by trying to combine the two > operations into one. I also think that trying to find the "least > delta" is extremely weird, and not always likely to produce anything > worthwhile. My SSTATE_DIR has weeks if not months of output from > hundreds of builds of different combinations of machine, distro, layer > selection, upstream vs mel, as well as between tons of build > environments with slight local alterations. Unless I'm > misunderstanding something, how exactly is it going to know what I > want it to compare against, and how it is going to keep finding the > "least delta" fast, amongst all that data? With the current tools, I > can explicitly use -S in one env and whatchanged in another (with > appropriate STAMPS_DIR) to compare between two exact build > environments, rather than letting bitbake try to automagically find > what I meant. Now that the two are combined into one, I'm not seeing > how to do what I'm currently doing. Can you explain this? Nothing changed in the data -S writes out. Yes, it does currently crash in some cases which is obviously a bug however assuming we fix that you can ignore what -S prints on the screen or writes to a file or whatever we end up doing and just do what you're doing now. On the other hand we have a huge hole in the current model when you don't have the two environments to compare. You have the current build and you want to know why the cache isn't being used. That is a valid user question which shouldn't need another build to compare against. So can we find the "least delta" fast? Its not actually that hard computationally or on resources, at least in the experiments I've made. We know the hashes of the current target's tasks and we can quickly tell which are in the cache and which are not (using the same function sstate uses for that purpose). That gives you a set of delta points with the sstate cache but you also know the recipe name and other details about the target of interest and that they should have common parent tasks. Based on that information you can come up with a list of possibilities of least difference and sort those by date (which bitbake-diffsigs -t uses) or whatever other mechanism we end up choosing to display the data. I agree its not always going to give useful information but its a start in the right direction and my local tests suggest its perhaps good enough to give the user the hints they want about cache reuse (or lack of). I agree we're not there yet, equally, to be quite honest the current debug of sstate sucks and we need to do something about it. Whilst you and I may understand it and are able to debug issues, many users are struggling with it :(. Cheers, Richard ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2014-01-01 23:38 ` Richard Purdie @ 2014-01-10 14:14 ` Martin Jansa 2014-01-27 8:52 ` Martin Jansa 2014-01-21 22:37 ` Richard Purdie 1 sibling, 1 reply; 16+ messages in thread From: Martin Jansa @ 2014-01-10 14:14 UTC (permalink / raw) To: Richard Purdie; +Cc: Chris Larson, bitbake-devel [-- Attachment #1: Type: text/plain, Size: 2166 bytes --] On Wed, Jan 01, 2014 at 11:38:01PM +0000, Richard Purdie wrote: > So can we find the "least delta" fast? Its not actually that hard > computationally or on resources, at least in the experiments I've made. > We know the hashes of the current target's tasks and we can quickly tell > which are in the cache and which are not (using the same function sstate > uses for that purpose). I've finally got some numbers to support my claim that it "feels" slower with this added functionality. My script is calling openembedded-core/scripts/sstate-diff-machines.sh --machines="qemuarm qemux86 qemux86copy" --targets=world --tmpdir=tmp-eglibc/; which in turn runs bitbake -S world three times (once for each machine). The script on jenkins doing world builds is completely crazy (I've killed it in last run assuming it was stuck, but it wasn't just terribly slow). http://jenkins.nas-admin.org/job/oe_world_compare_signatures/11/ Running 21 hours already and doing 2nd machine from 3 (maybe because longer SSTATE_MIRROR?). There is one python process using 100% cpu. In local builds without SSTATE_MIRROR it's not so bad, but still significantly slower with each release (dylan-dora-master). Master builds have that WIP patch included in order to finish. It's not very accurate, because it was running on my desktop and each one has different metadata, but at least the layers included are similar (only systems which got separated in dora and removal of meta-webos-backports which isn't needed for dora and newer). At least I've executed each test twice. dylan: real 9m37.271s user 6m37.104s sys 3m12.625s real 10m5.787s user 6m37.832s sys 3m23.614s dora: real 26m11.374s user 9m7.325s sys 3m0.048s real 28m10.606s user 8m27.895s sys 2m57.497s master: real 38m17.641s user 17m46.905s sys 5m12.061s real 36m12.748s user 16m32.961s sys 4m43.103s master-without-S-changes: real 27m34.228s user 8m6.811s sys 2m53.724s real 27m49.752s user 8m25.026s sys 2m50.921s -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2014-01-10 14:14 ` Martin Jansa @ 2014-01-27 8:52 ` Martin Jansa 0 siblings, 0 replies; 16+ messages in thread From: Martin Jansa @ 2014-01-27 8:52 UTC (permalink / raw) To: Richard Purdie; +Cc: Chris Larson, bitbake-devel [-- Attachment #1: Type: text/plain, Size: 2829 bytes --] On Fri, Jan 10, 2014 at 03:14:17PM +0100, Martin Jansa wrote: > On Wed, Jan 01, 2014 at 11:38:01PM +0000, Richard Purdie wrote: > > So can we find the "least delta" fast? Its not actually that hard > > computationally or on resources, at least in the experiments I've made. > > We know the hashes of the current target's tasks and we can quickly tell > > which are in the cache and which are not (using the same function sstate > > uses for that purpose). > > I've finally got some numbers to support my claim that it "feels" slower > with this added functionality. > > My script is calling > openembedded-core/scripts/sstate-diff-machines.sh --machines="qemuarm qemux86 qemux86copy" --targets=world --tmpdir=tmp-eglibc/; > > which in turn runs > bitbake -S world > three times (once for each machine). > > The script on jenkins doing world builds is completely crazy (I've > killed it in last run assuming it was stuck, but it wasn't just terribly > slow). > http://jenkins.nas-admin.org/job/oe_world_compare_signatures/11/ > Running 21 hours already and doing 2nd machine from 3 (maybe because > longer SSTATE_MIRROR?). There is one python process using 100% cpu. FWIW: this jenkins job was completed yesterday after 16 days small part of strace: http://logs.nslu2-linux.org/buildlogs/oe/oe-shr-core-branches/log.world.20140109_144047.log/log.strace complete log: http://logs.nslu2-linux.org/buildlogs/oe/oe-shr-core-branches/log.world.20140109_144047.log/signatures.sstate-diff-machines Previous runs of the same job took 25-40 minutes > In local builds without SSTATE_MIRROR it's not so bad, but still > significantly slower with each release (dylan-dora-master). > > Master builds have that WIP patch included in order to finish. > > It's not very accurate, because it was running on my desktop and each one has > different metadata, but at least the layers included are similar (only > systems which got separated in dora and removal > of meta-webos-backports which isn't needed for dora and newer). > > At least I've executed each test twice. > > dylan: > real 9m37.271s > user 6m37.104s > sys 3m12.625s > > real 10m5.787s > user 6m37.832s > sys 3m23.614s > > dora: > real 26m11.374s > user 9m7.325s > sys 3m0.048s > > real 28m10.606s > user 8m27.895s > sys 2m57.497s > > master: > real 38m17.641s > user 17m46.905s > sys 5m12.061s > > real 36m12.748s > user 16m32.961s > sys 4m43.103s > > master-without-S-changes: > real 27m34.228s > user 8m6.811s > sys 2m53.724s > > real 27m49.752s > user 8m25.026s > sys 2m50.921s > > -- > Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2014-01-01 23:38 ` Richard Purdie 2014-01-10 14:14 ` Martin Jansa @ 2014-01-21 22:37 ` Richard Purdie 2014-01-21 22:46 ` Martin Jansa 2014-01-27 13:23 ` Richard Purdie 1 sibling, 2 replies; 16+ messages in thread From: Richard Purdie @ 2014-01-21 22:37 UTC (permalink / raw) To: Chris Larson; +Cc: bitbake-devel On Wed, 2014-01-01 at 23:38 +0000, Richard Purdie wrote: > Nothing changed in the data -S writes out. > > Yes, it does currently crash in some cases which is obviously a bug > however assuming we fix that you can ignore what -S prints on the screen > or writes to a file or whatever we end up doing and just do what you're > doing now. > > On the other hand we have a huge hole in the current model when you > don't have the two environments to compare. You have the current build > and you want to know why the cache isn't being used. That is a valid > user question which shouldn't need another build to compare against. > > So can we find the "least delta" fast? Its not actually that hard > computationally or on resources, at least in the experiments I've made. > We know the hashes of the current target's tasks and we can quickly tell > which are in the cache and which are not (using the same function sstate > uses for that purpose). > > That gives you a set of delta points with the sstate cache but you also > know the recipe name and other details about the target of interest and > that they should have common parent tasks. Based on that information you > can come up with a list of possibilities of least difference and sort > those by date (which bitbake-diffsigs -t uses) or whatever other > mechanism we end up choosing to display the data. > > I agree its not always going to give useful information but its a start > in the right direction and my local tests suggest its perhaps good > enough to give the user the hints they want about cache reuse (or lack > of). > > I agree we're not there yet, equally, to be quite honest the current > debug of sstate sucks and we need to do something about it. Whilst you > and I may understand it and are able to debug issues, many users are > struggling with it :(. I appreciate its taken me a while to loop back around on this but I do now have an idea. What I'm proposing we do is add an optional parameter to -S. Without any parameter it would behave as it did classically. With a parameter, it could trigger the current "debug sstate" behaviour to stdout and it also allows it to trigger an sstate siggen specific behaviour as appripriate. I have some prototype code with writes out a "locked-sigs.inc" file for example, triggered from this same code block. I think this should let us do more creative things with sstate (even from the OE siggen class) yet also let it remain useful for different people. I don't have any code to do this yet but at least we have a plan assuming no strong objections. Cheers, Richard ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2014-01-21 22:37 ` Richard Purdie @ 2014-01-21 22:46 ` Martin Jansa 2014-01-27 13:23 ` Richard Purdie 1 sibling, 0 replies; 16+ messages in thread From: Martin Jansa @ 2014-01-21 22:46 UTC (permalink / raw) To: Richard Purdie; +Cc: Chris Larson, bitbake-devel [-- Attachment #1: Type: text/plain, Size: 2904 bytes --] On Tue, Jan 21, 2014 at 10:37:33PM +0000, Richard Purdie wrote: > On Wed, 2014-01-01 at 23:38 +0000, Richard Purdie wrote: > > Nothing changed in the data -S writes out. > > > > Yes, it does currently crash in some cases which is obviously a bug > > however assuming we fix that you can ignore what -S prints on the screen > > or writes to a file or whatever we end up doing and just do what you're > > doing now. > > > > On the other hand we have a huge hole in the current model when you > > don't have the two environments to compare. You have the current build > > and you want to know why the cache isn't being used. That is a valid > > user question which shouldn't need another build to compare against. > > > > So can we find the "least delta" fast? Its not actually that hard > > computationally or on resources, at least in the experiments I've made. > > We know the hashes of the current target's tasks and we can quickly tell > > which are in the cache and which are not (using the same function sstate > > uses for that purpose). > > > > That gives you a set of delta points with the sstate cache but you also > > know the recipe name and other details about the target of interest and > > that they should have common parent tasks. Based on that information you > > can come up with a list of possibilities of least difference and sort > > those by date (which bitbake-diffsigs -t uses) or whatever other > > mechanism we end up choosing to display the data. > > > > I agree its not always going to give useful information but its a start > > in the right direction and my local tests suggest its perhaps good > > enough to give the user the hints they want about cache reuse (or lack > > of). > > > > I agree we're not there yet, equally, to be quite honest the current > > debug of sstate sucks and we need to do something about it. Whilst you > > and I may understand it and are able to debug issues, many users are > > struggling with it :(. > > I appreciate its taken me a while to loop back around on this but I do > now have an idea. What I'm proposing we do is add an optional parameter > to -S. Without any parameter it would behave as it did classically. > > With a parameter, it could trigger the current "debug sstate" behaviour > to stdout and it also allows it to trigger an sstate siggen specific > behaviour as appripriate. I have some prototype code with writes out a > "locked-sigs.inc" file for example, triggered from this same code block. > > I think this should let us do more creative things with sstate (even > from the OE siggen class) yet also let it remain useful for different > people. > > I don't have any code to do this yet but at least we have a plan > assuming no strong objections. I strongly support this plan :). Cheers, -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2014-01-21 22:37 ` Richard Purdie 2014-01-21 22:46 ` Martin Jansa @ 2014-01-27 13:23 ` Richard Purdie 2014-01-27 13:39 ` Martin Jansa 1 sibling, 1 reply; 16+ messages in thread From: Richard Purdie @ 2014-01-27 13:23 UTC (permalink / raw) To: Chris Larson; +Cc: bitbake-devel On Tue, 2014-01-21 at 22:37 +0000, Richard Purdie wrote: > I appreciate its taken me a while to loop back around on this but I do > now have an idea. What I'm proposing we do is add an optional parameter > to -S. Without any parameter it would behave as it did classically. > > With a parameter, it could trigger the current "debug sstate" behaviour > to stdout and it also allows it to trigger an sstate siggen specific > behaviour as appripriate. I have some prototype code with writes out a > "locked-sigs.inc" file for example, triggered from this same code block. > > I think this should let us do more creative things with sstate (even > from the OE siggen class) yet also let it remain useful for different > people. So we have a small potential issue here. We use python's optparse which doesn't allow optional arguments. This means we can make -S take options but the options must always be specified. It has good reason for this as its near impossible for optparse to tell the difference between: bitbake image -S tracesigs and bitbake image -S image2 So our options are: a) Require a new parameter to -S always b) Hack it to use the option after -S as a parameter (meaning -S should always be last on the commandline). I do have a proof of concept but it makes me uneasy c) Add a new option with a different name In the interests of not doing something which binds us into a problem in future, I'm thinking -S should start to always take a parameter. The only issue is I'd like that parameter to be metadata (siggen) definable so we can't validate the option passed. Any opinions? Cheers, Richard ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2014-01-27 13:23 ` Richard Purdie @ 2014-01-27 13:39 ` Martin Jansa 2014-01-27 13:46 ` Richard Purdie 0 siblings, 1 reply; 16+ messages in thread From: Martin Jansa @ 2014-01-27 13:39 UTC (permalink / raw) To: Richard Purdie; +Cc: Chris Larson, bitbake-devel [-- Attachment #1: Type: text/plain, Size: 1969 bytes --] On Mon, Jan 27, 2014 at 01:23:25PM +0000, Richard Purdie wrote: > On Tue, 2014-01-21 at 22:37 +0000, Richard Purdie wrote: > > I appreciate its taken me a while to loop back around on this but I do > > now have an idea. What I'm proposing we do is add an optional parameter > > to -S. Without any parameter it would behave as it did classically. > > > > With a parameter, it could trigger the current "debug sstate" behaviour > > to stdout and it also allows it to trigger an sstate siggen specific > > behaviour as appripriate. I have some prototype code with writes out a > > "locked-sigs.inc" file for example, triggered from this same code block. > > > > I think this should let us do more creative things with sstate (even > > from the OE siggen class) yet also let it remain useful for different > > people. > > So we have a small potential issue here. We use python's optparse which > doesn't allow optional arguments. This means we can make -S take options > but the options must always be specified. > > It has good reason for this as its near impossible for optparse to tell > the difference between: > > bitbake image -S tracesigs > and > bitbake image -S image2 > > So our options are: > > a) Require a new parameter to -S always > b) Hack it to use the option after -S as a parameter (meaning -S should > always be last on the commandline). I do have a proof of concept but it > makes me uneasy > c) Add a new option with a different name > > In the interests of not doing something which binds us into a problem in > future, I'm thinking -S should start to always take a parameter. The > only issue is I'd like that parameter to be metadata (siggen) definable > so we can't validate the option passed. > > Any opinions? What about using some existing parameter in combination with -S? -S + -v shows tracesigs -S without -v old behavior -- Martin 'JaMa' Jansa jabber: Martin.Jansa@gmail.com [-- Attachment #2: Digital signature --] [-- Type: application/pgp-signature, Size: 205 bytes --] ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache 2014-01-27 13:39 ` Martin Jansa @ 2014-01-27 13:46 ` Richard Purdie 0 siblings, 0 replies; 16+ messages in thread From: Richard Purdie @ 2014-01-27 13:46 UTC (permalink / raw) To: Martin Jansa; +Cc: Chris Larson, bitbake-devel On Mon, 2014-01-27 at 14:39 +0100, Martin Jansa wrote: > On Mon, Jan 27, 2014 at 01:23:25PM +0000, Richard Purdie wrote: > > On Tue, 2014-01-21 at 22:37 +0000, Richard Purdie wrote: > > > I appreciate its taken me a while to loop back around on this but I do > > > now have an idea. What I'm proposing we do is add an optional parameter > > > to -S. Without any parameter it would behave as it did classically. > > > > > > With a parameter, it could trigger the current "debug sstate" behaviour > > > to stdout and it also allows it to trigger an sstate siggen specific > > > behaviour as appripriate. I have some prototype code with writes out a > > > "locked-sigs.inc" file for example, triggered from this same code block. > > > > > > I think this should let us do more creative things with sstate (even > > > from the OE siggen class) yet also let it remain useful for different > > > people. > > > > So we have a small potential issue here. We use python's optparse which > > doesn't allow optional arguments. This means we can make -S take options > > but the options must always be specified. > > > > It has good reason for this as its near impossible for optparse to tell > > the difference between: > > > > bitbake image -S tracesigs > > and > > bitbake image -S image2 > > > > So our options are: > > > > a) Require a new parameter to -S always > > b) Hack it to use the option after -S as a parameter (meaning -S should > > always be last on the commandline). I do have a proof of concept but it > > makes me uneasy > > c) Add a new option with a different name > > > > In the interests of not doing something which binds us into a problem in > > future, I'm thinking -S should start to always take a parameter. The > > only issue is I'd like that parameter to be metadata (siggen) definable > > so we can't validate the option passed. > > > > Any opinions? > > What about using some existing parameter in combination with -S? > > -S + -v shows tracesigs > -S without -v old behavior I have some use cases locally (such as locked sigs) which could really benefit from a string parameter which gets passed into the siggen code. I can see us needing more than two "behaviours" of -S... I also think using something like -v will lead to more confusion and the code won't be particularly nice either :/. Cheers, Richard ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2014-01-27 13:47 UTC | newest] Thread overview: 16+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2013-12-18 16:21 [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache Richard Purdie 2013-12-28 13:24 ` Martin Jansa 2013-12-30 16:18 ` Chris Larson 2013-12-30 16:25 ` [RFC][WIP][PATCH] runqueue: Don't fail without match found Martin Jansa 2013-12-30 16:29 ` [PATCH] runqueue: Add output for -S option for listing the changepoints compared with an sstate cache Chris Larson 2013-12-30 16:35 ` Martin Jansa 2013-12-30 22:10 ` Richard Purdie 2013-12-30 22:34 ` Chris Larson 2014-01-01 23:38 ` Richard Purdie 2014-01-10 14:14 ` Martin Jansa 2014-01-27 8:52 ` Martin Jansa 2014-01-21 22:37 ` Richard Purdie 2014-01-21 22:46 ` Martin Jansa 2014-01-27 13:23 ` Richard Purdie 2014-01-27 13:39 ` Martin Jansa 2014-01-27 13:46 ` Richard Purdie
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.