diff for duplicates of <1473970950.31073.23.camel@intel.com> diff --git a/a/1.txt b/N1/1.txt index e7d791b..36b829c 100644 --- a/a/1.txt +++ b/N1/1.txt @@ -1,156 +1,108 @@ -On Thu, 2016-09-15 at 10:59 -0700, Ben Greear wrote: -> On 09/15/2016 10:34 AM, Grumbach, Emmanuel wrote: -> > On Thu, 2016-09-15 at 08:14 -0700, Ben Greear wrote: -> > > On 09/15/2016 07:06 AM, Valo, Kalle wrote: -> > > > Ben Greear <greearb@candelatech.com> writes: -> > > > -> > > > > On 09/14/2016 07:18 AM, Valo, Kalle wrote: -> > > > > > greearb@candelatech.com writes: -> > > > > > -> > > > > > > From: Ben Greear <greearb@candelatech.com> -> > > > > > > -> > > > > > > This allows user-space tools to decode debug-log -> > > > > > > messages by parsing dmesg or /var/log/messages. -> > > > > > > -> > > > > > > Signed-off-by: Ben Greear <greearb@candelatech.com> -> > > > > > -> > > > > > Don't tracing points already provide the same information? -> > > > > -> > > > > Tracing tools are difficult to set up and may not be -> > > > > available on -> > > > > random embedded devices. And if we are dealing with bug -> > > > > reports -> > > > > from -> > > > > the field, most users will not be able to set it up -> > > > > regardless. -> > > > > -> > > > > There are similar ways to print out hex, but the logic below -> > > > > creates -> > > > > specific and parseable logs in the 'dmesg' output and -> > > > > similar. -> > > > > -> > > > > I have written a tool that can decode these messages into -> > > > > useful -> > > > > human-readable -> > > > > text so that I can debug firmware issues both locally and -> > > > > from -> > > > > field reports. -> > > > > -> > > > > Stock firmware generates similar logs and QCA could write -> > > > > their -> > > > > own decode logic -> > > > > for their firmware versions. -> > > > -> > > > Reinventing the wheel by using printk as the delivery mechanism -> > > > doesn't -> > > > sound like a good idea. IIRC Emmanuel talked about some kind of -> > > > firmware -> > > > debugging framework, he might have some ideas. -> > > -> > > Waiting for magical frameworks to fix problems is even worse. -> > > -> > It has been years since ath10k has been in the kernel. There is -> > > basically -> > > still no way to debug what the firmware is doing. -> > > -> > -> > I know the feeling :) I was in the same situation before I added -> > stuff -> > for iwlwifi. -> > -> > > My patch gives you something that can work right now, with the -> > > standard 'dmesg' -> > > framework found in virtually all kernels new and old, and it has -> > > been -> > > proven -> > > to be useful in the field. The messages are also nicely -> > > interleaved -> > > with the -> > > rest of the mac80211 stack messages and any other driver -> > > messages, so -> > > you have -> > > context. -> > > -> > > If someone wants to add support for a framework later, then by -> > > all -> > > means, post -> > > the patches when it is ready. -> > -> > From my experience, a strong and easy-to-use firmware debug -> > infrastructure is important because typically, the firmware is -> > written -> > by other people who have different priorities (and are not always -> > Linux -> > wizards) etc... Being able to give them good data is the only way -> > to -> > have them fix their bugs :) For us, it was really a game changer. -> > When -> > you work for a big corporate, having 2 groups work better together -> > always has a big impact. That's for the philosophical part :) -> > -> > FWIW: what I did has nothing to do with FW 'live tracing', but with -> > firmware dumps. One part of our firmware dumps include tracing. We -> > also -> > have "firmware prints", but we don't print them in the kernel log -> > and -> > they are not part of the firmware dump thing. We rather record them -> > in -> > tracepoints just like really *anything* that comes from the -> > firmware. -> > Basically, we have 2 layers, the transport layer (PCIe) and the -> > operation_mode layer. The first just brings the data from the -> > firmware -> > and in that layer we *blindly* record everything in tracepoints. In -> > the -> > operation_mode layer, we look at the data itself. In case of debug -> > prints from the firmware, we simply discard them, because we don't -> > really care of the meaning. All we want is to have them go through -> > the -> > PCIe layer so that they are recorded in the tracepoints. -> > When we finish recording the sequence we wanted with tracing (trace -> > -cmd), we parse the output and then, we parse the firmware prints. -> > IMHO, this is more reliable than kernel logs and you don't lose the -> > alignment with the driver traces as long as you have driver data in -> > tracepoints as well. -> -> I have other patches that remember the last 100 or so firmware log -> messages from -> the kernel and provide that in a binary dump image when firmware -> crashes. -> -> This is indeed very useful. -> -> But, when debugging non-crash occasions, it is still useful to see -> what -> the firmware is doing. -> - -For that, I have come up with the "triggers". Triggers are conditions -that can be detected by the driver and enabled by the user. So -basically, we can say: "Please dump the logs when you are deauth'ed by -the AP". Or when you get delBA, or when the stats that come up from the -firmware say such and such etc... There are hooks that I added in -mac80211 to let the driver know about events that are handled there -(MLME and friends). Then, even if your logs are stored in a cyclic -buffer, you don't miss them and you catch them at the right spot. -One of the most useful trigger we have is when a Tx packet is dropped. -You can take a look at struct iwl_fw_dbg_trigger_tlv in iwlwifi if you -want. - -> For instance, maybe it is reporting lots of tx-hangs and/or low-level -> resets. This gives you a clue as to why a user might report 'my wifi -> sucks'. -> > Since I am both FW and driver team for my firmware variant, -> and my approach has been working for me, then I feel it is certainly -> better than -> the current state. And just maybe the official upstream FW team -> could start -> using something similar as well. Currently, I don't see how they can -> ever make -> much progress on firmware crashes reported in stock kernels. -> > Thanks, -> Ben -_______________________________________________ -ath10k mailing list -ath10k@lists.infradead.org -http://lists.infradead.org/mailman/listinfo/ath10k +T24gVGh1LCAyMDE2LTA5LTE1IGF0IDEwOjU5IC0wNzAwLCBCZW4gR3JlZWFyIHdyb3RlOg0KPiBP +biAwOS8xNS8yMDE2IDEwOjM0IEFNLCBHcnVtYmFjaCwgRW1tYW51ZWwgd3JvdGU6DQo+ID4gT24g +VGh1LCAyMDE2LTA5LTE1IGF0IDA4OjE0IC0wNzAwLCBCZW4gR3JlZWFyIHdyb3RlOg0KPiA+ID4g +T24gMDkvMTUvMjAxNiAwNzowNiBBTSwgVmFsbywgS2FsbGUgd3JvdGU6DQo+ID4gPiA+IEJlbiBH +cmVlYXIgPGdyZWVhcmJAY2FuZGVsYXRlY2guY29tPiB3cml0ZXM6DQo+ID4gPiA+IA0KPiA+ID4g +PiA+IE9uIDA5LzE0LzIwMTYgMDc6MTggQU0sIFZhbG8sIEthbGxlIHdyb3RlOg0KPiA+ID4gPiA+ +ID4gZ3JlZWFyYkBjYW5kZWxhdGVjaC5jb20gd3JpdGVzOg0KPiA+ID4gPiA+ID4gDQo+ID4gPiA+ +ID4gPiA+IEZyb206IEJlbiBHcmVlYXIgPGdyZWVhcmJAY2FuZGVsYXRlY2guY29tPg0KPiA+ID4g +PiA+ID4gPiANCj4gPiA+ID4gPiA+ID4gVGhpcyBhbGxvd3MgdXNlci1zcGFjZSB0b29scyB0byBk +ZWNvZGUgZGVidWctbG9nDQo+ID4gPiA+ID4gPiA+IG1lc3NhZ2VzIGJ5IHBhcnNpbmcgZG1lc2cg +b3IgL3Zhci9sb2cvbWVzc2FnZXMuDQo+ID4gPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4gPiBTaWdu +ZWQtb2ZmLWJ5OiBCZW4gR3JlZWFyIDxncmVlYXJiQGNhbmRlbGF0ZWNoLmNvbT4NCj4gPiA+ID4g +PiA+IA0KPiA+ID4gPiA+ID4gRG9uJ3QgdHJhY2luZyBwb2ludHMgYWxyZWFkeSBwcm92aWRlIHRo +ZSBzYW1lIGluZm9ybWF0aW9uPw0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IFRyYWNpbmcgdG9vbHMg +YXJlIGRpZmZpY3VsdCB0byBzZXQgdXAgYW5kIG1heSBub3QgYmUNCj4gPiA+ID4gPiBhdmFpbGFi +bGUgb24NCj4gPiA+ID4gPiByYW5kb20gZW1iZWRkZWQgZGV2aWNlcy4gIEFuZCBpZiB3ZSBhcmUg +ZGVhbGluZyB3aXRoIGJ1Zw0KPiA+ID4gPiA+IHJlcG9ydHMNCj4gPiA+ID4gPiBmcm9tDQo+ID4g +PiA+ID4gdGhlIGZpZWxkLCBtb3N0IHVzZXJzIHdpbGwgbm90IGJlIGFibGUgdG8gc2V0IGl0IHVw +DQo+ID4gPiA+ID4gcmVnYXJkbGVzcy4NCj4gPiA+ID4gPiANCj4gPiA+ID4gPiBUaGVyZSBhcmUg +c2ltaWxhciB3YXlzIHRvIHByaW50IG91dCBoZXgsIGJ1dCB0aGUgbG9naWMgYmVsb3cNCj4gPiA+ +ID4gPiBjcmVhdGVzDQo+ID4gPiA+ID4gc3BlY2lmaWMgYW5kIHBhcnNlYWJsZSBsb2dzIGluIHRo +ZSAnZG1lc2cnIG91dHB1dCBhbmQNCj4gPiA+ID4gPiBzaW1pbGFyLg0KPiA+ID4gPiA+IA0KPiA+ +ID4gPiA+IEkgaGF2ZSB3cml0dGVuIGEgdG9vbCB0aGF0IGNhbiBkZWNvZGUgdGhlc2UgbWVzc2Fn +ZXMgaW50bw0KPiA+ID4gPiA+IHVzZWZ1bA0KPiA+ID4gPiA+IGh1bWFuLXJlYWRhYmxlDQo+ID4g +PiA+ID4gdGV4dCBzbyB0aGF0IEkgY2FuIGRlYnVnIGZpcm13YXJlIGlzc3VlcyBib3RoIGxvY2Fs +bHkgYW5kDQo+ID4gPiA+ID4gZnJvbQ0KPiA+ID4gPiA+IGZpZWxkIHJlcG9ydHMuDQo+ID4gPiA+ +ID4gDQo+ID4gPiA+ID4gU3RvY2sgZmlybXdhcmUgZ2VuZXJhdGVzIHNpbWlsYXIgbG9ncyBhbmQg +UUNBIGNvdWxkIHdyaXRlDQo+ID4gPiA+ID4gdGhlaXINCj4gPiA+ID4gPiBvd24gZGVjb2RlIGxv +Z2ljDQo+ID4gPiA+ID4gZm9yIHRoZWlyIGZpcm13YXJlIHZlcnNpb25zLg0KPiA+ID4gPiANCj4g +PiA+ID4gUmVpbnZlbnRpbmcgdGhlIHdoZWVsIGJ5IHVzaW5nIHByaW50ayBhcyB0aGUgZGVsaXZl +cnkgbWVjaGFuaXNtDQo+ID4gPiA+IGRvZXNuJ3QNCj4gPiA+ID4gc291bmQgbGlrZSBhIGdvb2Qg +aWRlYS4gSUlSQyBFbW1hbnVlbCB0YWxrZWQgYWJvdXQgc29tZSBraW5kIG9mDQo+ID4gPiA+IGZp +cm13YXJlDQo+ID4gPiA+IGRlYnVnZ2luZyBmcmFtZXdvcmssIGhlIG1pZ2h0IGhhdmUgc29tZSBp +ZGVhcy4NCj4gPiA+IA0KPiA+ID4gV2FpdGluZyBmb3IgbWFnaWNhbCBmcmFtZXdvcmtzIHRvIGZp +eCBwcm9ibGVtcyBpcyBldmVuIHdvcnNlLg0KPiA+ID4gDQo+ID4gSXQgaGFzIGJlZW4geWVhcnMg +c2luY2UgYXRoMTBrIGhhcyBiZWVuIGluIHRoZSBrZXJuZWwuICBUaGVyZSBpcw0KPiA+ID4gYmFz +aWNhbGx5DQo+ID4gPiBzdGlsbCBubyB3YXkgdG8gZGVidWcgd2hhdCB0aGUgZmlybXdhcmUgaXMg +ZG9pbmcuDQo+ID4gPiANCj4gPiANCj4gPiBJIGtub3cgdGhlIGZlZWxpbmcgOikgSSB3YXMgaW4g +dGhlIHNhbWUgc2l0dWF0aW9uIGJlZm9yZSBJIGFkZGVkDQo+ID4gc3R1ZmYNCj4gPiBmb3IgaXds +d2lmaS4NCj4gPiANCj4gPiA+IE15IHBhdGNoIGdpdmVzIHlvdSBzb21ldGhpbmcgdGhhdCBjYW4g +d29yayByaWdodCBub3csIHdpdGggdGhlDQo+ID4gPiBzdGFuZGFyZCAnZG1lc2cnDQo+ID4gPiBm +cmFtZXdvcmsgZm91bmQgaW4gdmlydHVhbGx5IGFsbCBrZXJuZWxzIG5ldyBhbmQgb2xkLCBhbmQg +aXQgaGFzDQo+ID4gPiBiZWVuDQo+ID4gPiBwcm92ZW4NCj4gPiA+IHRvIGJlIHVzZWZ1bCBpbiB0 +aGUgZmllbGQuICBUaGUgbWVzc2FnZXMgYXJlIGFsc28gbmljZWx5DQo+ID4gPiBpbnRlcmxlYXZl +ZA0KPiA+ID4gd2l0aCB0aGUNCj4gPiA+IHJlc3Qgb2YgdGhlIG1hYzgwMjExIHN0YWNrIG1lc3Nh +Z2VzIGFuZCBhbnkgb3RoZXIgZHJpdmVyDQo+ID4gPiBtZXNzYWdlcywgc28NCj4gPiA+IHlvdSBo +YXZlDQo+ID4gPiBjb250ZXh0Lg0KPiA+ID4gDQo+ID4gPiBJZiBzb21lb25lIHdhbnRzIHRvIGFk +ZCBzdXBwb3J0IGZvciBhIGZyYW1ld29yayBsYXRlciwgdGhlbiBieQ0KPiA+ID4gYWxsDQo+ID4g +PiBtZWFucywgcG9zdA0KPiA+ID4gdGhlIHBhdGNoZXMgd2hlbiBpdCBpcyByZWFkeS4NCj4gPiAN +Cj4gPiBGcm9tIG15IGV4cGVyaWVuY2UsIGEgc3Ryb25nIGFuZCBlYXN5LXRvLXVzZSBmaXJtd2Fy +ZSBkZWJ1Zw0KPiA+IGluZnJhc3RydWN0dXJlIGlzIGltcG9ydGFudCBiZWNhdXNlIHR5cGljYWxs +eSwgdGhlIGZpcm13YXJlIGlzDQo+ID4gd3JpdHRlbg0KPiA+IGJ5IG90aGVyIHBlb3BsZSB3aG8g +aGF2ZSBkaWZmZXJlbnQgcHJpb3JpdGllcyAoYW5kIGFyZSBub3QgYWx3YXlzDQo+ID4gTGludXgN +Cj4gPiB3aXphcmRzKSBldGMuLi4gQmVpbmcgYWJsZSB0byBnaXZlIHRoZW0gZ29vZCBkYXRhIGlz +IHRoZSBvbmx5IHdheQ0KPiA+IHRvDQo+ID4gaGF2ZSB0aGVtIGZpeCB0aGVpciBidWdzIDopIEZv +ciB1cywgaXQgd2FzIHJlYWxseSBhIGdhbWUgY2hhbmdlci4NCj4gPiBXaGVuDQo+ID4geW91IHdv +cmsgZm9yIGEgYmlnIGNvcnBvcmF0ZSwgaGF2aW5nIDIgZ3JvdXBzIHdvcmsgYmV0dGVyIHRvZ2V0 +aGVyDQo+ID4gYWx3YXlzIGhhcyBhIGJpZyBpbXBhY3QuIFRoYXQncyBmb3IgdGhlIHBoaWxvc29w +aGljYWwgcGFydCA6KQ0KPiA+IA0KPiA+IEZXSVc6IHdoYXQgSSBkaWQgaGFzIG5vdGhpbmcgdG8g +ZG8gd2l0aCBGVyAnbGl2ZSB0cmFjaW5nJywgYnV0IHdpdGgNCj4gPiBmaXJtd2FyZSBkdW1wcy4g +T25lIHBhcnQgb2Ygb3VyIGZpcm13YXJlIGR1bXBzIGluY2x1ZGUgdHJhY2luZy4gV2UNCj4gPiBh +bHNvDQo+ID4gaGF2ZSAiZmlybXdhcmUgcHJpbnRzIiwgYnV0IHdlIGRvbid0IHByaW50IHRoZW0g +aW4gdGhlIGtlcm5lbCBsb2cNCj4gPiBhbmQNCj4gPiB0aGV5IGFyZSBub3QgcGFydCBvZiB0aGUg +ZmlybXdhcmUgZHVtcCB0aGluZy4gV2UgcmF0aGVyIHJlY29yZCB0aGVtDQo+ID4gaW4NCj4gPiB0 +cmFjZXBvaW50cyBqdXN0IGxpa2UgcmVhbGx5ICphbnl0aGluZyogdGhhdCBjb21lcyBmcm9tIHRo +ZQ0KPiA+IGZpcm13YXJlLg0KPiA+IEJhc2ljYWxseSwgd2UgaGF2ZSAyIGxheWVycywgdGhlIHRy +YW5zcG9ydCBsYXllciAoUENJZSkgYW5kIHRoZQ0KPiA+IG9wZXJhdGlvbl9tb2RlIGxheWVyLiBU +aGUgZmlyc3QganVzdCBicmluZ3MgdGhlIGRhdGEgZnJvbSB0aGUNCj4gPiBmaXJtd2FyZQ0KPiA+ +IGFuZCBpbiB0aGF0IGxheWVyIHdlICpibGluZGx5KiByZWNvcmQgZXZlcnl0aGluZyBpbiB0cmFj +ZXBvaW50cy4gSW4NCj4gPiB0aGUNCj4gPiBvcGVyYXRpb25fbW9kZSBsYXllciwgd2UgbG9vayBh +dCB0aGUgZGF0YSBpdHNlbGYuIEluIGNhc2Ugb2YgZGVidWcNCj4gPiBwcmludHMgZnJvbSB0aGUg +ZmlybXdhcmUsIHdlIHNpbXBseSBkaXNjYXJkIHRoZW0sIGJlY2F1c2Ugd2UgZG9uJ3QNCj4gPiBy +ZWFsbHkgY2FyZSBvZiB0aGUgbWVhbmluZy4gQWxsIHdlIHdhbnQgaXMgdG8gaGF2ZSB0aGVtIGdv +IHRocm91Z2gNCj4gPiB0aGUNCj4gPiBQQ0llIGxheWVyIHNvIHRoYXQgdGhleSBhcmUgcmVjb3Jk +ZWQgaW4gdGhlIHRyYWNlcG9pbnRzLg0KPiA+IFdoZW4gd2UgZmluaXNoIHJlY29yZGluZyB0aGUg +c2VxdWVuY2Ugd2Ugd2FudGVkIHdpdGggdHJhY2luZyAodHJhY2UNCj4gPiAtY21kKSwgd2UgcGFy +c2UgdGhlIG91dHB1dCBhbmQgdGhlbiwgd2UgcGFyc2UgdGhlIGZpcm13YXJlIHByaW50cy4NCj4g +PiBJTUhPLCB0aGlzIGlzIG1vcmUgcmVsaWFibGUgdGhhbiBrZXJuZWwgbG9ncyBhbmQgeW91IGRv +bid0IGxvc2UgdGhlDQo+ID4gYWxpZ25tZW50IHdpdGggdGhlIGRyaXZlciB0cmFjZXMgYXMgbG9u +ZyBhcyB5b3UgaGF2ZSBkcml2ZXIgZGF0YSBpbg0KPiA+IHRyYWNlcG9pbnRzIGFzIHdlbGwuDQo+ +IA0KPiBJIGhhdmUgb3RoZXIgcGF0Y2hlcyB0aGF0IHJlbWVtYmVyIHRoZSBsYXN0IDEwMCBvciBz +byBmaXJtd2FyZSBsb2cNCj4gbWVzc2FnZXMgZnJvbQ0KPiB0aGUga2VybmVsIGFuZCBwcm92aWRl +IHRoYXQgaW4gYSBiaW5hcnkgZHVtcCBpbWFnZSB3aGVuIGZpcm13YXJlDQo+IGNyYXNoZXMuDQo+ +IA0KPiBUaGlzIGlzIGluZGVlZCB2ZXJ5IHVzZWZ1bC4NCj4gDQo+IEJ1dCwgd2hlbiBkZWJ1Z2dp +bmcgbm9uLWNyYXNoIG9jY2FzaW9ucywgaXQgaXMgc3RpbGwgdXNlZnVsIHRvIHNlZQ0KPiB3aGF0 +DQo+IHRoZSBmaXJtd2FyZSBpcyBkb2luZy4NCj4gDQoNCkZvciB0aGF0LCBJIGhhdmUgY29tZSB1 +cCB3aXRoIHRoZSAidHJpZ2dlcnMiLiBUcmlnZ2VycyBhcmUgY29uZGl0aW9ucw0KdGhhdCBjYW4g +YmUgZGV0ZWN0ZWQgYnkgdGhlIGRyaXZlciBhbmQgZW5hYmxlZCBieSB0aGUgdXNlci4gU28NCmJh +c2ljYWxseSwgd2UgY2FuIHNheTogIlBsZWFzZSBkdW1wIHRoZSBsb2dzIHdoZW4geW91IGFyZSBk +ZWF1dGgnZWQgYnkNCnRoZSBBUCIuIE9yIHdoZW4geW91IGdldCBkZWxCQSwgb3Igd2hlbiB0aGUg +c3RhdHMgdGhhdCBjb21lIHVwIGZyb20gdGhlDQpmaXJtd2FyZSBzYXkgc3VjaCBhbmQgc3VjaCBl +dGMuLi4gVGhlcmUgYXJlIGhvb2tzIHRoYXQgSSBhZGRlZCBpbg0KbWFjODAyMTEgdG8gbGV0IHRo +ZSBkcml2ZXIga25vdyBhYm91dCBldmVudHMgdGhhdCBhcmUgaGFuZGxlZCB0aGVyZQ0KKE1MTUUg +YW5kIGZyaWVuZHMpLiBUaGVuLCBldmVuIGlmIHlvdXIgbG9ncyBhcmUgc3RvcmVkIGluIGEgY3lj +bGljDQpidWZmZXIsIHlvdSBkb24ndCBtaXNzIHRoZW0gYW5kIHlvdSBjYXRjaCB0aGVtIGF0IHRo +ZSByaWdodCBzcG90Lg0KT25lIG9mIHRoZSBtb3N0IHVzZWZ1bCB0cmlnZ2VyIHdlIGhhdmUgaXMg +d2hlbiBhIFR4IHBhY2tldCBpcyBkcm9wcGVkLg0KWW91IGNhbiB0YWtlIGEgbG9vayBhdCBzdHJ1 +Y3QgaXdsX2Z3X2RiZ190cmlnZ2VyX3RsdiBpbiBpd2x3aWZpIGlmIHlvdQ0Kd2FudC4NCg0KPiBG +b3IgaW5zdGFuY2UsIG1heWJlIGl0IGlzIHJlcG9ydGluZyBsb3RzIG9mIHR4LWhhbmdzIGFuZC9v +ciBsb3ctbGV2ZWwNCj4gcmVzZXRzLiAgVGhpcyBnaXZlcyB5b3UgYSBjbHVlIGFzIHRvIHdoeSBh +IHVzZXIgbWlnaHQgcmVwb3J0ICdteSB3aWZpDQo+IHN1Y2tzJy4NCj4gPiBTaW5jZSBJIGFtIGJv +dGggRlcgYW5kIGRyaXZlciB0ZWFtIGZvciBteSBmaXJtd2FyZSB2YXJpYW50LA0KPiBhbmQgbXkg +YXBwcm9hY2ggaGFzIGJlZW4gd29ya2luZyBmb3IgbWUsIHRoZW4gSSBmZWVsIGl0IGlzIGNlcnRh +aW5seQ0KPiBiZXR0ZXIgdGhhbg0KPiB0aGUgY3VycmVudCBzdGF0ZS4gIEFuZCBqdXN0IG1heWJl +IHRoZSBvZmZpY2lhbCB1cHN0cmVhbSBGVyB0ZWFtDQo+IGNvdWxkIHN0YXJ0DQo+IHVzaW5nIHNv +bWV0aGluZyBzaW1pbGFyIGFzIHdlbGwuICBDdXJyZW50bHksIEkgZG9uJ3Qgc2VlIGhvdyB0aGV5 +IGNhbg0KPiBldmVyIG1ha2UNCj4gbXVjaCBwcm9ncmVzcyBvbiBmaXJtd2FyZSBjcmFzaGVzIHJl +cG9ydGVkIGluIHN0b2NrIGtlcm5lbHMuDQo+ID4gVGhhbmtzLA0KPiBCZW4NCg== diff --git a/a/content_digest b/N1/content_digest index c73a0c8..a201ed7 100644 --- a/a/content_digest +++ b/N1/content_digest @@ -15,161 +15,113 @@ " ath10k@lists.infradead.org <ath10k@lists.infradead.org>\0" "\00:1\0" "b\0" - "On Thu, 2016-09-15 at 10:59 -0700, Ben Greear wrote:\n" - "> On 09/15/2016 10:34 AM, Grumbach, Emmanuel wrote:\n" - "> > On Thu, 2016-09-15 at 08:14 -0700, Ben Greear wrote:\n" - "> > > On 09/15/2016 07:06 AM, Valo, Kalle wrote:\n" - "> > > > Ben Greear <greearb@candelatech.com> writes:\n" - "> > > > \n" - "> > > > > On 09/14/2016 07:18 AM, Valo, Kalle wrote:\n" - "> > > > > > greearb@candelatech.com writes:\n" - "> > > > > > \n" - "> > > > > > > From: Ben Greear <greearb@candelatech.com>\n" - "> > > > > > > \n" - "> > > > > > > This allows user-space tools to decode debug-log\n" - "> > > > > > > messages by parsing dmesg or /var/log/messages.\n" - "> > > > > > > \n" - "> > > > > > > Signed-off-by: Ben Greear <greearb@candelatech.com>\n" - "> > > > > > \n" - "> > > > > > Don't tracing points already provide the same information?\n" - "> > > > > \n" - "> > > > > Tracing tools are difficult to set up and may not be\n" - "> > > > > available on\n" - "> > > > > random embedded devices. And if we are dealing with bug\n" - "> > > > > reports\n" - "> > > > > from\n" - "> > > > > the field, most users will not be able to set it up\n" - "> > > > > regardless.\n" - "> > > > > \n" - "> > > > > There are similar ways to print out hex, but the logic below\n" - "> > > > > creates\n" - "> > > > > specific and parseable logs in the 'dmesg' output and\n" - "> > > > > similar.\n" - "> > > > > \n" - "> > > > > I have written a tool that can decode these messages into\n" - "> > > > > useful\n" - "> > > > > human-readable\n" - "> > > > > text so that I can debug firmware issues both locally and\n" - "> > > > > from\n" - "> > > > > field reports.\n" - "> > > > > \n" - "> > > > > Stock firmware generates similar logs and QCA could write\n" - "> > > > > their\n" - "> > > > > own decode logic\n" - "> > > > > for their firmware versions.\n" - "> > > > \n" - "> > > > Reinventing the wheel by using printk as the delivery mechanism\n" - "> > > > doesn't\n" - "> > > > sound like a good idea. IIRC Emmanuel talked about some kind of\n" - "> > > > firmware\n" - "> > > > debugging framework, he might have some ideas.\n" - "> > > \n" - "> > > Waiting for magical frameworks to fix problems is even worse.\n" - "> > > \n" - "> > It has been years since ath10k has been in the kernel. There is\n" - "> > > basically\n" - "> > > still no way to debug what the firmware is doing.\n" - "> > > \n" - "> > \n" - "> > I know the feeling :) I was in the same situation before I added\n" - "> > stuff\n" - "> > for iwlwifi.\n" - "> > \n" - "> > > My patch gives you something that can work right now, with the\n" - "> > > standard 'dmesg'\n" - "> > > framework found in virtually all kernels new and old, and it has\n" - "> > > been\n" - "> > > proven\n" - "> > > to be useful in the field. The messages are also nicely\n" - "> > > interleaved\n" - "> > > with the\n" - "> > > rest of the mac80211 stack messages and any other driver\n" - "> > > messages, so\n" - "> > > you have\n" - "> > > context.\n" - "> > > \n" - "> > > If someone wants to add support for a framework later, then by\n" - "> > > all\n" - "> > > means, post\n" - "> > > the patches when it is ready.\n" - "> > \n" - "> > From my experience, a strong and easy-to-use firmware debug\n" - "> > infrastructure is important because typically, the firmware is\n" - "> > written\n" - "> > by other people who have different priorities (and are not always\n" - "> > Linux\n" - "> > wizards) etc... Being able to give them good data is the only way\n" - "> > to\n" - "> > have them fix their bugs :) For us, it was really a game changer.\n" - "> > When\n" - "> > you work for a big corporate, having 2 groups work better together\n" - "> > always has a big impact. That's for the philosophical part :)\n" - "> > \n" - "> > FWIW: what I did has nothing to do with FW 'live tracing', but with\n" - "> > firmware dumps. One part of our firmware dumps include tracing. We\n" - "> > also\n" - "> > have \"firmware prints\", but we don't print them in the kernel log\n" - "> > and\n" - "> > they are not part of the firmware dump thing. We rather record them\n" - "> > in\n" - "> > tracepoints just like really *anything* that comes from the\n" - "> > firmware.\n" - "> > Basically, we have 2 layers, the transport layer (PCIe) and the\n" - "> > operation_mode layer. The first just brings the data from the\n" - "> > firmware\n" - "> > and in that layer we *blindly* record everything in tracepoints. In\n" - "> > the\n" - "> > operation_mode layer, we look at the data itself. In case of debug\n" - "> > prints from the firmware, we simply discard them, because we don't\n" - "> > really care of the meaning. All we want is to have them go through\n" - "> > the\n" - "> > PCIe layer so that they are recorded in the tracepoints.\n" - "> > When we finish recording the sequence we wanted with tracing (trace\n" - "> > -cmd), we parse the output and then, we parse the firmware prints.\n" - "> > IMHO, this is more reliable than kernel logs and you don't lose the\n" - "> > alignment with the driver traces as long as you have driver data in\n" - "> > tracepoints as well.\n" - "> \n" - "> I have other patches that remember the last 100 or so firmware log\n" - "> messages from\n" - "> the kernel and provide that in a binary dump image when firmware\n" - "> crashes.\n" - "> \n" - "> This is indeed very useful.\n" - "> \n" - "> But, when debugging non-crash occasions, it is still useful to see\n" - "> what\n" - "> the firmware is doing.\n" - "> \n" - "\n" - "For that, I have come up with the \"triggers\". Triggers are conditions\n" - "that can be detected by the driver and enabled by the user. So\n" - "basically, we can say: \"Please dump the logs when you are deauth'ed by\n" - "the AP\". Or when you get delBA, or when the stats that come up from the\n" - "firmware say such and such etc... There are hooks that I added in\n" - "mac80211 to let the driver know about events that are handled there\n" - "(MLME and friends). Then, even if your logs are stored in a cyclic\n" - "buffer, you don't miss them and you catch them at the right spot.\n" - "One of the most useful trigger we have is when a Tx packet is dropped.\n" - "You can take a look at struct iwl_fw_dbg_trigger_tlv in iwlwifi if you\n" - "want.\n" - "\n" - "> For instance, maybe it is reporting lots of tx-hangs and/or low-level\n" - "> resets. This gives you a clue as to why a user might report 'my wifi\n" - "> sucks'.\n" - "> > Since I am both FW and driver team for my firmware variant,\n" - "> and my approach has been working for me, then I feel it is certainly\n" - "> better than\n" - "> the current state. And just maybe the official upstream FW team\n" - "> could start\n" - "> using something similar as well. Currently, I don't see how they can\n" - "> ever make\n" - "> much progress on firmware crashes reported in stock kernels.\n" - "> > Thanks,\n" - "> Ben\n" - "_______________________________________________\n" - "ath10k mailing list\n" - "ath10k@lists.infradead.org\n" - http://lists.infradead.org/mailman/listinfo/ath10k + "T24gVGh1LCAyMDE2LTA5LTE1IGF0IDEwOjU5IC0wNzAwLCBCZW4gR3JlZWFyIHdyb3RlOg0KPiBP\n" + "biAwOS8xNS8yMDE2IDEwOjM0IEFNLCBHcnVtYmFjaCwgRW1tYW51ZWwgd3JvdGU6DQo+ID4gT24g\n" + "VGh1LCAyMDE2LTA5LTE1IGF0IDA4OjE0IC0wNzAwLCBCZW4gR3JlZWFyIHdyb3RlOg0KPiA+ID4g\n" + "T24gMDkvMTUvMjAxNiAwNzowNiBBTSwgVmFsbywgS2FsbGUgd3JvdGU6DQo+ID4gPiA+IEJlbiBH\n" + "cmVlYXIgPGdyZWVhcmJAY2FuZGVsYXRlY2guY29tPiB3cml0ZXM6DQo+ID4gPiA+IA0KPiA+ID4g\n" + "PiA+IE9uIDA5LzE0LzIwMTYgMDc6MTggQU0sIFZhbG8sIEthbGxlIHdyb3RlOg0KPiA+ID4gPiA+\n" + "ID4gZ3JlZWFyYkBjYW5kZWxhdGVjaC5jb20gd3JpdGVzOg0KPiA+ID4gPiA+ID4gDQo+ID4gPiA+\n" + "ID4gPiA+IEZyb206IEJlbiBHcmVlYXIgPGdyZWVhcmJAY2FuZGVsYXRlY2guY29tPg0KPiA+ID4g\n" + "PiA+ID4gPiANCj4gPiA+ID4gPiA+ID4gVGhpcyBhbGxvd3MgdXNlci1zcGFjZSB0b29scyB0byBk\n" + "ZWNvZGUgZGVidWctbG9nDQo+ID4gPiA+ID4gPiA+IG1lc3NhZ2VzIGJ5IHBhcnNpbmcgZG1lc2cg\n" + "b3IgL3Zhci9sb2cvbWVzc2FnZXMuDQo+ID4gPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4gPiBTaWdu\n" + "ZWQtb2ZmLWJ5OiBCZW4gR3JlZWFyIDxncmVlYXJiQGNhbmRlbGF0ZWNoLmNvbT4NCj4gPiA+ID4g\n" + "PiA+IA0KPiA+ID4gPiA+ID4gRG9uJ3QgdHJhY2luZyBwb2ludHMgYWxyZWFkeSBwcm92aWRlIHRo\n" + "ZSBzYW1lIGluZm9ybWF0aW9uPw0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IFRyYWNpbmcgdG9vbHMg\n" + "YXJlIGRpZmZpY3VsdCB0byBzZXQgdXAgYW5kIG1heSBub3QgYmUNCj4gPiA+ID4gPiBhdmFpbGFi\n" + "bGUgb24NCj4gPiA+ID4gPiByYW5kb20gZW1iZWRkZWQgZGV2aWNlcy4gIEFuZCBpZiB3ZSBhcmUg\n" + "ZGVhbGluZyB3aXRoIGJ1Zw0KPiA+ID4gPiA+IHJlcG9ydHMNCj4gPiA+ID4gPiBmcm9tDQo+ID4g\n" + "PiA+ID4gdGhlIGZpZWxkLCBtb3N0IHVzZXJzIHdpbGwgbm90IGJlIGFibGUgdG8gc2V0IGl0IHVw\n" + "DQo+ID4gPiA+ID4gcmVnYXJkbGVzcy4NCj4gPiA+ID4gPiANCj4gPiA+ID4gPiBUaGVyZSBhcmUg\n" + "c2ltaWxhciB3YXlzIHRvIHByaW50IG91dCBoZXgsIGJ1dCB0aGUgbG9naWMgYmVsb3cNCj4gPiA+\n" + "ID4gPiBjcmVhdGVzDQo+ID4gPiA+ID4gc3BlY2lmaWMgYW5kIHBhcnNlYWJsZSBsb2dzIGluIHRo\n" + "ZSAnZG1lc2cnIG91dHB1dCBhbmQNCj4gPiA+ID4gPiBzaW1pbGFyLg0KPiA+ID4gPiA+IA0KPiA+\n" + "ID4gPiA+IEkgaGF2ZSB3cml0dGVuIGEgdG9vbCB0aGF0IGNhbiBkZWNvZGUgdGhlc2UgbWVzc2Fn\n" + "ZXMgaW50bw0KPiA+ID4gPiA+IHVzZWZ1bA0KPiA+ID4gPiA+IGh1bWFuLXJlYWRhYmxlDQo+ID4g\n" + "PiA+ID4gdGV4dCBzbyB0aGF0IEkgY2FuIGRlYnVnIGZpcm13YXJlIGlzc3VlcyBib3RoIGxvY2Fs\n" + "bHkgYW5kDQo+ID4gPiA+ID4gZnJvbQ0KPiA+ID4gPiA+IGZpZWxkIHJlcG9ydHMuDQo+ID4gPiA+\n" + "ID4gDQo+ID4gPiA+ID4gU3RvY2sgZmlybXdhcmUgZ2VuZXJhdGVzIHNpbWlsYXIgbG9ncyBhbmQg\n" + "UUNBIGNvdWxkIHdyaXRlDQo+ID4gPiA+ID4gdGhlaXINCj4gPiA+ID4gPiBvd24gZGVjb2RlIGxv\n" + "Z2ljDQo+ID4gPiA+ID4gZm9yIHRoZWlyIGZpcm13YXJlIHZlcnNpb25zLg0KPiA+ID4gPiANCj4g\n" + "PiA+ID4gUmVpbnZlbnRpbmcgdGhlIHdoZWVsIGJ5IHVzaW5nIHByaW50ayBhcyB0aGUgZGVsaXZl\n" + "cnkgbWVjaGFuaXNtDQo+ID4gPiA+IGRvZXNuJ3QNCj4gPiA+ID4gc291bmQgbGlrZSBhIGdvb2Qg\n" + "aWRlYS4gSUlSQyBFbW1hbnVlbCB0YWxrZWQgYWJvdXQgc29tZSBraW5kIG9mDQo+ID4gPiA+IGZp\n" + "cm13YXJlDQo+ID4gPiA+IGRlYnVnZ2luZyBmcmFtZXdvcmssIGhlIG1pZ2h0IGhhdmUgc29tZSBp\n" + "ZGVhcy4NCj4gPiA+IA0KPiA+ID4gV2FpdGluZyBmb3IgbWFnaWNhbCBmcmFtZXdvcmtzIHRvIGZp\n" + "eCBwcm9ibGVtcyBpcyBldmVuIHdvcnNlLg0KPiA+ID4gDQo+ID4gSXQgaGFzIGJlZW4geWVhcnMg\n" + "c2luY2UgYXRoMTBrIGhhcyBiZWVuIGluIHRoZSBrZXJuZWwuICBUaGVyZSBpcw0KPiA+ID4gYmFz\n" + "aWNhbGx5DQo+ID4gPiBzdGlsbCBubyB3YXkgdG8gZGVidWcgd2hhdCB0aGUgZmlybXdhcmUgaXMg\n" + "ZG9pbmcuDQo+ID4gPiANCj4gPiANCj4gPiBJIGtub3cgdGhlIGZlZWxpbmcgOikgSSB3YXMgaW4g\n" + "dGhlIHNhbWUgc2l0dWF0aW9uIGJlZm9yZSBJIGFkZGVkDQo+ID4gc3R1ZmYNCj4gPiBmb3IgaXds\n" + "d2lmaS4NCj4gPiANCj4gPiA+IE15IHBhdGNoIGdpdmVzIHlvdSBzb21ldGhpbmcgdGhhdCBjYW4g\n" + "d29yayByaWdodCBub3csIHdpdGggdGhlDQo+ID4gPiBzdGFuZGFyZCAnZG1lc2cnDQo+ID4gPiBm\n" + "cmFtZXdvcmsgZm91bmQgaW4gdmlydHVhbGx5IGFsbCBrZXJuZWxzIG5ldyBhbmQgb2xkLCBhbmQg\n" + "aXQgaGFzDQo+ID4gPiBiZWVuDQo+ID4gPiBwcm92ZW4NCj4gPiA+IHRvIGJlIHVzZWZ1bCBpbiB0\n" + "aGUgZmllbGQuICBUaGUgbWVzc2FnZXMgYXJlIGFsc28gbmljZWx5DQo+ID4gPiBpbnRlcmxlYXZl\n" + "ZA0KPiA+ID4gd2l0aCB0aGUNCj4gPiA+IHJlc3Qgb2YgdGhlIG1hYzgwMjExIHN0YWNrIG1lc3Nh\n" + "Z2VzIGFuZCBhbnkgb3RoZXIgZHJpdmVyDQo+ID4gPiBtZXNzYWdlcywgc28NCj4gPiA+IHlvdSBo\n" + "YXZlDQo+ID4gPiBjb250ZXh0Lg0KPiA+ID4gDQo+ID4gPiBJZiBzb21lb25lIHdhbnRzIHRvIGFk\n" + "ZCBzdXBwb3J0IGZvciBhIGZyYW1ld29yayBsYXRlciwgdGhlbiBieQ0KPiA+ID4gYWxsDQo+ID4g\n" + "PiBtZWFucywgcG9zdA0KPiA+ID4gdGhlIHBhdGNoZXMgd2hlbiBpdCBpcyByZWFkeS4NCj4gPiAN\n" + "Cj4gPiBGcm9tIG15IGV4cGVyaWVuY2UsIGEgc3Ryb25nIGFuZCBlYXN5LXRvLXVzZSBmaXJtd2Fy\n" + "ZSBkZWJ1Zw0KPiA+IGluZnJhc3RydWN0dXJlIGlzIGltcG9ydGFudCBiZWNhdXNlIHR5cGljYWxs\n" + "eSwgdGhlIGZpcm13YXJlIGlzDQo+ID4gd3JpdHRlbg0KPiA+IGJ5IG90aGVyIHBlb3BsZSB3aG8g\n" + "aGF2ZSBkaWZmZXJlbnQgcHJpb3JpdGllcyAoYW5kIGFyZSBub3QgYWx3YXlzDQo+ID4gTGludXgN\n" + "Cj4gPiB3aXphcmRzKSBldGMuLi4gQmVpbmcgYWJsZSB0byBnaXZlIHRoZW0gZ29vZCBkYXRhIGlz\n" + "IHRoZSBvbmx5IHdheQ0KPiA+IHRvDQo+ID4gaGF2ZSB0aGVtIGZpeCB0aGVpciBidWdzIDopIEZv\n" + "ciB1cywgaXQgd2FzIHJlYWxseSBhIGdhbWUgY2hhbmdlci4NCj4gPiBXaGVuDQo+ID4geW91IHdv\n" + "cmsgZm9yIGEgYmlnIGNvcnBvcmF0ZSwgaGF2aW5nIDIgZ3JvdXBzIHdvcmsgYmV0dGVyIHRvZ2V0\n" + "aGVyDQo+ID4gYWx3YXlzIGhhcyBhIGJpZyBpbXBhY3QuIFRoYXQncyBmb3IgdGhlIHBoaWxvc29w\n" + "aGljYWwgcGFydCA6KQ0KPiA+IA0KPiA+IEZXSVc6IHdoYXQgSSBkaWQgaGFzIG5vdGhpbmcgdG8g\n" + "ZG8gd2l0aCBGVyAnbGl2ZSB0cmFjaW5nJywgYnV0IHdpdGgNCj4gPiBmaXJtd2FyZSBkdW1wcy4g\n" + "T25lIHBhcnQgb2Ygb3VyIGZpcm13YXJlIGR1bXBzIGluY2x1ZGUgdHJhY2luZy4gV2UNCj4gPiBh\n" + "bHNvDQo+ID4gaGF2ZSAiZmlybXdhcmUgcHJpbnRzIiwgYnV0IHdlIGRvbid0IHByaW50IHRoZW0g\n" + "aW4gdGhlIGtlcm5lbCBsb2cNCj4gPiBhbmQNCj4gPiB0aGV5IGFyZSBub3QgcGFydCBvZiB0aGUg\n" + "ZmlybXdhcmUgZHVtcCB0aGluZy4gV2UgcmF0aGVyIHJlY29yZCB0aGVtDQo+ID4gaW4NCj4gPiB0\n" + "cmFjZXBvaW50cyBqdXN0IGxpa2UgcmVhbGx5ICphbnl0aGluZyogdGhhdCBjb21lcyBmcm9tIHRo\n" + "ZQ0KPiA+IGZpcm13YXJlLg0KPiA+IEJhc2ljYWxseSwgd2UgaGF2ZSAyIGxheWVycywgdGhlIHRy\n" + "YW5zcG9ydCBsYXllciAoUENJZSkgYW5kIHRoZQ0KPiA+IG9wZXJhdGlvbl9tb2RlIGxheWVyLiBU\n" + "aGUgZmlyc3QganVzdCBicmluZ3MgdGhlIGRhdGEgZnJvbSB0aGUNCj4gPiBmaXJtd2FyZQ0KPiA+\n" + "IGFuZCBpbiB0aGF0IGxheWVyIHdlICpibGluZGx5KiByZWNvcmQgZXZlcnl0aGluZyBpbiB0cmFj\n" + "ZXBvaW50cy4gSW4NCj4gPiB0aGUNCj4gPiBvcGVyYXRpb25fbW9kZSBsYXllciwgd2UgbG9vayBh\n" + "dCB0aGUgZGF0YSBpdHNlbGYuIEluIGNhc2Ugb2YgZGVidWcNCj4gPiBwcmludHMgZnJvbSB0aGUg\n" + "ZmlybXdhcmUsIHdlIHNpbXBseSBkaXNjYXJkIHRoZW0sIGJlY2F1c2Ugd2UgZG9uJ3QNCj4gPiBy\n" + "ZWFsbHkgY2FyZSBvZiB0aGUgbWVhbmluZy4gQWxsIHdlIHdhbnQgaXMgdG8gaGF2ZSB0aGVtIGdv\n" + "IHRocm91Z2gNCj4gPiB0aGUNCj4gPiBQQ0llIGxheWVyIHNvIHRoYXQgdGhleSBhcmUgcmVjb3Jk\n" + "ZWQgaW4gdGhlIHRyYWNlcG9pbnRzLg0KPiA+IFdoZW4gd2UgZmluaXNoIHJlY29yZGluZyB0aGUg\n" + "c2VxdWVuY2Ugd2Ugd2FudGVkIHdpdGggdHJhY2luZyAodHJhY2UNCj4gPiAtY21kKSwgd2UgcGFy\n" + "c2UgdGhlIG91dHB1dCBhbmQgdGhlbiwgd2UgcGFyc2UgdGhlIGZpcm13YXJlIHByaW50cy4NCj4g\n" + "PiBJTUhPLCB0aGlzIGlzIG1vcmUgcmVsaWFibGUgdGhhbiBrZXJuZWwgbG9ncyBhbmQgeW91IGRv\n" + "bid0IGxvc2UgdGhlDQo+ID4gYWxpZ25tZW50IHdpdGggdGhlIGRyaXZlciB0cmFjZXMgYXMgbG9u\n" + "ZyBhcyB5b3UgaGF2ZSBkcml2ZXIgZGF0YSBpbg0KPiA+IHRyYWNlcG9pbnRzIGFzIHdlbGwuDQo+\n" + "IA0KPiBJIGhhdmUgb3RoZXIgcGF0Y2hlcyB0aGF0IHJlbWVtYmVyIHRoZSBsYXN0IDEwMCBvciBz\n" + "byBmaXJtd2FyZSBsb2cNCj4gbWVzc2FnZXMgZnJvbQ0KPiB0aGUga2VybmVsIGFuZCBwcm92aWRl\n" + "IHRoYXQgaW4gYSBiaW5hcnkgZHVtcCBpbWFnZSB3aGVuIGZpcm13YXJlDQo+IGNyYXNoZXMuDQo+\n" + "IA0KPiBUaGlzIGlzIGluZGVlZCB2ZXJ5IHVzZWZ1bC4NCj4gDQo+IEJ1dCwgd2hlbiBkZWJ1Z2dp\n" + "bmcgbm9uLWNyYXNoIG9jY2FzaW9ucywgaXQgaXMgc3RpbGwgdXNlZnVsIHRvIHNlZQ0KPiB3aGF0\n" + "DQo+IHRoZSBmaXJtd2FyZSBpcyBkb2luZy4NCj4gDQoNCkZvciB0aGF0LCBJIGhhdmUgY29tZSB1\n" + "cCB3aXRoIHRoZSAidHJpZ2dlcnMiLiBUcmlnZ2VycyBhcmUgY29uZGl0aW9ucw0KdGhhdCBjYW4g\n" + "YmUgZGV0ZWN0ZWQgYnkgdGhlIGRyaXZlciBhbmQgZW5hYmxlZCBieSB0aGUgdXNlci4gU28NCmJh\n" + "c2ljYWxseSwgd2UgY2FuIHNheTogIlBsZWFzZSBkdW1wIHRoZSBsb2dzIHdoZW4geW91IGFyZSBk\n" + "ZWF1dGgnZWQgYnkNCnRoZSBBUCIuIE9yIHdoZW4geW91IGdldCBkZWxCQSwgb3Igd2hlbiB0aGUg\n" + "c3RhdHMgdGhhdCBjb21lIHVwIGZyb20gdGhlDQpmaXJtd2FyZSBzYXkgc3VjaCBhbmQgc3VjaCBl\n" + "dGMuLi4gVGhlcmUgYXJlIGhvb2tzIHRoYXQgSSBhZGRlZCBpbg0KbWFjODAyMTEgdG8gbGV0IHRo\n" + "ZSBkcml2ZXIga25vdyBhYm91dCBldmVudHMgdGhhdCBhcmUgaGFuZGxlZCB0aGVyZQ0KKE1MTUUg\n" + "YW5kIGZyaWVuZHMpLiBUaGVuLCBldmVuIGlmIHlvdXIgbG9ncyBhcmUgc3RvcmVkIGluIGEgY3lj\n" + "bGljDQpidWZmZXIsIHlvdSBkb24ndCBtaXNzIHRoZW0gYW5kIHlvdSBjYXRjaCB0aGVtIGF0IHRo\n" + "ZSByaWdodCBzcG90Lg0KT25lIG9mIHRoZSBtb3N0IHVzZWZ1bCB0cmlnZ2VyIHdlIGhhdmUgaXMg\n" + "d2hlbiBhIFR4IHBhY2tldCBpcyBkcm9wcGVkLg0KWW91IGNhbiB0YWtlIGEgbG9vayBhdCBzdHJ1\n" + "Y3QgaXdsX2Z3X2RiZ190cmlnZ2VyX3RsdiBpbiBpd2x3aWZpIGlmIHlvdQ0Kd2FudC4NCg0KPiBG\n" + "b3IgaW5zdGFuY2UsIG1heWJlIGl0IGlzIHJlcG9ydGluZyBsb3RzIG9mIHR4LWhhbmdzIGFuZC9v\n" + "ciBsb3ctbGV2ZWwNCj4gcmVzZXRzLiAgVGhpcyBnaXZlcyB5b3UgYSBjbHVlIGFzIHRvIHdoeSBh\n" + "IHVzZXIgbWlnaHQgcmVwb3J0ICdteSB3aWZpDQo+IHN1Y2tzJy4NCj4gPiBTaW5jZSBJIGFtIGJv\n" + "dGggRlcgYW5kIGRyaXZlciB0ZWFtIGZvciBteSBmaXJtd2FyZSB2YXJpYW50LA0KPiBhbmQgbXkg\n" + "YXBwcm9hY2ggaGFzIGJlZW4gd29ya2luZyBmb3IgbWUsIHRoZW4gSSBmZWVsIGl0IGlzIGNlcnRh\n" + "aW5seQ0KPiBiZXR0ZXIgdGhhbg0KPiB0aGUgY3VycmVudCBzdGF0ZS4gIEFuZCBqdXN0IG1heWJl\n" + "IHRoZSBvZmZpY2lhbCB1cHN0cmVhbSBGVyB0ZWFtDQo+IGNvdWxkIHN0YXJ0DQo+IHVzaW5nIHNv\n" + "bWV0aGluZyBzaW1pbGFyIGFzIHdlbGwuICBDdXJyZW50bHksIEkgZG9uJ3Qgc2VlIGhvdyB0aGV5\n" + "IGNhbg0KPiBldmVyIG1ha2UNCj4gbXVjaCBwcm9ncmVzcyBvbiBmaXJtd2FyZSBjcmFzaGVzIHJl\n" + cG9ydGVkIGluIHN0b2NrIGtlcm5lbHMuDQo+ID4gVGhhbmtzLA0KPiBCZW4NCg== -128aa7df10115145498d6fd63974573db03222e5d8f8d969f51f42612afe0abe +80391695b3a185c04dc211426ae1b19457b3c8f309b75198a5ec26dbefce900c
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.