diff for duplicates of <1504796087.3561.7.camel@primarydata.com> diff --git a/a/1.txt b/N1/1.txt index 2cbe61b..4f60fdd 100644 --- a/a/1.txt +++ b/N1/1.txt @@ -1,197 +1,294 @@ -T24gVGh1LCAyMDE3LTA5LTA3IGF0IDA3OjM1IC0wNDAwLCBKZWZmIExheXRvbiB3cm90ZToNCj4g -T24gVGh1LCAyMDE3LTA5LTA3IGF0IDEzOjM3ICsxMDAwLCBOZWlsQnJvd24gd3JvdGU6DQo+ID4g -T24gVHVlLCBBdWcgMjkgMjAxNywgSmVmZiBMYXl0b24gd3JvdGU6DQo+ID4gDQo+ID4gPiBPbiBU -dWUsIDIwMTctMDgtMjkgYXQgMTE6MjMgKzEwMDAsIE5laWxCcm93biB3cm90ZToNCj4gPiA+ID4g -T24gTW9uLCBBdWcgMjggMjAxNywgSmVmZiBMYXl0b24gd3JvdGU6DQo+ID4gPiA+IA0KPiA+ID4g -PiA+IE9uIE1vbiwgMjAxNy0wOC0yOCBhdCAwOToyNCArMTAwMCwgTmVpbEJyb3duIHdyb3RlOg0K -PiA+ID4gPiA+ID4gT24gRnJpLCBBdWcgMjUgMjAxNywgSmVmZiBMYXl0b24gd3JvdGU6DQo+ID4g -PiA+ID4gPiANCj4gPiA+ID4gPiA+ID4gT24gVGh1LCAyMDE3LTA3LTIwIGF0IDE1OjQyIC0wNDAw -LCBKZWZmIExheXRvbiB3cm90ZToNCj4gPiA+ID4gPiA+ID4gPiBGcm9tOiBKZWZmIExheXRvbiA8 -amxheXRvbkByZWRoYXQuY29tPg0KPiA+ID4gPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4gPiA+IFRo -ZXJlIGlzIHNvbWUgYW1iaWd1aXR5IGluIG5mcyBhYm91dCBob3cgd3JpdGViYWNrDQo+ID4gPiA+ -ID4gPiA+ID4gZXJyb3JzIGFyZQ0KPiA+ID4gPiA+ID4gPiA+IHRyYWNrZWQuDQo+ID4gPiA+ID4g -PiA+ID4gDQo+ID4gPiA+ID4gPiA+ID4gRm9yIGluc3RhbmNlLCBuZnNfcGFnZWlvX2FkZF9yZXF1 -ZXN0IGNhbGxzDQo+ID4gPiA+ID4gPiA+ID4gbWFwcGluZ19zZXRfZXJyb3Igd2hlbg0KPiA+ID4g -PiA+ID4gPiA+IHRoZQ0KPiA+ID4gPiA+ID4gPiA+IGFkZCBmYWlscywgYnV0IHdlIHRyYWNrIGVy -cm9ycyB0aGF0IG9jY3VyIGFmdGVyIGFkZGluZw0KPiA+ID4gPiA+ID4gPiA+IHRoZQ0KPiA+ID4g -PiA+ID4gPiA+IHJlcXVlc3QNCj4gPiA+ID4gPiA+ID4gPiB3aXRoIGEgZGVkaWNhdGVkIGludCBl -cnJvciBpbiB0aGUgb3BlbiBjb250ZXh0Lg0KPiA+ID4gPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4g -PiA+IE5vdyB0aGF0IHdlIGhhdmUgYmV0dGVyIGluZnJhc3RydWN0dXJlIGZvciB0aGUgdmZzDQo+ -ID4gPiA+ID4gPiA+ID4gbGF5ZXIsIHRoaXMNCj4gPiA+ID4gPiA+ID4gPiBsYXR0ZXIgaW50IGlz -IG5vdyB1bm5lY2Vzc2FyeS4gSnVzdCBoYXZlDQo+ID4gPiA+ID4gPiA+ID4gbmZzX2NvbnRleHRf -c2V0X3dyaXRlX2Vycm9yIHNldA0KPiA+ID4gPiA+ID4gPiA+IHRoZSBlcnJvciBpbiB0aGUgbWFw -cGluZyB3aGVuIG9uZSBvY2N1cnMuDQo+ID4gPiA+ID4gPiA+ID4gDQo+ID4gPiA+ID4gPiA+ID4g -SGF2ZSBORlMgdXNlIGZpbGVfd3JpdGVfYW5kX3dhaXRfcmFuZ2UgdG8gaW5pdGlhdGUgYW5kDQo+ -ID4gPiA+ID4gPiA+ID4gd2FpdCBvbg0KPiA+ID4gPiA+ID4gPiA+IHdyaXRlYmFjaw0KPiA+ID4g -PiA+ID4gPiA+IG9mIHRoZSBkYXRhLCBhbmQgdGhlbiBjaGVjayBhZ2FpbiBhZnRlciBpc3N1aW5n -IHRoZQ0KPiA+ID4gPiA+ID4gPiA+IGNvbW1pdChzKS4NCj4gPiA+ID4gPiA+ID4gPiANCj4gPiA+ -ID4gPiA+ID4gPiBXaXRoIHRoaXMsIHdlIGFsc28gZG9uJ3QgbmVlZCB0byBwYXkgYXR0ZW50aW9u -IHRvIHRoZQ0KPiA+ID4gPiA+ID4gPiA+IEVSUk9SX1dSSVRFDQo+ID4gPiA+ID4gPiA+ID4gZmxh -ZyBmb3IgcmVwb3J0aW5nLCBhbmQganVzdCBjbGVhciBpdCB0byBpbmRpY2F0ZSB0bw0KPiA+ID4g -PiA+ID4gPiA+IHN1YnNlcXVlbnQNCj4gPiA+ID4gPiA+ID4gPiB3cml0ZXJzIHRoYXQgdGhleSBz -aG91bGQgdHJ5IHRvIGdvIGFzeW5jaHJvbm91cyBhZ2Fpbi4NCj4gPiA+ID4gPiA+ID4gPiANCj4g -PiA+ID4gPiA+ID4gPiBJbiBuZnNfcGFnZV9hc3luY19mbHVzaCwgc2FtcGxlIHRoZSBlcnJvciBi -ZWZvcmUNCj4gPiA+ID4gPiA+ID4gPiBsb2NraW5nIGFuZA0KPiA+ID4gPiA+ID4gPiA+IGpvaW5p -bmcNCj4gPiA+ID4gPiA+ID4gPiB0aGUgcmVxdWVzdHMsIGFuZCBjaGVjayBmb3IgZXJyb3JzIHNp -bmNlIHRoYXQgcG9pbnQuDQo+ID4gPiA+ID4gPiA+ID4gDQo+ID4gPiA+ID4gPiA+ID4gU2lnbmVk -LW9mZi1ieTogSmVmZiBMYXl0b24gPGpsYXl0b25AcmVkaGF0LmNvbT4NCj4gPiA+ID4gPiA+ID4g -PiAtLS0NCj4gPiA+ID4gPiA+ID4gPiAgZnMvbmZzL2ZpbGUuYyAgICAgICAgICB8IDI0ICsrKysr -KysrKysrLS0tLS0tLS0tLS0tLQ0KPiA+ID4gPiA+ID4gPiA+ICBmcy9uZnMvaW5vZGUuYyAgICAg -ICAgIHwgIDMgKy0tDQo+ID4gPiA+ID4gPiA+ID4gIGZzL25mcy93cml0ZS5jICAgICAgICAgfCAg -OCArKysrKystLQ0KPiA+ID4gPiA+ID4gPiA+ICBpbmNsdWRlL2xpbnV4L25mc19mcy5oIHwgIDEg -LQ0KPiA+ID4gPiA+ID4gPiA+ICA0IGZpbGVzIGNoYW5nZWQsIDE4IGluc2VydGlvbnMoKyksIDE4 -IGRlbGV0aW9ucygtKQ0KPiA+ID4gPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4gPiA+IEkgaGF2ZSBh -IGJhbGluZyB3aXJlIGFuZCBkdWN0IHRhcGUgc29sdXRpb24gZm9yIHRlc3RpbmcNCj4gPiA+ID4g -PiA+ID4gPiB0aGlzIHdpdGgNCj4gPiA+ID4gPiA+ID4gPiB4ZnN0ZXN0cyAodXNpbmcgaXB0YWJs -ZXMgUkVKRUNUIHRhcmdldHMgYW5kIHNvZnQNCj4gPiA+ID4gPiA+ID4gPiBtb3VudHMpLiBUaGlz -DQo+ID4gPiA+ID4gPiA+ID4gc2VlbXMgdG8NCj4gPiA+ID4gPiA+ID4gPiBtYWtlIG5mcyBkbyB0 -aGUgcmlnaHQgdGhpbmcuDQo+ID4gPiA+ID4gPiA+ID4gDQo+ID4gPiA+ID4gPiA+ID4gZGlmZiAt -LWdpdCBhL2ZzL25mcy9maWxlLmMgYi9mcy9uZnMvZmlsZS5jDQo+ID4gPiA+ID4gPiA+ID4gaW5k -ZXggNTcxM2ViMzJhNDVlLi4xNWQzYzZmYWFmZDMgMTAwNjQ0DQo+ID4gPiA+ID4gPiA+ID4gLS0t -IGEvZnMvbmZzL2ZpbGUuYw0KPiA+ID4gPiA+ID4gPiA+ICsrKyBiL2ZzL25mcy9maWxlLmMNCj4g -PiA+ID4gPiA+ID4gPiBAQCAtMjEyLDI1ICsyMTIsMjMgQEAgbmZzX2ZpbGVfZnN5bmNfY29tbWl0 -KHN0cnVjdCBmaWxlDQo+ID4gPiA+ID4gPiA+ID4gKmZpbGUsDQo+ID4gPiA+ID4gPiA+ID4gbG9m -Zl90IHN0YXJ0LCBsb2ZmX3QgZW5kLCBpbnQgZGF0YXN5bmMpDQo+ID4gPiA+ID4gPiA+ID4gIHsN -Cj4gPiA+ID4gPiA+ID4gPiAgCXN0cnVjdCBuZnNfb3Blbl9jb250ZXh0ICpjdHggPQ0KPiA+ID4g -PiA+ID4gPiA+IG5mc19maWxlX29wZW5fY29udGV4dChmaWxlKTsNCj4gPiA+ID4gPiA+ID4gPiAg -CXN0cnVjdCBpbm9kZSAqaW5vZGUgPSBmaWxlX2lub2RlKGZpbGUpOw0KPiA+ID4gPiA+ID4gPiA+ -IC0JaW50IGhhdmVfZXJyb3IsIGRvX3Jlc2VuZCwgc3RhdHVzOw0KPiA+ID4gPiA+ID4gPiA+IC0J -aW50IHJldCA9IDA7DQo+ID4gPiA+ID4gPiA+ID4gKwlpbnQgZG9fcmVzZW5kLCBzdGF0dXM7DQo+ -ID4gPiA+ID4gPiA+ID4gKwlpbnQgcmV0Ow0KPiA+ID4gPiA+ID4gPiA+ICANCj4gPiA+ID4gPiA+ -ID4gPiAgCWRwcmludGsoIk5GUzogZnN5bmMgZmlsZSglcEQyKSBkYXRhc3luYyAlZFxuIiwNCj4g -PiA+ID4gPiA+ID4gPiBmaWxlLA0KPiA+ID4gPiA+ID4gPiA+IGRhdGFzeW5jKTsNCj4gPiA+ID4g -PiA+ID4gPiAgDQo+ID4gPiA+ID4gPiA+ID4gIAluZnNfaW5jX3N0YXRzKGlub2RlLCBORlNJT1Nf -VkZTRlNZTkMpOw0KPiA+ID4gPiA+ID4gPiA+ICAJZG9fcmVzZW5kID0NCj4gPiA+ID4gPiA+ID4g -PiB0ZXN0X2FuZF9jbGVhcl9iaXQoTkZTX0NPTlRFWFRfUkVTRU5EX1dSSVRFUywgJmN0eC0NCj4g -PiA+ID4gPiA+ID4gPiA+ZmxhZ3MpOw0KPiA+ID4gPiA+ID4gPiA+IC0JaGF2ZV9lcnJvciA9DQo+ -ID4gPiA+ID4gPiA+ID4gdGVzdF9hbmRfY2xlYXJfYml0KE5GU19DT05URVhUX0VSUk9SX1dSSVRF -LA0KPiA+ID4gPiA+ID4gPiA+ICZjdHgtPmZsYWdzKTsNCj4gPiA+ID4gPiA+ID4gPiAtCXN0YXR1 -cyA9IG5mc19jb21taXRfaW5vZGUoaW5vZGUsIEZMVVNIX1NZTkMpOw0KPiA+ID4gPiA+ID4gPiA+ -IC0JaGF2ZV9lcnJvciB8PQ0KPiA+ID4gPiA+ID4gPiA+IHRlc3RfYml0KE5GU19DT05URVhUX0VS -Uk9SX1dSSVRFLCAmY3R4LQ0KPiA+ID4gPiA+ID4gPiA+ID4gZmxhZ3MpOw0KPiA+ID4gPiA+ID4g -PiA+IA0KPiA+ID4gPiA+ID4gPiA+IC0JaWYgKGhhdmVfZXJyb3IpIHsNCj4gPiA+ID4gPiA+ID4g -PiAtCQlyZXQgPSB4Y2hnKCZjdHgtPmVycm9yLCAwKTsNCj4gPiA+ID4gPiA+ID4gPiAtCQlpZiAo -cmV0KQ0KPiA+ID4gPiA+ID4gPiA+IC0JCQlnb3RvIG91dDsNCj4gPiA+ID4gPiA+ID4gPiAtCX0N -Cj4gPiA+ID4gPiA+ID4gPiAtCWlmIChzdGF0dXMgPCAwKSB7DQo+ID4gPiA+ID4gPiA+ID4gKwlj -bGVhcl9iaXQoTkZTX0NPTlRFWFRfRVJST1JfV1JJVEUsICZjdHgtDQo+ID4gPiA+ID4gPiA+ID4g -PmZsYWdzKTsNCj4gPiA+ID4gPiA+ID4gPiArCXJldCA9IG5mc19jb21taXRfaW5vZGUoaW5vZGUs -IEZMVVNIX1NZTkMpOw0KPiA+ID4gPiA+ID4gPiA+ICsNCj4gPiA+ID4gPiA+ID4gPiArCS8qIFJl -Y2hlY2sgYW5kIGFkdmFuY2UgYWZ0ZXIgdGhlIGNvbW1pdCAqLw0KPiA+ID4gPiA+ID4gPiA+ICsJ -c3RhdHVzID0gZmlsZV9jaGVja19hbmRfYWR2YW5jZV93Yl9lcnIoZmlsZSk7DQo+ID4gPiA+ID4g -PiANCj4gPiA+ID4gPiA+IFRoaXMgY2hhbmdlIG1ha2VzIHRoZSBjb2RlIGluY29uc2lzdGVudCB3 -aXRoIHRoZSBjb21tZW50DQo+ID4gPiA+ID4gPiBhYm92ZSB0aGUNCj4gPiA+ID4gPiA+IGZ1bmN0 -aW9uLCB3aGljaCBzdGlsbCByZWZlcmVuY2VzIGN0eC0+ZXJyb3IuICBUaGUgaW50ZW50IG9mDQo+ -ID4gPiA+ID4gPiB0aGUNCj4gPiA+ID4gPiA+IGNvbW1lbnQNCj4gPiA+ID4gPiA+IGlzIHN0aWxs -IGNvcnJlY3QsIGJ1dCB0aGUgZGV0YWlscyBoYXZlIGNoYW5nZWQuDQo+ID4gPiA+ID4gPiANCj4g -PiA+ID4gPiANCj4gPiA+ID4gPiBHb29kIGNhdGNoLiBJJ2xsIGZpeCB0aGF0IHVwIGluIGEgcmVz -cGluLg0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4gQWxzbywgdGhlcmUgaXMgYSBjYWxsIHRvIG1h -cHBpbmdfc2V0X2Vycm9yKCkgaW4NCj4gPiA+ID4gPiA+IG5mc19wYWdlaW9fYWRkX3JlcXVlc3Qo -KS4NCj4gPiA+ID4gPiA+IEkgd29uZGVyIGlmIHRoYXQgc2hvdWxkIGJlIGNoYW5nZWQgdG8NCj4g -PiA+ID4gPiA+ICAgbmZzX2NvbnRleHRfc2V0X3dyaXRlX2Vycm9yKHJlcS0+d2JfY29udGV4dCwg -ZGVzYy0NCj4gPiA+ID4gPiA+ID5wZ19lcnJvcikNCj4gPiA+ID4gPiA+ID8/DQo+ID4gPiA+ID4g -PiANCj4gPiA+ID4gPiANCj4gPiA+ID4gPiBUcmlja2llciBxdWVzdGlvbi4uLg0KPiA+ID4gPiA+ -IA0KPiA+ID4gPiA+IEknbSBub3QgcXVpdGUgc3VyZSB3aGF0IHNlbWFudGljcyB3ZSdyZSBsb29r -aW5nIGZvciB3aXRoDQo+ID4gPiA+ID4gTkZTX0NPTlRFWFRfRVJST1JfV1JJVEUuIEkga25vdyB0 -aGF0IGl0IGZvcmNlcyB3cml0ZXMgdG8gYmUNCj4gPiA+ID4gPiBzeW5jaHJvbm91cywgYnV0IEkn -bSBub3QgcXVpdGUgc3VyZSB3aHkgaXQgZ2V0cyBjbGVhcmVkIHRoZQ0KPiA+ID4gPiA+IHdheSBp -dA0KPiA+ID4gPiA+IGRvZXMuIEl0J3Mgc2V0IG9uIGFueSBlcnJvciBidXQgY2xlYXJlZCBiZWZv -cmUgaXNzdWluZyBhDQo+ID4gPiA+ID4gY29tbWl0Lg0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IEkg -YWRkZWQgYSBzaW1pbGFyIGZsYWcgdG8gQ2VwaCBpbm9kZXMgcmVjZW50bHksIGJ1dCBvbmx5DQo+ -ID4gPiA+ID4gY2xlYXIgaXQgd2hlbg0KPiA+ID4gPiA+IGEgd3JpdGUgc3VjY2VlZHMuIFdvdWxk -bid0IHRoYXQgbWFrZSBtb3JlIHNlbnNlIGhlcmUgYXMgd2VsbD8NCj4gPiA+ID4gDQo+ID4gPiA+ -IEl0IGlzIGEgYml0IGhhcmQgdG8gd3JhcCBvbmUncyBtaW5kIGFyb3VuZC4NCj4gPiA+ID4gDQo+ -ID4gPiA+IEluIHRoZSBvcmlnaW5hbCBjb2RlIChjb21taXQgN2IxNTlmYzE4ZDQxNzk4MCkgaXQg -bG9va3MgbGlrZToNCj4gPiA+ID4gIC0gdGVzdC1hbmQtY2xlYXIgYml0DQo+ID4gPiA+ICAtIHdy -aXRlIGFuZCBzeW5jDQo+ID4gPiA+ICAtIHRlc3QtYml0DQo+ID4gPiA+IA0KPiA+ID4gPiBUaGlz -IGRvZXMsIEkgdGhpbmssIHNlZW0gc2FmZXIgdGhhbiAiY2xlYXIgb24gc3VjY2Vzc2Z1bCB3cml0 -ZSINCj4gPiA+ID4gYXMgdGhlDQo+ID4gPiA+IHdyaXRlcyBjb3VsZCBjb21wbGV0ZSBvdXQtb2Yt -b3JkZXIgYW5kIEkgd291bGRuJ3QgYmUgc3VycHJpc2VkDQo+ID4gPiA+IGlmIHRoZQ0KPiA+ID4g -PiB1bnN1Y2Nlc3NmdWwgb25lcyBjb21wbGV0ZWQgd2l0aCBhbiBlcnJvciBiZWZvcmUgdGhlIHN1 -Y2Nlc3NmdWwNCj4gPiA+ID4gb25lIC0NCj4gPiA+ID4gcGFydGljdWxhcmx5IHdpdGggYW4gZXJy -b3IgbGlrZSBFRFFVT1QuDQo+ID4gPiA+IA0KPiA+ID4gPiBIb3dldmVyIHRoZSBjdXJyZW50IGNv -ZGUgZG9lcyB0aGUgd3JpdGVzIGJlZm9yZSB0aGUgdGVzdC1hbmQtDQo+ID4gPiA+IGNsZWFyLCBh -bmQNCj4gPiA+ID4gb25seSBkb2VzIHRoZSBjb21taXQgYWZ0ZXJ3YXJkcy4gIFRoYXQgbWFrZXMg -aXQgbGVzcyBjbGVhciB3aHkNCj4gPiA+ID4gdGhlDQo+ID4gPiA+IGN1cnJlbnQgc2VxdWVuY2Ug -aXMgYSBnb29kIGlkZWEuDQo+ID4gPiA+IA0KPiA+ID4gPiBIb3dldmVyIC4uLiBuZnNfZmlsZV9m -c3luY19jb21taXQoKSBpcyBvbmx5IGNhbGxlZCBpZg0KPiA+ID4gPiBmaWxlbWFwX3dyaXRlX2Fu -ZF93YWl0X3JhbmdlKCkgcmV0dXJuZWQgd2l0aCBzdWNjZXNzLCBzbyB3ZQ0KPiA+ID4gPiBvbmx5 -IGNsZWFyDQo+ID4gPiA+IHRoZSBmbGFnIGFmdGVyIHN1Y2Nlc3NmdWwgd3JpdGVzKD8pLg0KPiA+ -ID4gPiANCj4gPiA+ID4gT2guLi4uDQo+ID4gPiA+IFRoaXMgcGF0Y2ggZnJvbSBtZToNCj4gPiA+ -ID4gDQo+ID4gPiA+IENvbW1pdDogMmVkYjZiYzM4NTJjICgiTkZTIC0gZml4IHJlY2VudCBicmVh -a2FnZSB0byBORlMgZXJyb3INCj4gPiA+ID4gaGFuZGxpbmcuIikNCj4gPiA+ID4gDQo+ID4gPiA+ -IHNlZW1zIHRvIGhhdmUgYmVlbiByZXZlcnRlZCBieQ0KPiA+ID4gPiANCj4gPiA+ID4gQ29tbWl0 -OiA3YjI4MWVlMDI2NTUgKCJORlM6IGZzeW5jKCkgbXVzdCBleGl0IHdpdGggYW4gZXJyb3IgaWYN -Cj4gPiA+ID4gcGFnZSB3cml0ZWJhY2sgZmFpbGVkIikNCj4gPiA+ID4gDQo+ID4gPiA+IHdoaWNo -IHByb2JhYmx5IGlzbid0IGdvb2QuICBJdCBhcHBlYXJzIHRoYXQgdGhpcyBjb2RlIGlzIHZlcnkN -Cj4gPiA+ID4gZnJhZ2lsZQ0KPiA+ID4gPiBhbmQgZWFzaWx5IGJyb2tlbi4NCj4gPiANCj4gPiBP -biBmdXJ0aGVyIGludmVzdGlnYXRpb24sIEkgdGhpbmsgdGhlIHByb2JsZW0gdGhhdCBJIGZpeGVk -IGFuZCB0aGVuDQo+ID4gd2UNCj4gPiByZWludHJvZHVjZWQgd2lsbCBiZSBmaXhlZCBhZ2FpbiAt -IG1vcmUgcGVybWFuZW50bHkgLSBieSB5b3VyDQo+ID4gcGF0Y2guDQo+ID4gVGhlIHJvb3QgcHJv -YmxlbSBpcyB0aGF0IG5mcyBrZWVwcyBlcnJvciBjb2RlcyBpbiBhIGRpZmZlcmVudCB3YXkNCj4g -PiB0byB0aGUNCj4gPiBNTSBjb3JlLiAgQnkgdW5pZnlpbmcgdGhvc2UsIHRoZSBwcm9ibGVtIGdv -ZXMuDQo+ID4gKFRoZSBzcGVjaWZpYyBwcm9ibGVtIGlzIHRoYXQgd3JpdGVzIHdoaWNoIGhpdCBF -RFFVT1Qgb24gdGhlIHNlcnZlcg0KPiA+IGNhbg0KPiA+ICByZXBvcnQgRUlPIG9uIHRoZSBjbGll -bnQpLg0KPiA+IA0KPiA+IA0KPiA+ID4gPiBNYXliZSB3ZSBuZWVkIHRvIHdvcmsgb3V0IGV4YWN0 -bHkgd2hhdCBpcyByZXF1aXJlZCwgYW5kDQo+ID4gPiA+IGRvY3VtZW50IGl0IC0gc28NCj4gPiA+ -ID4gd2UgY2FuIHN0b3AgYnJlYWtpbmcgaXQuDQo+ID4gPiA+IE9yIG1heWJlIHdlIG5lZWQgc29t -ZSB1bml0IHRlc3RzLi4uLi4NCj4gPiA+ID4gDQo+ID4gPiANCj4gPiA+IFllcywgbGF5aW5nIG91 -dCB3aGF0J3MgbmVjZXNzYXJ5IGZvciB0aGlzIHdvdWxkIGJlIHZlcnkgaGVscGZ1bC4NCj4gPiA+ -IFdlDQo+ID4gPiBjbGVhcmx5IHdhbnQgdG8gc2V0IHRoZSBmbGFnIHdoZW4gYW4gZXJyb3Igb2Nj -dXJzLiBVbmRlciB3aGF0DQo+ID4gPiBjaXJjdW1zdGFuY2VzIHNob3VsZCB3ZSBiZSBjbGVhcmlu -ZyBpdD8NCj4gPiANCj4gPiBXZWxsLi4uLiBsb29raW5nIGJhY2sgYXQgIDdiMTU5ZmMxOGQ0MTc5 -ODBmNTdhZSB3aGljaCBpbnRyb2R1Y2VkDQo+ID4gdGhlDQo+ID4gZmxhZywgcHJpb3IgdG8gdGhh -dCB3cml0ZSBlcnJvcnMgKGN0eC0+ZXJyb3IpIHdlcmUgb25seSByZXBvcnRlZCBieQ0KPiA+IG5m -c19maWxlX2ZsdXNoIGFuZCBuZnNfZnN5bmMsIHNvIG9ubHkgb25lIGNsb3NlKCkgYW5kIGZzeW5j -KCkuDQo+ID4gDQo+ID4gQWZ0ZXIgdGhhdCBjb21taXQsIHNldHRpbmcgdGhlIGZsYWcgd291bGQg -bWVhbiB0aGF0IGVycm9ycyBjb3VsZCBiZQ0KPiA+IHJldHVybmVkIGJ5ICd3cml0ZScuICBTbyBj -bGVhcmluZyBhcyBwYXJ0IG9mIHJldHVybmluZyB0aGUgZXJyb3INCj4gPiBtYWtlcw0KPiA+IHBl -cmZlY3Qgc2Vuc2UuDQo+ID4gDQo+ID4gQXMgbG9uZyBhcyB0aGUgZXJyb3IgZ2V0cyByZWNvcmRl -ZCwgYW5kIGdldHMgcmV0dXJuZWQgd2hlbiBpdCBpcw0KPiA+IHJlY29yZGVkLCBpdCBkb2Vzbid0 -IG11Y2ggbWF0dGVyIHdoZW4gdGhlIGZsYWcgaXMgY2xlYXJlZC4gIFdpdGgNCj4gPiB5b3VyDQo+ -ID4gcGF0Y2hlcyB3ZSBkb24ndCBuZWVkIHRvIGZsYWcgYW55IG1vcmUgdG8gZ2V0IGVycm9ycyBy -ZWxpYWJseQ0KPiA+IHJlcG9ydGVkLg0KPiA+IA0KPiA+IExlYXZpbmcgdGhlIGZsYWcgc2V0IG1l -YW5zIHRoYXQgd3JpdGVzIGdvIG1vcmUgc2xvd2x5IC0gd2UgZG9uJ3QNCj4gPiBnZXQNCj4gPiBs -YXJnZSBxdWV1ZSBvZiBiYWNrZ3JvdW5kIHJpZ2h0cyBidWlsZGluZyB1cCBidXQgZGVzdGluZWQg -Zm9yDQo+ID4gZmFpbHVyZS4NCj4gPiBUaGlzIGlzIHRoZSBtYWluIHBvaW50IG1hZGUgaW4gdGhl -IGNvbW1lbnQgbWVzc2FnZSB3aGVuIHRoZSBmbGFnDQo+ID4gd2FzDQo+ID4gaW50cm9kdWNlZC4N -Cj4gPiBPZiBjb3Vyc2UsIGJ5IHRoZSB0aW1lIHdlIGZpcnN0IGdldCBhbiBlcnJvciB0aGVyZSBj -b3VsZCBhbHJlYWR5DQo+ID4gYnkgYSBsYXJnZSBxdWV1ZSwgc28gd2UgcHJvYmFibHkgd2FudCB0 -aGF0IHRvIGRyYWluIGNvbXBsZXRlbHkNCj4gPiBiZWZvcmUNCj4gPiBhbGxvd2luZyBhc3luYyB3 -cml0ZXMgYWdhaW4uDQoNCldlIGFscmVhZHkgaGF2ZSB0aGlzIGZ1bmN0aW9uYWxpdHkgaW1wbGVt -ZW50ZWQgaW4gdGhlIGV4aXN0aW5nIGNvZGUuDQoNCj4gPiANCj4gPiBJdCBtaWdodCBtYWtlIHNl -bnNlIHRvIGhhdmUgMiBmbGFncy4gIE9uZSB3aGljaCBzYXlzICJ3cml0ZXMgc2hvdWxkDQo+ID4g -YmUNCj4gPiBzeW5jaHJvbm91cyIsIGFub3RoZXIgdGhhdCBzYXlzICJUaGVyZSB3YXMgYW4gZXJy -b3IgcmVjZW50bHkiLg0KPiA+IFdlIGNsZWFyIHRoZSBlcnJvciBmbGFnIGJlZm9yZSBjYWxsaW5n -IG5mc19mc3luYywgYW5kIGlmIGl0IGlzDQo+ID4gc3RpbGwNCj4gPiBjbGVhciBhZnRlcndhcmRz -LCB3ZSBjbGVhciB0aGUgc3luYy13cml0ZXMgZmxhZy4gIE1heWJlIHRoYXQgaXMNCj4gPiBtb3Jl -DQo+ID4gY29tcGxleCB0aGFuIG5lZWRlZCB0aG91Z2guDQo+ID4gDQoNCldlIGFsc28gbmVlZCB0 -byBwcmVzZXJ2ZSB0aGUgTkZTX0NPTlRFWFRfUkVTRU5EX1dSSVRFUyBmbGFnLiBJIGRvbid0DQpz -ZWUgYW55IGdsb2JhbCBtZWNoYW5pc20gdGhhdCB3aWxsIHJlcGxhY2UgdGhhdC4NCg0KPiA+IEkn -bSBsZWFuaW5nIHRvd2FyZHMgeW91ciBzdWdnZXN0aW9uIHRoYXQgaXQgZG9lc24ndCBtYXR0ZXIg -dmVyeQ0KPiA+IG11Y2gNCj4gPiB3aGVuIGl0IGdldHMgY2xlYXJlZCwgYW5kIGNsZWFyaW5nIGl0 -IG9uIGFueSBzdWNjZXNzZnVsIHdyaXRlIGlzDQo+ID4gc2ltcGxlc3QuDQo+ID4gDQo+ID4gU28g -SSdtIHN0aWxsIGluIGZhdm9yIG9mIHVzaW5nIG5mc19jb250ZXh0X3NldF93cml0ZV9lcnJvcigp -IGluDQo+ID4gbmZzX3BhZ2Vpb19hZGRfcmVxdWVzdCgpLCBwcmltYXJpbHkgYmVjYXVzZSBpdCBp -cyBtb3N0IGNvbnNpc3RlbnQgLQ0KPiA+IHdlDQo+ID4gZG9uJ3QgbmVlZCBleGNlcHRpb25zLg0K -PiANCj4gVGhhbmtzIGZvciB0YWtpbmcgYSBjbG9zZXIgbG9vay4gSSBjYW4gZWFzaWx5IG1ha2Ug -dGhlIGNoYW5nZSBhYm92ZSwNCj4gYW5kDQo+IEkgZG8gdGhpbmsgdGhhdCBrZWVwaW5nIHRoaXMg -bWVjaGFuaXNtIGFzIHNpbXBsZSBhcyBwb3NzaWJsZSB3aWxsDQo+IG1ha2UNCj4gaXQgZWFzaWVy -IHRvIHByZXZlbnQgYml0cm90Lg0KPiANCj4gVGhhdCBzYWlkLi4uIE5GU19DT05URVhUX0VSUk9S -X1dSSVRFIGlzIGEgcGVyIGN0eCBmbGFnLCBhbmQgdGhlIGN0eA0KPiBpcyBhDQo+IHBlciBvcGVu -IGZpbGUgZGVzY3JpcHRpb24gb2JqZWN0Lg0KPiANCj4gSXMgdGhhdCB0aGUgY29ycmVjdCB3YXkg -dG8gdHJhY2sgdGhpcz8gQWxsIG9mIHRoZSBjdHgncyB3aWxsIHNoYXJlDQo+IHRoZQ0KPiBzYW1l -IGlub2RlLiBJZiB3ZSdyZSBnZXR0aW5nIHdyaXRlYmFjayBlcnJvcnMgZm9yIG9uZSBjb250ZXh0 -LCBpdCdzDQo+IHF1aXRlIGxpa2VseSB0aGF0IHdlJ2xsIGJlIHNlZWluZyB0aGVtIHZpYSBvdGhl -cnMuDQo+IA0KPiBJIHN1cHBvc2UgdGhlIGNvdW50ZXJhcmd1bWVudCBpcyB3aGVuIHdlIGhhdmUg -dGhpbmdzIGxpa2UgZXhwaXJpbmcNCj4ga3JiNQ0KPiB0aWNrZXRzLiBXcml0ZSBmYWlsdXJlcyB2 -aWEgYW4gZXhwaXJpbmcgc2V0IG9mIGNyZWRzIG1heSBoYXZlIG5vDQo+IGVmZmVjdA0KPiBvbiB3 -cml0ZWJhY2sgdmlhIG90aGVyIGNyZWRzLg0KPiANCj4gU3RpbGwsIEkgdGhpbmsgYSBwZXItaW5v -ZGUgZmxhZyBtaWdodCBtYWtlIG1vcmUgc2Vuc2UgaGVyZS4NCj4gDQo+IFRob3VnaHRzPw0KDQpB -cyBmYXIgYXMgSSdtIGNvbmNlcm5lZCwgdGhhdCB3b3VsZCBiZSBhIHJlZ3Jlc3Npb24uIFRoZSBt -b3N0IGNvbW1vbg0KcHJvYmxlbSB3aGVuIGZsdXNoaW5nIHdyaXRlYmFjayBkYXRhIHRvIHRoZSBz -ZXJ2ZXIgYXNpZGUgZnJvbSBFTk9TUEMNCihhbmQgcG9zc2libHkgRVNUQUxFKSBpcyBFQUNDRVMs -IHdoaWNoIGlzIHBhcnRpY3VsYXIgdG8gdGhlIGZpbGUNCmRlc2NyaXB0b3IgdGhhdCBvcGVuZWQg -dGhlIGZpbGUuDQoNCkZpbGUgY29udGV4dHMsIGFuZCBORlNfQ09OVEVYVF9FUlJPUl9XUklURSBz -b2x2ZSB0aGF0IHByb2JsZW0gYnkgYmVpbmcNCnByaXZhdGUgdG8gdGhlIGZpbGUgZGVzY3JpcHRv -ci4NCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lciwg -UHJpbWFyeURhdGENCnRyb25kLm15a2xlYnVzdEBwcmltYXJ5ZGF0YS5jb20NCg== +On Thu, 2017-09-07 at 07:35 -0400, Jeff Layton wrote: +> On Thu, 2017-09-07 at 13:37 +1000, NeilBrown wrote: +> > On Tue, Aug 29 2017, Jeff Layton wrote: +> > +> > > On Tue, 2017-08-29 at 11:23 +1000, NeilBrown wrote: +> > > > On Mon, Aug 28 2017, Jeff Layton wrote: +> > > > +> > > > > On Mon, 2017-08-28 at 09:24 +1000, NeilBrown wrote: +> > > > > > On Fri, Aug 25 2017, Jeff Layton wrote: +> > > > > > +> > > > > > > On Thu, 2017-07-20 at 15:42 -0400, Jeff Layton wrote: +> > > > > > > > From: Jeff Layton <jlayton@redhat.com> +> > > > > > > > +> > > > > > > > There is some ambiguity in nfs about how writeback +> > > > > > > > errors are +> > > > > > > > tracked. +> > > > > > > > +> > > > > > > > For instance, nfs_pageio_add_request calls +> > > > > > > > mapping_set_error when +> > > > > > > > the +> > > > > > > > add fails, but we track errors that occur after adding +> > > > > > > > the +> > > > > > > > request +> > > > > > > > with a dedicated int error in the open context. +> > > > > > > > +> > > > > > > > Now that we have better infrastructure for the vfs +> > > > > > > > layer, this +> > > > > > > > latter int is now unnecessary. Just have +> > > > > > > > nfs_context_set_write_error set +> > > > > > > > the error in the mapping when one occurs. +> > > > > > > > +> > > > > > > > Have NFS use file_write_and_wait_range to initiate and +> > > > > > > > wait on +> > > > > > > > writeback +> > > > > > > > of the data, and then check again after issuing the +> > > > > > > > commit(s). +> > > > > > > > +> > > > > > > > With this, we also don't need to pay attention to the +> > > > > > > > ERROR_WRITE +> > > > > > > > flag for reporting, and just clear it to indicate to +> > > > > > > > subsequent +> > > > > > > > writers that they should try to go asynchronous again. +> > > > > > > > +> > > > > > > > In nfs_page_async_flush, sample the error before +> > > > > > > > locking and +> > > > > > > > joining +> > > > > > > > the requests, and check for errors since that point. +> > > > > > > > +> > > > > > > > Signed-off-by: Jeff Layton <jlayton@redhat.com> +> > > > > > > > --- +> > > > > > > > fs/nfs/file.c | 24 +++++++++++------------- +> > > > > > > > fs/nfs/inode.c | 3 +-- +> > > > > > > > fs/nfs/write.c | 8 ++++++-- +> > > > > > > > include/linux/nfs_fs.h | 1 - +> > > > > > > > 4 files changed, 18 insertions(+), 18 deletions(-) +> > > > > > > > +> > > > > > > > I have a baling wire and duct tape solution for testing +> > > > > > > > this with +> > > > > > > > xfstests (using iptables REJECT targets and soft +> > > > > > > > mounts). This +> > > > > > > > seems to +> > > > > > > > make nfs do the right thing. +> > > > > > > > +> > > > > > > > diff --git a/fs/nfs/file.c b/fs/nfs/file.c +> > > > > > > > index 5713eb32a45e..15d3c6faafd3 100644 +> > > > > > > > --- a/fs/nfs/file.c +> > > > > > > > +++ b/fs/nfs/file.c +> > > > > > > > @@ -212,25 +212,23 @@ nfs_file_fsync_commit(struct file +> > > > > > > > *file, +> > > > > > > > loff_t start, loff_t end, int datasync) +> > > > > > > > { +> > > > > > > > struct nfs_open_context *ctx = +> > > > > > > > nfs_file_open_context(file); +> > > > > > > > struct inode *inode = file_inode(file); +> > > > > > > > - int have_error, do_resend, status; +> > > > > > > > - int ret = 0; +> > > > > > > > + int do_resend, status; +> > > > > > > > + int ret; +> > > > > > > > +> > > > > > > > dprintk("NFS: fsync file(%pD2) datasync %d\n", +> > > > > > > > file, +> > > > > > > > datasync); +> > > > > > > > +> > > > > > > > nfs_inc_stats(inode, NFSIOS_VFSFSYNC); +> > > > > > > > do_resend = +> > > > > > > > test_and_clear_bit(NFS_CONTEXT_RESEND_WRITES, &ctx- +> > > > > > > > >flags); +> > > > > > > > - have_error = +> > > > > > > > test_and_clear_bit(NFS_CONTEXT_ERROR_WRITE, +> > > > > > > > &ctx->flags); +> > > > > > > > - status = nfs_commit_inode(inode, FLUSH_SYNC); +> > > > > > > > - have_error |= +> > > > > > > > test_bit(NFS_CONTEXT_ERROR_WRITE, &ctx- +> > > > > > > > > flags); +> > > > > > > > +> > > > > > > > - if (have_error) { +> > > > > > > > - ret = xchg(&ctx->error, 0); +> > > > > > > > - if (ret) +> > > > > > > > - goto out; +> > > > > > > > - } +> > > > > > > > - if (status < 0) { +> > > > > > > > + clear_bit(NFS_CONTEXT_ERROR_WRITE, &ctx- +> > > > > > > > >flags); +> > > > > > > > + ret = nfs_commit_inode(inode, FLUSH_SYNC); +> > > > > > > > + +> > > > > > > > + /* Recheck and advance after the commit */ +> > > > > > > > + status = file_check_and_advance_wb_err(file); +> > > > > > +> > > > > > This change makes the code inconsistent with the comment +> > > > > > above the +> > > > > > function, which still references ctx->error. The intent of +> > > > > > the +> > > > > > comment +> > > > > > is still correct, but the details have changed. +> > > > > > +> > > > > +> > > > > Good catch. I'll fix that up in a respin. +> > > > > +> > > > > > Also, there is a call to mapping_set_error() in +> > > > > > nfs_pageio_add_request(). +> > > > > > I wonder if that should be changed to +> > > > > > nfs_context_set_write_error(req->wb_context, desc- +> > > > > > >pg_error) +> > > > > > ?? +> > > > > > +> > > > > +> > > > > Trickier question... +> > > > > +> > > > > I'm not quite sure what semantics we're looking for with +> > > > > NFS_CONTEXT_ERROR_WRITE. I know that it forces writes to be +> > > > > synchronous, but I'm not quite sure why it gets cleared the +> > > > > way it +> > > > > does. It's set on any error but cleared before issuing a +> > > > > commit. +> > > > > +> > > > > I added a similar flag to Ceph inodes recently, but only +> > > > > clear it when +> > > > > a write succeeds. Wouldn't that make more sense here as well? +> > > > +> > > > It is a bit hard to wrap one's mind around. +> > > > +> > > > In the original code (commit 7b159fc18d417980) it looks like: +> > > > - test-and-clear bit +> > > > - write and sync +> > > > - test-bit +> > > > +> > > > This does, I think, seem safer than "clear on successful write" +> > > > as the +> > > > writes could complete out-of-order and I wouldn't be surprised +> > > > if the +> > > > unsuccessful ones completed with an error before the successful +> > > > one - +> > > > particularly with an error like EDQUOT. +> > > > +> > > > However the current code does the writes before the test-and- +> > > > clear, and +> > > > only does the commit afterwards. That makes it less clear why +> > > > the +> > > > current sequence is a good idea. +> > > > +> > > > However ... nfs_file_fsync_commit() is only called if +> > > > filemap_write_and_wait_range() returned with success, so we +> > > > only clear +> > > > the flag after successful writes(?). +> > > > +> > > > Oh.... +> > > > This patch from me: +> > > > +> > > > Commit: 2edb6bc3852c ("NFS - fix recent breakage to NFS error +> > > > handling.") +> > > > +> > > > seems to have been reverted by +> > > > +> > > > Commit: 7b281ee02655 ("NFS: fsync() must exit with an error if +> > > > page writeback failed") +> > > > +> > > > which probably isn't good. It appears that this code is very +> > > > fragile +> > > > and easily broken. +> > +> > On further investigation, I think the problem that I fixed and then +> > we +> > reintroduced will be fixed again - more permanently - by your +> > patch. +> > The root problem is that nfs keeps error codes in a different way +> > to the +> > MM core. By unifying those, the problem goes. +> > (The specific problem is that writes which hit EDQUOT on the server +> > can +> > report EIO on the client). +> > +> > +> > > > Maybe we need to work out exactly what is required, and +> > > > document it - so +> > > > we can stop breaking it. +> > > > Or maybe we need some unit tests..... +> > > > +> > > +> > > Yes, laying out what's necessary for this would be very helpful. +> > > We +> > > clearly want to set the flag when an error occurs. Under what +> > > circumstances should we be clearing it? +> > +> > Well.... looking back at 7b159fc18d417980f57ae which introduced +> > the +> > flag, prior to that write errors (ctx->error) were only reported by +> > nfs_file_flush and nfs_fsync, so only one close() and fsync(). +> > +> > After that commit, setting the flag would mean that errors could be +> > returned by 'write'. So clearing as part of returning the error +> > makes +> > perfect sense. +> > +> > As long as the error gets recorded, and gets returned when it is +> > recorded, it doesn't much matter when the flag is cleared. With +> > your +> > patches we don't need to flag any more to get errors reliably +> > reported. +> > +> > Leaving the flag set means that writes go more slowly - we don't +> > get +> > large queue of background rights building up but destined for +> > failure. +> > This is the main point made in the comment message when the flag +> > was +> > introduced. +> > Of course, by the time we first get an error there could already +> > by a large queue, so we probably want that to drain completely +> > before +> > allowing async writes again. + +We already have this functionality implemented in the existing code. + +> > +> > It might make sense to have 2 flags. One which says "writes should +> > be +> > synchronous", another that says "There was an error recently". +> > We clear the error flag before calling nfs_fsync, and if it is +> > still +> > clear afterwards, we clear the sync-writes flag. Maybe that is +> > more +> > complex than needed though. +> > + +We also need to preserve the NFS_CONTEXT_RESEND_WRITES flag. I don't +see any global mechanism that will replace that. + +> > I'm leaning towards your suggestion that it doesn't matter very +> > much +> > when it gets cleared, and clearing it on any successful write is +> > simplest. +> > +> > So I'm still in favor of using nfs_context_set_write_error() in +> > nfs_pageio_add_request(), primarily because it is most consistent - +> > we +> > don't need exceptions. +> +> Thanks for taking a closer look. I can easily make the change above, +> and +> I do think that keeping this mechanism as simple as possible will +> make +> it easier to prevent bitrot. +> +> That said... NFS_CONTEXT_ERROR_WRITE is a per ctx flag, and the ctx +> is a +> per open file description object. +> +> Is that the correct way to track this? All of the ctx's will share +> the +> same inode. If we're getting writeback errors for one context, it's +> quite likely that we'll be seeing them via others. +> +> I suppose the counterargument is when we have things like expiring +> krb5 +> tickets. Write failures via an expiring set of creds may have no +> effect +> on writeback via other creds. +> +> Still, I think a per-inode flag might make more sense here. +> +> Thoughts? + +As far as I'm concerned, that would be a regression. The most common +problem when flushing writeback data to the server aside from ENOSPC +(and possibly ESTALE) is EACCES, which is particular to the file +descriptor that opened the file. + +File contexts, and NFS_CONTEXT_ERROR_WRITE solve that problem by being +private to the file descriptor. + +-- +Trond Myklebust +Linux NFS client maintainer, PrimaryData +trond.myklebust@primarydata.com diff --git a/a/content_digest b/N1/content_digest index 0bbfc03..4b53a66 100644 --- a/a/content_digest +++ b/N1/content_digest @@ -17,202 +17,299 @@ " linux-fsdevel@vger.kernel.org <linux-fsdevel@vger.kernel.org>\0" "\00:1\0" "b\0" - "T24gVGh1LCAyMDE3LTA5LTA3IGF0IDA3OjM1IC0wNDAwLCBKZWZmIExheXRvbiB3cm90ZToNCj4g\n" - "T24gVGh1LCAyMDE3LTA5LTA3IGF0IDEzOjM3ICsxMDAwLCBOZWlsQnJvd24gd3JvdGU6DQo+ID4g\n" - "T24gVHVlLCBBdWcgMjkgMjAxNywgSmVmZiBMYXl0b24gd3JvdGU6DQo+ID4gDQo+ID4gPiBPbiBU\n" - "dWUsIDIwMTctMDgtMjkgYXQgMTE6MjMgKzEwMDAsIE5laWxCcm93biB3cm90ZToNCj4gPiA+ID4g\n" - "T24gTW9uLCBBdWcgMjggMjAxNywgSmVmZiBMYXl0b24gd3JvdGU6DQo+ID4gPiA+IA0KPiA+ID4g\n" - "PiA+IE9uIE1vbiwgMjAxNy0wOC0yOCBhdCAwOToyNCArMTAwMCwgTmVpbEJyb3duIHdyb3RlOg0K\n" - "PiA+ID4gPiA+ID4gT24gRnJpLCBBdWcgMjUgMjAxNywgSmVmZiBMYXl0b24gd3JvdGU6DQo+ID4g\n" - "PiA+ID4gPiANCj4gPiA+ID4gPiA+ID4gT24gVGh1LCAyMDE3LTA3LTIwIGF0IDE1OjQyIC0wNDAw\n" - "LCBKZWZmIExheXRvbiB3cm90ZToNCj4gPiA+ID4gPiA+ID4gPiBGcm9tOiBKZWZmIExheXRvbiA8\n" - "amxheXRvbkByZWRoYXQuY29tPg0KPiA+ID4gPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4gPiA+IFRo\n" - "ZXJlIGlzIHNvbWUgYW1iaWd1aXR5IGluIG5mcyBhYm91dCBob3cgd3JpdGViYWNrDQo+ID4gPiA+\n" - "ID4gPiA+ID4gZXJyb3JzIGFyZQ0KPiA+ID4gPiA+ID4gPiA+IHRyYWNrZWQuDQo+ID4gPiA+ID4g\n" - "PiA+ID4gDQo+ID4gPiA+ID4gPiA+ID4gRm9yIGluc3RhbmNlLCBuZnNfcGFnZWlvX2FkZF9yZXF1\n" - "ZXN0IGNhbGxzDQo+ID4gPiA+ID4gPiA+ID4gbWFwcGluZ19zZXRfZXJyb3Igd2hlbg0KPiA+ID4g\n" - "PiA+ID4gPiA+IHRoZQ0KPiA+ID4gPiA+ID4gPiA+IGFkZCBmYWlscywgYnV0IHdlIHRyYWNrIGVy\n" - "cm9ycyB0aGF0IG9jY3VyIGFmdGVyIGFkZGluZw0KPiA+ID4gPiA+ID4gPiA+IHRoZQ0KPiA+ID4g\n" - "PiA+ID4gPiA+IHJlcXVlc3QNCj4gPiA+ID4gPiA+ID4gPiB3aXRoIGEgZGVkaWNhdGVkIGludCBl\n" - "cnJvciBpbiB0aGUgb3BlbiBjb250ZXh0Lg0KPiA+ID4gPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4g\n" - "PiA+IE5vdyB0aGF0IHdlIGhhdmUgYmV0dGVyIGluZnJhc3RydWN0dXJlIGZvciB0aGUgdmZzDQo+\n" - "ID4gPiA+ID4gPiA+ID4gbGF5ZXIsIHRoaXMNCj4gPiA+ID4gPiA+ID4gPiBsYXR0ZXIgaW50IGlz\n" - "IG5vdyB1bm5lY2Vzc2FyeS4gSnVzdCBoYXZlDQo+ID4gPiA+ID4gPiA+ID4gbmZzX2NvbnRleHRf\n" - "c2V0X3dyaXRlX2Vycm9yIHNldA0KPiA+ID4gPiA+ID4gPiA+IHRoZSBlcnJvciBpbiB0aGUgbWFw\n" - "cGluZyB3aGVuIG9uZSBvY2N1cnMuDQo+ID4gPiA+ID4gPiA+ID4gDQo+ID4gPiA+ID4gPiA+ID4g\n" - "SGF2ZSBORlMgdXNlIGZpbGVfd3JpdGVfYW5kX3dhaXRfcmFuZ2UgdG8gaW5pdGlhdGUgYW5kDQo+\n" - "ID4gPiA+ID4gPiA+ID4gd2FpdCBvbg0KPiA+ID4gPiA+ID4gPiA+IHdyaXRlYmFjaw0KPiA+ID4g\n" - "PiA+ID4gPiA+IG9mIHRoZSBkYXRhLCBhbmQgdGhlbiBjaGVjayBhZ2FpbiBhZnRlciBpc3N1aW5n\n" - "IHRoZQ0KPiA+ID4gPiA+ID4gPiA+IGNvbW1pdChzKS4NCj4gPiA+ID4gPiA+ID4gPiANCj4gPiA+\n" - "ID4gPiA+ID4gPiBXaXRoIHRoaXMsIHdlIGFsc28gZG9uJ3QgbmVlZCB0byBwYXkgYXR0ZW50aW9u\n" - "IHRvIHRoZQ0KPiA+ID4gPiA+ID4gPiA+IEVSUk9SX1dSSVRFDQo+ID4gPiA+ID4gPiA+ID4gZmxh\n" - "ZyBmb3IgcmVwb3J0aW5nLCBhbmQganVzdCBjbGVhciBpdCB0byBpbmRpY2F0ZSB0bw0KPiA+ID4g\n" - "PiA+ID4gPiA+IHN1YnNlcXVlbnQNCj4gPiA+ID4gPiA+ID4gPiB3cml0ZXJzIHRoYXQgdGhleSBz\n" - "aG91bGQgdHJ5IHRvIGdvIGFzeW5jaHJvbm91cyBhZ2Fpbi4NCj4gPiA+ID4gPiA+ID4gPiANCj4g\n" - "PiA+ID4gPiA+ID4gPiBJbiBuZnNfcGFnZV9hc3luY19mbHVzaCwgc2FtcGxlIHRoZSBlcnJvciBi\n" - "ZWZvcmUNCj4gPiA+ID4gPiA+ID4gPiBsb2NraW5nIGFuZA0KPiA+ID4gPiA+ID4gPiA+IGpvaW5p\n" - "bmcNCj4gPiA+ID4gPiA+ID4gPiB0aGUgcmVxdWVzdHMsIGFuZCBjaGVjayBmb3IgZXJyb3JzIHNp\n" - "bmNlIHRoYXQgcG9pbnQuDQo+ID4gPiA+ID4gPiA+ID4gDQo+ID4gPiA+ID4gPiA+ID4gU2lnbmVk\n" - "LW9mZi1ieTogSmVmZiBMYXl0b24gPGpsYXl0b25AcmVkaGF0LmNvbT4NCj4gPiA+ID4gPiA+ID4g\n" - "PiAtLS0NCj4gPiA+ID4gPiA+ID4gPiAgZnMvbmZzL2ZpbGUuYyAgICAgICAgICB8IDI0ICsrKysr\n" - "KysrKysrLS0tLS0tLS0tLS0tLQ0KPiA+ID4gPiA+ID4gPiA+ICBmcy9uZnMvaW5vZGUuYyAgICAg\n" - "ICAgIHwgIDMgKy0tDQo+ID4gPiA+ID4gPiA+ID4gIGZzL25mcy93cml0ZS5jICAgICAgICAgfCAg\n" - "OCArKysrKystLQ0KPiA+ID4gPiA+ID4gPiA+ICBpbmNsdWRlL2xpbnV4L25mc19mcy5oIHwgIDEg\n" - "LQ0KPiA+ID4gPiA+ID4gPiA+ICA0IGZpbGVzIGNoYW5nZWQsIDE4IGluc2VydGlvbnMoKyksIDE4\n" - "IGRlbGV0aW9ucygtKQ0KPiA+ID4gPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4gPiA+IEkgaGF2ZSBh\n" - "IGJhbGluZyB3aXJlIGFuZCBkdWN0IHRhcGUgc29sdXRpb24gZm9yIHRlc3RpbmcNCj4gPiA+ID4g\n" - "PiA+ID4gPiB0aGlzIHdpdGgNCj4gPiA+ID4gPiA+ID4gPiB4ZnN0ZXN0cyAodXNpbmcgaXB0YWJs\n" - "ZXMgUkVKRUNUIHRhcmdldHMgYW5kIHNvZnQNCj4gPiA+ID4gPiA+ID4gPiBtb3VudHMpLiBUaGlz\n" - "DQo+ID4gPiA+ID4gPiA+ID4gc2VlbXMgdG8NCj4gPiA+ID4gPiA+ID4gPiBtYWtlIG5mcyBkbyB0\n" - "aGUgcmlnaHQgdGhpbmcuDQo+ID4gPiA+ID4gPiA+ID4gDQo+ID4gPiA+ID4gPiA+ID4gZGlmZiAt\n" - "LWdpdCBhL2ZzL25mcy9maWxlLmMgYi9mcy9uZnMvZmlsZS5jDQo+ID4gPiA+ID4gPiA+ID4gaW5k\n" - "ZXggNTcxM2ViMzJhNDVlLi4xNWQzYzZmYWFmZDMgMTAwNjQ0DQo+ID4gPiA+ID4gPiA+ID4gLS0t\n" - "IGEvZnMvbmZzL2ZpbGUuYw0KPiA+ID4gPiA+ID4gPiA+ICsrKyBiL2ZzL25mcy9maWxlLmMNCj4g\n" - "PiA+ID4gPiA+ID4gPiBAQCAtMjEyLDI1ICsyMTIsMjMgQEAgbmZzX2ZpbGVfZnN5bmNfY29tbWl0\n" - "KHN0cnVjdCBmaWxlDQo+ID4gPiA+ID4gPiA+ID4gKmZpbGUsDQo+ID4gPiA+ID4gPiA+ID4gbG9m\n" - "Zl90IHN0YXJ0LCBsb2ZmX3QgZW5kLCBpbnQgZGF0YXN5bmMpDQo+ID4gPiA+ID4gPiA+ID4gIHsN\n" - "Cj4gPiA+ID4gPiA+ID4gPiAgCXN0cnVjdCBuZnNfb3Blbl9jb250ZXh0ICpjdHggPQ0KPiA+ID4g\n" - "PiA+ID4gPiA+IG5mc19maWxlX29wZW5fY29udGV4dChmaWxlKTsNCj4gPiA+ID4gPiA+ID4gPiAg\n" - "CXN0cnVjdCBpbm9kZSAqaW5vZGUgPSBmaWxlX2lub2RlKGZpbGUpOw0KPiA+ID4gPiA+ID4gPiA+\n" - "IC0JaW50IGhhdmVfZXJyb3IsIGRvX3Jlc2VuZCwgc3RhdHVzOw0KPiA+ID4gPiA+ID4gPiA+IC0J\n" - "aW50IHJldCA9IDA7DQo+ID4gPiA+ID4gPiA+ID4gKwlpbnQgZG9fcmVzZW5kLCBzdGF0dXM7DQo+\n" - "ID4gPiA+ID4gPiA+ID4gKwlpbnQgcmV0Ow0KPiA+ID4gPiA+ID4gPiA+ICANCj4gPiA+ID4gPiA+\n" - "ID4gPiAgCWRwcmludGsoIk5GUzogZnN5bmMgZmlsZSglcEQyKSBkYXRhc3luYyAlZFxuIiwNCj4g\n" - "PiA+ID4gPiA+ID4gPiBmaWxlLA0KPiA+ID4gPiA+ID4gPiA+IGRhdGFzeW5jKTsNCj4gPiA+ID4g\n" - "PiA+ID4gPiAgDQo+ID4gPiA+ID4gPiA+ID4gIAluZnNfaW5jX3N0YXRzKGlub2RlLCBORlNJT1Nf\n" - "VkZTRlNZTkMpOw0KPiA+ID4gPiA+ID4gPiA+ICAJZG9fcmVzZW5kID0NCj4gPiA+ID4gPiA+ID4g\n" - "PiB0ZXN0X2FuZF9jbGVhcl9iaXQoTkZTX0NPTlRFWFRfUkVTRU5EX1dSSVRFUywgJmN0eC0NCj4g\n" - "PiA+ID4gPiA+ID4gPiA+ZmxhZ3MpOw0KPiA+ID4gPiA+ID4gPiA+IC0JaGF2ZV9lcnJvciA9DQo+\n" - "ID4gPiA+ID4gPiA+ID4gdGVzdF9hbmRfY2xlYXJfYml0KE5GU19DT05URVhUX0VSUk9SX1dSSVRF\n" - "LA0KPiA+ID4gPiA+ID4gPiA+ICZjdHgtPmZsYWdzKTsNCj4gPiA+ID4gPiA+ID4gPiAtCXN0YXR1\n" - "cyA9IG5mc19jb21taXRfaW5vZGUoaW5vZGUsIEZMVVNIX1NZTkMpOw0KPiA+ID4gPiA+ID4gPiA+\n" - "IC0JaGF2ZV9lcnJvciB8PQ0KPiA+ID4gPiA+ID4gPiA+IHRlc3RfYml0KE5GU19DT05URVhUX0VS\n" - "Uk9SX1dSSVRFLCAmY3R4LQ0KPiA+ID4gPiA+ID4gPiA+ID4gZmxhZ3MpOw0KPiA+ID4gPiA+ID4g\n" - "PiA+IA0KPiA+ID4gPiA+ID4gPiA+IC0JaWYgKGhhdmVfZXJyb3IpIHsNCj4gPiA+ID4gPiA+ID4g\n" - "PiAtCQlyZXQgPSB4Y2hnKCZjdHgtPmVycm9yLCAwKTsNCj4gPiA+ID4gPiA+ID4gPiAtCQlpZiAo\n" - "cmV0KQ0KPiA+ID4gPiA+ID4gPiA+IC0JCQlnb3RvIG91dDsNCj4gPiA+ID4gPiA+ID4gPiAtCX0N\n" - "Cj4gPiA+ID4gPiA+ID4gPiAtCWlmIChzdGF0dXMgPCAwKSB7DQo+ID4gPiA+ID4gPiA+ID4gKwlj\n" - "bGVhcl9iaXQoTkZTX0NPTlRFWFRfRVJST1JfV1JJVEUsICZjdHgtDQo+ID4gPiA+ID4gPiA+ID4g\n" - "PmZsYWdzKTsNCj4gPiA+ID4gPiA+ID4gPiArCXJldCA9IG5mc19jb21taXRfaW5vZGUoaW5vZGUs\n" - "IEZMVVNIX1NZTkMpOw0KPiA+ID4gPiA+ID4gPiA+ICsNCj4gPiA+ID4gPiA+ID4gPiArCS8qIFJl\n" - "Y2hlY2sgYW5kIGFkdmFuY2UgYWZ0ZXIgdGhlIGNvbW1pdCAqLw0KPiA+ID4gPiA+ID4gPiA+ICsJ\n" - "c3RhdHVzID0gZmlsZV9jaGVja19hbmRfYWR2YW5jZV93Yl9lcnIoZmlsZSk7DQo+ID4gPiA+ID4g\n" - "PiANCj4gPiA+ID4gPiA+IFRoaXMgY2hhbmdlIG1ha2VzIHRoZSBjb2RlIGluY29uc2lzdGVudCB3\n" - "aXRoIHRoZSBjb21tZW50DQo+ID4gPiA+ID4gPiBhYm92ZSB0aGUNCj4gPiA+ID4gPiA+IGZ1bmN0\n" - "aW9uLCB3aGljaCBzdGlsbCByZWZlcmVuY2VzIGN0eC0+ZXJyb3IuICBUaGUgaW50ZW50IG9mDQo+\n" - "ID4gPiA+ID4gPiB0aGUNCj4gPiA+ID4gPiA+IGNvbW1lbnQNCj4gPiA+ID4gPiA+IGlzIHN0aWxs\n" - "IGNvcnJlY3QsIGJ1dCB0aGUgZGV0YWlscyBoYXZlIGNoYW5nZWQuDQo+ID4gPiA+ID4gPiANCj4g\n" - "PiA+ID4gPiANCj4gPiA+ID4gPiBHb29kIGNhdGNoLiBJJ2xsIGZpeCB0aGF0IHVwIGluIGEgcmVz\n" - "cGluLg0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+ID4gQWxzbywgdGhlcmUgaXMgYSBjYWxsIHRvIG1h\n" - "cHBpbmdfc2V0X2Vycm9yKCkgaW4NCj4gPiA+ID4gPiA+IG5mc19wYWdlaW9fYWRkX3JlcXVlc3Qo\n" - "KS4NCj4gPiA+ID4gPiA+IEkgd29uZGVyIGlmIHRoYXQgc2hvdWxkIGJlIGNoYW5nZWQgdG8NCj4g\n" - "PiA+ID4gPiA+ICAgbmZzX2NvbnRleHRfc2V0X3dyaXRlX2Vycm9yKHJlcS0+d2JfY29udGV4dCwg\n" - "ZGVzYy0NCj4gPiA+ID4gPiA+ID5wZ19lcnJvcikNCj4gPiA+ID4gPiA+ID8/DQo+ID4gPiA+ID4g\n" - "PiANCj4gPiA+ID4gPiANCj4gPiA+ID4gPiBUcmlja2llciBxdWVzdGlvbi4uLg0KPiA+ID4gPiA+\n" - "IA0KPiA+ID4gPiA+IEknbSBub3QgcXVpdGUgc3VyZSB3aGF0IHNlbWFudGljcyB3ZSdyZSBsb29r\n" - "aW5nIGZvciB3aXRoDQo+ID4gPiA+ID4gTkZTX0NPTlRFWFRfRVJST1JfV1JJVEUuIEkga25vdyB0\n" - "aGF0IGl0IGZvcmNlcyB3cml0ZXMgdG8gYmUNCj4gPiA+ID4gPiBzeW5jaHJvbm91cywgYnV0IEkn\n" - "bSBub3QgcXVpdGUgc3VyZSB3aHkgaXQgZ2V0cyBjbGVhcmVkIHRoZQ0KPiA+ID4gPiA+IHdheSBp\n" - "dA0KPiA+ID4gPiA+IGRvZXMuIEl0J3Mgc2V0IG9uIGFueSBlcnJvciBidXQgY2xlYXJlZCBiZWZv\n" - "cmUgaXNzdWluZyBhDQo+ID4gPiA+ID4gY29tbWl0Lg0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IEkg\n" - "YWRkZWQgYSBzaW1pbGFyIGZsYWcgdG8gQ2VwaCBpbm9kZXMgcmVjZW50bHksIGJ1dCBvbmx5DQo+\n" - "ID4gPiA+ID4gY2xlYXIgaXQgd2hlbg0KPiA+ID4gPiA+IGEgd3JpdGUgc3VjY2VlZHMuIFdvdWxk\n" - "bid0IHRoYXQgbWFrZSBtb3JlIHNlbnNlIGhlcmUgYXMgd2VsbD8NCj4gPiA+ID4gDQo+ID4gPiA+\n" - "IEl0IGlzIGEgYml0IGhhcmQgdG8gd3JhcCBvbmUncyBtaW5kIGFyb3VuZC4NCj4gPiA+ID4gDQo+\n" - "ID4gPiA+IEluIHRoZSBvcmlnaW5hbCBjb2RlIChjb21taXQgN2IxNTlmYzE4ZDQxNzk4MCkgaXQg\n" - "bG9va3MgbGlrZToNCj4gPiA+ID4gIC0gdGVzdC1hbmQtY2xlYXIgYml0DQo+ID4gPiA+ICAtIHdy\n" - "aXRlIGFuZCBzeW5jDQo+ID4gPiA+ICAtIHRlc3QtYml0DQo+ID4gPiA+IA0KPiA+ID4gPiBUaGlz\n" - "IGRvZXMsIEkgdGhpbmssIHNlZW0gc2FmZXIgdGhhbiAiY2xlYXIgb24gc3VjY2Vzc2Z1bCB3cml0\n" - "ZSINCj4gPiA+ID4gYXMgdGhlDQo+ID4gPiA+IHdyaXRlcyBjb3VsZCBjb21wbGV0ZSBvdXQtb2Yt\n" - "b3JkZXIgYW5kIEkgd291bGRuJ3QgYmUgc3VycHJpc2VkDQo+ID4gPiA+IGlmIHRoZQ0KPiA+ID4g\n" - "PiB1bnN1Y2Nlc3NmdWwgb25lcyBjb21wbGV0ZWQgd2l0aCBhbiBlcnJvciBiZWZvcmUgdGhlIHN1\n" - "Y2Nlc3NmdWwNCj4gPiA+ID4gb25lIC0NCj4gPiA+ID4gcGFydGljdWxhcmx5IHdpdGggYW4gZXJy\n" - "b3IgbGlrZSBFRFFVT1QuDQo+ID4gPiA+IA0KPiA+ID4gPiBIb3dldmVyIHRoZSBjdXJyZW50IGNv\n" - "ZGUgZG9lcyB0aGUgd3JpdGVzIGJlZm9yZSB0aGUgdGVzdC1hbmQtDQo+ID4gPiA+IGNsZWFyLCBh\n" - "bmQNCj4gPiA+ID4gb25seSBkb2VzIHRoZSBjb21taXQgYWZ0ZXJ3YXJkcy4gIFRoYXQgbWFrZXMg\n" - "aXQgbGVzcyBjbGVhciB3aHkNCj4gPiA+ID4gdGhlDQo+ID4gPiA+IGN1cnJlbnQgc2VxdWVuY2Ug\n" - "aXMgYSBnb29kIGlkZWEuDQo+ID4gPiA+IA0KPiA+ID4gPiBIb3dldmVyIC4uLiBuZnNfZmlsZV9m\n" - "c3luY19jb21taXQoKSBpcyBvbmx5IGNhbGxlZCBpZg0KPiA+ID4gPiBmaWxlbWFwX3dyaXRlX2Fu\n" - "ZF93YWl0X3JhbmdlKCkgcmV0dXJuZWQgd2l0aCBzdWNjZXNzLCBzbyB3ZQ0KPiA+ID4gPiBvbmx5\n" - "IGNsZWFyDQo+ID4gPiA+IHRoZSBmbGFnIGFmdGVyIHN1Y2Nlc3NmdWwgd3JpdGVzKD8pLg0KPiA+\n" - "ID4gPiANCj4gPiA+ID4gT2guLi4uDQo+ID4gPiA+IFRoaXMgcGF0Y2ggZnJvbSBtZToNCj4gPiA+\n" - "ID4gDQo+ID4gPiA+IENvbW1pdDogMmVkYjZiYzM4NTJjICgiTkZTIC0gZml4IHJlY2VudCBicmVh\n" - "a2FnZSB0byBORlMgZXJyb3INCj4gPiA+ID4gaGFuZGxpbmcuIikNCj4gPiA+ID4gDQo+ID4gPiA+\n" - "IHNlZW1zIHRvIGhhdmUgYmVlbiByZXZlcnRlZCBieQ0KPiA+ID4gPiANCj4gPiA+ID4gQ29tbWl0\n" - "OiA3YjI4MWVlMDI2NTUgKCJORlM6IGZzeW5jKCkgbXVzdCBleGl0IHdpdGggYW4gZXJyb3IgaWYN\n" - "Cj4gPiA+ID4gcGFnZSB3cml0ZWJhY2sgZmFpbGVkIikNCj4gPiA+ID4gDQo+ID4gPiA+IHdoaWNo\n" - "IHByb2JhYmx5IGlzbid0IGdvb2QuICBJdCBhcHBlYXJzIHRoYXQgdGhpcyBjb2RlIGlzIHZlcnkN\n" - "Cj4gPiA+ID4gZnJhZ2lsZQ0KPiA+ID4gPiBhbmQgZWFzaWx5IGJyb2tlbi4NCj4gPiANCj4gPiBP\n" - "biBmdXJ0aGVyIGludmVzdGlnYXRpb24sIEkgdGhpbmsgdGhlIHByb2JsZW0gdGhhdCBJIGZpeGVk\n" - "IGFuZCB0aGVuDQo+ID4gd2UNCj4gPiByZWludHJvZHVjZWQgd2lsbCBiZSBmaXhlZCBhZ2FpbiAt\n" - "IG1vcmUgcGVybWFuZW50bHkgLSBieSB5b3VyDQo+ID4gcGF0Y2guDQo+ID4gVGhlIHJvb3QgcHJv\n" - "YmxlbSBpcyB0aGF0IG5mcyBrZWVwcyBlcnJvciBjb2RlcyBpbiBhIGRpZmZlcmVudCB3YXkNCj4g\n" - "PiB0byB0aGUNCj4gPiBNTSBjb3JlLiAgQnkgdW5pZnlpbmcgdGhvc2UsIHRoZSBwcm9ibGVtIGdv\n" - "ZXMuDQo+ID4gKFRoZSBzcGVjaWZpYyBwcm9ibGVtIGlzIHRoYXQgd3JpdGVzIHdoaWNoIGhpdCBF\n" - "RFFVT1Qgb24gdGhlIHNlcnZlcg0KPiA+IGNhbg0KPiA+ICByZXBvcnQgRUlPIG9uIHRoZSBjbGll\n" - "bnQpLg0KPiA+IA0KPiA+IA0KPiA+ID4gPiBNYXliZSB3ZSBuZWVkIHRvIHdvcmsgb3V0IGV4YWN0\n" - "bHkgd2hhdCBpcyByZXF1aXJlZCwgYW5kDQo+ID4gPiA+IGRvY3VtZW50IGl0IC0gc28NCj4gPiA+\n" - "ID4gd2UgY2FuIHN0b3AgYnJlYWtpbmcgaXQuDQo+ID4gPiA+IE9yIG1heWJlIHdlIG5lZWQgc29t\n" - "ZSB1bml0IHRlc3RzLi4uLi4NCj4gPiA+ID4gDQo+ID4gPiANCj4gPiA+IFllcywgbGF5aW5nIG91\n" - "dCB3aGF0J3MgbmVjZXNzYXJ5IGZvciB0aGlzIHdvdWxkIGJlIHZlcnkgaGVscGZ1bC4NCj4gPiA+\n" - "IFdlDQo+ID4gPiBjbGVhcmx5IHdhbnQgdG8gc2V0IHRoZSBmbGFnIHdoZW4gYW4gZXJyb3Igb2Nj\n" - "dXJzLiBVbmRlciB3aGF0DQo+ID4gPiBjaXJjdW1zdGFuY2VzIHNob3VsZCB3ZSBiZSBjbGVhcmlu\n" - "ZyBpdD8NCj4gPiANCj4gPiBXZWxsLi4uLiBsb29raW5nIGJhY2sgYXQgIDdiMTU5ZmMxOGQ0MTc5\n" - "ODBmNTdhZSB3aGljaCBpbnRyb2R1Y2VkDQo+ID4gdGhlDQo+ID4gZmxhZywgcHJpb3IgdG8gdGhh\n" - "dCB3cml0ZSBlcnJvcnMgKGN0eC0+ZXJyb3IpIHdlcmUgb25seSByZXBvcnRlZCBieQ0KPiA+IG5m\n" - "c19maWxlX2ZsdXNoIGFuZCBuZnNfZnN5bmMsIHNvIG9ubHkgb25lIGNsb3NlKCkgYW5kIGZzeW5j\n" - "KCkuDQo+ID4gDQo+ID4gQWZ0ZXIgdGhhdCBjb21taXQsIHNldHRpbmcgdGhlIGZsYWcgd291bGQg\n" - "bWVhbiB0aGF0IGVycm9ycyBjb3VsZCBiZQ0KPiA+IHJldHVybmVkIGJ5ICd3cml0ZScuICBTbyBj\n" - "bGVhcmluZyBhcyBwYXJ0IG9mIHJldHVybmluZyB0aGUgZXJyb3INCj4gPiBtYWtlcw0KPiA+IHBl\n" - "cmZlY3Qgc2Vuc2UuDQo+ID4gDQo+ID4gQXMgbG9uZyBhcyB0aGUgZXJyb3IgZ2V0cyByZWNvcmRl\n" - "ZCwgYW5kIGdldHMgcmV0dXJuZWQgd2hlbiBpdCBpcw0KPiA+IHJlY29yZGVkLCBpdCBkb2Vzbid0\n" - "IG11Y2ggbWF0dGVyIHdoZW4gdGhlIGZsYWcgaXMgY2xlYXJlZC4gIFdpdGgNCj4gPiB5b3VyDQo+\n" - "ID4gcGF0Y2hlcyB3ZSBkb24ndCBuZWVkIHRvIGZsYWcgYW55IG1vcmUgdG8gZ2V0IGVycm9ycyBy\n" - "ZWxpYWJseQ0KPiA+IHJlcG9ydGVkLg0KPiA+IA0KPiA+IExlYXZpbmcgdGhlIGZsYWcgc2V0IG1l\n" - "YW5zIHRoYXQgd3JpdGVzIGdvIG1vcmUgc2xvd2x5IC0gd2UgZG9uJ3QNCj4gPiBnZXQNCj4gPiBs\n" - "YXJnZSBxdWV1ZSBvZiBiYWNrZ3JvdW5kIHJpZ2h0cyBidWlsZGluZyB1cCBidXQgZGVzdGluZWQg\n" - "Zm9yDQo+ID4gZmFpbHVyZS4NCj4gPiBUaGlzIGlzIHRoZSBtYWluIHBvaW50IG1hZGUgaW4gdGhl\n" - "IGNvbW1lbnQgbWVzc2FnZSB3aGVuIHRoZSBmbGFnDQo+ID4gd2FzDQo+ID4gaW50cm9kdWNlZC4N\n" - "Cj4gPiBPZiBjb3Vyc2UsIGJ5IHRoZSB0aW1lIHdlIGZpcnN0IGdldCBhbiBlcnJvciB0aGVyZSBj\n" - "b3VsZCBhbHJlYWR5DQo+ID4gYnkgYSBsYXJnZSBxdWV1ZSwgc28gd2UgcHJvYmFibHkgd2FudCB0\n" - "aGF0IHRvIGRyYWluIGNvbXBsZXRlbHkNCj4gPiBiZWZvcmUNCj4gPiBhbGxvd2luZyBhc3luYyB3\n" - "cml0ZXMgYWdhaW4uDQoNCldlIGFscmVhZHkgaGF2ZSB0aGlzIGZ1bmN0aW9uYWxpdHkgaW1wbGVt\n" - "ZW50ZWQgaW4gdGhlIGV4aXN0aW5nIGNvZGUuDQoNCj4gPiANCj4gPiBJdCBtaWdodCBtYWtlIHNl\n" - "bnNlIHRvIGhhdmUgMiBmbGFncy4gIE9uZSB3aGljaCBzYXlzICJ3cml0ZXMgc2hvdWxkDQo+ID4g\n" - "YmUNCj4gPiBzeW5jaHJvbm91cyIsIGFub3RoZXIgdGhhdCBzYXlzICJUaGVyZSB3YXMgYW4gZXJy\n" - "b3IgcmVjZW50bHkiLg0KPiA+IFdlIGNsZWFyIHRoZSBlcnJvciBmbGFnIGJlZm9yZSBjYWxsaW5n\n" - "IG5mc19mc3luYywgYW5kIGlmIGl0IGlzDQo+ID4gc3RpbGwNCj4gPiBjbGVhciBhZnRlcndhcmRz\n" - "LCB3ZSBjbGVhciB0aGUgc3luYy13cml0ZXMgZmxhZy4gIE1heWJlIHRoYXQgaXMNCj4gPiBtb3Jl\n" - "DQo+ID4gY29tcGxleCB0aGFuIG5lZWRlZCB0aG91Z2guDQo+ID4gDQoNCldlIGFsc28gbmVlZCB0\n" - "byBwcmVzZXJ2ZSB0aGUgTkZTX0NPTlRFWFRfUkVTRU5EX1dSSVRFUyBmbGFnLiBJIGRvbid0DQpz\n" - "ZWUgYW55IGdsb2JhbCBtZWNoYW5pc20gdGhhdCB3aWxsIHJlcGxhY2UgdGhhdC4NCg0KPiA+IEkn\n" - "bSBsZWFuaW5nIHRvd2FyZHMgeW91ciBzdWdnZXN0aW9uIHRoYXQgaXQgZG9lc24ndCBtYXR0ZXIg\n" - "dmVyeQ0KPiA+IG11Y2gNCj4gPiB3aGVuIGl0IGdldHMgY2xlYXJlZCwgYW5kIGNsZWFyaW5nIGl0\n" - "IG9uIGFueSBzdWNjZXNzZnVsIHdyaXRlIGlzDQo+ID4gc2ltcGxlc3QuDQo+ID4gDQo+ID4gU28g\n" - "SSdtIHN0aWxsIGluIGZhdm9yIG9mIHVzaW5nIG5mc19jb250ZXh0X3NldF93cml0ZV9lcnJvcigp\n" - "IGluDQo+ID4gbmZzX3BhZ2Vpb19hZGRfcmVxdWVzdCgpLCBwcmltYXJpbHkgYmVjYXVzZSBpdCBp\n" - "cyBtb3N0IGNvbnNpc3RlbnQgLQ0KPiA+IHdlDQo+ID4gZG9uJ3QgbmVlZCBleGNlcHRpb25zLg0K\n" - "PiANCj4gVGhhbmtzIGZvciB0YWtpbmcgYSBjbG9zZXIgbG9vay4gSSBjYW4gZWFzaWx5IG1ha2Ug\n" - "dGhlIGNoYW5nZSBhYm92ZSwNCj4gYW5kDQo+IEkgZG8gdGhpbmsgdGhhdCBrZWVwaW5nIHRoaXMg\n" - "bWVjaGFuaXNtIGFzIHNpbXBsZSBhcyBwb3NzaWJsZSB3aWxsDQo+IG1ha2UNCj4gaXQgZWFzaWVy\n" - "IHRvIHByZXZlbnQgYml0cm90Lg0KPiANCj4gVGhhdCBzYWlkLi4uIE5GU19DT05URVhUX0VSUk9S\n" - "X1dSSVRFIGlzIGEgcGVyIGN0eCBmbGFnLCBhbmQgdGhlIGN0eA0KPiBpcyBhDQo+IHBlciBvcGVu\n" - "IGZpbGUgZGVzY3JpcHRpb24gb2JqZWN0Lg0KPiANCj4gSXMgdGhhdCB0aGUgY29ycmVjdCB3YXkg\n" - "dG8gdHJhY2sgdGhpcz8gQWxsIG9mIHRoZSBjdHgncyB3aWxsIHNoYXJlDQo+IHRoZQ0KPiBzYW1l\n" - "IGlub2RlLiBJZiB3ZSdyZSBnZXR0aW5nIHdyaXRlYmFjayBlcnJvcnMgZm9yIG9uZSBjb250ZXh0\n" - "LCBpdCdzDQo+IHF1aXRlIGxpa2VseSB0aGF0IHdlJ2xsIGJlIHNlZWluZyB0aGVtIHZpYSBvdGhl\n" - "cnMuDQo+IA0KPiBJIHN1cHBvc2UgdGhlIGNvdW50ZXJhcmd1bWVudCBpcyB3aGVuIHdlIGhhdmUg\n" - "dGhpbmdzIGxpa2UgZXhwaXJpbmcNCj4ga3JiNQ0KPiB0aWNrZXRzLiBXcml0ZSBmYWlsdXJlcyB2\n" - "aWEgYW4gZXhwaXJpbmcgc2V0IG9mIGNyZWRzIG1heSBoYXZlIG5vDQo+IGVmZmVjdA0KPiBvbiB3\n" - "cml0ZWJhY2sgdmlhIG90aGVyIGNyZWRzLg0KPiANCj4gU3RpbGwsIEkgdGhpbmsgYSBwZXItaW5v\n" - "ZGUgZmxhZyBtaWdodCBtYWtlIG1vcmUgc2Vuc2UgaGVyZS4NCj4gDQo+IFRob3VnaHRzPw0KDQpB\n" - "cyBmYXIgYXMgSSdtIGNvbmNlcm5lZCwgdGhhdCB3b3VsZCBiZSBhIHJlZ3Jlc3Npb24uIFRoZSBt\n" - "b3N0IGNvbW1vbg0KcHJvYmxlbSB3aGVuIGZsdXNoaW5nIHdyaXRlYmFjayBkYXRhIHRvIHRoZSBz\n" - "ZXJ2ZXIgYXNpZGUgZnJvbSBFTk9TUEMNCihhbmQgcG9zc2libHkgRVNUQUxFKSBpcyBFQUNDRVMs\n" - "IHdoaWNoIGlzIHBhcnRpY3VsYXIgdG8gdGhlIGZpbGUNCmRlc2NyaXB0b3IgdGhhdCBvcGVuZWQg\n" - "dGhlIGZpbGUuDQoNCkZpbGUgY29udGV4dHMsIGFuZCBORlNfQ09OVEVYVF9FUlJPUl9XUklURSBz\n" - "b2x2ZSB0aGF0IHByb2JsZW0gYnkgYmVpbmcNCnByaXZhdGUgdG8gdGhlIGZpbGUgZGVzY3JpcHRv\n" - "ci4NCg0KLS0gDQpUcm9uZCBNeWtsZWJ1c3QNCkxpbnV4IE5GUyBjbGllbnQgbWFpbnRhaW5lciwg\n" - UHJpbWFyeURhdGENCnRyb25kLm15a2xlYnVzdEBwcmltYXJ5ZGF0YS5jb20NCg== + "On Thu, 2017-09-07 at 07:35 -0400, Jeff Layton wrote:\n" + "> On Thu, 2017-09-07 at 13:37 +1000, NeilBrown wrote:\n" + "> > On Tue, Aug 29 2017, Jeff Layton wrote:\n" + "> > \n" + "> > > On Tue, 2017-08-29 at 11:23 +1000, NeilBrown wrote:\n" + "> > > > On Mon, Aug 28 2017, Jeff Layton wrote:\n" + "> > > > \n" + "> > > > > On Mon, 2017-08-28 at 09:24 +1000, NeilBrown wrote:\n" + "> > > > > > On Fri, Aug 25 2017, Jeff Layton wrote:\n" + "> > > > > > \n" + "> > > > > > > On Thu, 2017-07-20 at 15:42 -0400, Jeff Layton wrote:\n" + "> > > > > > > > From: Jeff Layton <jlayton@redhat.com>\n" + "> > > > > > > > \n" + "> > > > > > > > There is some ambiguity in nfs about how writeback\n" + "> > > > > > > > errors are\n" + "> > > > > > > > tracked.\n" + "> > > > > > > > \n" + "> > > > > > > > For instance, nfs_pageio_add_request calls\n" + "> > > > > > > > mapping_set_error when\n" + "> > > > > > > > the\n" + "> > > > > > > > add fails, but we track errors that occur after adding\n" + "> > > > > > > > the\n" + "> > > > > > > > request\n" + "> > > > > > > > with a dedicated int error in the open context.\n" + "> > > > > > > > \n" + "> > > > > > > > Now that we have better infrastructure for the vfs\n" + "> > > > > > > > layer, this\n" + "> > > > > > > > latter int is now unnecessary. Just have\n" + "> > > > > > > > nfs_context_set_write_error set\n" + "> > > > > > > > the error in the mapping when one occurs.\n" + "> > > > > > > > \n" + "> > > > > > > > Have NFS use file_write_and_wait_range to initiate and\n" + "> > > > > > > > wait on\n" + "> > > > > > > > writeback\n" + "> > > > > > > > of the data, and then check again after issuing the\n" + "> > > > > > > > commit(s).\n" + "> > > > > > > > \n" + "> > > > > > > > With this, we also don't need to pay attention to the\n" + "> > > > > > > > ERROR_WRITE\n" + "> > > > > > > > flag for reporting, and just clear it to indicate to\n" + "> > > > > > > > subsequent\n" + "> > > > > > > > writers that they should try to go asynchronous again.\n" + "> > > > > > > > \n" + "> > > > > > > > In nfs_page_async_flush, sample the error before\n" + "> > > > > > > > locking and\n" + "> > > > > > > > joining\n" + "> > > > > > > > the requests, and check for errors since that point.\n" + "> > > > > > > > \n" + "> > > > > > > > Signed-off-by: Jeff Layton <jlayton@redhat.com>\n" + "> > > > > > > > ---\n" + "> > > > > > > > fs/nfs/file.c | 24 +++++++++++-------------\n" + "> > > > > > > > fs/nfs/inode.c | 3 +--\n" + "> > > > > > > > fs/nfs/write.c | 8 ++++++--\n" + "> > > > > > > > include/linux/nfs_fs.h | 1 -\n" + "> > > > > > > > 4 files changed, 18 insertions(+), 18 deletions(-)\n" + "> > > > > > > > \n" + "> > > > > > > > I have a baling wire and duct tape solution for testing\n" + "> > > > > > > > this with\n" + "> > > > > > > > xfstests (using iptables REJECT targets and soft\n" + "> > > > > > > > mounts). This\n" + "> > > > > > > > seems to\n" + "> > > > > > > > make nfs do the right thing.\n" + "> > > > > > > > \n" + "> > > > > > > > diff --git a/fs/nfs/file.c b/fs/nfs/file.c\n" + "> > > > > > > > index 5713eb32a45e..15d3c6faafd3 100644\n" + "> > > > > > > > --- a/fs/nfs/file.c\n" + "> > > > > > > > +++ b/fs/nfs/file.c\n" + "> > > > > > > > @@ -212,25 +212,23 @@ nfs_file_fsync_commit(struct file\n" + "> > > > > > > > *file,\n" + "> > > > > > > > loff_t start, loff_t end, int datasync)\n" + "> > > > > > > > {\n" + "> > > > > > > > \tstruct nfs_open_context *ctx =\n" + "> > > > > > > > nfs_file_open_context(file);\n" + "> > > > > > > > \tstruct inode *inode = file_inode(file);\n" + "> > > > > > > > -\tint have_error, do_resend, status;\n" + "> > > > > > > > -\tint ret = 0;\n" + "> > > > > > > > +\tint do_resend, status;\n" + "> > > > > > > > +\tint ret;\n" + "> > > > > > > > \n" + "> > > > > > > > \tdprintk(\"NFS: fsync file(%pD2) datasync %d\\n\",\n" + "> > > > > > > > file,\n" + "> > > > > > > > datasync);\n" + "> > > > > > > > \n" + "> > > > > > > > \tnfs_inc_stats(inode, NFSIOS_VFSFSYNC);\n" + "> > > > > > > > \tdo_resend =\n" + "> > > > > > > > test_and_clear_bit(NFS_CONTEXT_RESEND_WRITES, &ctx-\n" + "> > > > > > > > >flags);\n" + "> > > > > > > > -\thave_error =\n" + "> > > > > > > > test_and_clear_bit(NFS_CONTEXT_ERROR_WRITE,\n" + "> > > > > > > > &ctx->flags);\n" + "> > > > > > > > -\tstatus = nfs_commit_inode(inode, FLUSH_SYNC);\n" + "> > > > > > > > -\thave_error |=\n" + "> > > > > > > > test_bit(NFS_CONTEXT_ERROR_WRITE, &ctx-\n" + "> > > > > > > > > flags);\n" + "> > > > > > > > \n" + "> > > > > > > > -\tif (have_error) {\n" + "> > > > > > > > -\t\tret = xchg(&ctx->error, 0);\n" + "> > > > > > > > -\t\tif (ret)\n" + "> > > > > > > > -\t\t\tgoto out;\n" + "> > > > > > > > -\t}\n" + "> > > > > > > > -\tif (status < 0) {\n" + "> > > > > > > > +\tclear_bit(NFS_CONTEXT_ERROR_WRITE, &ctx-\n" + "> > > > > > > > >flags);\n" + "> > > > > > > > +\tret = nfs_commit_inode(inode, FLUSH_SYNC);\n" + "> > > > > > > > +\n" + "> > > > > > > > +\t/* Recheck and advance after the commit */\n" + "> > > > > > > > +\tstatus = file_check_and_advance_wb_err(file);\n" + "> > > > > > \n" + "> > > > > > This change makes the code inconsistent with the comment\n" + "> > > > > > above the\n" + "> > > > > > function, which still references ctx->error. The intent of\n" + "> > > > > > the\n" + "> > > > > > comment\n" + "> > > > > > is still correct, but the details have changed.\n" + "> > > > > > \n" + "> > > > > \n" + "> > > > > Good catch. I'll fix that up in a respin.\n" + "> > > > > \n" + "> > > > > > Also, there is a call to mapping_set_error() in\n" + "> > > > > > nfs_pageio_add_request().\n" + "> > > > > > I wonder if that should be changed to\n" + "> > > > > > nfs_context_set_write_error(req->wb_context, desc-\n" + "> > > > > > >pg_error)\n" + "> > > > > > ??\n" + "> > > > > > \n" + "> > > > > \n" + "> > > > > Trickier question...\n" + "> > > > > \n" + "> > > > > I'm not quite sure what semantics we're looking for with\n" + "> > > > > NFS_CONTEXT_ERROR_WRITE. I know that it forces writes to be\n" + "> > > > > synchronous, but I'm not quite sure why it gets cleared the\n" + "> > > > > way it\n" + "> > > > > does. It's set on any error but cleared before issuing a\n" + "> > > > > commit.\n" + "> > > > > \n" + "> > > > > I added a similar flag to Ceph inodes recently, but only\n" + "> > > > > clear it when\n" + "> > > > > a write succeeds. Wouldn't that make more sense here as well?\n" + "> > > > \n" + "> > > > It is a bit hard to wrap one's mind around.\n" + "> > > > \n" + "> > > > In the original code (commit 7b159fc18d417980) it looks like:\n" + "> > > > - test-and-clear bit\n" + "> > > > - write and sync\n" + "> > > > - test-bit\n" + "> > > > \n" + "> > > > This does, I think, seem safer than \"clear on successful write\"\n" + "> > > > as the\n" + "> > > > writes could complete out-of-order and I wouldn't be surprised\n" + "> > > > if the\n" + "> > > > unsuccessful ones completed with an error before the successful\n" + "> > > > one -\n" + "> > > > particularly with an error like EDQUOT.\n" + "> > > > \n" + "> > > > However the current code does the writes before the test-and-\n" + "> > > > clear, and\n" + "> > > > only does the commit afterwards. That makes it less clear why\n" + "> > > > the\n" + "> > > > current sequence is a good idea.\n" + "> > > > \n" + "> > > > However ... nfs_file_fsync_commit() is only called if\n" + "> > > > filemap_write_and_wait_range() returned with success, so we\n" + "> > > > only clear\n" + "> > > > the flag after successful writes(?).\n" + "> > > > \n" + "> > > > Oh....\n" + "> > > > This patch from me:\n" + "> > > > \n" + "> > > > Commit: 2edb6bc3852c (\"NFS - fix recent breakage to NFS error\n" + "> > > > handling.\")\n" + "> > > > \n" + "> > > > seems to have been reverted by\n" + "> > > > \n" + "> > > > Commit: 7b281ee02655 (\"NFS: fsync() must exit with an error if\n" + "> > > > page writeback failed\")\n" + "> > > > \n" + "> > > > which probably isn't good. It appears that this code is very\n" + "> > > > fragile\n" + "> > > > and easily broken.\n" + "> > \n" + "> > On further investigation, I think the problem that I fixed and then\n" + "> > we\n" + "> > reintroduced will be fixed again - more permanently - by your\n" + "> > patch.\n" + "> > The root problem is that nfs keeps error codes in a different way\n" + "> > to the\n" + "> > MM core. By unifying those, the problem goes.\n" + "> > (The specific problem is that writes which hit EDQUOT on the server\n" + "> > can\n" + "> > report EIO on the client).\n" + "> > \n" + "> > \n" + "> > > > Maybe we need to work out exactly what is required, and\n" + "> > > > document it - so\n" + "> > > > we can stop breaking it.\n" + "> > > > Or maybe we need some unit tests.....\n" + "> > > > \n" + "> > > \n" + "> > > Yes, laying out what's necessary for this would be very helpful.\n" + "> > > We\n" + "> > > clearly want to set the flag when an error occurs. Under what\n" + "> > > circumstances should we be clearing it?\n" + "> > \n" + "> > Well.... looking back at 7b159fc18d417980f57ae which introduced\n" + "> > the\n" + "> > flag, prior to that write errors (ctx->error) were only reported by\n" + "> > nfs_file_flush and nfs_fsync, so only one close() and fsync().\n" + "> > \n" + "> > After that commit, setting the flag would mean that errors could be\n" + "> > returned by 'write'. So clearing as part of returning the error\n" + "> > makes\n" + "> > perfect sense.\n" + "> > \n" + "> > As long as the error gets recorded, and gets returned when it is\n" + "> > recorded, it doesn't much matter when the flag is cleared. With\n" + "> > your\n" + "> > patches we don't need to flag any more to get errors reliably\n" + "> > reported.\n" + "> > \n" + "> > Leaving the flag set means that writes go more slowly - we don't\n" + "> > get\n" + "> > large queue of background rights building up but destined for\n" + "> > failure.\n" + "> > This is the main point made in the comment message when the flag\n" + "> > was\n" + "> > introduced.\n" + "> > Of course, by the time we first get an error there could already\n" + "> > by a large queue, so we probably want that to drain completely\n" + "> > before\n" + "> > allowing async writes again.\n" + "\n" + "We already have this functionality implemented in the existing code.\n" + "\n" + "> > \n" + "> > It might make sense to have 2 flags. One which says \"writes should\n" + "> > be\n" + "> > synchronous\", another that says \"There was an error recently\".\n" + "> > We clear the error flag before calling nfs_fsync, and if it is\n" + "> > still\n" + "> > clear afterwards, we clear the sync-writes flag. Maybe that is\n" + "> > more\n" + "> > complex than needed though.\n" + "> > \n" + "\n" + "We also need to preserve the NFS_CONTEXT_RESEND_WRITES flag. I don't\n" + "see any global mechanism that will replace that.\n" + "\n" + "> > I'm leaning towards your suggestion that it doesn't matter very\n" + "> > much\n" + "> > when it gets cleared, and clearing it on any successful write is\n" + "> > simplest.\n" + "> > \n" + "> > So I'm still in favor of using nfs_context_set_write_error() in\n" + "> > nfs_pageio_add_request(), primarily because it is most consistent -\n" + "> > we\n" + "> > don't need exceptions.\n" + "> \n" + "> Thanks for taking a closer look. I can easily make the change above,\n" + "> and\n" + "> I do think that keeping this mechanism as simple as possible will\n" + "> make\n" + "> it easier to prevent bitrot.\n" + "> \n" + "> That said... NFS_CONTEXT_ERROR_WRITE is a per ctx flag, and the ctx\n" + "> is a\n" + "> per open file description object.\n" + "> \n" + "> Is that the correct way to track this? All of the ctx's will share\n" + "> the\n" + "> same inode. If we're getting writeback errors for one context, it's\n" + "> quite likely that we'll be seeing them via others.\n" + "> \n" + "> I suppose the counterargument is when we have things like expiring\n" + "> krb5\n" + "> tickets. Write failures via an expiring set of creds may have no\n" + "> effect\n" + "> on writeback via other creds.\n" + "> \n" + "> Still, I think a per-inode flag might make more sense here.\n" + "> \n" + "> Thoughts?\n" + "\n" + "As far as I'm concerned, that would be a regression. The most common\n" + "problem when flushing writeback data to the server aside from ENOSPC\n" + "(and possibly ESTALE) is EACCES, which is particular to the file\n" + "descriptor that opened the file.\n" + "\n" + "File contexts, and NFS_CONTEXT_ERROR_WRITE solve that problem by being\n" + "private to the file descriptor.\n" + "\n" + "-- \n" + "Trond Myklebust\n" + "Linux NFS client maintainer, PrimaryData\n" + trond.myklebust@primarydata.com -386d4ee7f04b7939bc78c69bf2bbb79e8378814bd9b60e4206ca17cc9f64485d +9313fc1e197b4dc155530691c549784cba5132e3dbdc5b248ec75dc47d96d6ce
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.