From: Trond Myklebust <trondmy@primarydata.com>
To: "mhocko@kernel.org" <mhocko@kernel.org>
Cc: "torvalds@linux-foundation.org" <torvalds@linux-foundation.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
"n.borisov.lkml@gmail.com" <n.borisov.lkml@gmail.com>
Subject: Re: [GIT PULL] Please pull NFS client fixes for 4.12
Date: Thu, 11 May 2017 13:10:03 +0000 [thread overview]
Message-ID: <1494508201.3207.5.camel@primarydata.com> (raw)
In-Reply-To: <20170511125612.GK26782@dhcp22.suse.cz>
T24gVGh1LCAyMDE3LTA1LTExIGF0IDE0OjU2ICswMjAwLCBNaWNoYWwgSG9ja28gd3JvdGU6DQo+
IE9uIFRodSAxMS0wNS0xNyAxMjo0NTowMCwgVHJvbmQgTXlrbGVidXN0IHdyb3RlOg0KPiA+IE9u
IFRodSwgMjAxNy0wNS0xMSBhdCAxNDoyNiArMDIwMCwgTWljaGFsIEhvY2tvIHdyb3RlOg0KPiA+
ID4gT24gVGh1IDExLTA1LTE3IDEyOjE2OjM3LCBUcm9uZCBNeWtsZWJ1c3Qgd3JvdGU6DQo+ID4g
PiA+IE9uIFRodSwgMjAxNy0wNS0xMSBhdCAwOTo1OSArMDIwMCwgTWljaGFsIEhvY2tvIHdyb3Rl
Og0KPiA+ID4gPiA+IE9uIFRodSAxMS0wNS0xNyAxMDo1MzoyNywgTmlrb2xheSBCb3Jpc292IHdy
b3RlOg0KPiA+ID4gPiA+ID4gDQo+ID4gPiA+ID4gPiANCj4gPiA+ID4gPiA+IE9uIDEwLjA1LjIw
MTcgMTk6NDcsIFRyb25kIE15a2xlYnVzdCB3cm90ZToNCj4gPiA+ID4gPiANCj4gPiA+ID4gPiBb
Li4uXQ0KPiA+ID4gPiA+ID4gPiAtIENsZWFudXAgYW5kIHJlbW92YWwgb2Ygc29tZSBtZW1vcnkg
ZmFpbHVyZSBwYXRocyBub3cNCj4gPiA+ID4gPiA+ID4gdGhhdA0KPiA+ID4gPiA+ID4gPiDCoCBH
RlBfTk9GUyBpcyBndWFyYW50ZWVkIHRvIG5ldmVyIGZhaWwuDQo+ID4gPiA+ID4gPiANCj4gPiA+
ID4gPiA+IFdoYXQgZ3VhcmFudGVlcyB0aGF0PyBTaW5jZSBpZiB0aGlzIGlzIHRoZSBjYXNlIHRo
ZW4gdGhpcw0KPiA+ID4gPiA+ID4gY2FuDQo+ID4gPiA+ID4gPiByZXN1bHQgaW4NCj4gPiA+ID4g
PiA+IGEgbG90IG9mIG9wcG9ydHVuaXRpZXMgZm9yIGNsZWFudXAgYWNyb3NzIHRoZSB3aG9sZSBr
ZXJuZWwNCj4gPiA+ID4gPiA+IHRyZWUuDQo+ID4gPiA+ID4gPiBBZnRlcg0KPiA+ID4gPiA+ID4g
ZGlzY3Vzc2luZyB3aXRoIG1ob2NrbyAoY2MnZWQpIGl0IHNlZW1zIHRoYXQgaW4gcHJhY3RpY2UN
Cj4gPiA+ID4gPiA+IGV2ZXJ5dGhpbmcNCj4gPiA+ID4gPiA+IGJlbG93IENPU1RMWV9PUkRFUiB3
aGljaCBhcmUgbm90IEdGUF9OT1JFVFJZIHdpbGwgbmV2ZXINCj4gPiA+ID4gPiA+IGZhaWwuDQo+
ID4gPiA+ID4gPiBCdXQNCj4gPiA+ID4gPiA+IHRoaXMNCj4gPiA+ID4gPiA+IHNlbWFudGljIGlz
IG5vdCB0aGUgc2FtZSBhcyBHRlBfTk9GQUlMLiBFLmcuIG5vdGhpbmcNCj4gPiA+ID4gPiA+IGd1
YXJhbnRlZXMNCj4gPiA+ID4gPiA+IHRoYXQNCj4gPiA+ID4gPiA+IHRoaXMgd2lsbCBzdGF5IGxp
a2UgdGhhdCBpbiB0aGUgZnV0dXJlPw0KPiA+ID4gPiA+IA0KPiA+ID4gPiA+IEluIHByYWN0aWNl
IGl0IGlzIGhhcmQgdG8gY2hhbmdlIHRoZSBzZW1hbnRpYyBvZiBzbWFsbA0KPiA+ID4gPiA+IGFs
bG9jYXRpb25zDQo+ID4gPiA+ID4gbmV2ZXINCj4gPiA+ID4gPiBmYWlsIF9wcmFjdGljYWxseV8u
IEJ1dCB0aGlzIGlzIGFic29sdXRlbHkgbm90IGd1YXJhbnRlZWQhDQo+ID4gPiA+ID4gVGhleQ0K
PiA+ID4gPiA+IGNhbg0KPiA+ID4gPiA+IGZhaWwNCj4gPiA+ID4gPiBlLmcuIHdoZW4gdGhlIGFs
bG9jYXRpb24gY29udGV4dCBpcyB0aGUgb29tIHZpY3RpbS4gUmVtb3ZpbmcNCj4gPiA+ID4gPiBl
cnJvcg0KPiA+ID4gPiA+IHBhdGhzDQo+ID4gPiA+ID4gZm9yIGFsbG9jYXRpb24gZmFpbHVyZXMg
aXMganVzdCB3cm9uZy4NCj4gPiA+ID4gDQo+ID4gPiA+IE9LLCB0aGlzIG1ha2VzIG5vIGZ1Y2tp
bmcgc2Vuc2UgYXQgYWxsLg0KPiA+ID4gPiANCj4gPiA+ID4gRWl0aGVyIGFsbG9jYXRpb25zIGNh
biBmYWlsIG9yIHRoZXkgY2FuJ3QuDQo+ID4gPiA+IDEpIElmIHRoZXkgY2FuJ3QgZmFpbCwgdGhl
biB3ZSBkb24ndCBuZWVkIHRoZSBjaGVja3MuDQo+ID4gPiA+IDIpIElmIHRoZXkgY2FuIGZhaWws
IHRoZW4gd2UgZG8gbmVlZCB0aGVtLCBhbmQgdGhpcyBoYW5kDQo+ID4gPiA+IHdyaW5naW5nDQo+
ID4gPiA+IGluDQo+ID4gPiA+IHRoZSBNTSBjb21tdW5pdHkgYWJvdXQgR0ZQXyogc2VtYW50aWNz
IGFuZCBob3cgd2UgbmVlZCB0bw0KPiA+ID4gPiBwcmV2ZW50DQo+ID4gPiA+IGZhaWx1cmUgaXMg
ZnVja2luZyBwb2ludGxlc3MuDQo+ID4gPiANCj4gPiA+IGV2ZXJ5dGhpbmcgd2hpY2ggaXMgbm90
IF9fR0ZQX05PRkFJTCBtaWdodCBmYWlsLiBXZSB0cnkgaGFyZCBub3QNCj4gPiA+IHRvDQo+ID4g
PiBmYWlsDQo+ID4gPiBzbWFsbCBhbGxvY2F0aW9ucyByZXF1ZXN0cyBhcyBtdWNoIGFzIHdlIGNh
biBpbiBnZW5lcmFsIGJ1dCB5b3UNCj4gPiA+IF9oYXZlXyB0bw0KPiA+ID4gY2hlY2sgZm9yIGZh
aWx1cmVzLiBUaGVyZSBpcyBzaW1wbHkgbm8gd2F5IHRvIGd1YXJhbnRlZSAibmV2ZXINCj4gPiA+
IGZhaWwiDQo+ID4gPiBzZW1hbnRpYyBmb3IgYWxsIGFsbG9jYXRpb24gcmVxdWVzdHMuIFRoaXMg
aGFzIGJlZW4gbGlrZSB0aGF0DQo+ID4gPiBiYXNpY2FsbHkNCj4gPiA+IHNpbmNlIHllYXJzLiBB
bmQgZXZlbiB0aGlzIHRyeS10by1iZS1ub2ZhaWxpbmcgZm9yIHNtYWxsDQo+ID4gPiBhbGxvY2F0
aW9ucw0KPiA+ID4gaGFzDQo+ID4gPiBiZWVuIFBJVEEgZm9yIHNvbWUgY29ybmVyIGNhc2VzLg0K
PiA+IA0KPiA+IEknbGwgdGFrZSB0aGF0IGFzIGEgdm90ZSBmb3IgKDIpLCB0aGVuLg0KPiA+IA0K
PiA+IEkga25vdyB0aGF0IGZhaWx1cmVzIGNvdWxkIG9jY3VyIGluIHRoZSBwYXN0LiBUaGF0J3Mg
d2h5IHRob3NlIGNvZGUNCj4gPiBwYXRocyB3ZXJlIHRoZXJlLiBUaGUgcHJvYmxlbSBpcyB0aGF0
IHRoZSBNTSBjb21tdW5pdHkgaGFzIGJlZW4NCj4gPiBtYWtpbmcNCj4gPiBsb3RzIG9mIG5vaXNl
IG9uIG1haWxpbmcgbGlzdHMsIGNvbmZlcmVuY2VzIGFuZCBMV04gYXJ0aWNsZXMgYWJvdXQNCj4g
PiBob3cNCj4gPiB3ZSBtdXN0IG5vdCBmYWlsIHNtYWxsIGFsbG9jYXRpb25zIGJlY2F1c2UgdGhl
IE1NIGNvbW11bml0eQ0KPiA+IGJlbGlldmVzDQo+ID4gdGhhdCBub2JvZHkgZXhwZWN0cyBpdC4g
VGhpcyBpcyBjb25mdXNpbmcgZXZlcnlvbmUuLi4NCj4gDQo+IEl0IHdhcyBleGFjdGx5IG90aGVy
IHdheSBhcm91bmQuIFdlIHdvdWxkIGxpa2UgdG8gX2dldF9yaWRfb2ZfIHRoaXMNCj4gZG8NCj4g
bm90IGZhaWwgYmVoYXZpb3IgYmVjYXVzZSBpdCBpcyBjYXVzaW5nIGEgbWFqb3IgaGVhZGFjaGVz
IGluIG91dCBvZg0KPiBtZW1vcnkgY29ybmVyIGNhc2VzLiBKdXN0IHRha2UgR0ZQX05PRlMgYXMg
YW4gZXhhbXBsZS4gSXQgaXMgYSB3ZWFrDQo+IHJlY2xhaW0gY29udGV4dCBiZWNhdXNlIHdlIGNh
bm5vdCByZWNsYWltIGZzIG1ldGFkYXRhIGFuZCB0aGF0IG1pZ2h0DQo+IGJlDQo+IGEgbG90IG9m
IG1lbW9yeSBzbyB3ZSBjYW5ub3QgdHJpZ2dlciB0aGUgT09NIGtpbGxlciBhbmQgaGF2ZSB0byBy
ZWx5DQo+IG9uDQo+IGEgZGlmZmVyZW50IGFsbG9jYXRpb24gY29udGV4dCBvciBrc3dhcGQgdG8g
bWFrZSBhIHByb2dyZXNzIG9uIG91cg0KPiBiZWhhbGYuIFdlIHdvdWxkIHJlYWxseSBsaWtlIHRv
IGZhaWwgdGhvc2UgcmVxdWVzdHMgaW5zdGVhZC4gSSd2ZQ0KPiB0cmllZA0KPiB0aGF0IGluIHRo
ZSBwYXN0IGJ1dCBpdCB3YXMgZGVlbWVkIHRvIGRhbmdlcm91cyBiZWNhdXNlIF9hbGxfIGtlcm5l
bA0KPiBwYXRocyB3b3VsZCBoYXZlIHRvIGJlIGNoZWNrZWQgZm9yIGEgc2FuZSBmYWlsdXJlIGJl
aGF2aW9yLiBTbyB3ZSBhcmUNCj4ga2VlcGluZyBzdGF0dXMgcXVvIGluc3RlYWQuDQoNCklmIHdl
IHN1c3BlY3QgdGhlIGV4aXN0ZW5jZSBvZiBhIGxvYWQgb2YgcG90ZW50aWFsIHRpbWUgYm9tYnMg
aW4gdGhlDQprZXJuZWwgZHVlIHRvIG1pc3NpbmcgY2hlY2tzLCB0aGVuIHRoZSBzdGF0dXMgcXVv
IGlzIG5vdCBnb29kIGVub3VnaC4NCldlIHNob3VsZCBiZSB3b3JraW5nIG9uIHRvb2xzIHRvIGlk
ZW50aWZ5IHRoZXNlIGNvZGUgcGF0aHMuDQoNClF1aXRlIGZyYW5rbHksIEknZCBsb3ZlIHRvIHNl
ZSBhIGZ1enplci1saWtlIHRvb2wgdGhhdCBjYW4gcmFuZG9tbHkNCmZhaWwgYWxsb2NhdGlvbnMu
IEkgY2FuIGVhc2lseSBtYWtlIG9uZSBmb3IgdGhlIE5GUyBjb2RlLCBidXQgaWYgdGhlcmUNCmlz
IGEgZ2VuZXJhbCBwcm9ibGVtIGlkZW50aWZ5aW5nIGJ1Z2d5IGNvZGUsIHRoZW4gcGVyaGFwcyBp
dCBzaG91bGQgYmUNCnNvbHZlZCBhdCB0aGUgTU0gbGF5ZXIgaXRzZWxmLg0KDQo+ID4gSXQgY29u
ZnVzZWQgTmVpbCBCcm93biwgd2hvIGNvbnRyaWJ1dGVkIHRoZXNlIHBhdGNoZXMsIGFuZCBpdA0K
PiA+IGNvbmZ1c2VkDQo+ID4gbWUgYW5kIGFsbCB0aGUgb3RoZXIgcmV2aWV3ZXJzIG9mIHRoZXNl
IHBhdGNoZXMgb24gdGhlIGxpbnV4LW5mcw0KPiA+IG1haWxpbmcgbGlzdC4NCj4gPiANCj4gPiBT
byBpZiBpbmRlZWQgKDIpIGlzIGNvcnJlY3QsIHRoZW4gcGxlYXNlIGNhbiB3ZSBoYXZlIGEgY2xl
YXINCj4gPiBzdGF0ZW1lbnQNCj4gPiBfd2hlbiBkaXNjdXNzaW5nIGltcHJvdmVtZW50cyB0byBt
ZW1vcnkgYWxsb2NhdGlvbiBzZW1hbnRpY3NfIHRoYXQNCj4gPiBHRlBfKiBzdGlsbCBjYW4gZmFp
bCwgc3RpbGwgd2lsbCBmYWlsLCBhbmQgdGhhdCBjYWxsZXJzIHNob3VsZA0KPiA+IGFzc3VtZQ0K
PiA+IGl0IHdpbGwgZmFpbCBhbmQgc2hvdWxkIHRlc3QgdGhlaXIgY29kZSBwYXRocyBhc3N1bWlu
ZyB0aGUgZmFpbHVyZQ0KPiA+IGNhc2UuDQo+IA0KPiBJIGRvIG5vdCBzZWUgYW55IGV4cGxpY2l0
IGRvY3VtZW50YXRpb24gd2hpY2ggd291bGQgZW5jb3VyYWdlIHVzZXJzDQo+IHRvDQo+IG5vdCBj
aGVjayBmb3IgdGhlIGFsbG9jYXRpb24gZmFpbHVyZS4gT25seSBfX0dGUF9OT0ZBSUwgaXMgZG9j
dW1lbnRlZA0KPiBpdA0KPiBfbXVzdF8gcmV0cnkgZm9yIGV2ZXIuIE9mIGNvdXJzZSBJIGFtIG9w
ZW4gZm9yIGFueSBkb2N1bWVudGF0aW9uDQo+IGltcHJvdmVtZW50cy4NCg0KQXMgSSBzYWlkLCB0
aGUgcHJvYmxlbSBoYXMgYmVlbiB0aGUgZGlzY3Vzc2lvbiwgYW5kIGhvdyBpdCBmb2N1c3NlcyBv
bg0KIm11c3Qgbm90IGZhaWwiLg0KDQotLSANClRyb25kIE15a2xlYnVzdA0KTGludXggTkZTIGNs
aWVudCBtYWludGFpbmVyLCBQcmltYXJ5RGF0YQ0KdHJvbmQubXlrbGVidXN0QHByaW1hcnlkYXRh
LmNvbQ0K
WARNING: multiple messages have this Message-ID (diff)
From: Trond Myklebust <trondmy@primarydata.com>
To: "mhocko@kernel.org" <mhocko@kernel.org>
Cc: "torvalds@linux-foundation.org" <torvalds@linux-foundation.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"linux-nfs@vger.kernel.org" <linux-nfs@vger.kernel.org>,
"n.borisov.lkml@gmail.com" <n.borisov.lkml@gmail.com>
Subject: Re: [GIT PULL] Please pull NFS client fixes for 4.12
Date: Thu, 11 May 2017 13:10:03 +0000 [thread overview]
Message-ID: <1494508201.3207.5.camel@primarydata.com> (raw)
In-Reply-To: <20170511125612.GK26782@dhcp22.suse.cz>
On Thu, 2017-05-11 at 14:56 +0200, Michal Hocko wrote:
> On Thu 11-05-17 12:45:00, Trond Myklebust wrote:
> > On Thu, 2017-05-11 at 14:26 +0200, Michal Hocko wrote:
> > > On Thu 11-05-17 12:16:37, Trond Myklebust wrote:
> > > > On Thu, 2017-05-11 at 09:59 +0200, Michal Hocko wrote:
> > > > > On Thu 11-05-17 10:53:27, Nikolay Borisov wrote:
> > > > > >
> > > > > >
> > > > > > On 10.05.2017 19:47, Trond Myklebust wrote:
> > > > >
> > > > > [...]
> > > > > > > - Cleanup and removal of some memory failure paths now
> > > > > > > that
> > > > > > > GFP_NOFS is guaranteed to never fail.
> > > > > >
> > > > > > What guarantees that? Since if this is the case then this
> > > > > > can
> > > > > > result in
> > > > > > a lot of opportunities for cleanup across the whole kernel
> > > > > > tree.
> > > > > > After
> > > > > > discussing with mhocko (cc'ed) it seems that in practice
> > > > > > everything
> > > > > > below COSTLY_ORDER which are not GFP_NORETRY will never
> > > > > > fail.
> > > > > > But
> > > > > > this
> > > > > > semantic is not the same as GFP_NOFAIL. E.g. nothing
> > > > > > guarantees
> > > > > > that
> > > > > > this will stay like that in the future?
> > > > >
> > > > > In practice it is hard to change the semantic of small
> > > > > allocations
> > > > > never
> > > > > fail _practically_. But this is absolutely not guaranteed!
> > > > > They
> > > > > can
> > > > > fail
> > > > > e.g. when the allocation context is the oom victim. Removing
> > > > > error
> > > > > paths
> > > > > for allocation failures is just wrong.
> > > >
> > > > OK, this makes no fucking sense at all.
> > > >
> > > > Either allocations can fail or they can't.
> > > > 1) If they can't fail, then we don't need the checks.
> > > > 2) If they can fail, then we do need them, and this hand
> > > > wringing
> > > > in
> > > > the MM community about GFP_* semantics and how we need to
> > > > prevent
> > > > failure is fucking pointless.
> > >
> > > everything which is not __GFP_NOFAIL might fail. We try hard not
> > > to
> > > fail
> > > small allocations requests as much as we can in general but you
> > > _have_ to
> > > check for failures. There is simply no way to guarantee "never
> > > fail"
> > > semantic for all allocation requests. This has been like that
> > > basically
> > > since years. And even this try-to-be-nofailing for small
> > > allocations
> > > has
> > > been PITA for some corner cases.
> >
> > I'll take that as a vote for (2), then.
> >
> > I know that failures could occur in the past. That's why those code
> > paths were there. The problem is that the MM community has been
> > making
> > lots of noise on mailing lists, conferences and LWN articles about
> > how
> > we must not fail small allocations because the MM community
> > believes
> > that nobody expects it. This is confusing everyone...
>
> It was exactly other way around. We would like to _get_rid_of_ this
> do
> not fail behavior because it is causing a major headaches in out of
> memory corner cases. Just take GFP_NOFS as an example. It is a weak
> reclaim context because we cannot reclaim fs metadata and that might
> be
> a lot of memory so we cannot trigger the OOM killer and have to rely
> on
> a different allocation context or kswapd to make a progress on our
> behalf. We would really like to fail those requests instead. I've
> tried
> that in the past but it was deemed to dangerous because _all_ kernel
> paths would have to be checked for a sane failure behavior. So we are
> keeping status quo instead.
If we suspect the existence of a load of potential time bombs in the
kernel due to missing checks, then the status quo is not good enough.
We should be working on tools to identify these code paths.
Quite frankly, I'd love to see a fuzzer-like tool that can randomly
fail allocations. I can easily make one for the NFS code, but if there
is a general problem identifying buggy code, then perhaps it should be
solved at the MM layer itself.
> > It confused Neil Brown, who contributed these patches, and it
> > confused
> > me and all the other reviewers of these patches on the linux-nfs
> > mailing list.
> >
> > So if indeed (2) is correct, then please can we have a clear
> > statement
> > _when discussing improvements to memory allocation semantics_ that
> > GFP_* still can fail, still will fail, and that callers should
> > assume
> > it will fail and should test their code paths assuming the failure
> > case.
>
> I do not see any explicit documentation which would encourage users
> to
> not check for the allocation failure. Only __GFP_NOFAIL is documented
> it
> _must_ retry for ever. Of course I am open for any documentation
> improvements.
As I said, the problem has been the discussion, and how it focusses on
"must not fail".
--
Trond Myklebust
Linux NFS client maintainer, PrimaryData
trond.myklebust@primarydata.com
next prev parent reply other threads:[~2017-05-11 13:10 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-05-10 16:47 [GIT PULL] Please pull NFS client fixes for 4.12 Trond Myklebust
2017-05-10 16:47 ` Trond Myklebust
2017-05-10 20:06 ` Linus Torvalds
2017-05-11 7:53 ` Nikolay Borisov
2017-05-11 7:59 ` Michal Hocko
2017-05-11 12:16 ` Trond Myklebust
2017-05-11 12:16 ` Trond Myklebust
2017-05-11 12:26 ` Michal Hocko
2017-05-11 12:45 ` Trond Myklebust
2017-05-11 12:45 ` Trond Myklebust
2017-05-11 12:56 ` Michal Hocko
2017-05-11 13:10 ` Trond Myklebust [this message]
2017-05-11 13:10 ` Trond Myklebust
2017-05-11 13:27 ` Michal Hocko
2017-05-16 15:15 ` Jonathan Corbet
2017-05-11 13:41 ` Trond Myklebust
2017-05-11 13:41 ` Trond Myklebust
2017-05-11 13:54 ` Michal Hocko
-- strict thread matches above, loose matches on Subject: below --
2017-06-28 14:19 Trond Myklebust
2017-06-28 14:19 ` Trond Myklebust
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=1494508201.3207.5.camel@primarydata.com \
--to=trondmy@primarydata.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=mhocko@kernel.org \
--cc=n.borisov.lkml@gmail.com \
--cc=torvalds@linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.