From: Alistair Popple <apopple@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: John Hubbard <jhubbard@nvidia.com>,
linux-mm@kvack.org, nouveau@lists.freedesktop.org,
bskeggs@redhat.com, akpm@linux-foundation.org,
linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org,
kvm-ppc@vger.kernel.org, dri-devel@lists.freedesktop.org,
rcampbell@nvidia.com, jglisse@redhat.com, hch@infradead.org,
daniel@ffwll.ch, willy@infradead.org,
Christoph Hellwig <hch@lst.de>
Subject: Re: [PATCH v7 3/8] mm/rmap: Split try_to_munlock from try_to_unmap
Date: Thu, 01 Apr 2021 04:36:18 +0000 [thread overview]
Message-ID: <2557539.O4bb4zRkYN@nvdebian> (raw)
In-Reply-To: <20210331115746.GA1463678@nvidia.com>
On Wednesday, 31 March 2021 10:57:46 PM AEDT Jason Gunthorpe wrote:
> On Wed, Mar 31, 2021 at 03:15:47PM +1100, Alistair Popple wrote:
> > On Wednesday, 31 March 2021 2:56:38 PM AEDT John Hubbard wrote:
> > > On 3/30/21 3:56 PM, Alistair Popple wrote:
> > > ...
> > > >> +1 for renaming "munlock*" items to "mlock*", where applicable. good
> > grief.
> > > >
> > > > At least the situation was weird enough to prompt further
investigation :)
> > > >
> > > > Renaming to mlock* doesn't feel like the right solution to me either
> > though. I
> > > > am not sure if you saw me responding to myself earlier but I am
thinking
> > > > renaming try_to_munlock() -> page_mlocked() and try_to_munlock_one() -
>
> > > > page_mlock_one() might be better. Thoughts?
> > > >
> > >
> > > Quite confused by this naming idea. Because: try_to_munlock() returns
> > > void, so a boolean-style name such as "page_mlocked()" is already not a
> > > good fit.
> > >
> > > Even more important, though, is that try_to_munlock() is mlock-ing the
> > > page, right? Is there some subtle point I'm missing? It really is doing
> > > an mlock to the best of my knowledge here. Although the kerneldoc
> > > comment for try_to_munlock() seems questionable too:
> >
> > It's mlocking the page if it turns out it still needs to be locked after
> > unlocking it. But I don't think you're missing anything.
>
> It is really searching all VMA's to see if the VMA flag is set and if
> any are found then it mlocks the page.
>
> But presenting this rountine in its simplified form raises lots of
> questions:
>
> - What locking is being used to read the VMA flag?
> - Why do we need to manipulate global struct page flags under the
> page table locks of a single VMA?
I was wondering that and questioned it in an earlier version of this series. I
have done some digging and the commit log for b87537d9e2fe ("mm: rmap use pte
lock not mmap_sem to set PageMlocked") provides the original justification.
It's fairly long so I won't quote it here but the summary seems to be that
among other things the combination of page lock and ptl makes this safe. I
have yet to verify if everything there still holds and is sensible, but the
last paragraph certainly is :-)
"Stopped short of separating try_to_munlock_one() from try_to_munmap_one()
on this occasion, but that's probably the sensible next step - with a
rename, given that try_to_munlock()'s business is to try to set Mlocked."
> - Why do we need to check for huge pages inside the VMA loop, not
> before going to the rmap? PageTransCompoundHead() is not sensitive to
> the PTEs. (and what happens if the huge page breaks up concurrently?)
> - Why do we clear the mlock bit then run around to try and set it?
I don't have an answer for that as I'm not (yet) across all the mlock code
paths, but I'm hoping this patch at least won't change anything.
> Feels racey.
>
> Jason
>
WARNING: multiple messages have this Message-ID (diff)
From: Alistair Popple <apopple@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: John Hubbard <jhubbard@nvidia.com>, <linux-mm@kvack.org>,
<nouveau@lists.freedesktop.org>, <bskeggs@redhat.com>,
<akpm@linux-foundation.org>, <linux-doc@vger.kernel.org>,
<linux-kernel@vger.kernel.org>, <kvm-ppc@vger.kernel.org>,
<dri-devel@lists.freedesktop.org>, <rcampbell@nvidia.com>,
<jglisse@redhat.com>, <hch@infradead.org>, <daniel@ffwll.ch>,
<willy@infradead.org>, Christoph Hellwig <hch@lst.de>
Subject: Re: [PATCH v7 3/8] mm/rmap: Split try_to_munlock from try_to_unmap
Date: Thu, 1 Apr 2021 15:36:18 +1100 [thread overview]
Message-ID: <2557539.O4bb4zRkYN@nvdebian> (raw)
In-Reply-To: <20210331115746.GA1463678@nvidia.com>
On Wednesday, 31 March 2021 10:57:46 PM AEDT Jason Gunthorpe wrote:
> On Wed, Mar 31, 2021 at 03:15:47PM +1100, Alistair Popple wrote:
> > On Wednesday, 31 March 2021 2:56:38 PM AEDT John Hubbard wrote:
> > > On 3/30/21 3:56 PM, Alistair Popple wrote:
> > > ...
> > > >> +1 for renaming "munlock*" items to "mlock*", where applicable. good
> > grief.
> > > >
> > > > At least the situation was weird enough to prompt further
investigation :)
> > > >
> > > > Renaming to mlock* doesn't feel like the right solution to me either
> > though. I
> > > > am not sure if you saw me responding to myself earlier but I am
thinking
> > > > renaming try_to_munlock() -> page_mlocked() and try_to_munlock_one() -
>
> > > > page_mlock_one() might be better. Thoughts?
> > > >
> > >
> > > Quite confused by this naming idea. Because: try_to_munlock() returns
> > > void, so a boolean-style name such as "page_mlocked()" is already not a
> > > good fit.
> > >
> > > Even more important, though, is that try_to_munlock() is mlock-ing the
> > > page, right? Is there some subtle point I'm missing? It really is doing
> > > an mlock to the best of my knowledge here. Although the kerneldoc
> > > comment for try_to_munlock() seems questionable too:
> >
> > It's mlocking the page if it turns out it still needs to be locked after
> > unlocking it. But I don't think you're missing anything.
>
> It is really searching all VMA's to see if the VMA flag is set and if
> any are found then it mlocks the page.
>
> But presenting this rountine in its simplified form raises lots of
> questions:
>
> - What locking is being used to read the VMA flag?
> - Why do we need to manipulate global struct page flags under the
> page table locks of a single VMA?
I was wondering that and questioned it in an earlier version of this series. I
have done some digging and the commit log for b87537d9e2fe ("mm: rmap use pte
lock not mmap_sem to set PageMlocked") provides the original justification.
It's fairly long so I won't quote it here but the summary seems to be that
among other things the combination of page lock and ptl makes this safe. I
have yet to verify if everything there still holds and is sensible, but the
last paragraph certainly is :-)
"Stopped short of separating try_to_munlock_one() from try_to_munmap_one()
on this occasion, but that's probably the sensible next step - with a
rename, given that try_to_munlock()'s business is to try to set Mlocked."
> - Why do we need to check for huge pages inside the VMA loop, not
> before going to the rmap? PageTransCompoundHead() is not sensitive to
> the PTEs. (and what happens if the huge page breaks up concurrently?)
> - Why do we clear the mlock bit then run around to try and set it?
I don't have an answer for that as I'm not (yet) across all the mlock code
paths, but I'm hoping this patch at least won't change anything.
> Feels racey.
>
> Jason
>
WARNING: multiple messages have this Message-ID (diff)
From: Alistair Popple <apopple@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: rcampbell@nvidia.com, willy@infradead.org,
linux-doc@vger.kernel.org, nouveau@lists.freedesktop.org,
dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org,
kvm-ppc@vger.kernel.org, hch@infradead.org, linux-mm@kvack.org,
bskeggs@redhat.com, daniel@ffwll.ch, akpm@linux-foundation.org,
Christoph Hellwig <hch@lst.de>
Subject: Re: [Nouveau] [PATCH v7 3/8] mm/rmap: Split try_to_munlock from try_to_unmap
Date: Thu, 1 Apr 2021 15:36:18 +1100 [thread overview]
Message-ID: <2557539.O4bb4zRkYN@nvdebian> (raw)
In-Reply-To: <20210331115746.GA1463678@nvidia.com>
On Wednesday, 31 March 2021 10:57:46 PM AEDT Jason Gunthorpe wrote:
> On Wed, Mar 31, 2021 at 03:15:47PM +1100, Alistair Popple wrote:
> > On Wednesday, 31 March 2021 2:56:38 PM AEDT John Hubbard wrote:
> > > On 3/30/21 3:56 PM, Alistair Popple wrote:
> > > ...
> > > >> +1 for renaming "munlock*" items to "mlock*", where applicable. good
> > grief.
> > > >
> > > > At least the situation was weird enough to prompt further
investigation :)
> > > >
> > > > Renaming to mlock* doesn't feel like the right solution to me either
> > though. I
> > > > am not sure if you saw me responding to myself earlier but I am
thinking
> > > > renaming try_to_munlock() -> page_mlocked() and try_to_munlock_one() -
>
> > > > page_mlock_one() might be better. Thoughts?
> > > >
> > >
> > > Quite confused by this naming idea. Because: try_to_munlock() returns
> > > void, so a boolean-style name such as "page_mlocked()" is already not a
> > > good fit.
> > >
> > > Even more important, though, is that try_to_munlock() is mlock-ing the
> > > page, right? Is there some subtle point I'm missing? It really is doing
> > > an mlock to the best of my knowledge here. Although the kerneldoc
> > > comment for try_to_munlock() seems questionable too:
> >
> > It's mlocking the page if it turns out it still needs to be locked after
> > unlocking it. But I don't think you're missing anything.
>
> It is really searching all VMA's to see if the VMA flag is set and if
> any are found then it mlocks the page.
>
> But presenting this rountine in its simplified form raises lots of
> questions:
>
> - What locking is being used to read the VMA flag?
> - Why do we need to manipulate global struct page flags under the
> page table locks of a single VMA?
I was wondering that and questioned it in an earlier version of this series. I
have done some digging and the commit log for b87537d9e2fe ("mm: rmap use pte
lock not mmap_sem to set PageMlocked") provides the original justification.
It's fairly long so I won't quote it here but the summary seems to be that
among other things the combination of page lock and ptl makes this safe. I
have yet to verify if everything there still holds and is sensible, but the
last paragraph certainly is :-)
"Stopped short of separating try_to_munlock_one() from try_to_munmap_one()
on this occasion, but that's probably the sensible next step - with a
rename, given that try_to_munlock()'s business is to try to set Mlocked."
> - Why do we need to check for huge pages inside the VMA loop, not
> before going to the rmap? PageTransCompoundHead() is not sensitive to
> the PTEs. (and what happens if the huge page breaks up concurrently?)
> - Why do we clear the mlock bit then run around to try and set it?
I don't have an answer for that as I'm not (yet) across all the mlock code
paths, but I'm hoping this patch at least won't change anything.
> Feels racey.
>
> Jason
>
_______________________________________________
Nouveau mailing list
Nouveau@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/nouveau
WARNING: multiple messages have this Message-ID (diff)
From: Alistair Popple <apopple@nvidia.com>
To: Jason Gunthorpe <jgg@nvidia.com>
Cc: rcampbell@nvidia.com, willy@infradead.org,
linux-doc@vger.kernel.org, nouveau@lists.freedesktop.org,
dri-devel@lists.freedesktop.org, linux-kernel@vger.kernel.org,
kvm-ppc@vger.kernel.org, hch@infradead.org, linux-mm@kvack.org,
jglisse@redhat.com, bskeggs@redhat.com,
John Hubbard <jhubbard@nvidia.com>,
akpm@linux-foundation.org, Christoph Hellwig <hch@lst.de>
Subject: Re: [PATCH v7 3/8] mm/rmap: Split try_to_munlock from try_to_unmap
Date: Thu, 1 Apr 2021 15:36:18 +1100 [thread overview]
Message-ID: <2557539.O4bb4zRkYN@nvdebian> (raw)
In-Reply-To: <20210331115746.GA1463678@nvidia.com>
On Wednesday, 31 March 2021 10:57:46 PM AEDT Jason Gunthorpe wrote:
> On Wed, Mar 31, 2021 at 03:15:47PM +1100, Alistair Popple wrote:
> > On Wednesday, 31 March 2021 2:56:38 PM AEDT John Hubbard wrote:
> > > On 3/30/21 3:56 PM, Alistair Popple wrote:
> > > ...
> > > >> +1 for renaming "munlock*" items to "mlock*", where applicable. good
> > grief.
> > > >
> > > > At least the situation was weird enough to prompt further
investigation :)
> > > >
> > > > Renaming to mlock* doesn't feel like the right solution to me either
> > though. I
> > > > am not sure if you saw me responding to myself earlier but I am
thinking
> > > > renaming try_to_munlock() -> page_mlocked() and try_to_munlock_one() -
>
> > > > page_mlock_one() might be better. Thoughts?
> > > >
> > >
> > > Quite confused by this naming idea. Because: try_to_munlock() returns
> > > void, so a boolean-style name such as "page_mlocked()" is already not a
> > > good fit.
> > >
> > > Even more important, though, is that try_to_munlock() is mlock-ing the
> > > page, right? Is there some subtle point I'm missing? It really is doing
> > > an mlock to the best of my knowledge here. Although the kerneldoc
> > > comment for try_to_munlock() seems questionable too:
> >
> > It's mlocking the page if it turns out it still needs to be locked after
> > unlocking it. But I don't think you're missing anything.
>
> It is really searching all VMA's to see if the VMA flag is set and if
> any are found then it mlocks the page.
>
> But presenting this rountine in its simplified form raises lots of
> questions:
>
> - What locking is being used to read the VMA flag?
> - Why do we need to manipulate global struct page flags under the
> page table locks of a single VMA?
I was wondering that and questioned it in an earlier version of this series. I
have done some digging and the commit log for b87537d9e2fe ("mm: rmap use pte
lock not mmap_sem to set PageMlocked") provides the original justification.
It's fairly long so I won't quote it here but the summary seems to be that
among other things the combination of page lock and ptl makes this safe. I
have yet to verify if everything there still holds and is sensible, but the
last paragraph certainly is :-)
"Stopped short of separating try_to_munlock_one() from try_to_munmap_one()
on this occasion, but that's probably the sensible next step - with a
rename, given that try_to_munlock()'s business is to try to set Mlocked."
> - Why do we need to check for huge pages inside the VMA loop, not
> before going to the rmap? PageTransCompoundHead() is not sensitive to
> the PTEs. (and what happens if the huge page breaks up concurrently?)
> - Why do we clear the mlock bit then run around to try and set it?
I don't have an answer for that as I'm not (yet) across all the mlock code
paths, but I'm hoping this patch at least won't change anything.
> Feels racey.
>
> Jason
>
_______________________________________________
dri-devel mailing list
dri-devel@lists.freedesktop.org
https://lists.freedesktop.org/mailman/listinfo/dri-devel
next prev parent reply other threads:[~2021-04-01 4:36 UTC|newest]
Thread overview: 124+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-03-26 0:07 [PATCH v7 0/8] Add support for SVM atomics in Nouveau Alistair Popple
2021-03-26 0:07 ` Alistair Popple
2021-03-26 0:07 ` [Nouveau] " Alistair Popple
2021-03-26 0:07 ` Alistair Popple
2021-03-26 0:07 ` [PATCH v7 1/8] mm: Remove special swap entry functions Alistair Popple
2021-03-26 0:07 ` Alistair Popple
2021-03-26 0:07 ` [Nouveau] " Alistair Popple
2021-03-26 0:07 ` Alistair Popple
2021-03-30 18:38 ` Jason Gunthorpe
2021-03-30 18:38 ` Jason Gunthorpe
2021-03-30 18:38 ` [Nouveau] " Jason Gunthorpe
2021-03-30 18:38 ` Jason Gunthorpe
2021-03-26 0:07 ` [PATCH v7 2/8] mm/swapops: Rework swap entry manipulation code Alistair Popple
2021-03-26 0:07 ` Alistair Popple
2021-03-26 0:07 ` [Nouveau] " Alistair Popple
2021-03-26 0:07 ` Alistair Popple
2021-03-26 0:08 ` [PATCH v7 3/8] mm/rmap: Split try_to_munlock from try_to_unmap Alistair Popple
2021-03-26 0:08 ` Alistair Popple
2021-03-26 0:08 ` [Nouveau] " Alistair Popple
2021-03-26 0:08 ` Alistair Popple
2021-03-30 18:49 ` Jason Gunthorpe
2021-03-30 18:49 ` Jason Gunthorpe
2021-03-30 18:49 ` [Nouveau] " Jason Gunthorpe
2021-03-30 18:49 ` Jason Gunthorpe
2021-03-30 22:09 ` Alistair Popple
2021-03-30 22:09 ` Alistair Popple
2021-03-30 22:09 ` [Nouveau] " Alistair Popple
2021-03-30 22:09 ` Alistair Popple
2021-03-30 22:16 ` Alistair Popple
2021-03-30 22:16 ` Alistair Popple
2021-03-30 22:16 ` [Nouveau] " Alistair Popple
2021-03-30 22:16 ` Alistair Popple
2021-03-30 22:24 ` Jason Gunthorpe
2021-03-30 22:24 ` Jason Gunthorpe
2021-03-30 22:24 ` [Nouveau] " Jason Gunthorpe
2021-03-30 22:24 ` Jason Gunthorpe
2021-03-30 22:43 ` John Hubbard
2021-03-30 22:43 ` John Hubbard
2021-03-30 22:43 ` [Nouveau] " John Hubbard
2021-03-30 22:43 ` John Hubbard
2021-03-30 22:56 ` Alistair Popple
2021-03-30 22:56 ` Alistair Popple
2021-03-30 22:56 ` [Nouveau] " Alistair Popple
2021-03-30 22:56 ` Alistair Popple
2021-03-31 3:56 ` John Hubbard
2021-03-31 3:56 ` John Hubbard
2021-03-31 3:56 ` [Nouveau] " John Hubbard
2021-03-31 3:56 ` John Hubbard
2021-03-31 4:09 ` John Hubbard
2021-03-31 4:09 ` John Hubbard
2021-03-31 4:09 ` [Nouveau] " John Hubbard
2021-03-31 4:09 ` John Hubbard
2021-03-31 4:15 ` Alistair Popple
2021-03-31 4:15 ` Alistair Popple
2021-03-31 4:15 ` [Nouveau] " Alistair Popple
2021-03-31 4:15 ` Alistair Popple
2021-03-31 11:57 ` Jason Gunthorpe
2021-03-31 11:57 ` Jason Gunthorpe
2021-03-31 11:57 ` [Nouveau] " Jason Gunthorpe
2021-03-31 11:57 ` Jason Gunthorpe
2021-04-01 4:36 ` Alistair Popple [this message]
2021-04-01 4:36 ` Alistair Popple
2021-04-01 4:36 ` [Nouveau] " Alistair Popple
2021-04-01 4:36 ` Alistair Popple
2021-04-01 19:21 ` Shakeel Butt
2021-04-01 19:21 ` Shakeel Butt
2021-04-01 19:21 ` [Nouveau] " Shakeel Butt
2021-04-01 19:21 ` Shakeel Butt
2021-03-26 0:08 ` [PATCH v7 4/8] mm/rmap: Split migration into its own function Alistair Popple
2021-03-26 0:08 ` Alistair Popple
2021-03-26 0:08 ` [Nouveau] " Alistair Popple
2021-03-26 0:08 ` Alistair Popple
2021-03-26 0:08 ` [PATCH v7 5/8] mm: Device exclusive memory access Alistair Popple
2021-03-26 0:08 ` Alistair Popple
2021-03-26 0:08 ` [Nouveau] " Alistair Popple
2021-03-26 0:08 ` Alistair Popple
2021-03-30 19:32 ` Jason Gunthorpe
2021-03-30 19:32 ` Jason Gunthorpe
2021-03-30 19:32 ` [Nouveau] " Jason Gunthorpe
2021-03-30 19:32 ` Jason Gunthorpe
2021-03-31 12:59 ` Alistair Popple
2021-03-31 12:59 ` Alistair Popple
2021-03-31 12:59 ` [Nouveau] " Alistair Popple
2021-03-31 12:59 ` Alistair Popple
2021-03-31 13:18 ` Jason Gunthorpe
2021-03-31 13:18 ` Jason Gunthorpe
2021-03-31 13:18 ` [Nouveau] " Jason Gunthorpe
2021-03-31 13:18 ` Jason Gunthorpe
2021-03-31 13:27 ` Alistair Popple
2021-03-31 13:27 ` Alistair Popple
2021-03-31 13:27 ` [Nouveau] " Alistair Popple
2021-03-31 13:27 ` Alistair Popple
2021-03-31 13:46 ` Jason Gunthorpe
2021-03-31 13:46 ` Jason Gunthorpe
2021-03-31 13:46 ` [Nouveau] " Jason Gunthorpe
2021-03-31 13:46 ` Jason Gunthorpe
2021-04-01 0:45 ` Alistair Popple
2021-04-01 0:45 ` Alistair Popple
2021-04-01 0:45 ` [Nouveau] " Alistair Popple
2021-04-01 0:45 ` Alistair Popple
2021-04-01 0:48 ` Jason Gunthorpe
2021-04-01 0:48 ` Jason Gunthorpe
2021-04-01 0:48 ` [Nouveau] " Jason Gunthorpe
2021-04-01 0:48 ` Jason Gunthorpe
2021-04-01 2:20 ` Alistair Popple
2021-04-01 2:20 ` Alistair Popple
2021-04-01 2:20 ` [Nouveau] " Alistair Popple
2021-04-01 2:20 ` Alistair Popple
2021-04-01 11:55 ` Jason Gunthorpe
2021-04-01 11:55 ` Jason Gunthorpe
2021-04-01 11:55 ` [Nouveau] " Jason Gunthorpe
2021-04-01 11:55 ` Jason Gunthorpe
2021-03-26 0:08 ` [PATCH v7 6/8] mm: Selftests for exclusive device memory Alistair Popple
2021-03-26 0:08 ` Alistair Popple
2021-03-26 0:08 ` [Nouveau] " Alistair Popple
2021-03-26 0:08 ` Alistair Popple
2021-03-26 0:08 ` [PATCH v7 7/8] nouveau/svm: Refactor nouveau_range_fault Alistair Popple
2021-03-26 0:08 ` Alistair Popple
2021-03-26 0:08 ` [Nouveau] " Alistair Popple
2021-03-26 0:08 ` Alistair Popple
2021-03-26 0:08 ` [PATCH v7 8/8] nouveau/svm: Implement atomic SVM access Alistair Popple
2021-03-26 0:08 ` Alistair Popple
2021-03-26 0:08 ` [Nouveau] " Alistair Popple
2021-03-26 0:08 ` Alistair Popple
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2557539.O4bb4zRkYN@nvdebian \
--to=apopple@nvidia.com \
--cc=akpm@linux-foundation.org \
--cc=bskeggs@redhat.com \
--cc=daniel@ffwll.ch \
--cc=dri-devel@lists.freedesktop.org \
--cc=hch@infradead.org \
--cc=hch@lst.de \
--cc=jgg@nvidia.com \
--cc=jglisse@redhat.com \
--cc=jhubbard@nvidia.com \
--cc=kvm-ppc@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=nouveau@lists.freedesktop.org \
--cc=rcampbell@nvidia.com \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.