* [PATCH v4 1/2] oe-setup-layers: support inline URI
@ 2025-12-19 16:13 Corentin Guillevic
2025-12-19 16:13 ` [PATCH v4 2/2] meta/files/layers.schema.json: use URI shortcut for some remotes Corentin Guillevic
0 siblings, 1 reply; 5+ messages in thread
From: Corentin Guillevic @ 2025-12-19 16:13 UTC (permalink / raw)
To: openembedded-core; +Cc: Corentin Guillevic
Most of the time, when we describe a remote, the layer data (also used by
the script bitbake-setup) looks like this:
"bitbake": {
"git-remote": {
"remotes": {
"origin": {
"uri": "https://git.openembedded.org/bitbake"
}
},
...
}
}
i.e. an URI with the common name 'origin'. Alternatively, we could simplify this, by
using a shorter structure with the property 'uri' only:
"bitbake": {
"git-remote": {
"uri": "https://git.openembedded.org/bitbake
",
...
}
}
These properties can be used together.
Signed-off-by: Corentin Guillevic <corentin.guillevic@smile.fr>
---
Changes in v4:
- Set r_name just before the break
scripts/oe-setup-layers | 31 ++++++++++++++++++++++++++++++-
1 file changed, 30 insertions(+), 1 deletion(-)
diff --git a/scripts/oe-setup-layers b/scripts/oe-setup-layers
index 31cb963251..4813d6f9dc 100755
--- a/scripts/oe-setup-layers
+++ b/scripts/oe-setup-layers
@@ -60,6 +60,34 @@ def _write_layer_list(dest, repodirs):
with open(layers_f, 'w') as f:
json.dump({"version":"1.0","layers":layers}, f, sort_keys=True, indent=4)
+def _get_remotes(r_remote):
+ remotes = {}
+
+ if not 'remotes' in r_remote and not 'uri' in r_remote:
+ raise Exception("Expected key(s): 'remotes', 'uri'")
+
+ if 'remotes' in r_remote:
+ remotes = r_remote['remotes'].copy()
+
+ if 'uri' in r_remote:
+ r_name = ''
+
+ if 'remotes' in r_remote:
+ if not 'origin' in r_remote['remotes']:
+ r_name = 'origin'
+ else:
+ import itertools
+ for i in itertools.count(start=1):
+ if not 'origin-{}'.format(i) in r_remote['remotes']:
+ r_name = 'origin-{}'.format(i)
+ break
+ else:
+ r_name = 'origin'
+
+ remotes.update({r_name: {'uri': r_remote['uri']}})
+
+ return remotes
+
def _do_checkout(args, json):
repos = json['sources']
repodirs = []
@@ -80,7 +108,8 @@ def _do_checkout(args, json):
if not desc:
desc = rev[:10]
branch = r_remote['branch']
- remotes = r_remote['remotes']
+
+ remotes = _get_remotes(r_remote)
print('\nSetting up source {}, revision {}, branch {}'.format(r_name, desc, branch))
if not _is_repo_git_repo(repodir):
--
2.51.0
^ permalink raw reply related [flat|nested] 5+ messages in thread* [PATCH v4 2/2] meta/files/layers.schema.json: use URI shortcut for some remotes
2025-12-19 16:13 [PATCH v4 1/2] oe-setup-layers: support inline URI Corentin Guillevic
@ 2025-12-19 16:13 ` Corentin Guillevic
2025-12-21 9:05 ` [OE-core] " Mathieu Dubois-Briand
0 siblings, 1 reply; 5+ messages in thread
From: Corentin Guillevic @ 2025-12-19 16:13 UTC (permalink / raw)
To: openembedded-core; +Cc: Corentin Guillevic
Among the three sources, only two have a single remote. So we can replace
their entire structure 'remote' -> 'origin' -> 'uri' with a shorter one
(property 'uri' only).
Signed-off-by: Corentin Guillevic <corentin.guillevic@smile.fr>
---
meta/files/layers.example.json | 12 ++----------
1 file changed, 2 insertions(+), 10 deletions(-)
diff --git a/meta/files/layers.example.json b/meta/files/layers.example.json
index f3b6522083..d990502205 100644
--- a/meta/files/layers.example.json
+++ b/meta/files/layers.example.json
@@ -5,11 +5,7 @@
"git-remote": {
"branch": "master",
"describe": "",
- "remotes": {
- "remote-alex": {
- "uri": "https://github.com/kanavin/meta-alex"
- }
- },
+ "uri": "https://github.com/kanavin/meta-alex",
"rev": "05b25605fb8b2399e4706d7323828676bf0da0b5"
},
"path": "meta-alex"
@@ -18,11 +14,7 @@
"git-remote": {
"branch": "master",
"describe": "15.0-hardknott-3.3-310-g0a96edae",
- "remotes": {
- "origin": {
- "uri": "git://git.yoctoproject.org/meta-intel"
- }
- },
+ "uri": "git://git.yoctoproject.org/meta-intel",
"rev": "0a96edae609a3f48befac36af82cf1eed6786b4a"
}
},
--
2.51.0
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [OE-core] [PATCH v4 2/2] meta/files/layers.schema.json: use URI shortcut for some remotes
2025-12-19 16:13 ` [PATCH v4 2/2] meta/files/layers.schema.json: use URI shortcut for some remotes Corentin Guillevic
@ 2025-12-21 9:05 ` Mathieu Dubois-Briand
2025-12-21 17:33 ` Yoann Congal
0 siblings, 1 reply; 5+ messages in thread
From: Mathieu Dubois-Briand @ 2025-12-21 9:05 UTC (permalink / raw)
To: corentin.guillevic, openembedded-core
On Fri Dec 19, 2025 at 5:13 PM CET, Corentin Guillevic via lists.openembedded.org wrote:
> Among the three sources, only two have a single remote. So we can replace
> their entire structure 'remote' -> 'origin' -> 'uri' with a shorter one
> (property 'uri' only).
>
> Signed-off-by: Corentin Guillevic <corentin.guillevic@smile.fr>
> ---
Hi Corentin,
It looks like this is breaking some selftest:
025-12-19 13:57:11,641 - oe-selftest - INFO - bblayers.BitbakeLayers.test_validate_examplelayersjson (subunit.RemotedTestCase)
2025-12-19 13:57:11,642 - oe-selftest - INFO - ... FAIL
Stderr:
2025-12-19 13:53:18,059 - oe-selftest - INFO - Adding: "include selftest.inc" in /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/conf/local.conf
2025-12-19 13:53:18,059 - oe-selftest - INFO - Adding: "include bblayers.inc" in bblayers.conf
2025-12-19 13:57:11,642 - oe-selftest - INFO - 6: 13/38 66/653 (0.30s) (0 failed) (bblayers.BitbakeLayers.test_validate_examplelayersjson)
2025-12-19 13:57:11,642 - oe-selftest - INFO - testtools.testresult.real._StringException: Traceback (most recent call last):
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/bblayers.py", line 163, in test_validate_examplelayersjson
self.validate_layersjson(json)
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/bblayers.py", line 159, in validate_layersjson
self.validate_json(json, "layers.schema.json")
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/bblayers.py", line 152, in validate_json
result = runCmd(
^^^^^^^
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/utils/commands.py", line 214, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/tmp/work/x86_64-linux/python3-jsonschema-native/4.25.1/recipe-sysroot-native/usr/bin/nativepython3 /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/tmp/work/x86_64-linux/python3-jsonschema-native/4.25.1/recipe-sysroot-native/usr/bin/jsonschema -i /srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/files/layers.example.json --base-uri file:///srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/bitbake/bin/../setup-schema/ /srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/bitbake/bin/../setup-schema/layers.schema.json' returned non-zero exit status 1:
/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/tmp/work/x86_64-linux/python3-jsonschema-native/4.25.1/recipe-sysroot-native/usr/bin/jsonschema:5: DeprecationWarning: The jsonschema CLI is deprecated and will be removed in a future version. Please use check-jsonschema instead, which can be installed from https://pypi.org/project/check-jsonschema/
from jsonschema.cli import main
{'branch': 'master', 'describe': '', 'uri': 'https://github.com/kanavin/meta-alex', 'rev': '05b25605fb8b2399e4706d7323828676bf0da0b5'}: Additional properties are not allowed ('uri' was unexpected)
{'branch': 'master', 'describe': '15.0-hardknott-3.3-310-g0a96edae', 'uri': 'git://git.yoctoproject.org/meta-intel', 'rev': '0a96edae609a3f48befac36af82cf1eed6786b4a'}: Additional properties are not allowed ('uri' was unexpected)
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2867
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2762
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3003
Can you have a look at this?
Thanks,
Mathieu
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [OE-core] [PATCH v4 2/2] meta/files/layers.schema.json: use URI shortcut for some remotes
2025-12-21 9:05 ` [OE-core] " Mathieu Dubois-Briand
@ 2025-12-21 17:33 ` Yoann Congal
2025-12-22 7:10 ` Mathieu Dubois-Briand
0 siblings, 1 reply; 5+ messages in thread
From: Yoann Congal @ 2025-12-21 17:33 UTC (permalink / raw)
To: mathieu.dubois-briand; +Cc: corentin.guillevic, openembedded-core
[-- Attachment #1: Type: text/plain, Size: 5545 bytes --]
Le dim. 21 déc. 2025 à 10:05, Mathieu Dubois-Briand via
lists.openembedded.org
<mathieu.dubois-briand=bootlin.com@lists.openembedded.org> a écrit :
>
> On Fri Dec 19, 2025 at 5:13 PM CET, Corentin Guillevic via
lists.openembedded.org wrote:
> > Among the three sources, only two have a single remote. So we can
replace
> > their entire structure 'remote' -> 'origin' -> 'uri' with a shorter one
> > (property 'uri' only).
> >
> > Signed-off-by: Corentin Guillevic <corentin.guillevic@smile.fr>
> > ---
>
> Hi Corentin,
>
> It looks like this is breaking some selftest:
>
> 025-12-19 13:57:11,641 - oe-selftest - INFO -
bblayers.BitbakeLayers.test_validate_examplelayersjson
(subunit.RemotedTestCase)
> 2025-12-19 13:57:11,642 - oe-selftest - INFO - ... FAIL
>
> Stderr:
> 2025-12-19 13:53:18,059 - oe-selftest - INFO - Adding: "include
selftest.inc" in
/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/conf/local.conf
> 2025-12-19 13:53:18,059 - oe-selftest - INFO - Adding: "include
bblayers.inc" in bblayers.conf
> 2025-12-19 13:57:11,642 - oe-selftest - INFO - 6: 13/38 66/653 (0.30s) (0
failed) (bblayers.BitbakeLayers.test_validate_examplelayersjson)
> 2025-12-19 13:57:11,642 - oe-selftest - INFO -
testtools.testresult.real._StringException: Traceback (most recent call
last):
> File
"/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/bblayers.py",
line 163, in test_validate_examplelayersjson
> self.validate_layersjson(json)
> File
"/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/bblayers.py",
line 159, in validate_layersjson
> self.validate_json(json, "layers.schema.json")
> File
"/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/bblayers.py",
line 152, in validate_json
> result = runCmd(
> ^^^^^^^
> File
"/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/utils/commands.py",
line 214, in runCmd
> raise AssertionError("Command '%s' returned non-zero exit status
%d:\n%s" % (command, result.status, exc_output))
> AssertionError: Command
'/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/tmp/work/x86_64-linux/python3-jsonschema-native/4.25.1/recipe-sysroot-native/usr/bin/nativepython3
/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/tmp/work/x86_64-linux/python3-jsonschema-native/4.25.1/recipe-sysroot-native/usr/bin/jsonschema
-i
/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/files/layers.example.json
--base-uri
file:///srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/bitbake/bin/../setup-schema/
/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/bitbake/bin/../setup-schema/layers.schema.json'
returned non-zero exit status 1:
>
/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/tmp/work/x86_64-linux/python3-jsonschema-native/4.25.1/recipe-sysroot-native/usr/bin/jsonschema:5:
DeprecationWarning: The jsonschema CLI is deprecated and will be removed in
a future version. Please use check-jsonschema instead, which can be
installed from https://pypi.org/project/check-jsonschema/
> from jsonschema.cli import main
> {'branch': 'master', 'describe': '', 'uri': '
https://github.com/kanavin/meta-alex', 'rev':
'05b25605fb8b2399e4706d7323828676bf0da0b5'}: Additional properties are not
allowed ('uri' was unexpected)
> {'branch': 'master', 'describe': '15.0-hardknott-3.3-310-g0a96edae',
'uri': 'git://git.yoctoproject.org/meta-intel', 'rev':
'0a96edae609a3f48befac36af82cf1eed6786b4a'}: Additional properties are not
allowed ('uri' was unexpected)
>
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2867
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2762
> https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3003
>
> Can you have a look at this?
Hello,
The bitbake commit for this run is 96d10e4702b8ef53d9893f9b9e0c1d9c4585667c
(the current mathieu/master-next-success).
It does not contain the needed patches:
* [PATCH v4 1/4] bitbake-setup: add inline URI
https://lists.openembedded.org/g/bitbake-devel/message/18624
* [PATCH v4 2/4] layers.schema.json: support 'uri'
https://lists.openembedded.org/g/bitbake-devel/message/18625
* [PATCH v4 3/4] bitbake-setup: use URI shortcut for all configurations
https://lists.openembedded.org/g/bitbake-devel/message/18626
* [PATCH v4 4/4] doc/bitbake-setup: document "uri" property
https://lists.openembedded.org/g/bitbake-devel/message/18627
Can you retry this patch with a bitbake branch containing these commits?
(In retrospect, the link between the series should have been made more
clear)
> Thanks,
> Mathieu
>
> --
> Mathieu Dubois-Briand, Bootlin
> Embedded Linux and Kernel engineering
> https://bootlin.com
>
>
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#228252):
https://lists.openembedded.org/g/openembedded-core/message/228252
> Mute This Topic: https://lists.openembedded.org/mt/116862096/4316185
> Group Owner: openembedded-core+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [
yoann.congal@smile.fr]
> -=-=-=-=-=-=-=-=-=-=-=-
>
--
Yoann Congal
Smile ECS
[-- Attachment #2: Type: text/html, Size: 7492 bytes --]
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [OE-core] [PATCH v4 2/2] meta/files/layers.schema.json: use URI shortcut for some remotes
2025-12-21 17:33 ` Yoann Congal
@ 2025-12-22 7:10 ` Mathieu Dubois-Briand
0 siblings, 0 replies; 5+ messages in thread
From: Mathieu Dubois-Briand @ 2025-12-22 7:10 UTC (permalink / raw)
To: Yoann Congal; +Cc: corentin.guillevic, openembedded-core
On Sun Dec 21, 2025 at 6:33 PM CET, Yoann Congal wrote:
> Le dim. 21 déc. 2025 à 10:05, Mathieu Dubois-Briand via
> lists.openembedded.org
> <mathieu.dubois-briand=bootlin.com@lists.openembedded.org> a écrit :
>>
>> On Fri Dec 19, 2025 at 5:13 PM CET, Corentin Guillevic via
> lists.openembedded.org wrote:
>> > Among the three sources, only two have a single remote. So we can
> replace
>> > their entire structure 'remote' -> 'origin' -> 'uri' with a shorter one
>> > (property 'uri' only).
>> >
>> > Signed-off-by: Corentin Guillevic <corentin.guillevic@smile.fr>
>> > ---
>>
>> Hi Corentin,
>>
>> It looks like this is breaking some selftest:
>>
>> 025-12-19 13:57:11,641 - oe-selftest - INFO -
> bblayers.BitbakeLayers.test_validate_examplelayersjson
> (subunit.RemotedTestCase)
>> 2025-12-19 13:57:11,642 - oe-selftest - INFO - ... FAIL
>>
>> Stderr:
>> 2025-12-19 13:53:18,059 - oe-selftest - INFO - Adding: "include
> selftest.inc" in
> /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/conf/local.conf
>> 2025-12-19 13:53:18,059 - oe-selftest - INFO - Adding: "include
> bblayers.inc" in bblayers.conf
>> 2025-12-19 13:57:11,642 - oe-selftest - INFO - 6: 13/38 66/653 (0.30s) (0
> failed) (bblayers.BitbakeLayers.test_validate_examplelayersjson)
>> 2025-12-19 13:57:11,642 - oe-selftest - INFO -
> testtools.testresult.real._StringException: Traceback (most recent call
> last):
>> File
> "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/bblayers.py",
> line 163, in test_validate_examplelayersjson
>> self.validate_layersjson(json)
>> File
> "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/bblayers.py",
> line 159, in validate_layersjson
>> self.validate_json(json, "layers.schema.json")
>> File
> "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/selftest/cases/bblayers.py",
> line 152, in validate_json
>> result = runCmd(
>> ^^^^^^^
>> File
> "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/lib/oeqa/utils/commands.py",
> line 214, in runCmd
>> raise AssertionError("Command '%s' returned non-zero exit status
> %d:\n%s" % (command, result.status, exc_output))
>> AssertionError: Command
> '/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/tmp/work/x86_64-linux/python3-jsonschema-native/4.25.1/recipe-sysroot-native/usr/bin/nativepython3
> /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/tmp/work/x86_64-linux/python3-jsonschema-native/4.25.1/recipe-sysroot-native/usr/bin/jsonschema
> -i
> /srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/openembedded-core/meta/files/layers.example.json
> --base-uri
> file:///srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/bitbake/bin/../setup-schema/
> /srv/pokybuild/yocto-worker/oe-selftest-debian/build/layers/bitbake/bin/../setup-schema/layers.schema.json'
> returned non-zero exit status 1:
>>
> /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-917910/tmp/work/x86_64-linux/python3-jsonschema-native/4.25.1/recipe-sysroot-native/usr/bin/jsonschema:5:
> DeprecationWarning: The jsonschema CLI is deprecated and will be removed in
> a future version. Please use check-jsonschema instead, which can be
> installed from https://pypi.org/project/check-jsonschema/
>> from jsonschema.cli import main
>> {'branch': 'master', 'describe': '', 'uri': '
> https://github.com/kanavin/meta-alex', 'rev':
> '05b25605fb8b2399e4706d7323828676bf0da0b5'}: Additional properties are not
> allowed ('uri' was unexpected)
>> {'branch': 'master', 'describe': '15.0-hardknott-3.3-310-g0a96edae',
> 'uri': 'git://git.yoctoproject.org/meta-intel', 'rev':
> '0a96edae609a3f48befac36af82cf1eed6786b4a'}: Additional properties are not
> allowed ('uri' was unexpected)
>>
>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/2867
>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/2762
>> https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3003
>>
>> Can you have a look at this?
>
> Hello,
>
> The bitbake commit for this run is 96d10e4702b8ef53d9893f9b9e0c1d9c4585667c
> (the current mathieu/master-next-success).
> It does not contain the needed patches:
> * [PATCH v4 1/4] bitbake-setup: add inline URI
> https://lists.openembedded.org/g/bitbake-devel/message/18624
> * [PATCH v4 2/4] layers.schema.json: support 'uri'
> https://lists.openembedded.org/g/bitbake-devel/message/18625
> * [PATCH v4 3/4] bitbake-setup: use URI shortcut for all configurations
> https://lists.openembedded.org/g/bitbake-devel/message/18626
> * [PATCH v4 4/4] doc/bitbake-setup: document "uri" property
> https://lists.openembedded.org/g/bitbake-devel/message/18627
>
> Can you retry this patch with a bitbake branch containing these commits?
> (In retrospect, the link between the series should have been made more
> clear)
>
Yes, I was a bit too quick and took the two series out of sync. I will
add this one back to the queue.
Thanks,
Mathieu
--
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2025-12-22 7:10 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-12-19 16:13 [PATCH v4 1/2] oe-setup-layers: support inline URI Corentin Guillevic
2025-12-19 16:13 ` [PATCH v4 2/2] meta/files/layers.schema.json: use URI shortcut for some remotes Corentin Guillevic
2025-12-21 9:05 ` [OE-core] " Mathieu Dubois-Briand
2025-12-21 17:33 ` Yoann Congal
2025-12-22 7:10 ` Mathieu Dubois-Briand
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.