All of lore.kernel.org
 help / color / mirror / Atom feed
From: kernel test robot <oliver.sang@intel.com>
To: Frederic Weisbecker <frederic@kernel.org>
Cc: <oe-lkp@lists.linux.dev>, <lkp@intel.com>,
	<linux-kernel@vger.kernel.org>,
	Dan Carpenter <dan.carpenter@linaro.org>, <kvm@vger.kernel.org>,
	<linux-crypto@vger.kernel.org>, <linux-pm@vger.kernel.org>,
	<dri-devel@lists.freedesktop.org>,
	<intel-gfx@lists.freedesktop.org>,
	<linux-arm-msm@vger.kernel.org>,
	<freedreno@lists.freedesktop.org>, <linux-media@vger.kernel.org>,
	<netdev@vger.kernel.org>, <intel-wired-lan@lists.osuosl.org>,
	<chrome-platform@lists.linux.dev>, <linux-spi@vger.kernel.org>,
	<linux-usb@vger.kernel.org>, <virtualization@lists.linux.dev>,
	<linux-watchdog@vger.kernel.org>, <linux-erofs@lists.ozlabs.org>,
	<rcu@vger.kernel.org>, <oliver.sang@intel.com>
Subject: [linus:master] [treewide]  b04e317b52: INFO:task_blocked_for_more_than#seconds
Date: Wed, 12 Feb 2025 12:57:07 +0800	[thread overview]
Message-ID: <202502121025.55bfa801-lkp@intel.com> (raw)



Hello,


we noticed the issue happens with a low rate on this commit, but keeps clean
on parent when we even run the tests up to 999 times. just FYI.

41f70d8e16349c65 b04e317b522630b46f78ee62ecb
---------------- ---------------------------
       fail:runs  %reproduction    fail:runs
           |             |             |
           :999          2%          16:999   dmesg.INFO:task_blocked_for_more_than#seconds
           :999          2%          16:999   dmesg.Kernel_panic-not_syncing:hung_task:blocked_tasks


kernel test robot noticed "INFO:task_blocked_for_more_than#seconds" on:

commit: b04e317b522630b46f78ee62ecbdc5734e8d43de ("treewide: Introduce kthread_run_worker[_on_cpu]()")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

[test failed on linus/master      7ee983c850b40043ac4751836fbd9a2b4d0c5937]
[test failed on linux-next/master ed58d103e6da15a442ff87567898768dc3a66987]

in testcase: rcuscale
version: 
with following parameters:

	runtime: 300s
	scale_type: rcu



config: i386-randconfig-052-20250205
compiler: gcc-12
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G

(please refer to attached dmesg/kmsg for entire log/backtrace)



If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202502121025.55bfa801-lkp@intel.com


[ 1023.501569][   T32] INFO: task UVCG:82 blocked for more than 491 seconds.
[ 1023.510932][   T32]       Tainted: G                T  6.13.0-rc2-00014-gb04e317b5226 #1
[ 1023.528506][   T32] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1023.551252][   T32] task:UVCG            state:D stack:0     pid:82    tgid:82    ppid:2      flags:0x00004000
[ 1023.582336][   T32] Call Trace:
[ 1023.599286][ T32] __schedule (kernel/sched/core.c:5372 kernel/sched/core.c:6756) 
[ 1023.617204][ T32] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) 
[ 1023.637326][ T32] schedule (kernel/sched/core.c:6834 kernel/sched/core.c:6848) 
[ 1023.653175][ T32] schedule_preempt_disabled (kernel/sched/core.c:6906) 
[ 1023.672095][ T32] kthread (kernel/kthread.c:453) 
[ 1023.684220][ T32] ? kthread_flush_work (kernel/kthread.c:970) 
[ 1023.700611][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1023.716756][ T32] ret_from_fork (arch/x86/kernel/process.c:153) 
[ 1023.732758][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1023.748605][ T32] ret_from_fork_asm (arch/x86/entry/entry_32.S:737) 
[ 1023.764004][ T32] entry_INT80_32 (arch/x86/entry/entry_32.S:942) 
[ 1023.776959][   T32]
[ 1023.776959][   T32] Showing all locks held in the system:
[ 1023.803410][   T32] 1 lock held by khungtaskd/32:
[ 1023.815913][ T32] #0: c3baa3d4 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire (include/linux/rcupdate.h:336) 
[ 1023.842061][   T32] 1 lock held by in:imklog/227:
[ 1023.849815][ T32] #0: ecd0e884 (&f->f_pos_lock){+.+.}-{4:4}, at: fdget_pos (fs/file.c:1194) 
[ 1023.877481][   T32] 1 lock held by dmesg/494:
[ 1023.885260][   T32] 2 locks held by depmod/567:
[ 1023.903871][   T32]
[ 1023.907007][   T32] =============================================
[ 1023.907007][   T32]
[ 1023.938278][   T32] Kernel panic - not syncing: hung_task: blocked tasks
[ 1023.951013][   T32] CPU: 0 UID: 0 PID: 32 Comm: khungtaskd Tainted: G                T  6.13.0-rc2-00014-gb04e317b5226 #1 78ab3595737b7bb7ccfbfed5c1dcb65e25af99a7
[ 1023.973770][   T32] Tainted: [T]=RANDSTRUCT
[ 1023.979010][   T32] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
[ 1023.991787][   T32] Call Trace:
[ 1023.995862][ T32] dump_stack_lvl (lib/dump_stack.c:122) 
[ 1024.001535][ T32] dump_stack (lib/dump_stack.c:130) 
[ 1024.006622][ T32] panic (kernel/panic.c:258 kernel/panic.c:375) 
[ 1024.011360][ T32] check_hung_uninterruptible_tasks (kernel/hung_task.c:239) 
[ 1024.018678][ T32] watchdog (kernel/hung_task.c:398) 
[ 1024.023798][ T32] kthread (kernel/kthread.c:466) 
[ 1024.028697][ T32] ? check_hung_uninterruptible_tasks (kernel/hung_task.c:380) 
[ 1024.036601][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1024.042763][ T32] ret_from_fork (arch/x86/kernel/process.c:153) 
[ 1024.048431][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1024.054588][ T32] ret_from_fork_asm (arch/x86/entry/entry_32.S:737) 
[ 1024.060422][ T32] entry_INT80_32 (arch/x86/entry/entry_32.S:942) 
[ 1024.067908][   T32] Kernel Offset: disabled



The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250212/202502121025.55bfa801-lkp@intel.com



-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki


WARNING: multiple messages have this Message-ID (diff)
From: kernel test robot <oliver.sang@intel.com>
To: Frederic Weisbecker <frederic@kernel.org>
Cc: kvm@vger.kernel.org, dri-devel@lists.freedesktop.org,
	Dan Carpenter <dan.carpenter@linaro.org>,
	chrome-platform@lists.linux.dev,
	intel-wired-lan@lists.osuosl.org, linux-media@vger.kernel.org,
	freedreno@lists.freedesktop.org, linux-watchdog@vger.kernel.org,
	linux-pm@vger.kernel.org, linux-arm-msm@vger.kernel.org,
	intel-gfx@lists.freedesktop.org, virtualization@lists.linux.dev,
	rcu@vger.kernel.org, netdev@vger.kernel.org,
	linux-usb@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-spi@vger.kernel.org, oliver.sang@intel.com,
	linux-crypto@vger.kernel.org, oe-lkp@lists.linux.dev,
	linux-erofs@lists.ozlabs.org
Subject: [Intel-wired-lan] [linus:master] [treewide] b04e317b52: INFO:task_blocked_for_more_than#seconds
Date: Wed, 12 Feb 2025 12:57:07 +0800	[thread overview]
Message-ID: <202502121025.55bfa801-lkp@intel.com> (raw)



Hello,


we noticed the issue happens with a low rate on this commit, but keeps clean
on parent when we even run the tests up to 999 times. just FYI.

41f70d8e16349c65 b04e317b522630b46f78ee62ecb
---------------- ---------------------------
       fail:runs  %reproduction    fail:runs
           |             |             |
           :999          2%          16:999   dmesg.INFO:task_blocked_for_more_than#seconds
           :999          2%          16:999   dmesg.Kernel_panic-not_syncing:hung_task:blocked_tasks


kernel test robot noticed "INFO:task_blocked_for_more_than#seconds" on:

commit: b04e317b522630b46f78ee62ecbdc5734e8d43de ("treewide: Introduce kthread_run_worker[_on_cpu]()")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

[test failed on linus/master      7ee983c850b40043ac4751836fbd9a2b4d0c5937]
[test failed on linux-next/master ed58d103e6da15a442ff87567898768dc3a66987]

in testcase: rcuscale
version: 
with following parameters:

	runtime: 300s
	scale_type: rcu



config: i386-randconfig-052-20250205
compiler: gcc-12
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G

(please refer to attached dmesg/kmsg for entire log/backtrace)



If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202502121025.55bfa801-lkp@intel.com


[ 1023.501569][   T32] INFO: task UVCG:82 blocked for more than 491 seconds.
[ 1023.510932][   T32]       Tainted: G                T  6.13.0-rc2-00014-gb04e317b5226 #1
[ 1023.528506][   T32] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1023.551252][   T32] task:UVCG            state:D stack:0     pid:82    tgid:82    ppid:2      flags:0x00004000
[ 1023.582336][   T32] Call Trace:
[ 1023.599286][ T32] __schedule (kernel/sched/core.c:5372 kernel/sched/core.c:6756) 
[ 1023.617204][ T32] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) 
[ 1023.637326][ T32] schedule (kernel/sched/core.c:6834 kernel/sched/core.c:6848) 
[ 1023.653175][ T32] schedule_preempt_disabled (kernel/sched/core.c:6906) 
[ 1023.672095][ T32] kthread (kernel/kthread.c:453) 
[ 1023.684220][ T32] ? kthread_flush_work (kernel/kthread.c:970) 
[ 1023.700611][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1023.716756][ T32] ret_from_fork (arch/x86/kernel/process.c:153) 
[ 1023.732758][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1023.748605][ T32] ret_from_fork_asm (arch/x86/entry/entry_32.S:737) 
[ 1023.764004][ T32] entry_INT80_32 (arch/x86/entry/entry_32.S:942) 
[ 1023.776959][   T32]
[ 1023.776959][   T32] Showing all locks held in the system:
[ 1023.803410][   T32] 1 lock held by khungtaskd/32:
[ 1023.815913][ T32] #0: c3baa3d4 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire (include/linux/rcupdate.h:336) 
[ 1023.842061][   T32] 1 lock held by in:imklog/227:
[ 1023.849815][ T32] #0: ecd0e884 (&f->f_pos_lock){+.+.}-{4:4}, at: fdget_pos (fs/file.c:1194) 
[ 1023.877481][   T32] 1 lock held by dmesg/494:
[ 1023.885260][   T32] 2 locks held by depmod/567:
[ 1023.903871][   T32]
[ 1023.907007][   T32] =============================================
[ 1023.907007][   T32]
[ 1023.938278][   T32] Kernel panic - not syncing: hung_task: blocked tasks
[ 1023.951013][   T32] CPU: 0 UID: 0 PID: 32 Comm: khungtaskd Tainted: G                T  6.13.0-rc2-00014-gb04e317b5226 #1 78ab3595737b7bb7ccfbfed5c1dcb65e25af99a7
[ 1023.973770][   T32] Tainted: [T]=RANDSTRUCT
[ 1023.979010][   T32] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
[ 1023.991787][   T32] Call Trace:
[ 1023.995862][ T32] dump_stack_lvl (lib/dump_stack.c:122) 
[ 1024.001535][ T32] dump_stack (lib/dump_stack.c:130) 
[ 1024.006622][ T32] panic (kernel/panic.c:258 kernel/panic.c:375) 
[ 1024.011360][ T32] check_hung_uninterruptible_tasks (kernel/hung_task.c:239) 
[ 1024.018678][ T32] watchdog (kernel/hung_task.c:398) 
[ 1024.023798][ T32] kthread (kernel/kthread.c:466) 
[ 1024.028697][ T32] ? check_hung_uninterruptible_tasks (kernel/hung_task.c:380) 
[ 1024.036601][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1024.042763][ T32] ret_from_fork (arch/x86/kernel/process.c:153) 
[ 1024.048431][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1024.054588][ T32] ret_from_fork_asm (arch/x86/entry/entry_32.S:737) 
[ 1024.060422][ T32] entry_INT80_32 (arch/x86/entry/entry_32.S:942) 
[ 1024.067908][   T32] Kernel Offset: disabled



The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250212/202502121025.55bfa801-lkp@intel.com



-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki


WARNING: multiple messages have this Message-ID (diff)
From: kernel test robot <oliver.sang@intel.com>
To: Frederic Weisbecker <frederic@kernel.org>
Cc: kvm@vger.kernel.org, dri-devel@lists.freedesktop.org,
	Dan Carpenter <dan.carpenter@linaro.org>,
	chrome-platform@lists.linux.dev, lkp@intel.com,
	intel-wired-lan@lists.osuosl.org, linux-media@vger.kernel.org,
	freedreno@lists.freedesktop.org, linux-watchdog@vger.kernel.org,
	linux-pm@vger.kernel.org, linux-arm-msm@vger.kernel.org,
	intel-gfx@lists.freedesktop.org, virtualization@lists.linux.dev,
	rcu@vger.kernel.org, netdev@vger.kernel.org,
	linux-usb@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-spi@vger.kernel.org, oliver.sang@intel.com,
	linux-crypto@vger.kernel.org, oe-lkp@lists.linux.dev,
	linux-erofs@lists.ozlabs.org
Subject: [linus:master] [treewide]  b04e317b52: INFO:task_blocked_for_more_than#seconds
Date: Wed, 12 Feb 2025 12:57:07 +0800	[thread overview]
Message-ID: <202502121025.55bfa801-lkp@intel.com> (raw)



Hello,


we noticed the issue happens with a low rate on this commit, but keeps clean
on parent when we even run the tests up to 999 times. just FYI.

41f70d8e16349c65 b04e317b522630b46f78ee62ecb
---------------- ---------------------------
       fail:runs  %reproduction    fail:runs
           |             |             |
           :999          2%          16:999   dmesg.INFO:task_blocked_for_more_than#seconds
           :999          2%          16:999   dmesg.Kernel_panic-not_syncing:hung_task:blocked_tasks


kernel test robot noticed "INFO:task_blocked_for_more_than#seconds" on:

commit: b04e317b522630b46f78ee62ecbdc5734e8d43de ("treewide: Introduce kthread_run_worker[_on_cpu]()")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master

[test failed on linus/master      7ee983c850b40043ac4751836fbd9a2b4d0c5937]
[test failed on linux-next/master ed58d103e6da15a442ff87567898768dc3a66987]

in testcase: rcuscale
version: 
with following parameters:

	runtime: 300s
	scale_type: rcu



config: i386-randconfig-052-20250205
compiler: gcc-12
test machine: qemu-system-x86_64 -enable-kvm -cpu SandyBridge -smp 2 -m 16G

(please refer to attached dmesg/kmsg for entire log/backtrace)



If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202502121025.55bfa801-lkp@intel.com


[ 1023.501569][   T32] INFO: task UVCG:82 blocked for more than 491 seconds.
[ 1023.510932][   T32]       Tainted: G                T  6.13.0-rc2-00014-gb04e317b5226 #1
[ 1023.528506][   T32] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ 1023.551252][   T32] task:UVCG            state:D stack:0     pid:82    tgid:82    ppid:2      flags:0x00004000
[ 1023.582336][   T32] Call Trace:
[ 1023.599286][ T32] __schedule (kernel/sched/core.c:5372 kernel/sched/core.c:6756) 
[ 1023.617204][ T32] ? __this_cpu_preempt_check (lib/smp_processor_id.c:67) 
[ 1023.637326][ T32] schedule (kernel/sched/core.c:6834 kernel/sched/core.c:6848) 
[ 1023.653175][ T32] schedule_preempt_disabled (kernel/sched/core.c:6906) 
[ 1023.672095][ T32] kthread (kernel/kthread.c:453) 
[ 1023.684220][ T32] ? kthread_flush_work (kernel/kthread.c:970) 
[ 1023.700611][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1023.716756][ T32] ret_from_fork (arch/x86/kernel/process.c:153) 
[ 1023.732758][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1023.748605][ T32] ret_from_fork_asm (arch/x86/entry/entry_32.S:737) 
[ 1023.764004][ T32] entry_INT80_32 (arch/x86/entry/entry_32.S:942) 
[ 1023.776959][   T32]
[ 1023.776959][   T32] Showing all locks held in the system:
[ 1023.803410][   T32] 1 lock held by khungtaskd/32:
[ 1023.815913][ T32] #0: c3baa3d4 (rcu_read_lock){....}-{1:3}, at: rcu_lock_acquire (include/linux/rcupdate.h:336) 
[ 1023.842061][   T32] 1 lock held by in:imklog/227:
[ 1023.849815][ T32] #0: ecd0e884 (&f->f_pos_lock){+.+.}-{4:4}, at: fdget_pos (fs/file.c:1194) 
[ 1023.877481][   T32] 1 lock held by dmesg/494:
[ 1023.885260][   T32] 2 locks held by depmod/567:
[ 1023.903871][   T32]
[ 1023.907007][   T32] =============================================
[ 1023.907007][   T32]
[ 1023.938278][   T32] Kernel panic - not syncing: hung_task: blocked tasks
[ 1023.951013][   T32] CPU: 0 UID: 0 PID: 32 Comm: khungtaskd Tainted: G                T  6.13.0-rc2-00014-gb04e317b5226 #1 78ab3595737b7bb7ccfbfed5c1dcb65e25af99a7
[ 1023.973770][   T32] Tainted: [T]=RANDSTRUCT
[ 1023.979010][   T32] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-debian-1.16.2-1 04/01/2014
[ 1023.991787][   T32] Call Trace:
[ 1023.995862][ T32] dump_stack_lvl (lib/dump_stack.c:122) 
[ 1024.001535][ T32] dump_stack (lib/dump_stack.c:130) 
[ 1024.006622][ T32] panic (kernel/panic.c:258 kernel/panic.c:375) 
[ 1024.011360][ T32] check_hung_uninterruptible_tasks (kernel/hung_task.c:239) 
[ 1024.018678][ T32] watchdog (kernel/hung_task.c:398) 
[ 1024.023798][ T32] kthread (kernel/kthread.c:466) 
[ 1024.028697][ T32] ? check_hung_uninterruptible_tasks (kernel/hung_task.c:380) 
[ 1024.036601][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1024.042763][ T32] ret_from_fork (arch/x86/kernel/process.c:153) 
[ 1024.048431][ T32] ? kthread_is_per_cpu (kernel/kthread.c:413) 
[ 1024.054588][ T32] ret_from_fork_asm (arch/x86/entry/entry_32.S:737) 
[ 1024.060422][ T32] entry_INT80_32 (arch/x86/entry/entry_32.S:942) 
[ 1024.067908][   T32] Kernel Offset: disabled



The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20250212/202502121025.55bfa801-lkp@intel.com



-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki


             reply	other threads:[~2025-02-12  4:57 UTC|newest]

Thread overview: 3+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-02-12  4:57 kernel test robot [this message]
2025-02-12  4:57 ` [linus:master] [treewide] b04e317b52: INFO:task_blocked_for_more_than#seconds kernel test robot
2025-02-12  4:57 ` [Intel-wired-lan] " kernel test robot

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=202502121025.55bfa801-lkp@intel.com \
    --to=oliver.sang@intel.com \
    --cc=chrome-platform@lists.linux.dev \
    --cc=dan.carpenter@linaro.org \
    --cc=dri-devel@lists.freedesktop.org \
    --cc=frederic@kernel.org \
    --cc=freedreno@lists.freedesktop.org \
    --cc=intel-gfx@lists.freedesktop.org \
    --cc=intel-wired-lan@lists.osuosl.org \
    --cc=kvm@vger.kernel.org \
    --cc=linux-arm-msm@vger.kernel.org \
    --cc=linux-crypto@vger.kernel.org \
    --cc=linux-erofs@lists.ozlabs.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-media@vger.kernel.org \
    --cc=linux-pm@vger.kernel.org \
    --cc=linux-spi@vger.kernel.org \
    --cc=linux-usb@vger.kernel.org \
    --cc=linux-watchdog@vger.kernel.org \
    --cc=lkp@intel.com \
    --cc=netdev@vger.kernel.org \
    --cc=oe-lkp@lists.linux.dev \
    --cc=rcu@vger.kernel.org \
    --cc=virtualization@lists.linux.dev \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.