* a bug in autofs
@ 2002-12-01 7:16 Andrey R. Urazov
2002-12-01 18:29 ` Bryan O'Sullivan
2002-12-02 3:25 ` Jeremy Fitzhardinge
0 siblings, 2 replies; 10+ messages in thread
From: Andrey R. Urazov @ 2002-12-01 7:16 UTC (permalink / raw)
To: linux-kernel
[-- Attachment #1: Type: text/plain, Size: 10215 bytes --]
Hello,
I'm constantly observing a bug in autofs.
[1.] One line summary of the problem:
the system hangs when some operations are performed on autounted volumes
[2.] Full description of the problem/report:
I observed the problem in 2 forms:
1) when I run XMMS and its playlist contains entries laying somewhere
under /misc/cdrom but there is no cd in the drive or the cd in the
drive is not the one whose entries are stored in the playlist, it
takes about half a minute for the system to hang. Before it hangs
absolutely I get numerous messages "invalid seek on /dev/hdc" on my
virtual consoles
2) under /misc/summer there resides an ntfs volume with thousands of
files. And when I run
find /misc/summer
the system becames unusable after some amount of files is scanned.
Usually it just hangs. But one time "find" terminated with the
segmentation fault and then after 5 seconds or so the system hung.
The problem does not existed if the volumes are mounted through "mount".
Only automounting causes problems.
[3.] Keywords (i.e., modules, networking, kernel):
kernel, modules, automounting, autofs
[4.] Kernel version (from /proc/version):
Linux version 2.4.20 (coola@ktulu) (gcc version 3.2 20020903 (Red Hat
Linux 8.0 3.2-7)) #2 Sun Dec 1 12:48:50 NOVT 2002
the problem persists on both 2.4.19 and 2.4.20 and with both versions of
autofs (autofs and autofs4)
[5.] Output of Oops.. message (if applicable) with symbolic information
resolved (see Documentation/oops-tracing.txt)
[6.] A small shell script or example program which triggers the problem
(if possible)
find <path to an automounted volume with large amount of files>
[7.] Environment
doesn't matter, I reckon
[7.1.] Software (add the output of the ver_linux script here)
Linux ktulu 2.4.20 #2 Sun Dec 1 12:48:50 NOVT 2002 i686 athlon i386
GNU/Linux
Gnu C 3.2
Gnu make 3.79.1
util-linux 2.11r
mount 2.11r
modutils 2.4.18
e2fsprogs 1.27
jfsutils 1.0.17
reiserfsprogs 3.6.2
PPP 2.4.1
Linux C Library 2.2.93
Dynamic linker (ldd) 2.2.93
Procps 2.0.7
Net-tools 1.60
Kbd 1.06
Sh-utils 2.0.12
Modules Loaded nls_cp866 vfat fat maestro3 ac97_codec soundcore
af_packet
[7.2.] Processor information (from /proc/cpuinfo):
processor : 0
vendor_id : AuthenticAMD
cpu family : 6
model : 4
model name : AMD Athlon(tm) Processor
stepping : 4
cpu MHz : 1340.225
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 mmx fxsr syscall mmxext 3dnowext 3dnow
bogomips : 2673.86
[7.3.] Module information (from /proc/modules):
nls_cp866 4604 0 (autoclean)
vfat 12908 0 (autoclean)
fat 38776 0 (autoclean) [vfat]
maestro3 30544 0 (autoclean)
ac97_codec 13576 0 (autoclean) [maestro3]
soundcore 6500 2 (autoclean) [maestro3]
af_packet 15176 0 (autoclean)
[7.4.] Loaded driver and hardware information (/proc/ioports,
/proc/iomem)
0000-001f : dma1
0020-003f : pic1
0040-005f : timer
0060-006f : keyboard
0080-008f : dma page reg
00a0-00bf : pic2
00c0-00df : dma2
00f0-00ff : fpu
0170-0177 : ide1
01f0-01f7 : ide0
02f8-02ff : serial(auto)
0376-0376 : ide1
03c0-03df : vga+
03f6-03f6 : ide0
03f8-03ff : serial(auto)
0cf8-0cff : PCI conf1
a000-afff : PCI Bus #01
d800-d81f : 3Com Corporation 3c590 10BaseT [Vortex]
dc00-dcff : ESS Technology ES1988 Allegro-1
dc00-dcff : maestro3
ff00-ff0f : VIA Technologies, Inc. VT82C586B PIPC Bus Master IDE
ff00-ff07 : ide0
ff08-ff0f : ide1
00000000-0009fbff : System RAM
0009fc00-0009ffff : reserved
000a0000-000bffff : Video RAM area
000c0000-000c7fff : Video ROM
000f0000-000fffff : System ROM
00100000-07feffff : System RAM
00100000-00270240 : Kernel code
00270241-002ca3bf : Kernel data
07ff0000-07ff7fff : ACPI Tables
07ff8000-07ffffff : ACPI Non-volatile Storage
dec00000-decfffff : PCI Bus #01
df000000-df7fffff : Matrox Graphics, Inc. MGA 2064W [Millennium]
dfe00000-dfefffff : PCI Bus #01
dfffc000-dfffffff : Matrox Graphics, Inc. MGA 2064W [Millennium]
e0000000-e3ffffff : VIA Technologies, Inc. VT8367 [KT266]
fec00000-fec00fff : reserved
fee00000-fee00fff : reserved
fff80000-ffffffff : reserved
[7.5.] PCI information ('lspci -vvv' as root)
00:00.0 Host bridge: VIA Technologies, Inc. VT8367 [KT266]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort+ >SERR- <PERR-
Latency: 0
Region 0: Memory at e0000000 (32-bit, prefetchable) [size=64M]
Capabilities: [a0] AGP version 2.0
Status: RQ=31 SBA+ 64bit- FW- Rate=x1,x2
Command: RQ=0 SBA- AGP- 64bit- FW- Rate=<none>
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:01.0 PCI bridge: VIA Technologies, Inc. VT8367 [KT333 AGP] (prog-if
00 [Normal decode])
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort+ >SERR- <PERR-
Latency: 0
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 0000a000-0000afff
Memory behind bridge: dfe00000-dfefffff
Prefetchable memory behind bridge: dec00000-decfffff
BridgeCtl: Parity- SERR- NoISA+ VGA- MAbort- >Reset- FastB2B-
Capabilities: [80] Power Management version 2
Flags: PMEClk- DSI- D1+ D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:09.0 VGA compatible controller: Matrox Graphics, Inc. MGA 2064W
[Millennium] (rev 01) (prog-if 00 [VGA])
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping+ SERR- FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Interrupt: pin A routed to IRQ 11
Region 0: Memory at dfffc000 (32-bit, non-prefetchable)
[size=16K]
Region 1: Memory at df000000 (32-bit, prefetchable) [size=8M]
Expansion ROM at dffe0000 [disabled] [size=64K]
00:0a.0 Multimedia audio controller: ESS Technology ES1988 Allegro-1
(rev 10)
Subsystem: ESS Technology ESS Allegro-1 Audiodrive
Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (500ns min, 6000ns max)
Interrupt: pin A routed to IRQ 10
Region 0: I/O ports at dc00 [size=256]
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI+ D1+ D2+ AuxCurrent=0mA
PME(D0-,D1+,D2+,D3hot+,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:0c.0 Ethernet controller: 3Com Corporation 3c590 10BaseT [Vortex]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR+ FastB2B-
Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 32 (750ns min, 2000ns max)
Interrupt: pin A routed to IRQ 9
Region 0: I/O ports at d800 [size=32]
Expansion ROM at dffd0000 [disabled] [size=64K]
00:11.0 ISA bridge: VIA Technologies, Inc. VT8233 PCI to ISA Bridge
Subsystem: VIA Technologies, Inc. VT8233 PCI to ISA Bridge
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping+ SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 0
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
00:11.1 IDE interface: VIA Technologies, Inc. VT82C586B PIPC Bus Master
IDE (rev 06) (prog-if 8a [Master SecP PriP])
Subsystem: VIA Technologies, Inc. VT82C586B PIPC Bus Master IDE
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
Latency: 32
Interrupt: pin A routed to IRQ 14
Region 4: I/O ports at ff00 [size=16]
Capabilities: [c0] Power Management version 2
Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
Status: D0 PME-Enable- DSel=0 DScale=0 PME-
[7.6.] SCSI information (from /proc/scsi/scsi)
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
Vendor: _NEC Model: NR-7900A Rev: 1.23
Type: CD-ROM ANSI SCSI revision: 02
[7.7.] Other information that might be relevant to the problem (please
look in /proc and include all information that you think to be
relevant):
[X.] Other notes, patches, fixes, workarounds:
workaround is to use manual mounting instead of automounting
Best regards,
Andrey Urazov
--
Phasers locked on target, Captain.
--
dimanche 01 décembre, 2002, 13:00:36 +0600 - Andrey R. Urazov (mailto:coola@ngs.ru)
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: a bug in autofs
2002-12-01 7:16 a bug in autofs Andrey R. Urazov
@ 2002-12-01 18:29 ` Bryan O'Sullivan
2002-12-02 4:03 ` Andrey R. Urazov
2002-12-02 3:25 ` Jeremy Fitzhardinge
1 sibling, 1 reply; 10+ messages in thread
From: Bryan O'Sullivan @ 2002-12-01 18:29 UTC (permalink / raw)
To: Andrey R. Urazov; +Cc: linux-kernel
On Sat, 2002-11-30 at 23:16, Andrey R. Urazov wrote:
> I'm constantly observing a bug in autofs.
Which version of the autofs userspace tools are you using?
<b
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: a bug in autofs
2002-12-01 18:29 ` Bryan O'Sullivan
@ 2002-12-02 4:03 ` Andrey R. Urazov
0 siblings, 0 replies; 10+ messages in thread
From: Andrey R. Urazov @ 2002-12-02 4:03 UTC (permalink / raw)
To: Bryan O'Sullivan; +Cc: linux-kernel
[-- Attachment #1: Type: text/plain, Size: 471 bytes --]
On Sun, Dec 01, 2002 at 10:29:23AM -0800, Bryan O'Sullivan wrote:
> On Sat, 2002-11-30 at 23:16, Andrey R. Urazov wrote:
>
> > I'm constantly observing a bug in autofs.
>
> Which version of the autofs userspace tools are you using?
3.1.7 as supplied in Red Hat 8.0
Best regards,
Andrey Urazov
--
Murray's Rule:
Any country with "democratic" in the title isn't.
--
lundi 02 décembre, 2002, 10:03:00 +0600 - Andrey R. Urazov (mailto:coola@ngs.ru)
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: a bug in autofs
2002-12-01 7:16 a bug in autofs Andrey R. Urazov
2002-12-01 18:29 ` Bryan O'Sullivan
@ 2002-12-02 3:25 ` Jeremy Fitzhardinge
2002-12-02 7:57 ` Andrey R. Urazov
1 sibling, 1 reply; 10+ messages in thread
From: Jeremy Fitzhardinge @ 2002-12-02 3:25 UTC (permalink / raw)
To: Andrey R. Urazov; +Cc: Linux Kernel List
On Sat, 2002-11-30 at 23:16, Andrey R. Urazov wrote:
> 1) when I run XMMS and its playlist contains entries laying somewhere
> under /misc/cdrom but there is no cd in the drive or the cd in the
> drive is not the one whose entries are stored in the playlist, it
> takes about half a minute for the system to hang. Before it hangs
> absolutely I get numerous messages "invalid seek on /dev/hdc" on my
> virtual consoles
What happens if you try to manually mount the cdrom when there's nothing
in the drive?
> 2) under /misc/summer there resides an ntfs volume with thousands of
> files. And when I run
>
> find /misc/summer
>
> the system becames unusable after some amount of files is scanned.
> Usually it just hangs. But one time "find" terminated with the
> segmentation fault and then after 5 seconds or so the system hung.
Can you reproduce this with some other filesystem type (something which
is less flaky than NTFS)?
How many files are on the NTFS filesystem?
> The problem does not existed if the volumes are mounted through "mount".
> Only automounting causes problems.
Does this comment also apply to the cdrom case?
What mount options are you using to mount these filesystems? Are they
the same when you do it manually and using autofs?
What does the "dentry_cache" line say in /proc/slabinfo while you're
running the find on the NTFS filesystem?
Thanks,
J
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: a bug in autofs
2002-12-02 3:25 ` Jeremy Fitzhardinge
@ 2002-12-02 7:57 ` Andrey R. Urazov
2002-12-02 8:39 ` Jeremy Fitzhardinge
0 siblings, 1 reply; 10+ messages in thread
From: Andrey R. Urazov @ 2002-12-02 7:57 UTC (permalink / raw)
To: Jeremy Fitzhardinge; +Cc: linux-kernel
[-- Attachment #1: Type: text/plain, Size: 3733 bytes --]
On Sun, Dec 01, 2002 at 07:25:32PM -0800, Jeremy Fitzhardinge wrote:
> What happens if you try to manually mount the cdrom when there's nothing
> in the drive?
[root@ktulu coola]# en mount -o ro -t iso9660 /dev/cdrom /mnt
mount: No medium found
with this attempt a new line reading `cdrom: open failed' is appended to
the dmesg output
> > 2) under /misc/summer there resides an ntfs volume with thousands of
> > files. And when I run
> >
> > find /misc/summer
> >
> > the system becames unusable after some amount of files is scanned.
> > Usually it just hangs. But one time "find" terminated with the
> > segmentation fault and then after 5 seconds or so the system hung.
>
> Can you reproduce this with some other filesystem type (something which
> is less flaky than NTFS)?
Tried with fat32 and found no problems, everything is okay in this case.
>
> How many files are on the NTFS filesystem?
To test for the bug I used the following command:
find /misc/summer | tee /dev/tty | wc
in most cases I didn't see the output of wc, but two times I managed to
(in this cases `find' managed to terminate before the system hung).
24849 files, 1712874 total characters (in filenames)
25087 files, 1737450 characters
> > The problem does not existed if the volumes are mounted through
> > "mount". Only automounting causes problems.
Sorry for misinforming you. When this was written I tried manual
mounting only once and all was okay during this trial. But when
I repeated the test, the system crashed as in the case of automounting.
And I didn't manage to perform a succesful find over ntfs volume once
more. So the problem here is probably in the ntfs driver, not autofs.
>
> Does this comment also apply to the cdrom case?
This applies to cdrom case. Just tested it. I.e. the system doesn't
crash when I invoke xmms with a playlist referring to nonexistent cd.
> What mount options are you using to mount these filesystems? Are they
> the same when you do it manually and using autofs?
Tried different combinations. Does not matter.
> What does the "dentry_cache" line say in /proc/slabinfo while you're
> running the find on the NTFS filesystem?
It varies so I'm including several snapshots:
[coola@ktulu coola]$ cat /proc/slabinfo|grep dentry
dentry_cache 2069 4380 128 146 146 1
[coola@ktulu coola]$ cat /proc/slabinfo|grep dentry
dentry_cache 5311 5340 128 178 178 1
[coola@ktulu coola]$ cat /proc/slabinfo|grep dentry
dentry_cache 2901 4200 128 140 140 1
[coola@ktulu coola]$ cat /proc/slabinfo|grep dentry
dentry_cache 5383 5400 128 180 180 1
[coola@ktulu coola]$ cat /proc/slabinfo|grep dentry
dentry_cache 1358 4080 128 136 136 1
[coola@ktulu coola]$ cat /proc/slabinfo|grep dentry
dentry_cache 3110 4080 128 136 136 1
[coola@ktulu coola]$ cat /proc/slabinfo|grep dentry
dentry_cache 3731 4920 128 164 164 1
[coola@ktulu coola]$ cat /proc/slabinfo|grep dentry
dentry_cache 4068 4920 128 164 164 1
[coola@ktulu coola]$ cat /proc/slabinfo|grep dentry
dentry_cache 4132 4920 128 164 164 1
[coola@ktulu coola]$ cat /proc/slabinfo|grep dentry
dentry_cache 4192 4920 128 164 164 1
[coola@ktulu coola]$ cat /proc/slabinfo|grep dentry
dentry_cache 4258 4920 128 164 164 1
> Thanks,
> J
That's me who should thank you.
Best regards,
Andrey Urazov
--
Lying is an indispensable part of making life tolerable.
-- Bergan Evans
--
lundi 02 décembre, 2002, 10:06:15 +0600 - Andrey R. Urazov (mailto:coola@ngs.ru)
[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 10+ messages in thread* Re: a bug in autofs
2002-12-02 7:57 ` Andrey R. Urazov
@ 2002-12-02 8:39 ` Jeremy Fitzhardinge
2002-12-02 15:17 ` Andrey R. Urazov
0 siblings, 1 reply; 10+ messages in thread
From: Jeremy Fitzhardinge @ 2002-12-02 8:39 UTC (permalink / raw)
To: Andrey R. Urazov; +Cc: Linux Kernel List
On Sun, 2002-12-01 at 23:57, Andrey R. Urazov wrote:
> On Sun, Dec 01, 2002 at 07:25:32PM -0800, Jeremy Fitzhardinge wrote:
> > What happens if you try to manually mount the cdrom when there's nothing
> > in the drive?
> [root@ktulu coola]# en mount -o ro -t iso9660 /dev/cdrom /mnt
> mount: No medium found
>
> with this attempt a new line reading `cdrom: open failed' is appended to
> the dmesg output
Don't really know about this. Autofs just uses mount to mount the
filesystem, so whatever happens when you manually mount should also
happen when you automount. What's "en"? Is this the same mount command
line as automount execs? What's in your auto.master? Which module are
you using?
> Sorry for misinforming you. When this was written I tried manual
> mounting only once and all was okay during this trial. But when
> I repeated the test, the system crashed as in the case of automounting.
> And I didn't manage to perform a succesful find over ntfs volume once
> more. So the problem here is probably in the ntfs driver, not autofs.
That's what I suspected was happening, but I wanted to be sure.
J
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: a bug in autofs
2002-12-02 8:39 ` Jeremy Fitzhardinge
@ 2002-12-02 15:17 ` Andrey R. Urazov
[not found] ` <1038847726.2560.51.camel@ixodes.goop.org>
0 siblings, 1 reply; 10+ messages in thread
From: Andrey R. Urazov @ 2002-12-02 15:17 UTC (permalink / raw)
To: Jeremy Fitzhardinge
On Mon, Dec 02, 2002 at 12:39:06AM -0800, Jeremy Fitzhardinge wrote:
> > > What happens if you try to manually mount the cdrom when there's
> > > nothing in the drive?
> > [root@ktulu coola]# en mount -o ro -t iso9660 /dev/cdrom /mnt
> > mount: No medium found
> >
> > with this attempt a new line reading `cdrom: open failed' is appended to
> > the dmesg output
>
> Don't really know about this. Autofs just uses mount to mount the
> filesystem, so whatever happens when you manually mount should also
> happen when you automount. What's "en"? Is this the same mount command
> line as automount execs? What's in your auto.master? Which module are
> you using?
"en" is script changing locale to en_US so that you see message in
English.
auto.master:
# $Id: auto.master,v 1.2 1997/10/06 21:52:03 hpa Exp $
# Sample auto.master file
# Format of this file:
# mountpoint map options
# For details of the format look at autofs(8).
/misc /etc/auto.misc --timeout=60
Just tested that the system hangs when the autofs4 module is in use.
autofs (without 4) module does not cause any problems.
What is characteristic is that before the system hangs I get numerous
messages reading `invalid seek on drive /dev/hdc' on my virtual console.
Best regards,
Andrey Urazov
--
F: When into a room I plunge, I
Sometimes find some VIOLET FUNGI.
Then I linger, darkly brooding
On the poison they're exuding.
-- The Roguelet's ABC
--
lundi 02 décembre, 2002, 21:05:18 +0600 - Andrey R. Urazov (mailto:coola@ngs.ru)
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2002-12-03 5:03 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2002-12-01 7:16 a bug in autofs Andrey R. Urazov
2002-12-01 18:29 ` Bryan O'Sullivan
2002-12-02 4:03 ` Andrey R. Urazov
2002-12-02 3:25 ` Jeremy Fitzhardinge
2002-12-02 7:57 ` Andrey R. Urazov
2002-12-02 8:39 ` Jeremy Fitzhardinge
2002-12-02 15:17 ` Andrey R. Urazov
[not found] ` <1038847726.2560.51.camel@ixodes.goop.org>
2002-12-02 17:50 ` Andrey R. Urazov
2002-12-02 18:35 ` Jeremy Fitzhardinge
2002-12-03 5:07 ` Andrey R. Urazov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox