* RE: [linux-lvm] lvcreate distributed extent option
@ 2002-11-30 8:03 ZECHA,HOLGER (HP-Germany,ex1)
2002-11-30 9:08 ` James Hawtin
` (2 more replies)
0 siblings, 3 replies; 11+ messages in thread
From: ZECHA,HOLGER (HP-Germany,ex1) @ 2002-11-30 8:03 UTC (permalink / raw)
To: linux-lvm, oolon; +Cc: lvm-devel
Hello,
what do you mean with
> How about striping with the strip size equal to that of a logical extent?
Can you give an example for the lvcreate command.
The reason to do this is to distribute the disk I/O accross all disks. the
logical extent 1 is on disk1, logical extent 2 on disk 2 and so on.
This means during read of logical extent 1, the heade of the second disk can
already be moved to prepare to read logical extent 2.
This allows us to do more I/O's per second
Holger
-----Original Message-----
From: James Hawtin [mailto:oolon@pzat.meep.org]
Sent: Samstag, 30. November 2002 14:40
To: linux-lvm@sistina.com
Cc: lvm-devel@sistina.com
Subject: Re: [linux-lvm] lvcreate distributed extent option
On Sat, 30 Nov 2002, ZECHA,HOLGER (HP-Germany,ex1) wrote:
>
> Hello,
>
> I want to create a logical volume of 40GB Size using a physical 9GB
disks,is
> there a way to create a logical volume in the way, that the first logical
> extend is on disk one, the second on disk two, ... the 7th on disk one and
> so on ?
>
> Meaning all logical extends are distributed accross all disks.
>
> In HP-UX lvdisplay wwill look like this:
> --- Logical extents ---
> LE PV1 PE1 Status 1
> 00000 /dev/dsk/c16t0d0 00012 current
> 00001 /dev/dsk/c16t0d1 00012 current
> 00002 /dev/dsk/c17t0d2 00000 current
> 00003 /dev/dsk/c17t0d3 00000 current
> 00004 /dev/dsk/c16t0d4 00000 current
> 00005 /dev/dsk/c16t0d5 00000 current
> 00006 /dev/dsk/c17t0d6 00000 current
> 00007 /dev/dsk/c17t0d7 00000 current
> 00008 /dev/dsk/c16t1d0 00000 current
> 00009 /dev/dsk/c16t1d1 00000 current
> 00010 /dev/dsk/c17t1d2 00000 current
> 00011 /dev/dsk/c17t1d3 00000 current
> 00012 /dev/dsk/c16t1d4 00000 current
>
> Holger
>
How about striping with the strip size equal to that of a logical extent?
otherwise if you have quite alot of disk space on another drive you could
create your lv on that one then move the extents one at a time to the
destinations drives using pvmove.....
One does wonder why you want to do this!
James
_______________________________________________
linux-lvm mailing list
linux-lvm@sistina.com
http://lists.sistina.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
^ permalink raw reply [flat|nested] 11+ messages in thread* RE: [linux-lvm] lvcreate distributed extent option
2002-11-30 8:03 [linux-lvm] lvcreate distributed extent option ZECHA,HOLGER (HP-Germany,ex1)
@ 2002-11-30 9:08 ` James Hawtin
2002-11-30 9:24 ` toby
2002-11-30 10:01 ` Ragnar Kjørstad
2 siblings, 0 replies; 11+ messages in thread
From: James Hawtin @ 2002-11-30 9:08 UTC (permalink / raw)
To: ZECHA,HOLGER (HP-Germany,ex1); +Cc: linux-lvm, lvm-devel
> This allows us to do more I/O's per second
>
yes... Thats striping why use a stripe size equal to that of a logical
extent I would have expected you to want them much smaller than that!
the default for striping on lvm is 8k. I my LE size is 16 meg,
16 meg strips would be a bit pointless in my book. Anyway using -I you can
set it to what you want.
lvcreate -i <num> -I <K> .....
from the lvcreate man page
-i, --stripes Stripes
Gives the number of stripes. This is equal to the
number of physical volumes to scatter the logical
volume.
-I, --stripesize StripeSize
Gives the number of kilobytes for the granularity
of the stripes.
StripeSize must be 2^n (n = 2 to 9)
James
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [linux-lvm] lvcreate distributed extent option
2002-11-30 8:03 [linux-lvm] lvcreate distributed extent option ZECHA,HOLGER (HP-Germany,ex1)
2002-11-30 9:08 ` James Hawtin
@ 2002-11-30 9:24 ` toby
2002-12-03 5:19 ` Heinz J . Mauelshagen
2002-11-30 10:01 ` Ragnar Kjørstad
2 siblings, 1 reply; 11+ messages in thread
From: toby @ 2002-11-30 9:24 UTC (permalink / raw)
To: linux-lvm
"ZECHA,HOLGER (HP-Germany,ex1)" wrote:
>
> Hello,
>
> what do you mean with
> > How about striping with the strip size equal to that of a logical extent?
>
> Can you give an example for the lvcreate command.
>
> The reason to do this is to distribute the disk I/O accross all disks. the
> logical extent 1 is on disk1, logical extent 2 on disk 2 and so on.
> This means during read of logical extent 1, the heade of the second disk can
> already be moved to prepare to read logical extent 2.
>
> This allows us to do more I/O's per second
>
> Holger
>
Once you create a striped lv, you cannot extend it by adding more disks. Current
lvm version anyway.
I wanted mine to be more flexible, so I first made a software raid set then made
the pv on top of the raid. That was in Jan this year. Last week I doubled the
size by adding another software raid set. I expanded two of the ext3 formatted
lvs in the process.
Perhaps not the fastest system in terms of throughput, but quite convenient to
resize.
-tkb
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] lvcreate distributed extent option
2002-11-30 9:24 ` toby
@ 2002-12-03 5:19 ` Heinz J . Mauelshagen
0 siblings, 0 replies; 11+ messages in thread
From: Heinz J . Mauelshagen @ 2002-12-03 5:19 UTC (permalink / raw)
To: linux-lvm
On Sat, Nov 30, 2002 at 10:24:06AM -0500, toby wrote:
> "ZECHA,HOLGER (HP-Germany,ex1)" wrote:
> >
> > Hello,
> >
> > what do you mean with
> > > How about striping with the strip size equal to that of a logical extent?
> >
> > Can you give an example for the lvcreate command.
> >
> > The reason to do this is to distribute the disk I/O accross all disks. the
> > logical extent 1 is on disk1, logical extent 2 on disk 2 and so on.
> > This means during read of logical extent 1, the heade of the second disk can
> > already be moved to prepare to read logical extent 2.
> >
> > This allows us to do more I/O's per second
> >
> > Holger
> >
>
>
> Once you create a striped lv, you cannot extend it by adding more disks. Current
> lvm version anyway.
That constraint doesn't exits any longer in LVM2/device-mapper
using the new LVM2 ondisk metadata format.
>
> I wanted mine to be more flexible, so I first made a software raid set then made
> the pv on top of the raid. That was in Jan this year. Last week I doubled the
> size by adding another software raid set. I expanded two of the ext3 formatted
> lvs in the process.
>
> Perhaps not the fastest system in terms of throughput, but quite convenient to
> resize.
>
>
> -tkb
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
--
Regards,
Heinz -- The LVM Guy --
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
Mauelshagen@Sistina.com +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] lvcreate distributed extent option
2002-11-30 8:03 [linux-lvm] lvcreate distributed extent option ZECHA,HOLGER (HP-Germany,ex1)
2002-11-30 9:08 ` James Hawtin
2002-11-30 9:24 ` toby
@ 2002-11-30 10:01 ` Ragnar Kjørstad
2002-11-30 11:25 ` James Hawtin
2 siblings, 1 reply; 11+ messages in thread
From: Ragnar Kjørstad @ 2002-11-30 10:01 UTC (permalink / raw)
To: linux-lvm; +Cc: oolon, lvm-devel
On Sat, Nov 30, 2002 at 03:02:38PM +0100, ZECHA,HOLGER (HP-Germany,ex1) wrote:
> > How about striping with the strip size equal to that of a logical extent?
>
> This allows us to do more I/O's per second
Only if you're only using a single or a few files at the time. If you're
using many files in parallel you will get more I/O's per second by
concatenating the devices instead if striping them. (that's what LVM
does by default)
--
Ragnar Kjørstad
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] lvcreate distributed extent option
2002-11-30 10:01 ` Ragnar Kjørstad
@ 2002-11-30 11:25 ` James Hawtin
0 siblings, 0 replies; 11+ messages in thread
From: James Hawtin @ 2002-11-30 11:25 UTC (permalink / raw)
To: linux-lvm; +Cc: lvm-devel
On Sat, 30 Nov 2002, [iso-8859-1] Ragnar Kjørstad wrote:
> Only if you're only using a single or a few files at the time. If you're
> using many files in parallel you will get more I/O's per second by
> concatenating the devices instead if striping them. (that's what LVM
> does by default)
That of course assumes that your filesystem spreads files across the whole
lv in random and even fashion, and of cos if the contation uses two
different parts from the same disk its much slower... I have a feeling
that files are more oftain clustered towards the front of the disk. But
like with all things how you setup your disks is very "use" dependant and
requires some degree of black magik ;-) Or atleast trying what works best
for you. Personally I have found stripping does not make my system much
faster. (Mostly I do compiling) Probably for me if I wanted to get IO,
mirroring include paths would be best, which is the best for reading,
"normal" for writting. And striping for where the data is being written,
but I guess thats why people use the jack of all trades raid 5 ;-)
I wish Lvm supported raid 5 and Mirroring...
James
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [linux-lvm] lvcreate distributed extent option
@ 2002-11-30 14:10 ZECHA,HOLGER (HP-Germany,ex1)
0 siblings, 0 replies; 11+ messages in thread
From: ZECHA,HOLGER (HP-Germany,ex1) @ 2002-11-30 14:10 UTC (permalink / raw)
To: linux-lvm; +Cc: lvm-devel
Hello,
to clarify why we need this setup:
this is the setup for the oracle directory where the datafiles and
indexfiles are located.
we don't setup dedicated filesystem for data and indexfiles and we don't use
raw devices. We mix everything in one directory which is setup up with the
distributed option from HP-UX LVM on HP-UX. With this setup extents are
distributed accross all LUN's and therefore disk IO is also distributed
across all LUN's and administration of this setup i quit easier the using
dedicated filesystems for datafiles and index files.
We want to do this in the same way on Linux, therfore my question.
I wrote a shell script which does excatly this distribution for Linux.
Holger
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [linux-lvm] lvcreate distributed extent option
@ 2002-11-30 12:02 ZECHA,HOLGER (HP-Germany,ex1)
2002-11-30 14:09 ` James Hawtin
0 siblings, 1 reply; 11+ messages in thread
From: ZECHA,HOLGER (HP-Germany,ex1) @ 2002-11-30 12:02 UTC (permalink / raw)
To: linux-lvm; +Cc: lvm-devel
Hello,
there still seems to be a missunderstanding
I want to have the following:
--- Logical extents ---
LE PV PE reads writes
00000 /dev/sdd1 00001 0 1
00001 /dev/sde1 00001 0 0
00002 /dev/sdf1 00001 0 0
00003 /dev/sdg1 00001 0 0
00004 /dev/sdh1 00001 0 0
00005 /dev/sdi1 00001 0 0
00006 /dev/sdd1 00002 0 0
00007 /dev/sde1 00002 0 0
00008 /dev/sdf1 00002 0 0
00009 /dev/sdg1 00002 0 0
00010 /dev/sdh1 00002 0 0
00011 /dev/sdi1 00002 0 0
00012 /dev/sdd1 00003 0 0
00013 /dev/sde1 00003 0 0
00014 /dev/sdf1 00003 0 0
.....
but with lvcreate -I 8 -L 40000M -n lvdata /dev/vgtest /dev/sdd1 /dev/sde1
.... I get
--- Logical extents ---
LE PV PE reads writes
00000 /dev/sdd1 00063 0 1
00001 /dev/sdd1 00064 0 0
00002 /dev/sdd1 00065 0 0
00003 /dev/sdd1 00066 0 0
00004 /dev/sdd1 00067 0 0
00005 /dev/sdd1 00068 0 0
00006 /dev/sdd1 00069 0 0
00007 /dev/sdd1 00070 0 0
00008 /dev/sdd1 00071 0 0
00009 /dev/sdd1 00072 0 0
00010 /dev/sdd1 00073 0 0
00011 /dev/sdd1 00074 0 0
00012 /dev/sdd1 00075 0 0
00013 /dev/sdd1 00076 0 0
00014 /dev/sdd1 00077 0 0
...
02105 /dev/sde1 00063 0 0
02106 /dev/sde1 00064 0 0
02107 /dev/sde1 00065 0 0
02108 /dev/sde1 00066 0 0
02109 /dev/sde1 00067 0 0
02110 /dev/sde1 00068 0 0
02111 /dev/sde1 00069 0 0
02112 /dev/sde1 00070 0 0
A distribution like described above looks like this on HP-UX LVM:
00000 /dev/dsk/c16t0d0 00012 current
00001 /dev/dsk/c16t0d1 00012 current
00002 /dev/dsk/c17t0d2 00000 current
00003 /dev/dsk/c17t0d3 00000 current
00004 /dev/dsk/c16t0d4 00000 current
00005 /dev/dsk/c16t0d5 00000 current
00006 /dev/dsk/c17t0d6 00000 current
00007 /dev/dsk/c17t0d7 00000 current
00008 /dev/dsk/c16t1d0 00000 current
00009 /dev/dsk/c16t1d1 00000 current
00010 /dev/dsk/c17t1d2 00000 current
00011 /dev/dsk/c17t1d3 00000 current
00012 /dev/dsk/c16t1d4 00000 current
Holger
-----Original Message-----
From: James Hawtin [mailto:oolon@ankh.org]
Sent: Samstag, 30. November 2002 18:24
To: linux-lvm@sistina.com
Cc: lvm-devel@sistina.com
Subject: Re: [linux-lvm] lvcreate distributed extent option
This message uses a character set that is not supported by the Internet
Service. To view the original message content, open the attached message.
If the text doesn't display correctly, save the attachment to disk, and then
open it using a viewer that can display the original character set.
^ permalink raw reply [flat|nested] 11+ messages in thread
* RE: [linux-lvm] lvcreate distributed extent option
2002-11-30 12:02 ZECHA,HOLGER (HP-Germany,ex1)
@ 2002-11-30 14:09 ` James Hawtin
0 siblings, 0 replies; 11+ messages in thread
From: James Hawtin @ 2002-11-30 14:09 UTC (permalink / raw)
To: linux-lvm; +Cc: lvm-devel
On Sat, 30 Nov 2002, ZECHA,HOLGER (HP-Germany,ex1) wrote:
> Hello,
>
> there still seems to be a missunderstanding
> I want to have the following:
>
Hmm your worring to much about how your physical data is being layout! The
reason you see that difference between HP lvm and linux lvm is because
they implement striping differently.
On, HP, they take the modulo of the number of stripes and allocate the
disk in that fasion ie.
a 0 0%2 =0
b 1 1%2 =1
a 2 2%2 =0
b 1 3%2 =1
On linux its
LE Stripe Size (LSS) = Number of LE / Number of strips
a = 0 - (LSS-1)
b = LSS - (LSS*2-1)
But thats just how the Logical extends are mapped LE Stripe Size != real
stripe size on HP or Linux its just how the strip pools are layed out.
The LVM system still has to re cut things LEs up to make real stripes.
Advantage of the way HP does it...
No problems extending things....
Disadvantage
Harder on the maths?
Difficult to specify which LEs to pick when moving them from one PV to
another.
Advantage of Linux way
Easy to specify and check when moving LEs from one PV to another.
Maths easier?
Disadvantage.
LEs have to be renumbered if the LV is extended or reduced, hence not
possible on linux.
Still don't believe me?
Try this....
--- Logical volume ---
LV Name /dev/sunset_vg/oolon4_lv
VG Name sunset_vg
LV Write Access read/write
LV Status available
LV # 5
# open 0
LV Size 18.38 GB
Current LE 588
Allocated LE 588
Stripes 2
Stripe size (KByte) 16
Allocation next free
Read ahead sectors 120
Block device 58:4
--- Distribution of logical volume on 2 physical volumes ---
PV Name PE on PV reads writes
/dev/ide/host2/bus0/target0/lun0/part5 294 0 0
/dev/ide/host2/bus0/target1/lun0/part2 294 0 0
--- logical volume i/o statistic ---
0 reads 0 writes
--- Logical extents ---
LE PV PE reads writes
00000 /dev/ide/host2/bus0/target0/lun0/part5 00060 0 0
00001 /dev/ide/host2/bus0/target0/lun0/part5 00061 0 0
00002 /dev/ide/host2/bus0/target0/lun0/part5 00062 0 0
...
00294 /dev/ide/host2/bus0/target1/lun0/part2 00060 0 0
00295 /dev/ide/host2/bus0/target1/lun0/part2 00061 0 0
00296 /dev/ide/host2/bus0/target1/lun0/part2 00062 0 0
...
bash# dd if=/dev/sunset_vg/oolon4_lv of=/dev/null bs=16k count=2
2+0 records in
2+0 records out
bash# dd if=/dev/sunset_vg/oolon4_lv of=/dev/null bs=16k count=2
2+0 records in
2+0 records out
bash# dd if=/dev/sunset_vg/oolon4_lv of=/dev/null bs=16k count=2
2+0 records in
2+0 records out
lvdisplay -vv /dev/sunset_vg/oolon4_lv
....
--- Distribution of logical volume on 2 physical volumes ---
PV Name PE on PV reads writes
/dev/ide/host2/bus0/target0/lun0/part5 294 240 0
/dev/ide/host2/bus0/target1/lun0/part2 294 216 0
--- logical volume i/o statistic ---
456 reads 0 writes
--- Logical extents ---
LE PV PE reads writes
00000 /dev/ide/host2/bus0/target0/lun0/part5 00060 240 0
00001 /dev/ide/host2/bus0/target0/lun0/part5 00061 0 0
00002 /dev/ide/host2/bus0/target0/lun0/part5 00062 0 0
....
00294 /dev/ide/host2/bus0/target1/lun0/part2 00060 216 0
00295 /dev/ide/host2/bus0/target1/lun0/part2 00061 0 0
00296 /dev/ide/host2/bus0/target1/lun0/part2 00062 0 0
....
As you see my stripe size was 16k and I grabbed the first 32k of the
device 3 times.
You will notice the read figures are relatively even... not quite sure why
they are not the same I assume LVM might have some kind of header on the
LV. Which would account for the difference...
(I did this after rebooting the system to get clean figures).
You will however notice the stripping did work, it did use both disks not
as you suggest just 1 (if that had been the case the second disk would
have shown 0 reads), test it yourself, as I explained above the
"difference" you are seeing is just down to how the implementation of the
strip parts has been made. Though personally I agree with you the HP
system (due to the fact it can be resized) is supperior!
James
^ permalink raw reply [flat|nested] 11+ messages in thread
* [linux-lvm] lvcreate distributed extent option
@ 2002-11-30 3:53 ZECHA,HOLGER (HP-Germany,ex1)
2002-11-30 7:41 ` James Hawtin
0 siblings, 1 reply; 11+ messages in thread
From: ZECHA,HOLGER (HP-Germany,ex1) @ 2002-11-30 3:53 UTC (permalink / raw)
To: linux-lvm, lvm-devel
Hello,
I want to create a logical volume of 40GB Size using a physical 9GB disks,is
there a way to create a logical volume in the way, that the first logical
extend is on disk one, the second on disk two, ... the 7th on disk one and
so on ?
Meaning all logical extends are distributed accross all disks.
In HP-UX lvdisplay wwill look like this:
--- Logical extents ---
LE PV1 PE1 Status 1
00000 /dev/dsk/c16t0d0 00012 current
00001 /dev/dsk/c16t0d1 00012 current
00002 /dev/dsk/c17t0d2 00000 current
00003 /dev/dsk/c17t0d3 00000 current
00004 /dev/dsk/c16t0d4 00000 current
00005 /dev/dsk/c16t0d5 00000 current
00006 /dev/dsk/c17t0d6 00000 current
00007 /dev/dsk/c17t0d7 00000 current
00008 /dev/dsk/c16t1d0 00000 current
00009 /dev/dsk/c16t1d1 00000 current
00010 /dev/dsk/c17t1d2 00000 current
00011 /dev/dsk/c17t1d3 00000 current
00012 /dev/dsk/c16t1d4 00000 current
Holger
^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [linux-lvm] lvcreate distributed extent option
2002-11-30 3:53 ZECHA,HOLGER (HP-Germany,ex1)
@ 2002-11-30 7:41 ` James Hawtin
0 siblings, 0 replies; 11+ messages in thread
From: James Hawtin @ 2002-11-30 7:41 UTC (permalink / raw)
To: linux-lvm; +Cc: lvm-devel
On Sat, 30 Nov 2002, ZECHA,HOLGER (HP-Germany,ex1) wrote:
>
> Hello,
>
> I want to create a logical volume of 40GB Size using a physical 9GB disks,is
> there a way to create a logical volume in the way, that the first logical
> extend is on disk one, the second on disk two, ... the 7th on disk one and
> so on ?
>
> Meaning all logical extends are distributed accross all disks.
>
> In HP-UX lvdisplay wwill look like this:
> --- Logical extents ---
> LE PV1 PE1 Status 1
> 00000 /dev/dsk/c16t0d0 00012 current
> 00001 /dev/dsk/c16t0d1 00012 current
> 00002 /dev/dsk/c17t0d2 00000 current
> 00003 /dev/dsk/c17t0d3 00000 current
> 00004 /dev/dsk/c16t0d4 00000 current
> 00005 /dev/dsk/c16t0d5 00000 current
> 00006 /dev/dsk/c17t0d6 00000 current
> 00007 /dev/dsk/c17t0d7 00000 current
> 00008 /dev/dsk/c16t1d0 00000 current
> 00009 /dev/dsk/c16t1d1 00000 current
> 00010 /dev/dsk/c17t1d2 00000 current
> 00011 /dev/dsk/c17t1d3 00000 current
> 00012 /dev/dsk/c16t1d4 00000 current
>
> Holger
>
How about striping with the strip size equal to that of a logical extent?
otherwise if you have quite alot of disk space on another drive you could
create your lv on that one then move the extents one at a time to the
destinations drives using pvmove.....
One does wonder why you want to do this!
James
^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2002-12-03 5:19 UTC | newest]
Thread overview: 11+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2002-11-30 8:03 [linux-lvm] lvcreate distributed extent option ZECHA,HOLGER (HP-Germany,ex1)
2002-11-30 9:08 ` James Hawtin
2002-11-30 9:24 ` toby
2002-12-03 5:19 ` Heinz J . Mauelshagen
2002-11-30 10:01 ` Ragnar Kjørstad
2002-11-30 11:25 ` James Hawtin
-- strict thread matches above, loose matches on Subject: below --
2002-11-30 14:10 ZECHA,HOLGER (HP-Germany,ex1)
2002-11-30 12:02 ZECHA,HOLGER (HP-Germany,ex1)
2002-11-30 14:09 ` James Hawtin
2002-11-30 3:53 ZECHA,HOLGER (HP-Germany,ex1)
2002-11-30 7:41 ` James Hawtin
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.