[linux-lvm] Questions

[linux-lvm] Questions

[Stuart D. Gathman]

Which display commands display on-disk metadata, which display 
lvmtab/lvmtab.d, and which display kernel data structures?  I'd like to 
know if the metadata is trashed on my original two PVs, or just on
the new one I tried to add.

Given my situation described previously (where vgextend/me screwed up and 
kernel data is fine, VG is active, but disk metadata seems to be toast),
should I run vgcfgrestore on the live VG in the hope that things might 
boot?  I have Centos-3  /etc/lvmconf backups of metadata (I think) for
prior to the attempt to vgextend.

What is the equivalent of pvremove (which is what I needed) in LVM1?
I tried using pvcreate "I'm really really sure", but that seems to have
screwed up.  If VG metadata is stored at the beginning of a PV, then
I guess dd might be the answer.

Does the last /etc/lvmconf entry respresent the result of the last 
command?  Or is it a backup of things just before the last autobackup
command?


-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Questions

[Heinz Mauelshagen]

On Sat, Mar 03, 2007 at 08:59:38PM -0500, Stuart D. Gathman wrote:
> Which display commands display on-disk metadata, which display 
> lvmtab/lvmtab.d, and which display kernel data structures?  I'd like to 
> know if the metadata is trashed on my original two PVs, or just on
> the new one I tried to add.

Because you're refering to LVM1, you see kernel metadata unless you
provide the -D option to the *display commands.

> 
> Given my situation described previously (where vgextend/me screwed up and 
> kernel data is fine, VG is active, but disk metadata seems to be toast),
> should I run vgcfgrestore on the live VG in the hope that things might 
> boot?  I have Centos-3  /etc/lvmconf backups of metadata (I think) for
> prior to the attempt to vgextend.

Make sure that your /etc/lvmconf backup is alright for your case
by running "vgcfgrestore -ll -f ...".
If so, vgcfgrestore it after initializing the PVs with pvcreate -f.

> 
> What is the equivalent of pvremove (which is what I needed) in LVM1?
> I tried using pvcreate "I'm really really sure", but that seems to have
> screwed up.  If VG metadata is stored at the beginning of a PV, then
> I guess dd might be the answer.

There's no pveremove in LVM1, so dd of 512 bytes of zeroes will do.

> 
> Does the last /etc/lvmconf entry respresent the result of the last 
> command?

/etc/lvmconf contains a backup history including the actual
VG configuration.

> Or is it a backup of things just before the last autobackup
> command?
> 
> 
> -- 
> 	      Stuart D. Gathman <stuart@bmsi.com>
>     Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
> "Confutatis maledictis, flammis acribus addictis" - background song for
> a Microsoft sponsored "Where do you want to go from here?" commercial.
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@redhat.com
> https://www.redhat.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Red Hat GmbH
Consulting Development Engineer                   Am Sonnenhang 11
Storage Development                               56242 Marienrachdorf
                                                  Germany
Mauelshagen@RedHat.com                            PHONE +49  171 7803392
                                                  FAX   +49 2626 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] Questions

[Stuart D. Gathman]

On Mon, 5 Mar 2007, Heinz Mauelshagen wrote:

> On Sat, Mar 03, 2007 at 08:59:38PM -0500, Stuart D. Gathman wrote:
> > Which display commands display on-disk metadata, which display 
> > lvmtab/lvmtab.d, and which display kernel data structures?  I'd like to 
> > know if the metadata is trashed on my original two PVs, or just on
> > the new one I tried to add.
> 
> Because you're refering to LVM1, you see kernel metadata unless you
> provide the -D option to the *display commands.

Why did vgdisplay refuse to display anything until I restored
/etc/lvmtab.d/rootvg from the backup (prior to the attempted vgextend)?

What is the relationship between /etc/lvmconf backups and /etc/lvmtab.d?
Are they the same or different format?  I notice that the size is identical,
but there where byte differences when I compared them on the backup of the
working system.

What does this mean?

[root@cmslax root]# vgdisplay -D
vgdisplay -- ERROR: not all physical volumes of volume group "rootvg" online

And sure enough, 
[root@cmslax root]# pvdisplay /dev/md3
--- Physical volume ---
PV Name               /dev/md3
VG Name               rootvg
PV Size               16.87 GB [35374848 secs] / NOT usable 32.19 MB [LVM: 130 KB]
PV#                   3
PV Status             NOT available
Allocatable           yes
Cur LV                1
PE Size (KByte)       32768
Total PE              538
Free PE               481
Allocated PE          57
PV UUID               YfvIpi-x3Cg-xKrR-j1T2-Gutz-u1tp-sl8t01

/dev/md3 is most assuredly available and readable/writable.

> Make sure that your /etc/lvmconf backup is alright for your case
> by running "vgcfgrestore -ll -f ...".
> If so, vgcfgrestore it after initializing the PVs with pvcreate -f.

Why is the pvcreate necessary?  The PVs all have metadata displayable
with pvdisplay.  Pvdisplay has no -D option, so I assume it displays
on disk metadata.

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

Re: [linux-lvm] Questions

[Stuart D. Gathman]

On Mon, 5 Mar 2007, Heinz Mauelshagen wrote:

> Make sure that your /etc/lvmconf backup is alright for your case
> by running "vgcfgrestore -ll -f ...".

Does -ll imply -t?  (I.e., is it read-only?)

-- 
	      Stuart D. Gathman <stuart@bmsi.com>
    Business Management Systems Inc.  Phone: 703 591-0911 Fax: 703 591-6154
"Confutatis maledictis, flammis acribus addictis" - background song for
a Microsoft sponsored "Where do you want to go from here?" commercial.

[linux-lvm] questions

[Magosanyi Arpad]

Hi!

What would you check on disk related to snapshots? How?
Is there any upper meaning of the last word of my last lv entry is being 0xffff?

-- 
GNU GPL: csak tiszta forrásból

Re: [linux-lvm] questions

[Wolfgang Weisselberg]

Hi, Magosanyi!

Magosanyi Arpad (mag@bunuel.tii.matav.hu) wrote 14 lines:

> What would you check on disk related to snapshots? 

There shall be no snapshot of a snapshot.  (Yes, I have that
right here on my disk.  Can't remove the snapshot-of-snapshot
(segfault), can't remove the snapshot (under snapshot)).

Does the snapshot point to the correct device?  (Happened to
me, too.  Usually after an (unrelated?) crash.  No idea how
to check -- but all Devices Under Snapshot shall have at
least 1 snapshot.)

Does the persistent snapshot-map look like it could work on
that device claims to be against? (bigger than the device
in question?)

Ability to turn persistent snapshots into non-persistent ones
(cleaned up after reboot) -- so that rogue ones can be
removed during a reboot.

-Wolfgang

Re: [linux-lvm] questions

[Heinz J . Mauelshagen]

On Thu, Nov 08, 2001 at 02:14:52AM +0100, Wolfgang Weisselberg wrote:
> Hi, Magosanyi!
> 
> Magosanyi Arpad (mag@bunuel.tii.matav.hu) wrote 14 lines:
> 
> > What would you check on disk related to snapshots? 
> 
> There shall be no snapshot of a snapshot.  (Yes, I have that
> right here on my disk.  Can't remove the snapshot-of-snapshot
> (segfault), can't remove the snapshot (under snapshot)).

Wolfgang,

with which LVM version do you have trouble removing snapshots?
Which compiler did you use to compile LVM? With or without optimization?

> 
> Does the snapshot point to the correct device?  (Happened to
> me, too.  Usually after an (unrelated?) crash.  No idea how
> to check -- but all Devices Under Snapshot shall have at
> least 1 snapshot.)

Do you mean, that a snapshot didn't refer to the correct device after
a crash any longer?

> 
> Does the persistent snapshot-map look like it could work on
> that device claims to be against? (bigger than the device
> in question?)

?

> 
> Ability to turn persistent snapshots into non-persistent ones
> (cleaned up after reboot) -- so that rogue ones can be
> removed during a reboot.

Well, this wouldn't make a difference because just the copy on write
exception tables would be removed but the rest of the snapshots metadata
where still in place making it most likely still impossible to remove it
in case we have a real bug related here.

> 
> -Wolfgang
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://www.sistina.com/lvm/Pages/howto.html

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] questions

[Wolfgang Weisselberg]

Hi, Heinz!

Heinz J . Mauelshagen (mauelshagen@sistina.com) wrote 75 lines:

> On Thu, Nov 08, 2001 at 02:14:52AM +0100, Wolfgang Weisselberg wrote:
> > Magosanyi Arpad (mag@bunuel.tii.matav.hu) wrote 14 lines:

> > > What would you check on disk related to snapshots? 

> > There shall be no snapshot of a snapshot.  (Yes, I have that
> > right here on my disk.  Can't remove the snapshot-of-snapshot
> > (segfault), can't remove the snapshot (under snapshot)).

> Wolfgang,

> with which LVM version do you have trouble removing snapshots?

As far as I can see/remember, all from 1.0 up to the current
(2001-11-11) CVS.  Specially at least:
lvm-1.0
lvm-1.0.1_rc4_2001_10_14        (CVS)
lvm-1.0.1_rc4_2001_10_23        (CVS)
lvm-1.0.1_rc4_2001_10_30        (CVS)
lvm-1.0.1_rc4_2001_11_11        (CVS)

> Which compiler did you use to compile LVM? With or without optimization?

SuSE's gcc-2.95.2-98 for their 7.0:
$ gcc -v
Reading specs from /usr/lib/gcc-lib/i486-suse-linux/2.95.2/specs
gcc version 2.95.2 19991024 (release)

Optimization is as the CVS says, i.e. off for newer ones.

The LVs were created over the time, but mostly before 0.9;
the system was converted to IOP11 as part of the upgrade to 1.0.

lvscan says something very... interesting:

|$ lvscan
|lvscan -- ACTIVE   Original "/dev/main_vg/amanda_holding_disk" [6.00 GB]
|lvscan -- ACTIVE            "/dev/main_vg/usr_src_packages" [1.22 GB]
|lvscan -- ACTIVE   Original "/dev/main_vg/usr" [3.00 GB]
|lvscan -- ACTIVE            "/dev/main_vg/home" [1.00 GB]
|[... snip a couple OK ones ...]
|lvscan -- ACTIVE            "/dev/main_vg/usr_local_games_Heroes3" [392.00 MB]
|lvscan -- inactive Original "/dev/main_vg/SCHROTT2" [51.19 MB] of /dev/main_vg/amanda_holding_disk
|lvscan -- ACTIVE   Original "/dev/main_vg/SCHROTT4" [51.19 MB] of /dev/main_vg/amanda_holding_disk
|lvscan -- ACTIVE   Original "/dev/main_vg/var_lib_amanda" [300.00 MB]
|lvscan -- inactive Snapshot "/dev/main_vg/SCHROTT1" [19.69 MB] of /dev/main_vg/SCHROTT2
|lvscan -- ACTIVE   Snapshot "/dev/main_vg/SCHROTT3" [287.44 MB] of /dev/main_vg/SCHROTT4
|lvscan -- 23 logical volumes with 23.82 GB total in 1 volume group
|lvscan -- 21 active / 2 inactive logical volumes

The /dev/main_vg/SCHROTT[1-4] are the failed snapshots.
At least one of them was against /dev/main_vg/usr, but they
became snapped against /dev/main_vg/amanda_holding_disk.

SCHROTT[13] segfault on lvremove:
|lvremove -- do you really want to remove "/dev/main_vg/SCHROTT1"? [y/n]: y
|<1> lv_release -- CALLED with /dev/main_vg/SCHROTT1
|<22> lv_check_name -- CALLED with lv_name: "/dev/main_vg/SCHROTT1"
|<333> lvm_check_chars -- CALLED with name: "/dev/main_vg/SCHROTT1"
|<333> lvm_check_chars -- LEAVING with ret: 0
|<333> vg_check_name -- CALLED with VG: main_vg
|<4444> lvm_check_chars -- CALLED with name: "main_vg"
|<4444> lvm_check_chars -- LEAVING with ret: 0
|<333> vg_check_name -- LEAVING with ret: 0
|<22> lv_check_name -- LEAVING with ret: 0
|<1> lv_release -- after search for /dev/main_vg/SCHROTT1
|<1> lv_release -- /dev/main_vg/SCHROTT1 found
|Segmentation fault

and SCHROTT[24] won't allow to be removed.

At the moment it's even worse: 
lvm-1.0.1_rc4_2001_11_11's vgscan is unable to scan the VGs
completely, it sees about 1/3, then segfaults, but the (old)
1.0 executable (same library!) works.  Funny enough, once the
system is up & running, it works...

> > Does the snapshot point to the correct device?  (Happened to
> > me, too.  Usually after an (unrelated?) crash.  No idea how
> > to check -- but all Devices Under Snapshot shall have at
> > least 1 snapshot.)

> Do you mean, that a snapshot didn't refer to the correct device after
> a crash any longer?

Bingo.  Exactly that.
Then I was able to rename them, but not able to remove them.
Sometimes (seldom) I could, but that was not repeatable.

Note that these crashes were most likely *not* the fault of lvm,
and did not occur while an lvm-operation was running.

Do you need a dump/dd/whatever?  Just tell me, I'll try my
best to provide you with the data.

> > Does the persistent snapshot-map look like it could work on
> > that device claims to be against? (bigger than the device
> > in question?)

> ?

Assume you have a snapshot of 1GB.  Assume the claimed original
is 400M.  Since you cannot produce a snapshot larger than the
original, this looks like an invalid snapshot, doesn't it?

> > Ability to turn persistent snapshots into non-persistent ones
> > (cleaned up after reboot) -- so that rogue ones can be
> > removed during a reboot.

> Well, this wouldn't make a difference because just the copy on write
> exception tables would be removed but the rest of the snapshots metadata
> where still in place making it most likely still impossible to remove it
> in case we have a real bug related here.

True enough.  Probably we need an interface to the metadata,
so the user can (at his own risk!) change them.  Sorta like
fdisk or debugfs.

-Wolfgang

[linux-lvm] questions

[Ragnar Kjørstad]

Hi

Is there a limit for the maximum size of a LVM diskgroup?
(internally in LVM-code, or because of other limits in the kernel)
The 2TB (1 TB) maximum devicesize in the linux-kernel doesn't apply, does
it?

How does LVM handle that one of it's devices changes size? E.g. if one
of the physical disks is a RAIDset, and more disks are added to it?
Is there any way to tell LVM to use more or less of the disk?


Thanks.


-- 
Ragnar Kj�rstad
BigStorage

Re: [linux-lvm] questions

[Heinz J. Mauelshagen]

On Thu, Feb 08, 2001 at 08:55:51PM +0100, Ragnar Kj�rstad wrote:
> Hi
> 
> Is there a limit for the maximum size of a LVM diskgroup?
> (internally in LVM-code, or because of other limits in the kernel)
> The 2TB (1 TB) maximum devicesize in the linux-kernel doesn't apply, does
> it?

256 (which is limited to 128 SCSI disks and 20? IDE discs) * maximum PV size
which is limited to 2 TB --> 296 TB in case you'll find 2 TB IDE disks ;-)

> 
> How does LVM handle that one of it's devices changes size? E.g. if one
> of the physical disks is a RAIDset, and more disks are added to it?
> Is there any way to tell LVM to use more or less of the disk?

LVM can't handle size changes directly today.
You need to go with

   a. additional partitions on the size increased device
   b. additional LUNs exposing the added capacity (typical with some
      HW RAID device models)

and create additional PVs on them.

Basically we could implement an enhancement to pvchange(8) in order
to support growing of PVs.

> 
> 
> Thanks.
> 
> 
> -- 
> Ragnar Kj�rstad
> BigStorage
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Re: [linux-lvm] questions

[Andreas Dilger]

Ragnar Kj_rstad writes:
> Is there a limit for the maximum size of a LVM diskgroup?
> (internally in LVM-code, or because of other limits in the kernel)
> The 2TB (1 TB) maximum devicesize in the linux-kernel doesn't apply, does
> it?

Yes, even LVM has the 2TB limit at this point.  This is because LVM
presents itself to the rest of the kernel as a block device (just like
any other), so the kernel 2TB limit for block devices still holds.

Heinz talked about allowing LVM to have PAGE_SIZE (4k) blocks (unlike
the rest of the kernel, which uses 512 byte blocks) so it may be possible
to go up to 16TB with LVM without a huge amount of kernel redesign.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] questions

[Ragnar Kjørstad]

On Thu, Feb 08, 2001 at 03:44:33PM -0700, Andreas Dilger wrote:
> Ragnar Kj_rstad writes:
> > Is there a limit for the maximum size of a LVM diskgroup?
> > (internally in LVM-code, or because of other limits in the kernel)
> > The 2TB (1 TB) maximum devicesize in the linux-kernel doesn't apply, does
> > it?
> 
> Yes, even LVM has the 2TB limit at this point.  This is because LVM
> presents itself to the rest of the kernel as a block device (just like
> any other), so the kernel 2TB limit for block devices still holds.

For the volume, but not for the diskgroup, right?

> Heinz talked about allowing LVM to have PAGE_SIZE (4k) blocks (unlike
> the rest of the kernel, which uses 512 byte blocks) so it may be possible
> to go up to 16TB with LVM without a huge amount of kernel redesign.

This sounds really interesting. 
What exactly would be needed to do that?
I believe there have been issues with lvm block sizes and XFS (maybe
other filesystems too). Would changing it to 4k create new issues?
Would it be possible to increese it even futher? if the filesystem used
more than 4k blocksize?


Thanks.



-- 
Ragnar Kj�rstad
BigStorage

Re: [linux-lvm] questions

[Andreas Dilger]

Ragnar Kj_rstad writes:
> On Thu, Feb 08, 2001 at 03:44:33PM -0700, Andreas Dilger wrote:
> > Yes, even LVM has the 2TB limit at this point.  This is because LVM
> > presents itself to the rest of the kernel as a block device (just like
> > any other), so the kernel 2TB limit for block devices still holds.
> 
> For the volume, but not for the diskgroup, right?

Yes, the 2TB limit is for a single LV.  You can have up to 255 LVs in
total I think, so 512TB total usable space with LVM, assuming your
configuration works in this way.  Probably the 255 LV limit will be
a problem before the 512TB limit is...

> > Heinz talked about allowing LVM to have PAGE_SIZE (4k) blocks (unlike
> > the rest of the kernel, which uses 512 byte blocks) so it may be possible
> > to go up to 16TB with LVM without a huge amount of kernel redesign.
> 
> This sounds really interesting.  What exactly would be needed to do that?

Change all of the code paths in the kernel that assume block size is 512
bytes...

> I believe there have been issues with lvm block sizes and XFS (maybe
> other filesystems too). Would changing it to 4k create new issues?

Yes, probably.

> Would it be possible to increese it even futher? if the filesystem used
> more than 4k blocksize?

Probably not.  There will likely be lots of problems for block size >
PAGE_SIZE.  Of course on ia64 and alpha PAGE_SIZE = 8kB, so you can go
to 32TB I think, assuming the rest of the issues are fixed.

Cheers, Andreas
-- 
Andreas Dilger  \ "If a man ate a pound of pasta and a pound of antipasto,
                 \  would they cancel out, leaving him still hungry?"
http://www-mddsp.enel.ucalgary.ca/People/adilger/               -- Dogbert

Re: [linux-lvm] questions

[Christoph Hellwig]

On Thu, Feb 08, 2001 at 05:05:36PM -0700, Andreas Dilger wrote:
> Change all of the code paths in the kernel that assume block size is 512
> bytes...

IIRC some S/390 devices already use 4K block size.

	Christoph

-- 
Of course it doesn't work. We've performed a software upgrade.

Re: [linux-lvm] questions

[Heinz J. Mauelshagen]

On Fri, Feb 09, 2001 at 09:34:05AM +0100, Christoph Hellwig wrote:
> On Thu, Feb 08, 2001 at 05:05:36PM -0700, Andreas Dilger wrote:
> > Change all of the code paths in the kernel that assume block size is 512
> > bytes...
> 
> IIRC some S/390 devices already use 4K block size.

Yes. They have so called DASD drives which are 4K hard sectored.

Anyway. if you change the BLOCK_SIZE definition in lvm.h from 1 to 4k
you can build LVs larger than 2TB (or even 4TB which is the default with the
1k block size used by LVM today).
You can't go beyond PAGE_SIZE though because a whole pile of code in
the kernel doesn't support that.
That means that you could even go to 8k block size on machines supporting
that page size.
Remember: you loose comapatibility with 4k page sized systems in this case.

> 
> 	Christoph
> 
> -- 
> Of course it doesn't work. We've performed a software upgrade.
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-