[linux-lvm] Problems running my home-built LVM2

[linux-lvm] Problems running my home-built LVM2

[Stefan Monnier]

I'm trying to use LVM2 on a small MIPS machine.  This machine uses
uClibc-0.9.29 with Linux-2.6.25.20 and has limited RAM and CPU power,
but does come with a 1TB drive which I'd rather manage with LVM.

I had no particular problems building the latest version of LVM2
for it.  And as far as I can tell, it's not fully broken,
but it's unusable: every run of `lvm' gives different results.
E.g. `lvm pvs' sometimes gives the expected output, sometimes lists
nothing at all, sometimes complains about missing volume groups,
sometimes "Incorrect metadata area header checksum" [as mentioned in an
earlier message on this list, incidentally].

I've taken a look at the compile log and didn't see any funny warning.
I tried to compile other versions of LVM (2.02.44, 2.02.43, 2.02.39),
but they all give the same result.

Then I installed Debian in a chroot, and Debian's 2.02.39 version of
`lvm' works just fine.

Does anyone have any idea what might be going on?

As a temporary workaround, I use "chroot /debian lvm" instead of "lvm",
but I really would like to get the "native" lvm working.


        Stefan

Re: [linux-lvm] Problems running my home-built LVM2

[Milan Broz]

Stefan Monnier wrote:
> I'm trying to use LVM2 on a small MIPS machine.  This machine uses
> uClibc-0.9.29 with Linux-2.6.25.20 and has limited RAM and CPU power,
> but does come with a 1TB drive which I'd rather manage with LVM.

> I had no particular problems building the latest version of LVM2
> for it.  And as far as I can tell, it's not fully broken,
> but it's unusable: every run of `lvm' gives different results.
> E.g. `lvm pvs' sometimes gives the expected output, sometimes lists
> nothing at all, sometimes complains about missing volume groups,
> sometimes "Incorrect metadata area header checksum" [as mentioned in an
> earlier message on this list, incidentally].

I guess that using uClibc is the problem.
Can you try to link it to standard glibc (if possible; you can try to use
static build also)?
(Just to test that using uClibc is the problem or not.)

> Then I installed Debian in a chroot, and Debian's 2.02.39 version of
> `lvm' works just fine.

That version uses glibc, right?

Milan
--
mbroz@redhat.com

[linux-lvm] Re: Problems running my home-built LVM2

[Stefan Monnier]

>> I'm trying to use LVM2 on a small MIPS machine.  This machine uses
>> uClibc-0.9.29 with Linux-2.6.25.20 and has limited RAM and CPU power,
>> but does come with a 1TB drive which I'd rather manage with LVM.

>> I had no particular problems building the latest version of LVM2
>> for it.  And as far as I can tell, it's not fully broken,
>> but it's unusable: every run of `lvm' gives different results.
>> E.g. `lvm pvs' sometimes gives the expected output, sometimes lists
>> nothing at all, sometimes complains about missing volume groups,
>> sometimes "Incorrect metadata area header checksum" [as mentioned in an
>> earlier message on this list, incidentally].

> I guess that using uClibc is the problem.

It sounds like a likely culprit, indeed.  But Googling indicates that
LVM2 has been compiled and used with uClibc for initramfs purposes so it
should work fine.  Tho maybe the problem is that those references I see
are old: i.e. it used to work OK but not anymore.

Is someone successfully using a recent LVM2 with a recent uClibc?

> Can you try to link it to standard glibc (if possible; you can try to use
> static build also)?

That's probably beyond my ability for now (the target machine is under
powered, so I need to cross-compile, which makes everything a lot more
... interesting).

>> Then I installed Debian in a chroot, and Debian's 2.02.39 version of
>> `lvm' works just fine.
> That version uses glibc, right?

Yes, indeed.


        Stefan

Re: [linux-lvm] Re: Problems running my home-built LVM2

[Bryn M. Reeves]

Stefan Monnier wrote:
>>> I'm trying to use LVM2 on a small MIPS machine.  This machine uses
>>> uClibc-0.9.29 with Linux-2.6.25.20 and has limited RAM and CPU power,
>>> but does come with a 1TB drive which I'd rather manage with LVM.
> 
>>> I had no particular problems building the latest version of LVM2
>>> for it.  And as far as I can tell, it's not fully broken,
>>> but it's unusable: every run of `lvm' gives different results.
>>> E.g. `lvm pvs' sometimes gives the expected output, sometimes lists
>>> nothing at all, sometimes complains about missing volume groups,
>>> sometimes "Incorrect metadata area header checksum" [as mentioned in an
>>> earlier message on this list, incidentally].
> 
>> I guess that using uClibc is the problem.
> 
> It sounds like a likely culprit, indeed.  But Googling indicates that
> LVM2 has been compiled and used with uClibc for initramfs purposes so it
> should work fine.  Tho maybe the problem is that those references I see

AFAIK, nobody is testing this regularly, especially not for the latest 
releases. Also many (most?) distributions are now shipping an 
initramfs with a full glibc environment so this use is becoming less 
common.

Regards,
Bryn.

[linux-lvm] Re: Problems running my home-built LVM2

[Stefan Monnier]

>> It sounds like a likely culprit, indeed.  But Googling indicates that
>> LVM2 has been compiled and used with uClibc for initramfs purposes so it
>> should work fine.  Tho maybe the problem is that those references I see

> AFAIK, nobody is testing this regularly, especially not for the latest
> releases.  Also many (most?) distributions are now shipping an
> initramfs with a full glibc environment so this use is becoming
> less common.

I think I've finally "tracked it down".  Or at least I've found
a workaround: if I pass "--disable-o_direct" to configure, everything
works fine.

While trying to track it down, I saw that calls to "read" (from within
_io in dev-io.c) would often return garbage (tho not always).  Not sure
if it's a problem in LVM2, or in uClibc, or in the kernel, but I think
I've already wasted more than enough time on this, so I'll just use
the --disable-o_direct workaround.


        Stefan

Re: [linux-lvm] Re: Problems running my home-built LVM2

[Alasdair G Kergon]

On Mon, Feb 16, 2009 at 06:20:32PM -0500, Stefan Monnier wrote:
> _io in dev-io.c) would often return garbage (tho not always).  Not sure
> if it's a problem in LVM2, or in uClibc, or in the kernel, but I think
> I've already wasted more than enough time on this, so I'll just use
> the --disable-o_direct workaround.
 
We had a problem with O-direct several years ago which we blamed on a rogue
version of gcc.

Alasdair
-- 
agk@redhat.com

[linux-lvm] Re: Problems running my home-built LVM2

[Stefan Monnier]

>> _io in dev-io.c) would often return garbage (tho not always).  Not sure
>> if it's a problem in LVM2, or in uClibc, or in the kernel, but I think
>> I've already wasted more than enough time on this, so I'll just use
>> the --disable-o_direct workaround.
> We had a problem with O-direct several years ago which we blamed on a rogue
> version of gcc.

Based on my experience I'd say that it seems to still be around.
At least on `mipsel'.  I have since discovered that Debian's `lvm2'
package disables O_DIRECT with the following lines:

   ifneq (,$(findstring $(DEB_HOST_ARCH), arm armeb hppa mips mipsel))
   CONFIGURE_FLAGS += --disable-o_direct
   endif

so that's probably why the Debian package worked fine.  It's probably
unrelated to uClibc.  My wild gut feeling is that it's a bug in the
kernel's handling of cache coherency.

Related Debian bugs are #351632 #330166 #267157 #247896.


        Stefan

Re: [linux-lvm] Re: Problems running my home-built LVM2

[Milan Broz]

Stefan Monnier wrote:
>>> _io in dev-io.c) would often return garbage (tho not always).  Not sure
>>> if it's a problem in LVM2, or in uClibc, or in the kernel, but I think
>>> I've already wasted more than enough time on this, so I'll just use
>>> the --disable-o_direct workaround.
>> We had a problem with O-direct several years ago which we blamed on a rogue
>> version of gcc.
> 
> Based on my experience I'd say that it seems to still be around.
> At least on `mipsel'.  I have since discovered that Debian's `lvm2'
> package disables O_DIRECT with the following lines:
> 
>    ifneq (,$(findstring $(DEB_HOST_ARCH), arm armeb hppa mips mipsel))
>    CONFIGURE_FLAGS += --disable-o_direct
>    endif

Ah, ok. That explains why my test on i686 arch & uClibc works:)

This really seems like problem unrelated to LVM2,
is possible that O_DIRECT is just buggy on ARM & similar archs
see http://lkml.org/lkml/2008/11/19/300 for example...

Milan 
--
mbroz@redhat.com

[linux-lvm] Re: Problems running my home-built LVM2

[Stefan Monnier]

> This really seems like problem unrelated to LVM2,
> is possible that O_DIRECT is just buggy on ARM & similar archs
> see http://lkml.org/lkml/2008/11/19/300 for example...

Yes, that's also my impression.  It would be good to add a clear
note/warning about it somewhere in the README, or even change the
`configure.in' to test O_DIRECT (or even disable it on the affected
platforms).
I spent a lot of time trying to figure out why my LVM2 build
didn't work and how to fix it, and I hope nobody else needs to go
through that.


        Stefan