2.6.9-mm1: LVM stopped working

2.6.9-mm1: LVM stopped working

[Mathieu Segaud]

Well, I gave a try to last -mm tree. The bot seemed good till it got to
LVM stuff. Vgchange does not find any volume groups. I can't say much because
lvm is pretty "early stuff" on this box; so it is pretty unusable. All I know
for now, as I changed a little my boot scripts to be more verbose, is that
vgchange -avvv y returns this kind of message: 
hdXN: cannot read LABEL
and this message for all parts it can test....
As I need this box up and running, I came back to 2.6.9-rc3-mm3 (it works
pretty well). I will be able to run more tests on it, tomorrow but for now
that's all I can provide.

Oh and dmesg didn't have any oops or BUG in it, and seemed quite usual,
in IDE detection and settings messages and device-mapper messages.

However, I use dm-crypt to encrypt my / (no initrd, just initramfs) and
it works under 2.6.9-mm1, so the bug is likely to be in IDE stuff.

Sorry, for not being able to provide more infos. I will see if I can try on
another LVM'ed box but not for critical stuff.

Mathieu

-- 
Lots of luck ... please pass your crack pipe arounds so the rest of us
idiots can see your vision or lack of ... 

	- Andre Hedrick on linux-kernel

Re: 2.6.9-mm1: LVM stopped working

[Christophe Saout]

[-- Attachment #1: Type: text/plain, Size: 1165 bytes --]

Am Sonntag, den 24.10.2004, 01:06 +0200 schrieb Mathieu Segaud:

> Well, I gave a try to last -mm tree. The bot seemed good till it got to
> LVM stuff. Vgchange does not find any volume groups. I can't say much because
> lvm is pretty "early stuff" on this box; so it is pretty unusable. All I know
> for now, as I changed a little my boot scripts to be more verbose, is that
> vgchange -avvv y returns this kind of message: 
> hdXN: cannot read LABEL
> and this message for all parts it can test....
> As I need this box up and running, I came back to 2.6.9-rc3-mm3 (it works
> pretty well). I will be able to run more tests on it, tomorrow but for now
> that's all I can provide.
> 
> Oh and dmesg didn't have any oops or BUG in it, and seemed quite usual,
> in IDE detection and settings messages and device-mapper messages.
> 
> However, I use dm-crypt to encrypt my / (no initrd, just initramfs) and
> it works under 2.6.9-mm1, so the bug is likely to be in IDE stuff.

Are you encrypting your PV or your LVs?

There's some new dm-crypt code in -mm1 along with some API changes, but
backward compatibility is provided and should work.


[-- Attachment #2: Dies ist ein digital signierter Nachrichtenteil --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.9-mm1: LVM stopped working

[Mathieu Segaud]

Christophe Saout <christophe@saout.de> disait dernièrement que :

> Are you encrypting your PV or your LVs?

non they are not encrypted, I thought about the new iv but my aes-encrypted
/ still boot :)
I wonder if it comes from some ide changes, as the messages from vgscan and
vgchange indicate that LABEL areas are detected, but cannot be read....

quite weird as anything else works quite well...

>
> There's some new dm-crypt code in -mm1 along with some API changes, but
> backward compatibility is provided and should work.

Best regards,

Mathieu

-- 
printk("----------- [cut here ] --------- [please bite here ] ---------\n");
        linux-2.6.6/arch/x86_64/kernel/traps.

Re: 2.6.9-mm1: LVM stopped working

[Mathieu Segaud]

Christophe Saout <christophe@saout.de> disait dernièrement que :

> Are you encrypting your PV or your LVs?
>
> There's some new dm-crypt code in -mm1 along with some API changes, but
> backward compatibility is provided and should work.

I tried 2.6.9-mm1, reverting all the new dm-crypt stuff, and it didn't make it.
So it is not related to these patches.
Will look further into it later; for now I must go working on my PhD :)

Best regards,

-- 
<riel> google rules
<google> rules: http://www.law.cornell.edu/rules/fre/overview.html

	- Rik van Riel chatting with the bots on #kernelnewbies

Re: 2.6.9-mm1: LVM stopped working

[jfannin1]

On Mon, Oct 25, 2004 at 09:03:22PM +0200, Christophe Saout wrote:
> Am Sonntag, den 24.10.2004, 01:06 +0200 schrieb Mathieu Segaud:
> 
>>Well, I gave a try to last -mm tree. The bot seemed good till it got to
>>LVM stuff. Vgchange does not find any volume groups. I can't say much because
>>lvm is pretty "early stuff" on this box; so it is pretty unusable.

    LVM doesn't work with 2.6.9-mm1 here either, complaining that it
can't find all the pv's. I'm not using any sort of encryption. Here,
pvdisplay reports:

  --- Physical volume ---
  PV Name               /dev/hda2
  VG Name               home
  PV Size               24.52 GB / not usable 0   
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              6277
  Free PE               0
  Allocated PE          6277
  PV UUID               M8tcls-7Tp7-sAYe-ypH3-if50-00JH-hvvXSL
   
  --- Physical volume ---
  PV Name               unknown device
  VG Name               home
  PV Size               70.47 GB / not usable 0   
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              18040
  Free PE               0
  Allocated PE          18040
  PV UUID               SmreB9-Q3dp-DBBc-q0N9-v762-o6UB-1VUgYw
   
  --- Physical volume ---
  PV Name               unknown device
  VG Name               home
  PV Size               25.12 GB / not usable 0   
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              6431
  Free PE               0
  Allocated PE          6431
  PV UUID               sbbFSh-0MP8-jtir-Jcyx-VtcE-TxNh-tfNwNe

    I can open the device nodes for the 'missing' pv's in a hexeditor and see the
uuid magic; if I reboot into 2.6.9-rc4-mm1 they are found without a
problem, and everything works.

    Whether or not I'll have time to try to narrow down the change
that causes this depends on things that are out of my control ATM. :-/

-- 
Joseph Fannin
jfannin1@columbus.rr.com

Re: 2.6.9-mm1: LVM stopped working

[Jeff Chua]

On Tue, 26 Oct 2004 jfannin1@columbus.rr.com wrote:

> On Mon, Oct 25, 2004 at 09:03:22PM +0200, Christophe Saout wrote:
>> Am Sonntag, den 24.10.2004, 01:06 +0200 schrieb Mathieu Segaud:
>>
>>> Well, I gave a try to last -mm tree. The bot seemed good till it got to
>>> LVM stuff. Vgchange does not find any volume groups. I can't say much because
>>> lvm is pretty "early stuff" on this box; so it is pretty unusable.
>
>    LVM doesn't work with 2.6.9-mm1 here either, complaining that it
> can't find all the pv's. I'm not using any sort of encryption. Here,
> pvdisplay reports:

It doesn't work on 2.6.10-rc1 either. Works fine on 2.6.9 and 2.4.8-rc1.

   device-mapper ioctl cmd 0 failed: Inappropriate ioctl for device
   striped: Required device-mapper target(s) not detected in your kernel
   lvcreate: Create a logical volume



Jeff.

Re: 2.6.9-mm1: LVM stopped working

[Alasdair G Kergon]

On Tue, Oct 26, 2004 at 08:36:51AM -0400, jfannin1@columbus.rr.com wrote:
> LVM doesn't work with 2.6.9-mm1 here either, complaining that it
> can't find all the pv's. I'm not using any sort of encryption. Here,
> pvdisplay reports:

> I can open the device nodes for the 'missing' pv's in a hexeditor and see the
> uuid magic; if I reboot into 2.6.9-rc4-mm1 they are found without a
> problem, and everything works.

Firstly enable lvm debugging.  lvm.conf: log { file="/tmp/lvm2.log" level=7 }
Compare the lvm log files for the kernels to see where it's going wrong.
Then take complete straces (incl. read/write data) of the lvm process 
with each kernel and again compare them. [Or put files on web and send URLs.]

[To check for repeat of old problems with related symptoms:]
  Were both kernels compiled with the same compiler version? Which version?
  Does it make any difference if you rebuild lvm with --disable-o_direct?
 
Alasdair
-- 
agk@redhat.com

Re: 2.6.9-mm1: LVM stopped working

[Bartlomiej Zolnierkiewicz]

On Sun, 24 Oct 2004 01:06:07 +0200, Mathieu Segaud
<matt@minas-morgul.org> wrote:
> 
> Well, I gave a try to last -mm tree. The bot seemed good till it got to
> LVM stuff. Vgchange does not find any volume groups. I can't say much because
> lvm is pretty "early stuff" on this box; so it is pretty unusable. All I know
> for now, as I changed a little my boot scripts to be more verbose, is that
> vgchange -avvv y returns this kind of message:
> hdXN: cannot read LABEL
> and this message for all parts it can test....
> As I need this box up and running, I came back to 2.6.9-rc3-mm3 (it works
> pretty well). I will be able to run more tests on it, tomorrow but for now
> that's all I can provide.
> 
> Oh and dmesg didn't have any oops or BUG in it, and seemed quite usual,
> in IDE detection and settings messages and device-mapper messages.
> 
> However, I use dm-crypt to encrypt my / (no initrd, just initramfs) and
> it works under 2.6.9-mm1, so the bug is likely to be in IDE stuff.

prove it ;)

There were only minor IDE changes from 2.6.9-rc3-mm3 to 2.6.9-mm1,
I don't see any obvious suspects...

Re: 2.6.9-mm1: LVM stopped working

[Alasdair G Kergon]

On Tue, Oct 26, 2004 at 09:55:38PM +0800, Jeff Chua wrote:
> It doesn't work on 2.6.10-rc1 either. Works fine on 2.6.9 and 2.4.8-rc1.
>   device-mapper ioctl cmd 0 failed: Inappropriate ioctl for device

Do you get any corresponding kernel messages?
Check /dev/mapper/control corresponds to  /proc/devices & /proc/misc.
(See device-mapper scripts/devmap_mknod.sh)
Use 'dmsetup version' and 'dmsetup targets' to test.

Alasdair
-- 
agk@redhat.com

Re: 2.6.9-mm1: LVM stopped working

[Bartlomiej Zolnierkiewicz]

On Tue, 26 Oct 2004 16:00:47 +0200, Bartlomiej Zolnierkiewicz
<bzolnier@gmail.com> wrote:
> On Sun, 24 Oct 2004 01:06:07 +0200, Mathieu Segaud
> <matt@minas-morgul.org> wrote:
> >
> > Well, I gave a try to last -mm tree. The bot seemed good till it got to
> > LVM stuff. Vgchange does not find any volume groups. I can't say much because
> > lvm is pretty "early stuff" on this box; so it is pretty unusable. All I know
> > for now, as I changed a little my boot scripts to be more verbose, is that
> > vgchange -avvv y returns this kind of message:
> > hdXN: cannot read LABEL
> > and this message for all parts it can test....
> > As I need this box up and running, I came back to 2.6.9-rc3-mm3 (it works
> > pretty well). I will be able to run more tests on it, tomorrow but for now
> > that's all I can provide.
> >
> > Oh and dmesg didn't have any oops or BUG in it, and seemed quite usual,
> > in IDE detection and settings messages and device-mapper messages.
> >
> > However, I use dm-crypt to encrypt my / (no initrd, just initramfs) and
> > it works under 2.6.9-mm1, so the bug is likely to be in IDE stuff.
> 
> prove it ;)

To make this task easier I prepared 2.6.9-rc3-mm3 to 2.6.9-mm1 IDE patch:

http://home.elka.pw.edu.pl/~bzolnier/ide-2.6.9-rc3-mm3-to-2.6.9-mm1.patch.bz2

Just revert it from 2.6.9-mm1.

> There were only minor IDE changes from 2.6.9-rc3-mm3 to 2.6.9-mm1,
> I don't see any obvious suspects...

Re: 2.6.9-mm1: LVM stopped working

[Mathieu Segaud]

Bartlomiej Zolnierkiewicz <bzolnier@gmail.com> disait dernièrement que :

> To make this task easier I prepared 2.6.9-rc3-mm3 to 2.6.9-mm1 IDE patch:
>
> http://home.elka.pw.edu.pl/~bzolnier/ide-2.6.9-rc3-mm3-to-2.6.9-mm1.patch.bz2
>
> Just revert it from 2.6.9-mm1.

thx, I will test it soon.
I have just made straces of vgchange processes in success and failure cases
(there is little difference in the fact that in the failure case, I added
-v verbose option but that's all)

vgchange tries to read 2 chunks of data from the partition:
- the first 2048 bytes,
- and after closing device, and reopening it, the 512 next ones.

in the failure case, the first read succeeds with just 1536 bytes read,
which causes the process to issue another read syscall to read the "missing"
512 bytes, which fails...

for now, that's all I can see
I will enable lvm debugging, for the next try

the straces are:
http://www.crans.org/~segaud/vgchange.failure
http://www.crans.org/~segaud/vgchange.succeeded
(names are obvious)

Best regards,

Mathieu

-- 
"I am a living example of someone who took on an issue and benefited from it."

George W. Bush
April 25, 2001
Speaking to John King of CNN.

Re: 2.6.9-mm1: LVM stopped working

[Mathieu Segaud]

Bartlomiej Zolnierkiewicz <bzolnier@gmail.com> disait dernièrement que :


>> > However, I use dm-crypt to encrypt my / (no initrd, just initramfs) and
>> > it works under 2.6.9-mm1, so the bug is likely to be in IDE stuff.
>> 
>> prove it ;)
>
> To make this task easier I prepared 2.6.9-rc3-mm3 to 2.6.9-mm1 IDE patch:
>
> http://home.elka.pw.edu.pl/~bzolnier/ide-2.6.9-rc3-mm3-to-2.6.9-mm1.patch.bz2
>
> Just revert it from 2.6.9-mm1.

reverting ide changes do not change anything....
error is still here
The only changes I can see now, are the md changes. I will try reverting it,
and if I get no positive results, I give up (for today :))

-- 
printk("What? oldfid != cii->c_fid. Call 911.\n");
        linux-2.4.3/fs/coda/cnode.c

Re: 2.6.9-mm1: LVM stopped working

[Mathieu Segaud]

Mathieu Segaud <matt@minas-morgul.org> disait dernièrement que :

> reverting ide changes do not change anything....
> error is still here
> The only changes I can see now, are the md changes. I will try reverting it,
> and if I get no positive results, I give up (for today :))

obviously, md changes cannot have caused this failure....my mistake

I will dig again and again, if need be
-- 
dprintk(5, KERN_DEBUG "Jotti is een held!\n");
        linux-2.6.6/drivers/media/video/zoran_card.c

Re: 2.6.9-mm1: LVM stopped working

[Laurent Riffard]

[-- Attachment #1: Type: text/plain, Size: 2019 bytes --]


Hello,

Lvm2 stopped working since 2.6.9-mm1 for me too : 2.6.9-rc4-mm1 was 
fine, 2.6.9-mm1 to 2.6.10-mm2 break lvm2. Reverting 
dio-handle-eof.patch on these kernel solves the problem.

I have a simple test case here.

With 2.6.9-rc4-mm1, "pvdisplay /dev/hda4" shows :
 --- Physical volume ---
 PV Name               /dev/hda4
 VG Name               vglinux1
 PV Size               19,07 GB / not usable 0
 Allocatable           yes
 PE Size (KByte)       4096
 Total PE              4882
 Free PE               3424
 Allocated PE          1458
 PV UUID               Kvi5oA-d8NL-DU0n-vJpt-TKb3-RmDP-nrZoaz

With later -mm kernel, "pvdisplay /dev/hda4" shows :
 No physical volume label read from /dev/hda4
 Failed to read physical volume "/dev/hda4"

I tracked down the problem to this code section in fs/direct-io.c (function direct_io_worker) :

  1012                 dio->total_pages = 0;
  1013                 if (user_addr & (PAGE_SIZE-1)) {
  1014                         dio->total_pages++;
  1015                         bytes -= PAGE_SIZE - (user_addr & (PAGE_SIZE - 1));
  1016                 }
  1017                 dio->total_pages += (bytes + PAGE_SIZE - 1) / PAGE_SIZE;
  1018                 dio->curr_user_address = user_addr;
  1019
  1020                 ret = do_direct_IO(dio);
  1021
  1022                 dio->result += bytes -
  1023                         ((dio->final_block_in_request - dio->block_in_file) <<
  1024                                         blkbits);

In my case, direct_io_worker is called to read 2048 bytes at the beginning of /dev/hda4 :
 user_addr=0xbfff9800 (half page aligned)
 bytes=2048 (half page)
So "bytes" is zeroed line 1015.
And dio->result is zeroed line 1023.
As a result, direct_io_worker returns 0.

Before dio-handle-eof.patch, line 1022 was :
	dio->result += iov[seg].iov_len -

What is the semantic of "bytes" line 1015 : bytes to read on the next page ?
Did I miss something ?

hope this helps... 
I will do some tests if needed.

-- 
laurent



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 252 bytes --]

Re: 2.6.9-mm1: LVM stopped working

[Laurent Riffard]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


I applied the following patch [http://lkml.org/lkml/2004/11/2/129]
from Jens Axboe  onto kernel 2.6.10-rc1-mm2 (succeeded with offset 2
lines) and it solved the problem.

Thanks.

BTW, is there a mean to reply to a post in lkml witout being
subscribed ?
- --
laurent


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFBiAr6UqUFrirTu6IRAh8OAKC3bMrwHy8BFxOoEhg03VbQp9R5hACgu5pn
ZWguH7/6uODFa8NuhwinreQ=
=qXXZ
-----END PGP SIGNATURE-----