dm-mirror/kmirrord oops

dm-mirror/kmirrord oops

[Cyril Chaboisseau]

I just had a big problem lately when I wanted to migrate my PV from one
disk to another

to do so, I first created a LVM partition on my second disk,

then I initialized the partition (pvcreate)
# pvcreate /dev/sdb1

I extended my vg to the newly PV
# vgextend vg /dev/sdb1

and then I tried to move the data from the old PV to the newly created
# pvmove -v /dev/sda4 /dev/sdb1

everything went fine for maybe half an hour, but at 21% it froze

I waited for maybe 15/20mn and couldn't do anything
the machine seemed to respond (KDE running with clock and gkrellm
showing progress) but I couldn't launch any command : everytime I typed
one nothing happened

so, I ended up rebooting the machine (reset)


after that, every attempt to boot multi-user or Single failed with an
oops on dm-mirror

at that point I was running 2.6.11.11 + grsec (BTW, the processor is amd64)

I finally stabilized things by commenting some LVM partitions on my fstab
(everything except /usr and /var) but every time I tried to backup
things onto another disk it would crash again

so I tried 2.6.12.1 with CONFIG_DM_MIRROR built-in instead of module

it worked a little but again after some time it crashed again with the
following :




Unable to handle kernel paging request at ffffc2000015c128 RIP: 
<ffffffff802964a8>{core_in_sync+8}
PGD 3ff92067 PUD 3ff91067 PMD 3ff90067 PTE 0
Oops: 0000 [1] 
CPU 0 
Modules linked in: af_packet ipv6 ip_nat_ftp ip_conntrack_ftp iptable_mangle iptable_nat snd_via82xx gameport snd_ac97_codec snd_pcm_oss snd_mixer_oss snd_pcm snd_timer snd_page_alloc snd_mpu401_uart snd_rawmidi snd_seq_device snd soundcore ehci_hcd tsdev eth1394 usbhid uhci_hcd 3c59x rt2500 ohci1394 quota_v2 cpufreq_powersave cpufreq_ondemand powernow_k8 freq_table processor w83627hf i2c_sensor i2c_isa i2c_core sr_mod sbp2 ieee1394 ide_cd cdrom evdev unix
Pid: 848, comm: kmirrord/0 Not tainted 2.6.12.1
RIP: 0010:[core_in_sync+8/32] <ffffffff802964a8>{core_in_sync+8}
RSP: 0018:ffff8100022afbf0  EFLAGS: 00010246
RAX: ffffc2000015b000 RBX: ffff81003b42e2d8 RCX: ffffc2000015f1a0
RDX: 0000000000000001 RSI: 0000000000008940 RDI: ffff81000ff93440
RBP: 0000000000008940 R08: 0000000000008940 R09: ffff810001b71f38
R10: 0000000000000001 R11: ffff810001000000 R12: 0000000000000001
R13: ffff81003b42e2c0 R14: ffff81003b42e2d8 R15: ffff81000ff93440
FS:  00002aaaab00d4a0(0000) GS:ffffffff8047a9c0(0000) knlGS:0000000000000000
CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
CR2: ffffc2000015c128 CR3: 000000002475a000 CR4: 00000000000006e0
Process kmirrord/0 (pid: 848, threadinfo ffff8100022ae000, task ffff810002264130)
Stack: ffffffff802969da ffff81000bb56b00 0000000000000000 ffff8100022afd28 
       ffffffff802972c7 ffff81003f7c3e00 32c932e0cc32cc37 9300814000002400 
       ffff81003b42e308 0000000000000002 
Call Trace:<ffffffff802969da>{rh_state+58} <ffffffff802972c7>{do_work+1735}
       <ffffffff801536c0>{mempool_free_slab+0} <ffffffff80306184>{thread_return+0}
       <ffffffff8013e5a8>{worker_thread+440} <ffffffff8012c1e0>{default_wake_function+0}
       <ffffffff8012c1e0>{default_wake_function+0} <ffffffff8013e3f0>{worker_thread+0}
       <ffffffff8014241d>{kthread+205} <ffffffff8010f173>{child_rip+8}
       <ffffffff801e4b70>{vgacon_cursor+0} <ffffffff80142460>{keventd_create_kthread+0}
       <ffffffff80142350>{kthread+0} <ffffffff8010f16b>{child_rip+0}
       

Code: 0f a3 30 19 f6 31 c0 85 f6 0f 95 c0 c3 66 66 66 90 66 66 66 
RIP <ffffffff802964a8>{core_in_sync+8} RSP <ffff8100022afbf0>
CR2: ffffc2000015c128



after several attemps to move the PV to another disk, I finally rebooted
multi-user and under a graphical console I realized that a lot of disk
activity was going on plus a /dev/vg/pmove0 LV was created

so I waited until the pvmove finalized and after that, I could copy,
delete, remove LV and so on without any glitch or freeze !




from my understanding, there is a locking problem and if a process tries
to do something on a block while it's being moved, then dm-mirror (or
kmirrod if built-in) crashes

has anyone experienced the same problem ?

could this be related to the 64 bits arch ? (amd64)


thanks for your help



PS : I finally destroyed the whole LVM and reconstructed another one
from backup so I cannot provide anymore information nor reproduce the
bug

-- 
	Cyril Chaboisseau

Re: dm-mirror/kmirrord oops

[Jakob Oestergaard]

On Mon, Jun 27, 2005 at 09:07:09AM +0200, Cyril Chaboisseau wrote:
> I just had a big problem lately when I wanted to migrate my PV from one
> disk to another
> 
> to do so, I first created a LVM partition on my second disk,
> 
> then I initialized the partition (pvcreate)
> # pvcreate /dev/sdb1
> 
> I extended my vg to the newly PV
> # vgextend vg /dev/sdb1
> 
> and then I tried to move the data from the old PV to the newly created
> # pvmove -v /dev/sda4 /dev/sdb1
> 
> everything went fine for maybe half an hour, but at 21% it froze

Is your root filesystem mounted on an LV in 'vg' ?

As far as I know, this is a known problem - you cannot pvmove your root
filesystem.

But on another note; you should not have your root (or at least /etc)
filesystem on LVM - because the LVM backup/recovery files are written
there, and if LVM screws up completely, you'll then have no way of
recovering (since the recovery files you need to get LVM going are
stored via. LVM).

I always create a partition for / and keep /boot, /sbin, /lib and /etc
there.

Then, I create another partition and make it a PV, put /var, /usr and
everything else on LVM.

That way I can pvmove anything without running into the bug you
(probably) saw, and I will be able to recover the LVM in case it screws
up (which I haven't seen yet).

-- 

 / jakob

Re: dm-mirror/kmirrord oops

[Cyril Chaboisseau]

 Le 27 June vers 13:57, Jakob Oestergaard écrivait:
> Is your root filesystem mounted on an LV in 'vg' ?

no it's not

everything but / and /boot were in LVM

Sys. de fich.        1K-blocs       Occupé Disponible Capacité Monté sur
/dev/sda1               918322    263155    606171  31% /
tmpfs                   511064         0    511064   0% /dev/shm
/dev/sda3               233366     31130    189786  15% /boot
/dev/mapper/vg-tmp     1032088     32960    946700   4% /tmp
/dev/mapper/vg-usr     6192704   3577264   2300868  61% /usr
/dev/mapper/vg-var     2064208    679576   1279776  35% /var
/dev/mapper/vg-vcache
                       2064208    102568   1856784   6% /var/cache
/dev/mapper/vg-ulocal
                       6192704   2311668   3629380  39% /usr/local
/dev/mapper/vg-home   16513960   4829176  10845924  31% /home
tmpfs                    10240      3332      6908  33% /dev

this is the new layout but it was pretty much the way things were
organised


> But on another note; you should not have your root (or at least /etc)
> filesystem on LVM - because the LVM backup/recovery files are written
> there, and if LVM screws up completely, you'll then have no way of
> recovering (since the recovery files you need to get LVM going are
> stored via. LVM).

I thing that not having / in LVM saved my datas !
(I was indeed able to boot with a Knoppix and comment out the
problematic partitions)

> I always create a partition for / and keep /boot, /sbin, /lib and /etc
> there.
> 
> Then, I create another partition and make it a PV, put /var, /usr and
> everything else on LVM.

indeed

> That way I can pvmove anything without running into the bug you
> (probably) saw, and I will be able to recover the LVM in case it screws
> up (which I haven't seen yet).

in my case, I really thing that there was a problem with pvmove/kmirrord
when it tried to move data when they were being accessed

now, this is just a _wild_ guess after spending the whole afternoon
trying to recover my disks and I have no hard figures to back this 
speculation

I just realized that when I finally let pvmove finish it's work
everything went fine while previously every single attempt to copy data
to a new disk failed one way or another the same way : Oops in the
kernel

-- 
	Cyril Chaboisseau