Re: Oops when starting md multipath on a 2.4 kernel

[James Pearson <james-p@moving-picture.com>]

Luciano Chavez wrote:
> On Thu, 2005-07-14 at 11:09 +0100, James Pearson wrote:
> 
>>Mike Tran wrote:
>>
>>>James Pearson wrote:
>>>
>>>
>>>>We have an existing system runing a 2.4.27 based kernel that uses md 
>>>>multipath and external fibre channel arrays.
>>>>
>>>>We need to add more internal disks to the system, which means the 
>>>>external drives change device names.
>>>>
>>>>When I tried to start the md multipath device using mdadm, the kernel 
>>>>Oops'd. Removing the new internal disks and going back the original 
>>>>setup, I can start the multipath device - as this machine is in 
>>>>production, I can't do any more tests.
>>>>
>>>>However, I can reproduce the problem on test system by creating an md 
>>>>multipath device on an external SCSI disk, using /dev/sda1, stopping 
>>>>the multipath device, rmmod'ing the SCSI driver, pluging in a couple 
>>>>of USB storage devices which become /dev/sda and /dev/sdb and then 
>>>>modprobing the SCSI driver, so the original /dev/sda1 is now /dev/sdc1.
>>>>
>>>>When I run 'mdadm -A -s', I get the following Oops:
>>>>
>>>> [events: 00000004]
>>>>md: bind<sdc1,1>
>>>>md: sdc1's event counter: 00000004
>>>>md0: former device sda1 is unavailable, removing from array!
>>>>md: unbind<sdc1,0>
>>>>md: export_rdev(sdc1)
>>>>md: RAID level -4 does not need chunksize! Continuing anyway.
>>>>md: multipath personality registered as nr 7
>>>>md0: max total readahead window set to 124k
>>>>md0: 1 data-disks, max readahead per data-disk: 124k
>>>>Unable to handle kernel NULL pointer dereference at virtual address 
>>>>00000040
>>>> printing eip:
>>>>e096527e
>>>>*pde = 00000000
>>>>Oops: 0000
>>>>CPU:    0
>>>>EIP:    0010:[<e096527e>]    Not tainted
>>>>EFLAGS: 00010246
>>>>eax: deb62a94   ebx: 00000000   ecx: dd65b400   edx: 00000000
>>>>esi: 0000001c   edi: deb62a94   ebp: 00000000   esp: dd5fbdbc
>>>>ds: 0018   es: 0018   ss: 0018
>>>>Process mdadm (pid: 1389, stackpage=dd5fb000)
>>>>Stack: dd4c4000 dfa96000 c035ad00 00000000 00000286 dd4c4000 00000000 
>>>>00000000
>>>>       deb62a94 dd5fbe5c dd4c6000 c02a6e10 dd65b400 c035ef1f 0000007c 
>>>>00000000
>>>>       0000000a ffffffff 00000002 00002e2e c0118b49 00002e2e 00002e2e 
>>>>00000286
>>>>Call Trace:    [<c02a6e10>] [<c0118b49>] [<c0118cc4>] [<c024a88c>] 
>>>>[<c024abb6>]
>>>>  [<c0118cc4>] [<c024907e>] [<c024b6f2>] [<c024c60c>] [<c014a326>] 
>>>>[<c013c483>]
>>>>  [<c013ca18>] [<c01375ac>] [<c013ca63>] [<c01439b6>] [<c01087c7>]
>>>>
>>>>Code: 8b 45 40 85 c0 0f 84 c2 01 00 00 6a 00 ff b4 24 cc 00 00 00
>>>>
>>>>Running through ksymoops gives:
>>>>
>>>>Unable to handle kernel NULL pointer dereference at virtual address 
>>>>00000040
>>>>e096527e
>>>>*pde = 00000000
>>>>Oops: 0000
>>>>CPU:    0
>>>>EIP:    0010:[<e096527e>]    Not tainted
>>>>Using defaults from ksymoops -t elf32-i386 -a i386
>>>>EFLAGS: 00010246
>>>>eax: deb62a94   ebx: 00000000   ecx: dd65b400   edx: 00000000
>>>>esi: 0000001c   edi: deb62a94   ebp: 00000000   esp: dd5fbdbc
>>>>ds: 0018   es: 0018   ss: 0018
>>>>Process mdadm (pid: 1389, stackpage=dd5fb000)
>>>>Stack: dd4c4000 dfa96000 c035ad00 00000000 00000286 dd4c4000 00000000 
>>>>00000000
>>>>       deb62a94 dd5fbe5c dd4c6000 c02a6e10 dd65b400 c035ef1f 0000007c 
>>>>00000000
>>>>       0000000a ffffffff 00000002 00002e2e c0118b49 00002e2e 00002e2e 
>>>>00000286
>>>>Call Trace:    [<c02a6e10>] [<c0118b49>] [<c0118cc4>] [<c024a88c>] 
>>>>[<c024abb6>]
>>>>  [<c0118cc4>] [<c024907e>] [<c024b6f2>] [<c024c60c>] [<c014a326>] 
>>>>[<c013c483>]
>>>>  [<c013ca18>] [<c01375ac>] [<c013ca63>] [<c01439b6>] [<c01087c7>]
>>>>Code: 8b 45 40 85 c0 0f 84 c2 01 00 00 6a 00 ff b4 24 cc 00 00 00
>>>>
>>>>
>>>>>>EIP; e096527e <[multipath]multipath_run+2be/6c0>   <=====
>>>>
>>>>Trace; c02a6e10 <vsnprintf+2e0/450>
>>>>Trace; c0118b49 <call_console_drivers+e9/f0>
>>>>Trace; c0118cc4 <printk+104/110>
>>>>Trace; c024a88c <device_size_calculation+19c/1f0>
>>>>Trace; c024abb6 <do_md_run+2d6/360>
>>>>Trace; c0118cc4 <printk+104/110>
>>>>Trace; c024907e <bind_rdev_to_array+9e/b0>
>>>>Trace; c024b6f2 <add_new_disk+132/290>
>>>>Trace; c024c60c <md_ioctl+6fc/790>
>>>>Trace; c014a326 <iput+236/240>
>>>>Trace; c013c483 <bdput+93/a0>
>>>>Trace; c013ca18 <blkdev_put+98/a0>
>>>>Trace; c01375ac <fput+bc/e0>
>>>>Trace; c013ca63 <blkdev_ioctl+23/30>
>>>>Trace; c01439b6 <sys_ioctl+216/230>
>>>>Trace; c01087c7 <system_call+33/38>
>>>>Code;  e096527e <[multipath]multipath_run+2be/6c0>
>>>>00000000 <_EIP>:
>>>>Code;  e096527e <[multipath]multipath_run+2be/6c0>   <=====
>>>>   0:   8b 45 40                  mov    0x40(%ebp),%eax   <=====
>>>>Code;  e0965281 <[multipath]multipath_run+2c1/6c0>
>>>>   3:   85 c0                     test   %eax,%eax
>>>>Code;  e0965283 <[multipath]multipath_run+2c3/6c0>
>>>>   5:   0f 84 c2 01 00 00         je     1cd <_EIP+0x1cd> e096544b 
>>>><[multipath]m
>>>>ultipath_run+48b/6c0>
>>>>Code;  e0965289 <[multipath]multipath_run+2c9/6c0>
>>>>   b:   6a 00                     push   $0x0
>>>>Code;  e096528b <[multipath]multipath_run+2cb/6c0>
>>>>   d:   ff b4 24 cc 00 00 00      pushl  0xcc(%esp,1)
>>>>
>>>>My /etc/mdadm.conf contains:
>>>>
>>>>DEVICE /dev/sd?1
>>>>ARRAY /dev/md0 level=multipath num-devices=1
>>>>  UUID=277e4ba5:6c23c087:e17c877c:da642955
>>>>
>>>>
>>>>Should md multipath be able to handle changes like this with the 
>>>>underlying devices?
>>>>
>>>>
>>>>Thanks
>>>>
>>>>James Pearson
>>>>
>>>
>>>Hi James,
>>>
>>>My co-worker and I just happened to run into this problem a few days 
>>>ago. So, I would like to share with you what we know.
>>>
>>>The device major/minor numbers no longer match up values recorded in the 
>>>descriptor array in the md superblock. Because of the exception made in 
>>>the current code, the descriptor entries are removed and although the 
>>>real devices are present and accounted for, they are kicked out from the 
>>>array. This leaves the array with zero devices. When multipath_run() is 
>>>invoked, it blows up expecting to have had some disks.
>>>
>>>Lars Marowsky-Brée suggested some patches for md multipath in 2002 but 
>>>never made it to mainline 2.4 kernel:
>>>http://marc.theaimsgroup.com/?l=linux-kernel&m=103355467608953&w=2
>>>
>>>That patch is large and most of it is not requried for this particular 
>>>problem.  The section that reinitializes the descriptor array from 
>>>current rdevs for the case of multipath will resolve this issue of 
>>>device names shift.
>>>
>>>Lars, Is it ok with you if I compose a patch from your original patch 
>>>and post it here?
>>
>>Thanks - that patch applies OK to more recent 2.4 kernels and appears to 
>>'fix' this problem.
>>
>>However, if you have a cut down patch that fixes just this problem, then 
>>I would appreciate it if you could make it available.
>>
>>Thanks
>>
>>James Pearson
>>
> 
> 
> James,
> 
> Here is the reduced patch from the patch that Lars originally produced
> that worked for us for that particular problem with the multipath disks
> major/minor numbers shifting. Hopefully, Marcelo can review it and
> consider it for inclusion in 2.4 mainline. Let us know if this works for
> you. The credit, of couse, should still go to Lars. We simply picked out
> the part that fixes that particular issue.

Patch appears to work fine.

Thanks

James Pearson
-
To unsubscribe from this list: send the line "unsubscribe linux-raid" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html