[linux-lvm] Strange LVM Error With AoE Disks

[linux-lvm] Strange LVM Error With AoE Disks

[Jayson Vantuyl]

[-- Attachment #1: Type: text/plain, Size: 2656 bytes --]

Hello,

We have been using Coraid's ATA-Over-Ethernet shelves for a while  
with much success.

Recently, we added a second shelf (numbered 1) to our first shelf  
(numbered 0).  CLVM has been running on the old shelf perfectly fine.

As soon as I added the second shelf, attempting to lvcreate a new lv  
utilizing the new disks generated roughly the following errors:

   Error locking on node ey00-02: Internal lvm error, check syslog
   Error locking on node ey00-05: Internal lvm error, check syslog
   Error locking on node ey00-01: Internal lvm error, check syslog
   Error locking on node ey00-00: Internal lvm error, check syslog
   Error locking on node ey00-04: Internal lvm error, check syslog
   Error locking on node ey00-03: Internal lvm error, check syslog
   Failed to activate new LV.

All of the nodes show the following errors in syslog:

Feb  7 06:09:36 ey00-00 lvm[4869]: Couldn't find all physical volumes  
for volume group ey00-data.
Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find device with uuid  
'0Cot9Z-BHjK-2Nkw-eEdy-fbFF-Wh1q-qhRaut'.
Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find all physical volumes  
for volume group ey00-data.
Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find device with uuid  
'0Cot9Z-BHjK-2Nkw-eEdy-fbFF-Wh1q-qhRaut'.
Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find all physical volumes  
for volume group ey00-data.
Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find device with uuid  
'0Cot9Z-BHjK-2Nkw-eEdy-fbFF-Wh1q-qhRaut'.
Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find all physical volumes  
for volume group ey00-data.
Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find device with uuid  
'0Cot9Z-BHjK-2Nkw-eEdy-fbFF-Wh1q-qhRaut'.
Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find all physical volumes  
for volume group ey00-data.
Feb  7 06:09:37 ey00-00 lvm[4869]: Volume group for uuid not found:  
WWbD8SXOsAJzDYRCFQiciQho84Rl99nVF7QbO0ArRxnH4cZeKgzG0Nx4gbEhgALU

Inspecting the lvm.conf shows that these devices are the new ones  
that were added.

Even more bizarre, pvscan finds them just fine on all nodes.

The only thing I can note about these devices that is particularly  
different is that they appear to be using minor numbers above 256.   
Note this ls output:

brw-rw---- 1 root disk 152, 288 Feb  7 04:43 /dev/etherd/e1.2
brw-rw---- 1 root disk 152, 289 Feb  7 06:10 /dev/etherd/e1.2p1
brw-rw---- 1 root disk 152, 304 Feb  7 04:44 /dev/etherd/e1.3
brw-rw---- 1 root disk 152, 305 Feb  7 06:10 /dev/etherd/e1.3p1

Is there a known problem with LVM or CLVM related to large device  
minor numbers?

-- 
Jayson Vantuyl
Systems Architect
Engine Yard
jvantuyl@engineyard.com



[-- Attachment #2: Type: text/html, Size: 5927 bytes --]

Re: [linux-lvm] Strange LVM Error With AoE Disks

[Patrick Caulfield]

Jayson Vantuyl wrote:
> Hello,
> 
> We have been using Coraid's ATA-Over-Ethernet shelves for a while with
> much success.
> 
> Recently, we added a second shelf (numbered 1) to our first shelf
> (numbered 0).  CLVM has been running on the old shelf perfectly fine.
> 
> As soon as I added the second shelf, attempting to lvcreate a new lv
> utilizing the new disks generated roughly the following errors:
> 
>   Error locking on node ey00-02: Internal lvm error, check syslog
>   Error locking on node ey00-05: Internal lvm error, check syslog
>   Error locking on node ey00-01: Internal lvm error, check syslog
>   Error locking on node ey00-00: Internal lvm error, check syslog
>   Error locking on node ey00-04: Internal lvm error, check syslog
>   Error locking on node ey00-03: Internal lvm error, check syslog
>   Failed to activate new LV.
> 
> All of the nodes show the following errors in syslog:
> 
> Feb  7 06:09:36 ey00-00 lvm[4869]: Couldn't find all physical volumes
> for volume group ey00-data.
> Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find device with uuid
> '0Cot9Z-BHjK-2Nkw-eEdy-fbFF-Wh1q-qhRaut'.
> Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find all physical volumes
> for volume group ey00-data.
> Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find device with uuid
> '0Cot9Z-BHjK-2Nkw-eEdy-fbFF-Wh1q-qhRaut'.
> Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find all physical volumes
> for volume group ey00-data.
> Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find device with uuid
> '0Cot9Z-BHjK-2Nkw-eEdy-fbFF-Wh1q-qhRaut'.
> Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find all physical volumes
> for volume group ey00-data.
> Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find device with uuid
> '0Cot9Z-BHjK-2Nkw-eEdy-fbFF-Wh1q-qhRaut'.
> Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find all physical volumes
> for volume group ey00-data.
> Feb  7 06:09:37 ey00-00 lvm[4869]: Volume group for uuid not found:
> WWbD8SXOsAJzDYRCFQiciQho84Rl99nVF7QbO0ArRxnH4cZeKgzG0Nx4gbEhgALU
> 
> Inspecting the lvm.conf shows that these devices are the new ones that
> were added.
> 
> Even more bizarre, pvscan finds them just fine on all nodes.
> 
> The only thing I can note about these devices that is particularly
> different is that they appear to be using minor numbers above 256.  Note
> this ls output:
> 
> brw-rw---- 1 root disk 152, 288 Feb  7 04:43 /dev/etherd/e1.2
> brw-rw---- 1 root disk 152, 289 Feb  7 06:10 /dev/etherd/e1.2p1
> brw-rw---- 1 root disk 152, 304 Feb  7 04:44 /dev/etherd/e1.3
> brw-rw---- 1 root disk 152, 305 Feb  7 06:10 /dev/etherd/e1.3p1
> 
> Is there a known problem with LVM or CLVM related to large device minor
> numbers?


Did you restart clvmd after adding the new disks? There is a bug where it
doesn't always recognise new hardware.

-- 

patrick

Re: [linux-lvm] Strange LVM Error With AoE Disks

[Jayson Vantuyl]

[-- Attachment #1: Type: text/plain, Size: 311 bytes --]

No, I didn't, actually.  That worked.  Thanks a MILLION!

On Feb 7, 2007, at 8:44 AM, Patrick Caulfield wrote:

> Did you restart clvmd after adding the new disks? There is a bug  
> where it
> doesn't always recognise new hardware.



-- 
Jayson Vantuyl
Systems Architect
Engine Yard
jvantuyl@engineyard.com



[-- Attachment #2: Type: text/html, Size: 3077 bytes --]

[linux-lvm] Re: Strange LVM Error With AoE Disks

[Ed L. Cashin]

On Wed, Feb 07, 2007 at 08:16:23AM -0600, Jayson Vantuyl wrote:
...
> All of the nodes show the following errors in syslog:
> 
> Feb  7 06:09:36 ey00-00 lvm[4869]: Couldn't find all physical volumes  
> for volume group ey00-data.
> Feb  7 06:09:37 ey00-00 lvm[4869]: Couldn't find device with uuid  
> '0Cot9Z-BHjK-2Nkw-eEdy-fbFF-Wh1q-qhRaut'.
...
> Even more bizarre, pvscan finds them just fine on all nodes.
> 
> The only thing I can note about these devices that is particularly  
> different is that they appear to be using minor numbers above 256.   
...
> Is there a known problem with LVM or CLVM related to large device  
> minor numbers?

Hello, Jayson Vantuyl.  I just wanted to chime in to say that we are
not aware of such a problem, but we wouldn't necessarily have heard
about it, since the population using CLVM on AoE is probably not large
yet.  (Many use LVM, but I haven't heard much feedback from folks
using CLVM.)

I haven't looked at the CLVM sources really recently, but we have seen
the large minor device number problem before in older versions of LVM
where glibc's macros are used to work with the major and minor parts
that are encoded within device numbers.  So if there's a problem, it
might be possible to find it by looking for those macros.

-- 
  Ed L Cashin <ecashin@coraid.com>