[linux-lvm] lvcreate with a 52 drive volume

[linux-lvm] lvcreate with a 52 drive volume

[markw]

I'm having trouble creating a striped logical volume across all 52 disks
in a volume group.  This is what happens when I try to create the
logical volume:

# lvcreate -i52 -I4 -l450996 -n mydata data_volume
  device-mapper ioctl cmd 9 failed: Invalid argument
  Couldn't load device 'data_volume-mydata'.
  Failed to activate new LV.

I see these two lines echoed to the console:
device-mapper: couldn't split parameters
device-mapper: error adding target to table

Here's the output for vgdisplay:

  --- Volume group ---
  VG Name               data_volume
  System ID             
  Format                lvm2
  Metadata Areas        52
  Metadata Sequence No  1
  VG Access             read/write
  VG Status             resizable
  MAX LV                256
  Cur LV                0
  Open LV               0
  Max PV                256
  Cur PV                52
  Act PV                52
  VG Size               1.72 TB
  PE Size               4.00 MB
  Total PE              450996
  Alloc PE / Size       0 / 0   
  Free  PE / Size       450996 / 1.72 TB
  VG UUID               kKlfs8-CWAA-zMg5-ELOm-GmI7-zOhx-KsXyAA


I reduced the number of disks in the stipe set by 4 drives (and the
logical volume size appropriately) until I could create a logical
volume with these parameters:
	lvcreate -i12 -I4 -l104076 -n mydata data_volume

Any suggestions?

Thanks!
-- 
Mark Wong - - markw@osdl.org
Open Source Development Lab Inc - A non-profit corporation
12725 SW Millikan Way - Suite 400 - Beaverton, OR 97005
(503) 626-2455 x 32 (office)
(503) 626-2436      (fax)
http://www.osdl.org/archive/markw/

Re: [linux-lvm] lvcreate with a 52 drive volume

[Alasdair G Kergon]

On Fri, Aug 15, 2003 at 01:11:51PM -0700, markw@osdl.org wrote:
> # lvcreate -i52 -I4 -l450996 -n mydata data_volume
>   device-mapper ioctl cmd 9 failed: Invalid argument
>   Couldn't load device 'data_volume-mydata'.
>   Failed to activate new LV.

Mark, please would you enable the debug messages
as described in the docs,
viz. the following settings in lvm.conf:

  log {
          file="/tmp/lvm2.log"
	  level=7
	  activation=1
  }

then look at /tmp/lvm2.log after running the
problematic command?
 
> device-mapper: couldn't split parameters
> device-mapper: error adding target to table

The log file will show you the actual parameters 
that were passed to device-mapper leading to
the error messages.
 
Please put the log file(s) somewhere on the web and 
post URL(s) to the list - or mail the log file(s)
to me privately as a *compressed* attachment.

Alasdair
-- 
agk@uk.sistina.com

Re: [linux-lvm] lvcreate with a 52 drive volume

[markw]

Ok, I've put the log file:

http://developer.osdl.org/markw/lvm2.log

Compressed:
http://developer.osdl.org/markw/lvm2.log.gz

Thanks,
Mark

On 15 Aug, Alasdair G Kergon wrote:
> On Fri, Aug 15, 2003 at 01:11:51PM -0700, markw@osdl.org wrote:
>> # lvcreate -i52 -I4 -l450996 -n mydata data_volume
>>   device-mapper ioctl cmd 9 failed: Invalid argument
>>   Couldn't load device 'data_volume-mydata'.
>>   Failed to activate new LV.
> 
> Mark, please would you enable the debug messages
> as described in the docs,
> viz. the following settings in lvm.conf:
> 
>   log {
>           file="/tmp/lvm2.log"
> 	  level=7
> 	  activation=1
>   }
> 
> then look at /tmp/lvm2.log after running the
> problematic command?
>  
>> device-mapper: couldn't split parameters
>> device-mapper: error adding target to table
> 
> The log file will show you the actual parameters 
> that were passed to device-mapper leading to
> the error messages.
>  
> Please put the log file(s) somewhere on the web and 
> post URL(s) to the list - or mail the log file(s)
> to me privately as a *compressed* attachment.
> 
> Alasdair

Re: [linux-lvm] lvcreate with a 52 drive volume

[Alasdair G Kergon]

On Fri, Aug 15, 2003 at 04:04:37PM -0700, markw@osdl.org wrote:
> http://developer.osdl.org/markw/lvm2.log.gz
Thanks.
 
The line of interest is:

  0 3694559232 striped 52 8 /dev/rd/c0d0 384 /dev/rd/c0d1 384 /dev/rd/c0d2
  384 /dev/rd/c0d3 384 /dev/rd/c0d4 384 /dev/rd/c0d5 384 /dev/rd/c0d6 384
  /dev/rd/c0d7 384 /dev/rd/c0d8 384 /dev/rd/c0d9 384 /dev/rd/c0d10 384
  /dev/rd/c0d11 384 /dev/rd/c0d12 384 /dev/rd/c0d14 384 /dev/rd/c0d15 384
  /dev/rd/c0d16 384 /dev/rd/c0d17 384 /dev/rd/c0d18 384 /dev/rd/c0d19 384
  /dev/rd/c0d20 384 /dev/rd/c0d21 384 /dev/rd/c0d22 384 /dev/rd/c0d23 384
  /dev/rd/c0d24 384 /dev/rd/c0d25 384 /dev/rd/c0d26 384 /dev/rd/c1d0 384
  /dev/rd/c1d1 384 /dev/rd/c1d2 384 /dev/rd/c1d3 384 /dev/rd/c1d4 384
  /dev/rd/c1d5 384 /dev/rd/c1d6 384 /dev/rd/c1d7 384 /dev/rd/c1d8 384
  /dev/rd/c1d9 384 /dev/rd/c1d10 384 /dev/rd/c1d11 384 /dev/rd/c1d12 384
  /dev/rd/c1d14 384 /dev/rd/c1d15 384 /dev/rd/c1d16 384 /dev/rd/c1d17 384
  /dev/rd/c1d18 384 /dev/rd/c1d19 384 /dev/rd/c1d20 384 /dev/rd/c1d21 384
  /dev/rd/c1d22 384 /dev/rd/c1d23 384 /dev/rd/c1d24 384 /dev/rd/c1d25 384
  /dev/rd/c1d26 384

The error you got was:
  device-mapper: couldn't split parameters

Look at dm-table.c:498,
  char *argv[32];
which suggests a hard-coded limit of 32 [tokens after "striped"?]
slipped into the code:-(

Try recompiling with a larger number there.

Alasdair
-- 
agk@uk.sistina.com

Re: [linux-lvm] lvcreate with a 52 drive volume

[markw]

On 16 Aug, Alasdair G Kergon wrote:
> Look at dm-table.c:498,
>   char *argv[32];
> which suggests a hard-coded limit of 32 [tokens after "striped"?]
> slipped into the code:-(
> 
> Try recompiling with a larger number there.

Hmmm, I changed the number, but when I do a 'make clean' and a 'make',
I don't see dm-table.c compiled anywhere in the device-mapper tree or
the LVM2 tools.  Am I neglecting something obvious?

Thanks,
Mark

Re: [linux-lvm] lvcreate with a 52 drive volume

[Jan Niehusmann]

On Mon, Aug 18, 2003 at 10:32:42AM -0700, markw@osdl.org wrote:
> Hmmm, I changed the number, but when I do a 'make clean' and a 'make',
> I don't see dm-table.c compiled anywhere in the device-mapper tree or
> the LVM2 tools.  Am I neglecting something obvious?

It should get compiled into dm-mod.o, if you compile dm as a module.

Jan

Re: [linux-lvm] lvcreate with a 52 drive volume

[Alasdair G Kergon]

On Mon, Aug 18, 2003 at 10:32:42AM -0700, markw@osdl.org wrote:
> Hmmm, I changed the number, but when I do a 'make clean' and a 'make',
> I don't see dm-table.c compiled anywhere in the device-mapper tree or
> the LVM2 tools.  Am I neglecting something obvious?

It's hard-coded into the kernel I'm afraid, not the tools:

Are you editing <your_kernel_source_dir>/drivers/md/dm-table.c then
rebuilding the kernel (or module, if it's a module)?

Alasdair
-- 
agk@uk.sistina.com

Re: [linux-lvm] lvcreate with a 52 drive volume

[markw]

On 18 Aug, Alasdair G Kergon wrote:
> On Mon, Aug 18, 2003 at 10:32:42AM -0700, markw@osdl.org wrote:
>> Hmmm, I changed the number, but when I do a 'make clean' and a 'make',
>> I don't see dm-table.c compiled anywhere in the device-mapper tree or
>> the LVM2 tools.  Am I neglecting something obvious?
> 
> It's hard-coded into the kernel I'm afraid, not the tools:
> 
> Are you editing <your_kernel_source_dir>/drivers/md/dm-table.c then
> rebuilding the kernel (or module, if it's a module)?

Aha, no I'm not.  I will try that now.  :)

Thanks,
Mark

Re: [linux-lvm] lvcreate with a 52 drive volume

[markw]

Ok, I increased the size of argv to 2048 in the dm-table.c file that's
part of the linux-2.6.0-test3 tree, and then I got the following kernel
oops when I ran lvcreate:

Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:                                                              
02142cf9      
*pde = 00000000
Oops: 0000 [#1]
SMP            
CPU:    2048
EIP:    0060:[<02142cf9>]    Not tainted VLI
EFLAGS: 00010046                            
EIP is at __kmalloc+0x4d/0x7a
eax: 00000800   ebx: d5e06000   ecx: d9eef520   edx: 00000000
esi: 00000246   edi: 000000d0   ebp: 00000000   esp: d5e07ea0
ds: 007b   es: 007b   ss: 0068                               
Process  (pid: 0, threadinfo=d5e06000 task=5b782032)
Stack: 65746365 d5e07f08 d5e07f08 da93c080 00000008 022b0290 000001b0 000000d0 
       0000000a 3a363164 043c2000 00000007 da93714c 00000034 da937160 da93c080 
       d5c2d280 da93c0ac d5e07f08 022af594 da93c080 0000006a d5e07f08 da93715c 
Call Trace:                                                                    
 [<022b0290>] stripe_ctr+0x1da/0x1e2
 [<022af594>] dm_table_add_target+0x156/0x1ac
                                             
Code: e3 39 c1 76 12 83 c2 0c 8b 02 85 c0 75 f3 31 c0 83 c4 08 5b 5e 5f c3 f7 c7 01 00 00 00 74 34 8b 4a 08 9c 5e fa 8b 43 10 8b 14 81 <8b> 02 85 c0 74 14 83 e8 01 c7 42 0c 01 00 00 00 89 02 8b 44 82 
Unable to handle kernel NULL pointer dereference at virtual address 00000000
 printing eip:                                                              
0212a6a9      
*pde = 00000000

Re: [linux-lvm] lvcreate with a 52 drive volume

[Alasdair G Kergon]

On Mon, Aug 18, 2003 at 02:09:18PM -0700, markw@osdl.org wrote:
> Ok, I increased the size of argv to 2048 in the dm-table.c file that's

2048 is probably too big for the kernel stack.

Set it only as big as you need it.

We're working on a proper fix.

Alasdair
-- 
agk@uk.sistina.com

Re: [linux-lvm] lvcreate with a 52 drive volume

[markw]

On 18 Aug, Alasdair G Kergon wrote:
> On Mon, Aug 18, 2003 at 02:09:18PM -0700, markw@osdl.org wrote:
>> Ok, I increased the size of argv to 2048 in the dm-table.c file that's
> 
> 2048 is probably too big for the kernel stack.
> 
> Set it only as big as you need it.
> 
> We're working on a proper fix.
> 
> Alasdair

Yeah, I got greedy, 1024 worked for me. :)

Thanks again,
Mark

Re: [linux-lvm] lvcreate with a 52 drive volume

[Alasdair G Kergon]

[-- Attachment #1: Type: text/plain, Size: 567 bytes --]

On Mon, Aug 18, 2003 at 05:00:12PM -0700, markw@osdl.org wrote:
> On 18 Aug, Alasdair G Kergon wrote:
> > On Mon, Aug 18, 2003 at 02:09:18PM -0700, markw@osdl.org wrote:
> >> Ok, I increased the size of argv to 2048 in the dm-table.c file that's
> > 2048 is probably too big for the kernel stack.
> > Set it only as big as you need it.
> Yeah, I got greedy, 1024 worked for me. :)
 
Here's a patch to try that removes the hard-coded limit of 32.

(Plus one to fix the error message if you give a linear
target more than 2 arguments.)

Alasdair
-- 
agk@uk.sistina.com

[-- Attachment #2: dm_argv.patch --]
[-- Type: text/plain, Size: 1910 bytes --]

Remove hard-coded limit of 32 arguments per target.
--- linux-2.4.21/drivers/md/dm-table.c	Tue Jul  1 20:17:31 2003
+++ linux/drivers/md/dm-table.c	Tue Aug 19 15:43:50 2003
@@ -17,6 +17,7 @@
 #define NODE_SIZE L1_CACHE_BYTES
 #define KEYS_PER_NODE (NODE_SIZE / sizeof(sector_t))
 #define CHILDREN_PER_NODE (KEYS_PER_NODE + 1)
+#define MAX_TARGET_ARGS 64
 
 struct dm_table {
 	atomic_t holders;
@@ -443,10 +444,16 @@
 /*
  * Destructively splits up the argument list to pass to ctr.
  */
-static int split_args(int max, int *argc, char **argv, char *input)
+static int split_args(int *argc, char ***argvp, char *input)
 {
 	char *start, *end = input, *out;
+	char **argv;
+	int max_args = MAX_TARGET_ARGS;
+
 	*argc = 0;
+	argv = kmalloc(sizeof(*argv) * max_args, GFP_NOIO);
+	if (!argv)
+		return -ENOMEM;
 
 	while (1) {
 		start = end;
@@ -475,8 +482,20 @@
 		}
 
 		/* have we already filled the array ? */
-		if ((*argc + 1) > max)
-			return -EINVAL;
+		if ((*argc + 1) > max_args) {
+			char **argv2;
+			
+			max_args *= 2;
+			argv2 = kmalloc(sizeof(*argv2) * max_args, GFP_NOIO);
+			if (!argv2) {
+				kfree(argv);
+				return -ENOMEM;
+			}
+
+			memcpy(argv2, argv, sizeof(*argv) * *argc);
+			kfree(argv);
+			argv = argv2;
+		}
 
 		/* we know this is whitespace */
 		if (*end)
@@ -488,6 +507,7 @@
 		(*argc)++;
 	}
 
+	*argvp = argv;
 	return 0;
 }
 
@@ -495,7 +515,7 @@
 			sector_t start, sector_t len, char *params)
 {
 	int r = -EINVAL, argc;
-	char *argv[32];
+	char **argv;
 	struct dm_target *tgt;
 
 	if ((r = check_space(t)))
@@ -524,13 +544,14 @@
 		goto bad;
 	}
 
-	r = split_args(ARRAY_SIZE(argv), &argc, argv, params);
+	r = split_args(&argc, &argv, params);
 	if (r) {
-		tgt->error = "couldn't split parameters";
+		tgt->error = "couldn't split parameters (insufficient memory)";
 		goto bad;
 	}
 
 	r = tgt->type->ctr(tgt, argc, argv);
+	kfree(argv);
 	if (r)
 		goto bad;
 

[-- Attachment #3: dm_linear_errmsg.patch --]
[-- Type: text/plain, Size: 385 bytes --]

Fix error message when linear targets gets handed more than 2 arguments.
--- linux-2.4.21/drivers/md/dm-linear.c	Fri Jul  4 18:56:24 2003
+++ linux/drivers/md/dm-linear.c	Tue Aug 19 16:09:38 2003
@@ -27,7 +27,7 @@
 	struct linear_c *lc;
 
 	if (argc != 2) {
-		ti->error = "dm-linear: Not enough arguments";
+		ti->error = "dm-linear: Invalid argument count";
 		return -EINVAL;
 	}
 

Re: [linux-lvm] lvcreate with a 52 drive volume

[markw]

On 19 Aug, Alasdair G Kergon wrote:
> On Mon, Aug 18, 2003 at 05:00:12PM -0700, markw@osdl.org wrote:
>> On 18 Aug, Alasdair G Kergon wrote:
>> > On Mon, Aug 18, 2003 at 02:09:18PM -0700, markw@osdl.org wrote:
>> >> Ok, I increased the size of argv to 2048 in the dm-table.c file that's
>> > 2048 is probably too big for the kernel stack.
>> > Set it only as big as you need it.
>> Yeah, I got greedy, 1024 worked for me. :)
>  
> Here's a patch to try that removes the hard-coded limit of 32.
> 
> (Plus one to fix the error message if you give a linear
> target more than 2 arguments.)

I've applied it to 2.6.0-test3-mm3 and it works.

Thanks,
Mark