LVM on md0: raid0_make_request bug: can't convert block across chunks or bigger than 64k

LVM on md0: raid0_make_request bug: can't convert block across chunks or bigger than 64k

[Karl Vogel]

I'm getting the following kernel messages on V2.6.0-test8-mm1 (I've also
tried plain -test7 and some kernels before that) when copying moderately
sized files from a raid-0/LVM volume:

--- snip ---
raid0_make_request bug: can't convert block across chunks or bigger than
64k 24081064 64
raid0_make_request bug: can't convert block across chunks or bigger than
64k 24080656 64
raid0_make_request bug: can't convert block across chunks or bigger than
64k 24080784 64
raid0_make_request bug: can't convert block across chunks or bigger than
64k 24080928 64
--- snip ---

The data was created on a V2.4 kernel. The partition contains 1 LVM
volume, which is using a RAID-0 md0 as the physical volume:

# ./lvm lvdisplay
  --- Logical volume ---
  LV Name                /dev/system/root
  VG Name                system
  LV UUID                000000-0000-0000-0000-0000-0000-000000
  LV Write Access        read/write
  LV Status              available
  # open                 2
  LV Size                11.58 GB
  Current LE             2964
  Segments               1
  Allocation             next free
  Read ahead sectors     10000
  Block device           254:0

# ./lvm pvdisplay
  --- Physical volume ---
  PV Name               /dev/md0
  VG Name               system
  PV Size               11.58 GB / not usable 4.31 MB
  Allocatable           yes (but full)
  PE Size (KByte)       4096
  Total PE              2964
  Free PE               0
  Allocated PE          2964
  PV UUID               uYHQYU-TmrF-I3Ws-674P-v3Rx-4CPz-mOe5Tu

# ./lvm vgdisplay
  --- Volume group ---
  VG Name               system
  System ID             kvo.local.org1047770061
  Format                lvm1
  VG Access             read/write
  VG Status             resizable
  MAX LV                256
  Cur LV                1
  Open LV               0
  Max PV                256
  Cur PV                1
  Act PV                1
  VG Size               11.58 GB
  PE Size               4.00 MB
  Total PE              2964
  Alloc PE / Size       2964 / 11.58 GB
  Free  PE / Size       0 / 0
  VG UUID               AE3SH5-7pbd-itL5-firC-Sxis-0IXE-qMa5dD

The RAID-0 is setup as:

# cat /etc/raidtab
raiddev /dev/md0
        raid-level              0
        nr-raid-disks           2
        persistent-superblock   0
        chunk-size              8

        device                  /dev/hdd1
        raid-disk               0
        device                  /dev/hdc2
        raid-disk               1

The boot message of the md0 initializing looks like:

Oct 20 20:44:55 kvo kernel: md: Autodetecting RAID arrays.
Oct 20 20:44:55 kvo kernel: md: autorun ...
Oct 20 20:44:55 kvo kernel: md: considering hdd1 ...
Oct 20 20:44:55 kvo kernel: md:  adding hdd1 ...
Oct 20 20:44:55 kvo kernel: md:  adding hdc2 ...
Oct 20 20:44:55 kvo kernel: md: created md0
Oct 20 20:44:55 kvo kernel: md: bind<hdc2>
Oct 20 20:44:55 kvo kernel: md: bind<hdd1>
Oct 20 20:44:55 kvo kernel: md: running: <hdd1><hdc2>
Oct 20 20:44:55 kvo kernel: md0: setting max_sectors to 128, segment
boundary to 32767
Oct 20 20:44:55 kvo kernel: raid0: looking at hdd1
Oct 20 20:44:55 kvo kernel: raid0:   comparing hdd1(6345600) with
hdd1(6345600)
Oct 20 20:44:55 kvo kernel: raid0:   END
Oct 20 20:44:55 kvo syslog: klogd startup succeeded
Oct 20 20:44:55 kvo kernel: raid0:   ==> UNIQUE
Oct 20 20:44:55 kvo kernel: raid0: 1 zones
Oct 20 20:44:55 kvo kernel: raid0: looking at hdc2
Oct 20 20:44:55 kvo kernel: raid0:   comparing hdc2(5799360) with
hdd1(6345600)
Oct 20 20:44:55 kvo kernel: raid0:   NOT EQUAL
Oct 20 20:44:55 kvo kernel: raid0:   comparing hdc2(5799360) with
hdc2(5799360)
Oct 20 20:44:55 kvo kernel: raid0:   END
Oct 20 20:44:55 kvo kernel: raid0:   ==> UNIQUE
Oct 20 20:44:55 kvo kernel: raid0: 2 zones
Oct 20 20:44:55 kvo kernel: raid0: FINAL 2 zones
Oct 20 20:44:55 kvo kernel: raid0: zone 1
Oct 20 20:44:55 kvo kernel: raid0: checking hdc2 ... nope.
Oct 20 20:44:55 kvo kernel: raid0: checking hdd1 ... contained as device
0
Oct 20 20:44:55 kvo kernel:   (6345600) is smallest!.
Oct 20 20:44:55 kvo kernel: raid0: zone->nb_dev: 1, size: 546240
Oct 20 20:44:55 kvo kernel: raid0: current zone offset: 6345600
Oct 20 20:44:55 kvo kernel: raid0: done.
Oct 20 20:44:55 kvo kernel: raid0 : md_size is 12144960 blocks.
Oct 20 20:44:55 kvo kernel: raid0 : conf->hash_spacing is 11598720
blocks.
Oct 20 20:44:55 kvo kernel: raid0 : nb_zone is 2.
Oct 20 20:44:55 kvo kernel: raid0 : Allocating 8 bytes for hash.
Oct 20 20:44:55 kvo kernel: md: ... autorun DONE.

I'm using LVM2.1.95.15 with the V1 interface (am getting errors when
compiling the latest LVM2 tools, but that's a different story :)


Please use reply-all if you require more details.

Regards,
Karl

Re: LVM on md0: raid0_make_request bug: can't convert block across chunks or bigger than 64k

[Kevin Corry]

On Monday 20 October 2003 15:35, Karl Vogel wrote:
> I'm getting the following kernel messages on V2.6.0-test8-mm1 (I've also
> tried plain -test7 and some kernels before that) when copying moderately
> sized files from a raid-0/LVM volume:
>
> --- snip ---
> raid0_make_request bug: can't convert block across chunks or bigger than
> 64k 24081064 64
> raid0_make_request bug: can't convert block across chunks or bigger than
> 64k 24080656 64
> raid0_make_request bug: can't convert block across chunks or bigger than
> 64k 24080784 64
> raid0_make_request bug: can't convert block across chunks or bigger than
> 64k 24080928 64

Looks like this was just recently fixed on the linux-raid list.

http://marc.theaimsgroup.com/?l=linux-raid&m=106661294929434

-- 
Kevin Corry
kevcorry@us.ibm.com
http://evms.sourceforge.net/

Re: LVM on md0: raid0_make_request bug: can't convert block across chunks or bigger than 64k

[Karl Vogel]

On Mon, 2003-10-20 at 23:06, Kevin Corry wrote:
> On Monday 20 October 2003 15:35, Karl Vogel wrote:
> > I'm getting the following kernel messages on V2.6.0-test8-mm1 (I've also
> > tried plain -test7 and some kernels before that) when copying moderately
> > sized files from a raid-0/LVM volume:
> >
> > --- snip ---
> > raid0_make_request bug: can't convert block across chunks or bigger than
> > 64k 24081064 64
> > raid0_make_request bug: can't convert block across chunks or bigger than
> > 64k 24080656 64
> > raid0_make_request bug: can't convert block across chunks or bigger than
> > 64k 24080784 64
> > raid0_make_request bug: can't convert block across chunks or bigger than
> > 64k 24080928 64
> 
> Looks like this was just recently fixed on the linux-raid list.
> 
> http://marc.theaimsgroup.com/?l=linux-raid&m=106661294929434

Applied the patch on 2.6.0-test8-mm1 but it made no difference.

Somebody else referred to this posting:
  http://marc.theaimsgroup.com/?l=linux-kernel&m=103369952814053&w=2

but that patch doesn't apply cleanly anymore and I'm not familiar with
the code to be confident to fix it up myself. (that post was from almost
exactly 1 year ago, so alot changed probably :)

Re: LVM on md0: raid0_make_request bug: can't convert block across chunks or bigger than 64k

[Neil Brown]

On Monday October 20, karl.vogel@seagha.com wrote:
> On Mon, 2003-10-20 at 23:06, Kevin Corry wrote:
> > On Monday 20 October 2003 15:35, Karl Vogel wrote:
> > > I'm getting the following kernel messages on V2.6.0-test8-mm1 (I've also
> > > tried plain -test7 and some kernels before that) when copying moderately
> > > sized files from a raid-0/LVM volume:
> > >
> > > --- snip ---
> > > raid0_make_request bug: can't convert block across chunks or bigger than
> > > 64k 24081064 64
> > > raid0_make_request bug: can't convert block across chunks or bigger than
> > > 64k 24080656 64
> > > raid0_make_request bug: can't convert block across chunks or bigger than
> > > 64k 24080784 64
> > > raid0_make_request bug: can't convert block across chunks or bigger than
> > > 64k 24080928 64
> > 
> > Looks like this was just recently fixed on the linux-raid list.
> > 
> > http://marc.theaimsgroup.com/?l=linux-raid&m=106661294929434
> 
> Applied the patch on 2.6.0-test8-mm1 but it made no difference.

no, thats a completely different problem.

> 
> Somebody else referred to this posting:
>   http://marc.theaimsgroup.com/?l=linux-kernel&m=103369952814053&w=2
> 
> but that patch doesn't apply cleanly anymore and I'm not familiar with
> the code to be confident to fix it up myself. (that post was from almost
> exactly 1 year ago, so alot changed probably :)

That patch is already included.

The problem is that dm is not honouring the merge_bvec_fn that
raid0 has set.

This patch might fix it, but I'm not very familiar with the dm code,
so I make no promises.

(I wonder why you are running LVM on top of raid0 given that lvm
contains raid0 functionality).

NeilBrown



 ----------- Diffstat output ------------
 ./drivers/md/dm-table.c |    5 +++++
 1 files changed, 5 insertions(+)

diff ./drivers/md/dm-table.c~current~ ./drivers/md/dm-table.c
--- ./drivers/md/dm-table.c~current~	2003-10-21 10:05:29.000000000 +1000
+++ ./drivers/md/dm-table.c	2003-10-21 10:06:27.000000000 +1000
@@ -489,6 +489,11 @@ int dm_get_device(struct dm_target *ti, 
 		rs->max_sectors =
 			min_not_zero(rs->max_sectors, q->max_sectors);
 
+		if (q->merge_bvec_fn)
+			rs->max_sectors =
+				min_not_zero(rs->max_sectors, PAGE_SIZE>>9);
+			
+
 		rs->max_phys_segments =
 			min_not_zero(rs->max_phys_segments,
 				     q->max_phys_segments);

Re: LVM on md0: raid0_make_request bug: can't convert block across chunks or bigger than 64k

[Kevin Corry]

On Monday 20 October 2003 19:09, Neil Brown wrote:
> On Monday October 20, karl.vogel@seagha.com wrote:
> > On Mon, 2003-10-20 at 23:06, Kevin Corry wrote:
> > > On Monday 20 October 2003 15:35, Karl Vogel wrote:
> > > > I'm getting the following kernel messages on V2.6.0-test8-mm1 (I've
> > > > also tried plain -test7 and some kernels before that) when copying
> > > > moderately sized files from a raid-0/LVM volume:
> > > >
> > > > --- snip ---
> > > > raid0_make_request bug: can't convert block across chunks or bigger
> > > > than 64k 24081064 64
> > > > raid0_make_request bug: can't convert block across chunks or bigger
> > > > than 64k 24080656 64
> > > > raid0_make_request bug: can't convert block across chunks or bigger
> > > > than 64k 24080784 64
> > > > raid0_make_request bug: can't convert block across chunks or bigger
> > > > than 64k 24080928 64
> > >
> > > Looks like this was just recently fixed on the linux-raid list.
> > >
> > > http://marc.theaimsgroup.com/?l=linux-raid&m=106661294929434
> >
> > Applied the patch on 2.6.0-test8-mm1 but it made no difference.
>
> no, thats a completely different problem.

Sorry for the confusion. Karl's error messages looked just like ones that were 
reported on the evms-devel list, which were supposedly fixed by the above 
patch.

> The problem is that dm is not honouring the merge_bvec_fn that
> raid0 has set.
>
> This patch might fix it, but I'm not very familiar with the dm code,
> so I make no promises.
>
>  ----------- Diffstat output ------------
>  ./drivers/md/dm-table.c |    5 +++++
>  1 files changed, 5 insertions(+)
>
> diff ./drivers/md/dm-table.c~current~ ./drivers/md/dm-table.c
> --- ./drivers/md/dm-table.c~current~	2003-10-21 10:05:29.000000000 +1000
> +++ ./drivers/md/dm-table.c	2003-10-21 10:06:27.000000000 +1000
> @@ -489,6 +489,11 @@ int dm_get_device(struct dm_target *ti,
>  		rs->max_sectors =
>  			min_not_zero(rs->max_sectors, q->max_sectors);
>
> +		if (q->merge_bvec_fn)
> +			rs->max_sectors =
> +				min_not_zero(rs->max_sectors, PAGE_SIZE>>9);
> +
> +
>  		rs->max_phys_segments =
>  			min_not_zero(rs->max_phys_segments,
>  				     q->max_phys_segments);

This will probably work, as long as raid0 can split a one-page request that 
spans a chunk boundary. I'll be interested to see if this solves Karl's 
problem.

Joe Thornber was telling me about an idea he had to solve the 
Device-Mapper-on-top-of-MD problem using the dm-io code in his latest 
Device-Mapper patchset 
(http://people.sistina.com/~thornber/patches/2.6/2.6.0-test6/2.6.0-t6-mm1-dm2.tar.bz2). 
I believe he's away for this week, but I'll ask him about it again the next 
time I talk to him.

-- 
Kevin Corry
kevcorry@us.ibm.com
http://evms.sourceforge.net/

Re: LVM on md0: raid0_make_request bug: can't convert block across chunks or bigger than 64k

[Karl Vogel]

On Tue, 2003-10-21 at 15:41, Kevin Corry wrote:
> On Monday 20 October 2003 19:09, Neil Brown wrote:
> > On Monday October 20, karl.vogel@seagha.com wrote:
> > > On Mon, 2003-10-20 at 23:06, Kevin Corry wrote:
> > > > On Monday 20 October 2003 15:35, Karl Vogel wrote:
> > > > > I'm getting the following kernel messages on V2.6.0-test8-mm1 (I've
> > > > > also tried plain -test7 and some kernels before that) when copying
> > > > > moderately sized files from a raid-0/LVM volume:
> > > > >
> > > > > --- snip ---
> > > > > raid0_make_request bug: can't convert block across chunks or bigger
> > > > > than 64k 24081064 64
> > > > > raid0_make_request bug: can't convert block across chunks or bigger
> > > > > than 64k 24080656 64
> > > > > raid0_make_request bug: can't convert block across chunks or bigger
> > > > > than 64k 24080784 64
> > > > > raid0_make_request bug: can't convert block across chunks or bigger
> > > > > than 64k 24080928 64
> > > >
> > > > Looks like this was just recently fixed on the linux-raid list.
> > > >
> > > > http://marc.theaimsgroup.com/?l=linux-raid&m=106661294929434
> > >
> > > Applied the patch on 2.6.0-test8-mm1 but it made no difference.
> >
> > no, thats a completely different problem.
> 
> Sorry for the confusion. Karl's error messages looked just like ones that were 
> reported on the evms-devel list, which were supposedly fixed by the above 
> patch.
> 
> > The problem is that dm is not honouring the merge_bvec_fn that
> > raid0 has set.
> >
> > This patch might fix it, but I'm not very familiar with the dm code,
> > so I make no promises.
> >
> >  ----------- Diffstat output ------------
> >  ./drivers/md/dm-table.c |    5 +++++
> >  1 files changed, 5 insertions(+)
> >
> > diff ./drivers/md/dm-table.c~current~ ./drivers/md/dm-table.c
> > --- ./drivers/md/dm-table.c~current~	2003-10-21 10:05:29.000000000 +1000
> > +++ ./drivers/md/dm-table.c	2003-10-21 10:06:27.000000000 +1000
> > @@ -489,6 +489,11 @@ int dm_get_device(struct dm_target *ti,
> >  		rs->max_sectors =
> >  			min_not_zero(rs->max_sectors, q->max_sectors);
> >
> > +		if (q->merge_bvec_fn)
> > +			rs->max_sectors =
> > +				min_not_zero(rs->max_sectors, PAGE_SIZE>>9);
> > +
> > +
> >  		rs->max_phys_segments =
> >  			min_not_zero(rs->max_phys_segments,
> >  				     q->max_phys_segments);
> 
> This will probably work, as long as raid0 can split a one-page request that 
> spans a chunk boundary. I'll be interested to see if this solves Karl's 
> problem.

Good news... it solves the problem with my setup. I was able to copy
files off the logical volume (did an md5sum compare to make sure I got
the complete files.)


> Joe Thornber was telling me about an idea he had to solve the 
> Device-Mapper-on-top-of-MD problem using the dm-io code in his latest 
> Device-Mapper patchset 
> (http://people.sistina.com/~thornber/patches/2.6/2.6.0-test6/2.6.0-t6-mm1-dm2.tar.bz2). 
> I believe he's away for this week, but I'll ask him about it again the next 
> time I talk to him.

Re: LVM on md0: raid0_make_request bug: can't convert block across chunks or bigger than 64k

[Karl Vogel]

> > >  ----------- Diffstat output ------------
> > >  ./drivers/md/dm-table.c |    5 +++++
> > >  1 files changed, 5 insertions(+)
> > >
> > > diff ./drivers/md/dm-table.c~current~ ./drivers/md/dm-table.c
> > > --- ./drivers/md/dm-table.c~current~	2003-10-21 10:05:29.000000000 +1000
> > > +++ ./drivers/md/dm-table.c	2003-10-21 10:06:27.000000000 +1000
> > > @@ -489,6 +489,11 @@ int dm_get_device(struct dm_target *ti,
> > >  		rs->max_sectors =
> > >  			min_not_zero(rs->max_sectors, q->max_sectors);
> > >
> > > +		if (q->merge_bvec_fn)
> > > +			rs->max_sectors =
> > > +				min_not_zero(rs->max_sectors, PAGE_SIZE>>9);
> > > +
> > > +
> > >  		rs->max_phys_segments =
> > >  			min_not_zero(rs->max_phys_segments,
> > >  				     q->max_phys_segments);
> > 
> > This will probably work, as long as raid0 can split a one-page request that 
> > spans a chunk boundary. I'll be interested to see if this solves Karl's 
> > problem.
> 
> Good news... it solves the problem with my setup. I was able to copy
> files off the logical volume (did an md5sum compare to make sure I got
> the complete files.)

Forgot to mention that it generates a compiler warning:

  CC      drivers/md/dm-table.o
drivers/md/dm-table.c: In function `dm_get_device':
drivers/md/dm-table.c:494: warning: comparison of distinct pointer types
lacks a cast

Using:
$ gcc -v
Reading specs from /usr/lib/gcc-lib/i386-redhat-linux/3.3.1/specs
Configured with: ../configure --prefix=/usr --mandir=/usr/share/man
--infodir=/usr/share/info --enable-shared --enable-threads=posix
--disable-checking --with-system-zlib --enable-__cxa_atexit
--host=i386-redhat-linux
Thread model: posix
gcc version 3.3.1 20030930 (Red Hat Linux 3.3.1-6)

Re: LVM on md0: raid0_make_request bug: can't convert block acros s chunks or bigger than 64k

[Karl Vogel]

On Tue, 2003-10-21 at 02:09, Neil Brown wrote:

> The problem is that dm is not honouring the merge_bvec_fn that
> raid0 has set.
> 
> This patch might fix it, but I'm not very familiar with the dm code,
> so I make no promises.
> 
> (I wonder why you are running LVM on top of raid0 given that lvm
> contains raid0 functionality).
> 
> NeilBrown
> 
> 
> 
>  ----------- Diffstat output ------------
>  ./drivers/md/dm-table.c |    5 +++++
>  1 files changed, 5 insertions(+)
> 
> diff ./drivers/md/dm-table.c~current~ ./drivers/md/dm-table.c
> --- ./drivers/md/dm-table.c~current~	2003-10-21 10:05:29.000000000
> +1000
> +++ ./drivers/md/dm-table.c	2003-10-21 10:06:27.000000000 +1000
> @@ -489,6 +489,11 @@ int dm_get_device(struct dm_target *ti, 
>  		rs->max_sectors =
>  			min_not_zero(rs->max_sectors, q->max_sectors);
>  
> +		if (q->merge_bvec_fn)
> +			rs->max_sectors =
> +				min_not_zero(rs->max_sectors,
> PAGE_SIZE>>9);
> +			
> +
>  		rs->max_phys_segments =
>  			min_not_zero(rs->max_phys_segments,
>  				     q->max_phys_segments);


I noticed in the 2.6.0-test9 notes the following:

---
Neil Brown:
  o md -  Use sector rather than block numbers when splitting raid0
    requests
---

I'm not sure if this is related to the problem I was experiencing?!
Anyway this doesn't fix the problem I was having. I still get the errors
with -test9. Above patch to dm-table.c works for me.

Just thought I'd mention this..

Re: LVM on md0: raid0_make_request bug: can't convert block acros s chunks or bigger than 64k

[Neil Brown]

On Saturday October 25, karl.vogel@seagha.com wrote:
> 
> 
> I noticed in the 2.6.0-test9 notes the following:
> 
> ---
> Neil Brown:
>   o md -  Use sector rather than block numbers when splitting raid0
>     requests
> ---
> 
> I'm not sure if this is related to the problem I was experiencing?!
> Anyway this doesn't fix the problem I was having. I still get the errors
> with -test9. Above patch to dm-table.c works for me.
> 
> Just thought I'd mention this..
> 
This addresses a different problem: raid0 would not correctly handle
some requests that are not 1K aligned.  XFS does some non-1K-aligned
requests where reading it's journal and caught this bug.
It has nothing to do with LVM or your problem.

NeilBrown