Re: [Evms-devel] 2.6.10-ac10 + 2.6.10udm1 + latest evms => not nice

Re: [Evms-devel] 2.6.10-ac10 + 2.6.10udm1 + latest evms => not nice

[Lars Marowsky-Bree]

[-- Attachment #1: Type: text/plain, Size: 641 bytes --]

On 2005-01-21T15:13:26, syrius.ml@no-log.org wrote:

> Don't know if it's related to evms or to dm.
> kernel is a 2.6.10-ac10 + 2.6.10-udm1 + evms kernel patches.
> (bd_claims + dm bbr + dm multipath)
> 
> Here is the nice welcome message I've just got :)

This looks like md on DM breakage, which Jens has just fixed in our
kernel. I'm not sure whether he has submitted it upstream yet, but you
can give it a try.

Does the attached patch apply & fix the problem for you?


Sincerely,
    Lars Marowsky-Brée <lmb@suse.de>

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business


[-- Attachment #2: md-bio-on-stack --]
[-- Type: text/plain, Size: 1518 bytes --]

From: Jens Axboe <axboe@suse.de>
Subject: Fix md using bio on stack with bio clones
Patch-mainline: 
References: 49931

If md resides on top of a driver using bio_clone() (such as dm), it will
oops the kernel due to md submitting a botched bio that has a veclist but
doesn't have bio->bi_max_vecs set.

Acked-by: 
Signed-off-by: Jens Axboe <axboe@suse.de>

===== drivers/md/md.c 1.231 vs edited =====
--- 1.231/drivers/md/md.c	2004-12-01 09:13:51 +01:00
+++ edited/drivers/md/md.c	2005-01-19 13:23:30 +01:00
@@ -332,29 +332,26 @@
 static int sync_page_io(struct block_device *bdev, sector_t sector, int size,
 		   struct page *page, int rw)
 {
-	struct bio bio;
-	struct bio_vec vec;
+	struct bio *bio = bio_alloc(GFP_KERNEL, 1);
 	struct completion event;
+	int ret;
+
+	bio_get(bio);
 
 	rw |= (1 << BIO_RW_SYNC);
 
-	bio_init(&bio);
-	bio.bi_io_vec = &vec;
-	vec.bv_page = page;
-	vec.bv_len = size;
-	vec.bv_offset = 0;
-	bio.bi_vcnt = 1;
-	bio.bi_idx = 0;
-	bio.bi_size = size;
-	bio.bi_bdev = bdev;
-	bio.bi_sector = sector;
+	bio->bi_bdev = bdev;
+	bio->bi_sector = sector;
+	bio_add_page(bio, page, size, 0);
 	init_completion(&event);
-	bio.bi_private = &event;
-	bio.bi_end_io = bi_complete;
-	submit_bio(rw, &bio);
+	bio->bi_private = &event;
+	bio->bi_end_io = bi_complete;
+	submit_bio(rw, bio);
 	wait_for_completion(&event);
 
-	return test_bit(BIO_UPTODATE, &bio.bi_flags);
+	ret = test_bit(BIO_UPTODATE, &bio->bi_flags);
+	bio_put(bio);
+	return ret;
 }
 
 static int read_disk_sb(mdk_rdev_t * rdev)

Re: [Evms-devel] 2.6.10-ac10 + 2.6.10udm1 + latest evms => not nice

[syrius.ml]

Lars Marowsky-Bree <lmb@suse.de> writes:

> [...]
> Does the attached patch apply & fix the problem for you?

unfortunatly it did not.

here is the new oops (not sure it will help more)

device-mapper: 4.3.0-ioctl (2004-09-30) initialised: dm-devel@redhat.com
md: md driver 0.90.1 MAX_MD_DEVS=256, MD_SB_DISKS=27
md: bind<dm-0>
md: bind<dm-1>
md: bind<dm-2>
md: bind<dm-3>
md: bind<dm-4>
raid5: automatically using best checksumming function: pIII_sse
   pIII_sse  :   988.000 MB/sec
raid5: using function: pIII_sse (988.000 MB/sec)
md: raid5 personality registered as nr 4
raid5: device dm-4 operational as raid disk 4
raid5: device dm-3 operational as raid disk 3
raid5: device dm-2 operational as raid disk 2
raid5: device dm-1 operational as raid disk 1
raid5: device dm-0 operational as raid disk 0
raid5: allocated 5235kB for md0
raid5: raid level 5 set md0 active with 5 out of 5 devices, algorithm 2
RAID5 conf printout:
 --- rd:5 wd:5 fd:0
 disk 0, o:1, dev:dm-0
 disk 1, o:1, dev:dm-1
 disk 2, o:1, dev:dm-2
 disk 3, o:1, dev:dm-3
 disk 4, o:1, dev:dm-4
Unable to handle kernel NULL pointer dereference at virtual address 00000004
 printing eip:
c02214eb
*pde = 00000000
Oops: 0000 [#1]
PREEMPT SMP
Modules linked in: raid5 xor md dm_mod md5 ipv6 adm1021 eeprom i2c_piix4 i2c_isa
 lm75 lm78 w83781d i2c_sensor i2c_core agpgart scsi_mod 3c59x
CPU:    1
EIP:    0060:[<c02214eb>]    Not tainted VLI
EFLAGS: 00010202   (2.6.10-ac10)
eax: 00000000   ebx: f7c33488   ecx: 00000000   edx: c19374e0
esi: f7c33488   edi: 00000000   ebp: c19374e0   esp: f7b7f81c
ds: 007b   es: 007b   ss: 0068
Process evms_activate (pid: 1455, threadinfo=f7b7e000 task=f78f4a80)
Stack: c18ee3e0 00000200 00000000 f78f4a80 000000bf f7b7f850 00000008 00000000
       f7c33488 f79c9c80 c01317f0 c19374e0 c0221d77 f7c33488 c19374e0 00000000
       00000000 00000000 f78f4a80 c01317f0 f7b7f88c f7b7f88c c015f110 c18fd8a0
Call Trace:
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c0221d77>] generic_make_request+0x157/0x1f0
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c015f110>] bio_alloc+0x20/0x1e0
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c015f3af>] bio_clone+0x9f/0xc0
 [<f88b835a>] __map_bio+0x4a/0x120 [dm_mod]
 [<f88b87bd>] __clone_and_map+0x2ad/0x2c0 [dm_mod]
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<f88b8888>] __split_bio+0xb8/0x140 [dm_mod]
 [<c0103bcc>] apic_timer_interrupt+0x1c/0x24
 [<f88b898c>] dm_request+0x7c/0xb0 [dm_mod]
 [<c0221d77>] generic_make_request+0x157/0x1f0
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<f88d6ce0>] compute_blocknr+0xb0/0x120 [raid5]
 [<f88d7dbe>] handle_stripe+0x79e/0xea0 [raid5]
 [<f88d6a74>] raid5_build_block+0x74/0x90 [raid5]
 [<f88d6311>] get_active_stripe+0x71/0x2f0 [raid5]
 [<c0125599>] __mod_timer+0xf9/0x140
 [<f88d88f8>] make_request+0x178/0x270 [raid5]
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c0221d77>] generic_make_request+0x157/0x1f0
 [<c0112db3>] unmask_IO_APIC_irq+0x23/0x40
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c015f3c1>] bio_clone+0xb1/0xc0
 [<f88b835a>] __map_bio+0x4a/0x120 [dm_mod]
 [<f88b87bd>] __clone_and_map+0x2ad/0x2c0 [dm_mod]
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<f88b8888>] __split_bio+0xb8/0x140 [dm_mod]
 [<f88b898c>] dm_request+0x7c/0xb0 [dm_mod]
 [<c0221d77>] generic_make_request+0x157/0x1f0
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c013181f>] autoremove_wake_function+0x2f/0x60
 [<c013de63>] mempool_alloc+0x73/0x140
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c0221e80>] submit_bio+0x70/0x120
 [<c015f1d7>] bio_alloc+0xe7/0x1e0
 [<c015bba0>] end_buffer_async_read+0x0/0xe0
 [<c015ea06>] submit_bh+0xd6/0x130
 [<c015d894>] block_read_full_page+0x1b4/0x2e0
 [<c013a69d>] add_to_page_cache+0xbd/0xe0
 [<c0142264>] read_pages+0xf4/0x140
 [<c0161930>] blkdev_get_block+0x0/0x60
 [<c013f3a4>] __alloc_pages+0x1d4/0x370
 [<f88b8a90>] dm_any_congested+0x30/0x60 [dm_mod]
 [<c01425a9>] do_page_cache_readahead+0x119/0x190
 [<c014271a>] page_cache_readahead+0xfa/0x1f0
 [<c013ae67>] do_generic_mapping_read+0x127/0x500
 [<c013b542>] __generic_file_aio_read+0x212/0x250
 [<c013b240>] file_read_actor+0x0/0xf0
 [<c02248d6>] get_disk+0x66/0xd0
 [<c013b6b4>] generic_file_read+0xb4/0xd0
 [<c0162376>] do_open+0x96/0x360
 [<c0162704>] blkdev_open+0x34/0x70
 [<c015928c>] dentry_open+0x11c/0x260
 [<c01317f0>] autoremove_wake_function+0x0/0x60
 [<c0159f18>] vfs_read+0xb8/0x130
 [<c015b072>] fget_light+0x82/0x90
 [<c015a201>] sys_read+0x51/0x80
 [<c0103179>] sysenter_past_esp+0x52/0x75
Code: 55 57 31 ff 56 53 83 ec 20 8b 54 24 38 8b 74 24 34 8b 02 89 44 24 10 8b 42 1c c1 e8 09 89 44 24 18 8b 4a 2c 0f b7 42 16 8d 04 40 <8b> 44 81 04 c1 e8 09 89 44 24 14 8b 5a 10 8d 44 24 38 89 44 24


Thanks.

-- 

Re: 2.6.10-ac10 + 2.6.10udm1 + latest evms => not nice

[Kevin Corry]

On Friday 21 January 2005 8:47 am, Lars Marowsky-Bree wrote:
> On 2005-01-21T15:13:26, syrius.ml@no-log.org wrote:
> > Don't know if it's related to evms or to dm.
> > kernel is a 2.6.10-ac10 + 2.6.10-udm1 + evms kernel patches.
> > (bd_claims + dm bbr + dm multipath)
> >
> > Here is the nice welcome message I've just got :)
>
> This looks like md on DM breakage, which Jens has just fixed in our
> kernel. I'm not sure whether he has submitted it upstream yet, but you
> can give it a try.
>
> Does the attached patch apply & fix the problem for you?

>    If md resides on top of a driver using bio_clone() (such as dm), it will
>    oops the kernel due to md submitting a botched bio that has a veclist but
>    doesn't have bio->bi_max_vecs set.

Is there a specific scenario that causes the kernel to oops if an MD device is 
made of DM devices? EVMS has been using this kind of configuration for a 
couple years, and we've never noticed a problem with it. Also, syrius 
mentioned on IRC that this problem only started occurring with -ac10, and 
that -ac8 worked fine. I'm checking out the diffs between those two versions 
to see what else might have changed.

-- 
Kevin Corry
kevcorry@us.ibm.com
http://evms.sourceforge.net/


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Evms-devel mailing list
Evms-devel@lists.sourceforge.net
To subscribe/unsubscribe, please visit:
https://lists.sourceforge.net/lists/listinfo/evms-devel

Re: 2.6.10-ac10 + 2.6.10udm1 + latest evms => not nice

[Kevin Corry]

On Friday 21 January 2005 9:01 am, Kevin Corry wrote:
> On Friday 21 January 2005 8:47 am, Lars Marowsky-Bree wrote:
> > This looks like md on DM breakage, which Jens has just fixed in our
> > kernel. I'm not sure whether he has submitted it upstream yet, but you
> > can give it a try.
> >
> > Does the attached patch apply & fix the problem for you?
> >
> >    If md resides on top of a driver using bio_clone() (such as dm), it
> > will oops the kernel due to md submitting a botched bio that has a
> > veclist but doesn't have bio->bi_max_vecs set.
>
> Is there a specific scenario that causes the kernel to oops if an MD device
> is made of DM devices? EVMS has been using this kind of configuration for a
> couple years, and we've never noticed a problem with it. Also, syrius
> mentioned on IRC that this problem only started occurring with -ac10, and
> that -ac8 worked fine. I'm checking out the diffs between those two
> versions to see what else might have changed.

Ok, here's the excerpt from the -ac10 patch that probably exposed the bug in 
MD. So it looks like a cloned bio no longer uses the original bio's bvec, 
which makes Lars' explination make a bit more sense. :) So, try Lars' patch 
with your -ac10 kernel and see if that fixes it. If so, we should forward 
that to Alan and make sure he includes it. I'll also check the latest -mm and 
-bk to see if these patches need to be merged upstream before 2.6.11 final.


diff -Naurp linux-2.6.10-ac9/fs/bio.c linux-2.6.10-ac10/fs/bio.c
--- linux-2.6.10-ac9/fs/bio.c 2004-12-24 15:35:00.000000000 -0600
+++ linux-2.6.10-ac10/fs/bio.c 2005-01-21 08:50:27.000000000 -0600
@@ -98,12 +98,7 @@ void bio_destructor(struct bio *bio)
 
  BIO_BUG_ON(pool_idx >= BIOVEC_NR_POOLS);
 
- /*
-  * cloned bio doesn't own the veclist
-  */
- if (!bio_flagged(bio, BIO_CLONED))
-  mempool_free(bio->bi_io_vec, bp->pool);
-
+ mempool_free(bio->bi_io_vec, bp->pool);
  mempool_free(bio, bio_pool);
 }
 
@@ -210,7 +205,9 @@ inline int bio_hw_segments(request_queue
  */
 inline void __bio_clone(struct bio *bio, struct bio *bio_src)
 {
- bio->bi_io_vec = bio_src->bi_io_vec;
+ request_queue_t *q = bdev_get_queue(bio_src->bi_bdev);
+
+ memcpy(bio->bi_io_vec, bio_src->bi_io_vec, bio_src->bi_max_vecs * 
sizeof(struct bio_vec));
 
  bio->bi_sector = bio_src->bi_sector;
  bio->bi_bdev = bio_src->bi_bdev;
@@ -222,21 +219,9 @@ inline void __bio_clone(struct bio *bio,
   * for the clone
   */
  bio->bi_vcnt = bio_src->bi_vcnt;
- bio->bi_idx = bio_src->bi_idx;
- if (bio_flagged(bio, BIO_SEG_VALID)) {
-  bio->bi_phys_segments = bio_src->bi_phys_segments;
-  bio->bi_hw_segments = bio_src->bi_hw_segments;
-  bio->bi_flags |= (1 << BIO_SEG_VALID);
- }
  bio->bi_size = bio_src->bi_size;
-
- /*
-  * cloned bio does not own the bio_vec, so users cannot fiddle with
-  * it. clear bi_max_vecs and clear the BIO_POOL_BITS to make this
-  * apparent
-  */
- bio->bi_max_vecs = 0;
- bio->bi_flags &= (BIO_POOL_MASK - 1);
+ bio_phys_segments(q, bio);
+ bio_hw_segments(q, bio);
 }
 
 /**
@@ -248,7 +233,7 @@ inline void __bio_clone(struct bio *bio,
  */
 struct bio *bio_clone(struct bio *bio, int gfp_mask)
 {
- struct bio *b = bio_alloc(gfp_mask, 0);
+ struct bio *b = bio_alloc(gfp_mask, bio->bi_max_vecs);
 
  if (b)
   __bio_clone(b, bio);

-- 
Kevin Corry
kevcorry@us.ibm.com
http://evms.sourceforge.net/


-------------------------------------------------------
This SF.Net email is sponsored by: IntelliVIEW -- Interactive Reporting
Tool for open source databases. Create drag-&-drop reports. Save time
by over 75%! Publish reports on the web. Export to DOC, XLS, RTF, etc.
Download a FREE copy at http://www.intelliview.com/go/osdn_nl
_______________________________________________
Evms-devel mailing list
Evms-devel@lists.sourceforge.net
To subscribe/unsubscribe, please visit:
https://lists.sourceforge.net/lists/listinfo/evms-devel