* [linux-lvm] Oops when running snapshots
@ 2004-01-05 4:36 Steve McIntyre
2004-01-12 9:36 ` Steve McIntyre
0 siblings, 1 reply; 9+ messages in thread
From: Steve McIntyre @ 2004-01-05 4:36 UTC (permalink / raw)
To: linux-lvm
Has there been any progress on the issue reported by Andrew Patterson
in
http://lists.sistina.com/pipermail/linux-lvm/2003-July/014422.html
???
I've got a brand new server that seems to have just shown the same
problem over the Christmas period: twin P4 Xeon, 1GB RAM (configured
for 4GB highmem), kernel 2.4.23. Oops message:
ksymoops 2.4.5 on i686 2.4.23. Options used
-V (default)
-k /proc/ksyms (default)
-l /proc/modules (default)
-o /lib/modules/2.4.23/ (default)
-m /boot/System.map-2.4.23-20031224 (specified)
No modules in ksyms, skipping objects
Warning (read_lsmod): no symbols in lsmod, is /proc/modules a valid lsmod file?
Warning (compare_maps): ksyms_base symbol IO_APIC_get_PCI_irq_vector_R__ver_IO_APIC_get_PCI_irq_vector not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __br_write_lock_R__ver___br_write_lock not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __br_write_unlock_R__ver___br_write_unlock not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __brlock_array_R__ver___brlock_array not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __global_cli_R__ver___global_cli not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __global_restore_flags_R__ver___global_restore_flags not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __global_save_flags_R__ver___global_save_flags not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __global_sti_R__ver___global_sti not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol atomic_dec_and_lock_R__ver_atomic_dec_and_lock not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol cpu_data_R__ver_cpu_data not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol cpu_online_map_R__ver_cpu_online_map not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol create_bounce_R__ver_create_bounce not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol del_timer_sync_R__ver_del_timer_sync not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol drive_info_R__ver_drive_info not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol flush_tlb_page_R__ver_flush_tlb_page not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol global_irq_holder_R__ver_global_irq_holder not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol highmem_start_page_R__ver_highmem_start_page not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol ide_pci_register_host_proc_R__ver_ide_pci_register_host_proc not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol kernel_flag_cacheline_R__ver_kernel_flag_cacheline not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol kmap_high_R__ver_kmap_high not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol kmap_prot_R__ver_kmap_prot not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol kmap_pte_R__ver_kmap_pte not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol kunmap_high_R__ver_kunmap_high not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol set_cpus_allowed_R__ver_set_cpus_allowed not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol smp_call_function_R__ver_smp_call_function not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol smp_num_cpus_R__ver_smp_num_cpus not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol synchronize_irq_R__ver_synchronize_irq not found in System.map. Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol tqueue_lock_R__ver_tqueue_lock not found in System.map. Ignoring ksyms_base entry
Dec 29 03:04:59 cetus kernel: c0287ed5
Dec 29 03:04:59 cetus kernel: Oops: 0002
Dec 29 03:04:59 cetus kernel: CPU: 3
Dec 29 03:04:59 cetus kernel: EIP: 0010:[<c0287ed5>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Dec 29 03:04:59 cetus kernel: EFLAGS: 00010202
Dec 29 03:04:59 cetus kernel: eax: 00000000 ebx: f91e89a8 ecx: f8b09450 edx: f8b02778
Dec 29 03:04:59 cetus kernel: esi: 01fb8200 edi: d8858800 ebp: 00000831 esp: f6abfc3c
Dec 29 03:04:59 cetus kernel: ds: 0018 es: 0018 ss: 0018
Dec 29 03:04:59 cetus kernel: Process nfsd (pid: 286, stackpage=f6abf000)
Dec 29 03:04:59 cetus kernel: Stack: d8858800 d8858970 f7dee770 05a00000 00000080 00000002 00000070 00000000
Dec 29 03:04:59 cetus kernel: c0284951 f6abfc9a f6abfc9c 01fb8180 d8858800 00003a00 e5ddfa00 076800f0
Dec 29 03:04:59 cetus kernel: 05a00000 dea60780 02dc00e8 f7dee600 f786d000 01fb8180 0003b400 08310000
Dec 29 03:04:59 cetus kernel: Call Trace: [<c0284951>] [<c0284a37>] [<c01e3b8c>] [<c01e3bf2>] [<c013d06c>]
Dec 29 03:04:59 cetus kernel: [<c013d1a7>] [<c013b309>] [<c0130f72>] [<c01312e0>] [<c0131350>] [<c0131e78>]
Dec 29 03:04:59 cetus kernel: [<c014e7fd>] [<c0132182>] [<c0131e2e>] [<c01294ce>] [<c0129be5>] [<c0129e3c>]
Dec 29 03:04:59 cetus kernel: [<c012a4df>] [<c012a36c>] [<c0191f84>] [<c019714b>] [<c018e5d3>] [<c02fec67>]
Dec 29 03:04:59 cetus kernel: [<c018e3bf>] [<c0105694>]
Dec 29 03:04:59 cetus kernel: Code: 89 10 c7 01 00 00 00 00 c7 41 04 00 00 00 00 8b 03 89 48 04
>>EIP; c0287ed5 <lvm_snapshot_remap_block+a9/f8> <=====
>>ebx; f91e89a8 <END_OF_CODE+38d5cae8/????>
>>ecx; f8b09450 <END_OF_CODE+3867d590/????>
>>edx; f8b02778 <END_OF_CODE+386768b8/????>
>>esi; 01fb8200 Before first symbol
>>edi; d8858800 <END_OF_CODE+183cc940/????>
>>ebp; 00000831 Before first symbol
>>esp; f6abfc3c <END_OF_CODE+36633d7c/????>
Trace; c0284951 <lvm_map+3b9/490>
Trace; c0284a37 <lvm_make_request_fn+f/1c>
Trace; c01e3b8c <generic_make_request+120/130>
Trace; c01e3bf2 <submit_bh+56/e0>
Trace; c013d06c <sync_page_buffers+94/ac>
Trace; c013d1a7 <try_to_free_buffers+123/148>
Trace; c013b309 <try_to_release_page+49/4c>
Trace; c0130f72 <shrink_cache+21e/3f4>
Trace; c01312e0 <shrink_caches+3c/48>
Trace; c0131350 <try_to_free_pages_zone+64/e4>
Trace; c0131e78 <balance_classzone+48/1c8>
Trace; c014e7fd <iget4+4d/e8>
Trace; c0132182 <__alloc_pages+18a/28c>
Trace; c0131e2e <_alloc_pages+16/18>
Trace; c01294ce <page_cache_read+7a/d0>
Trace; c0129be5 <generic_file_readahead+105/13c>
Trace; c0129e3c <do_generic_file_read+1f0/494>
Trace; c012a4df <generic_file_read+93/190>
Trace; c012a36c <file_read_actor+0/e0>
Trace; c0191f84 <nfsd_read+1bc/260>
Trace; c019714b <nfsd3_proc_read+127/184>
Trace; c018e5d3 <nfsd_dispatch+d3/19a>
Trace; c02fec67 <svc_process+28f/4f0>
Trace; c018e3bf <nfsd+1f3/334>
Trace; c0105694 <arch_kernel_thread+28/38>
Code; c0287ed5 <lvm_snapshot_remap_block+a9/f8>
00000000 <_EIP>:
Code; c0287ed5 <lvm_snapshot_remap_block+a9/f8> <=====
0: 89 10 mov %edx,(%eax) <=====
Code; c0287ed7 <lvm_snapshot_remap_block+ab/f8>
2: c7 01 00 00 00 00 movl $0x0,(%ecx)
Code; c0287edd <lvm_snapshot_remap_block+b1/f8>
8: c7 41 04 00 00 00 00 movl $0x0,0x4(%ecx)
Code; c0287ee4 <lvm_snapshot_remap_block+b8/f8>
f: 8b 03 mov (%ebx),%eax
Code; c0287ee6 <lvm_snapshot_remap_block+ba/f8>
11: 89 48 04 mov %ecx,0x4(%eax)
Jan 2 08:50:33 cetus kernel: ACPI: LAPIC_NMI (acpi_id[0x00] polarity[0x1] trigger[0x3] lint[0x1])
Jan 2 08:50:33 cetus kernel: ACPI: LAPIC_NMI (acpi_id[0x01] polarity[0x1] trigger[0x3] lint[0x1])
Jan 2 08:50:33 cetus kernel: cpu: 0, clocks: 1329112, slice: 265822
Jan 2 08:50:33 cetus kernel: cpu: 1, clocks: 1329112, slice: 265822
Jan 2 08:50:33 cetus kernel: cpu: 3, clocks: 1329112, slice: 265822
Jan 2 08:50:33 cetus kernel: cpu: 2, clocks: 1329112, slice: 265822
Jan 2 08:50:33 cetus kernel: e1000: eth0 NIC Link is Up 100 Mbps Full Duplex
29 warnings issued. Results may not be reliable.
--
Steve McIntyre, Plasmon smcintyre@software.plasmon.com
Oh My God! They Killed init! You Bastards!
"Can't keep my eyes from the circling sky,
"Tongue-tied & twisted, Just an earth-bound misfit, I..."
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [linux-lvm] Oops when running snapshots 2004-01-05 4:36 [linux-lvm] Oops when running snapshots Steve McIntyre @ 2004-01-12 9:36 ` Steve McIntyre 2004-01-14 8:37 ` Andrew Patterson 0 siblings, 1 reply; 9+ messages in thread From: Steve McIntyre @ 2004-01-12 9:36 UTC (permalink / raw) To: linux-lvm On Mon, Jan 05, 2004 at 10:35:00AM +0000, Me @ Plasmon wrote: >Has there been any progress on the issue reported by Andrew Patterson >in > >http://lists.sistina.com/pipermail/linux-lvm/2003-July/014422.html > >??? Apparently not. Should I expect a response here, or ask on the main kernel list? -- Steve McIntyre, Plasmon smcintyre@software.plasmon.com "Can't keep my eyes from the circling sky, "Tongue-tied & twisted, Just an earth-bound misfit, I..." ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Oops when running snapshots 2004-01-12 9:36 ` Steve McIntyre @ 2004-01-14 8:37 ` Andrew Patterson 2004-01-21 5:35 ` Steve McIntyre 0 siblings, 1 reply; 9+ messages in thread From: Andrew Patterson @ 2004-01-14 8:37 UTC (permalink / raw) To: linux-lvm; +Cc: smcintyre [-- Attachment #1: Type: text/plain, Size: 9222 bytes --] On Mon, 2004-01-12 at 07:35, Steve McIntyre wrote: > On Mon, Jan 05, 2004 at 10:35:00AM +0000, Me @ Plasmon wrote: > >Has there been any progress on the issue reported by Andrew Patterson > >in > > > >http://lists.sistina.com/pipermail/linux-lvm/2003-July/014422.html > > > >??? > > Apparently not. Should I expect a response here, or ask on the main > kernel list? We worked over this problem with Heinz on LVM 1.07. One of our kernel hackers (along with Heinz) came up with several possible solutions to a race condition in the snapshot code. They are presented below. We found that option #1 worked, but sometimes it could take a long time to create multiple snapshots of the same LV under extremely heavy load. The first snapshot is fine, but the second or more can take hours. Option #2 created a huge performance penalty to writes on the original LV (90-99%). I don't remember that option #3 helped any, and we never tried #4. We settled with option #1 and solved the long snapshot creation time by stopping I/O while creating the snapshot. The LVM snapshot problem is the same one I had with the nfs export hash table (classic reader/writer problem). Read and write semaphores are being used to allow concurrent read access to the snapshot hash table while allowing exclusive write access. However, lvm_snapshot_remap_block -> lvm_find_exception_table does the hash table optimization (i.e. writing to the hash table) regardless of whether the read or write semaphore is held. The snapshot hash table has 128K entries, so there's more potential for performance improvement. Here's the call tree for lvm_find_exception_table: lvm_find_exception_table - lvm_snapshot_remap_block -- __remap_snapshot -- _remap_snapshot -- lvm_map lvm_map, _remap_snapshot, and __remap_snapshot all call lvm_snapshot_remap_block. 1) lvm_map lvm_map() { down_read(&lv->lv_lock) ... if (lvm_snapshot_remap_block(...)) ... up_read(&lv->lv_lock) } This is bad, because lvm_snapshot_remap_block will call lvm_find_exception_table, which could write to the hash table while only holding the read semaphore during the optimization step. 2) _remap_snapshot + __remap_snapshot lvm_map() { down_read(&lv->lv_lock) ... _remap_snapshot() { down_read(&lv->lock) // again! r = lvm_snapshot_remap_block() up_read(&lv->lock) if (!r) __remap_snapshot() { down_write(&lv->lock) // good idea, but we already hold read! if (!lvm_snapshot_remap_block()) {} up_write(&lv->lock) } } up_read(&lv->lv_lock) } This is bad, because down_read(&lv->lock) is called twice unnecessarily. Also, lvm_snapshot_remap_block will still possibly write to the hash table while only holding the read semaphore. I don't know how lvm_map->_remap_snapshot->__remap_snapshot works at all. down_write should block until all the readers have released the semaphore. Since this kernel process already holds the read semaphore, it should never get the chance to get the write semaphore. Options to fix (included as attachments and inline): 1) don't do the hash table optimization in lvm_find_exception_table, which we tried last night. No crashes, but slow. Doesn't fix the "get the write semaphore while holding the read semaphore" problem in __remap_snapshot(). --- lvm-snap.c~ Mon Aug 25 15:28:50 2003 +++ lvm-snap.c-erik Thu Aug 28 13:33:28 2003 @@ -130,11 +130,13 @@ static inline lv_block_exception_t *lvm_ exception = list_entry(next, lv_block_exception_t, hash); if (exception->rsector_org == org_start && exception->rdev_org == org_dev) { +#if 0 if (i) { /* fun, isn't it? :) */ list_del(next); list_add(next, hash_table); } +#endif ret = exception; break; } 2) clean up the locking: always get write semaphore before going into lvm_snapshot_remap_block, realize we already hold the read semaphore: --- lvm.c~ Fri Aug 29 13:56:26 2003 +++ lvm.c-erik Fri Aug 29 14:08:30 2003 @@ -1183,6 +1183,7 @@ static void __remap_snapshot(kdev_t rdev { /* copy a chunk from the origin to a snapshot device */ + up_read(&lv->lv_lock); down_write(&lv->lv_lock); /* we must redo lvm_snapshot_remap_block in order to avoid a @@ -1192,6 +1193,7 @@ static void __remap_snapshot(kdev_t rdev lvm_write_COW_table_block(vg, lv); up_write(&lv->lv_lock); + down_read(&lv->lv_lock); } static inline void _remap_snapshot(kdev_t rdev, ulong rsector, @@ -1200,9 +1202,11 @@ static inline void _remap_snapshot(kdev_ int r; /* check to see if this chunk is already in the snapshot */ - down_read(&lv->lv_lock); - r = lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv); up_read(&lv->lv_lock); + down_write(&lv->lv_lock); + r = lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv); + up_write(&lv->lv_lock); + down_read(&lv->lv_lock); if (!r) /* we haven't yet copied this block to the snapshot */ @@ -1340,9 +1344,16 @@ static int lvm_map(struct buffer_head *b goto out; if (lv->lv_access & LV_SNAPSHOT) { /* remap snapshot */ + up_read(&lv->lv_lock); + down_write(&lv->lv_lock); if (lvm_snapshot_remap_block(&rdev_map, &rsector_map, - pe_start, lv) < 0) + pe_start, lv) < 0) { + up_write(&lv->lv_lock); + down_read(&lv->lv_lock); goto bad; + } + up_write(&lv->lv_lock); + down_read(&lv->lv_lock); } else if (rw == WRITE || rw == WRITEA) { /* snapshot origin */ lv_t *snap; 3) clean up the locking: grab the write semaphore in lvm_map and don't get any other semaphores. This isn't recommended and I'm sure Heinz will cry when he sees this. This defeats much of the purpose of having separate read and write semaphores, and lvm_map might will hold the write semaphore for a long time. This will kill concurrent access to a volume, and will probably hurt performance. --- lvm.c~ Fri Aug 29 13:56:26 2003 +++ lvm.c-erik Fri Aug 29 14:16:01 2003 @@ -1183,7 +1183,6 @@ static void __remap_snapshot(kdev_t rdev { /* copy a chunk from the origin to a snapshot device */ - down_write(&lv->lv_lock); /* we must redo lvm_snapshot_remap_block in order to avoid a race condition in the gap where no lock was held */ @@ -1191,7 +1190,6 @@ static void __remap_snapshot(kdev_t rdev !lvm_snapshot_COW(rdev, rsector, pe_start, rsector, vg, lv)) lvm_write_COW_table_block(vg, lv); - up_write(&lv->lv_lock); } static inline void _remap_snapshot(kdev_t rdev, ulong rsector, @@ -1200,9 +1198,7 @@ static inline void _remap_snapshot(kdev_ int r; /* check to see if this chunk is already in the snapshot */ - down_read(&lv->lv_lock); r = lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv); - up_read(&lv->lv_lock); if (!r) /* we haven't yet copied this block to the snapshot */ @@ -1253,7 +1249,7 @@ static int lvm_map(struct buffer_head *b lv_t *lv = vg_this->lv[LV_BLK(minor)]; - down_read(&lv->lv_lock); + down_write(&lv->lv_lock); if (!(lv->lv_status & LV_ACTIVE)) { printk(KERN_ALERT "%s - lvm_map: ll_rw_blk for inactive LV %s\n", @@ -1327,7 +1323,7 @@ static int lvm_map(struct buffer_head *b if (_defer_extent(bh, rw, rdev_map, rsector_map, vg_this->pe_size)) { - up_read(&lv->lv_lock); + up_write(&lv->lv_lock); return 0; } @@ -1365,13 +1361,13 @@ static int lvm_map(struct buffer_head *b out: bh->b_rdev = rdev_map; bh->b_rsector = rsector_map; - up_read(&lv->lv_lock); + up_write(&lv->lv_lock); return 1; bad: if (bh->b_end_io) buffer_IO_error(bh); - up_read(&lv->lv_lock); + up_write(&lv->lv_lock); return -1; } /* lvm_map() */ 4) in lvm_snapshot_remap_block, somehow determine if we hold the read semaphore and magically turn it into a write semaphore before doing the optimization. I'm not going to attempt this one. -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Andrew Patterson Voice: (970) 898-3261 Hewlett-Packard Company Email: andrew@fc.hp.com [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 189 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Oops when running snapshots 2004-01-14 8:37 ` Andrew Patterson @ 2004-01-21 5:35 ` Steve McIntyre 0 siblings, 0 replies; 9+ messages in thread From: Steve McIntyre @ 2004-01-21 5:35 UTC (permalink / raw) To: andrew; +Cc: linux-lvm On Tue, Jan 13, 2004 at 11:35:46AM -0700, Andrew Patterson wrote: > >We worked over this problem with Heinz on LVM 1.07. One of our kernel >hackers (along with Heinz) came up with several possible solutions to a >race condition in the snapshot code. They are presented below. We >found that option #1 worked, but sometimes it could take a long time to >create multiple snapshots of the same LV under extremely heavy load. >The first snapshot is fine, but the second or more can take hours. >Option #2 created a huge performance penalty to writes on the original >LV (90-99%). I don't remember that option #3 helped any, and we never >tried #4. We settled with option #1 and solved the long snapshot >creation time by stopping I/O while creating the snapshot. Ok, thanks. We had another hard lockup overnight caused by this problem. I've just applied #1; I'll let you know how it goes. <snip> >Options to fix (included as attachments and inline): > >1) don't do the hash table optimization in lvm_find_exception_table, >which >we tried last night. No crashes, but slow. Doesn't fix the "get the >write >semaphore while holding the read semaphore" problem in >__remap_snapshot(). > >--- lvm-snap.c~ Mon Aug 25 15:28:50 2003 >+++ lvm-snap.c-erik Thu Aug 28 13:33:28 2003 >@@ -130,11 +130,13 @@ static inline lv_block_exception_t *lvm_ > exception = list_entry(next, lv_block_exception_t, >hash); > if (exception->rsector_org == org_start && > exception->rdev_org == org_dev) { >+#if 0 > if (i) { > /* fun, isn't it? :) */ > list_del(next); > list_add(next, hash_table); > } >+#endif > ret = exception; > break; > } -- Steve McIntyre, Plasmon smcintyre@software.plasmon.com Getting a SCSI chain working is perfectly simple if you remember that there must be exactly three terminations: one on one end of the cable, one on the far end, and the goat, terminated over the SCSI chain with a silver-handled knife whilst burning *black* candles. --- Anthony DeBoer ^ permalink raw reply [flat|nested] 9+ messages in thread
* RE: [linux-lvm] Oops when running snapshots
@ 2004-01-12 10:27 Little, Chris
0 siblings, 0 replies; 9+ messages in thread
From: Little, Chris @ 2004-01-12 10:27 UTC (permalink / raw)
To: 'linux-lvm@sistina.com'
Personally, I have had no responses concerning snapshot issues. Perhaps
snapshot is very low priority, or maybe I'm just really far back in the
queue.
> -----Original Message-----
> From: Steve McIntyre [mailto:smcintyre@software.plasmon.com]
> Sent: Monday, January 12, 2004 8:35 AM
> To: linux-lvm@sistina.com
> Subject: Re: [linux-lvm] Oops when running snapshots
>
>
> On Mon, Jan 05, 2004 at 10:35:00AM +0000, Me @ Plasmon wrote:
> >Has there been any progress on the issue reported by Andrew Patterson
> >in
> >
> >http://lists.sistina.com/pipermail/linux-lvm/2003-July/014422.html
> >
> >???
>
> Apparently not. Should I expect a response here, or ask on the main
> kernel list?
>
> --
> Steve McIntyre, Plasmon
> smcintyre@software.plasmon.com
> "Can't keep my eyes from the circling sky,
> "Tongue-tied & twisted, Just an earth-bound misfit, I..."
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
>
^ permalink raw reply [flat|nested] 9+ messages in thread* [linux-lvm] Oops when running snapshots
@ 2003-07-09 17:27 Andrew Patterson
2003-07-10 4:34 ` Heinz J . Mauelshagen
0 siblings, 1 reply; 9+ messages in thread
From: Andrew Patterson @ 2003-07-09 17:27 UTC (permalink / raw)
To: linux-lvm
Ran a load test last night where we created snapshots two snapshots
every 2 hours under heavy file system load. We got the following oops:
Unable to handle kernel NULL pointer dereference at virtual address
00000000
802895f5
*pde = 55bd7001
Oops: 0002
CPU: 2
EIP: 0010:[<802895f5>] Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010206
eax: 00000000 ebx: f8ab2118 ecx: f8a79048 edx: f8a79030
esi: 00810180 edi: b4db0000 ebp: 00000820 esp: a2765c4c
ds: 0018 es: 0018 ss: 0018
Process vsftpd (pid: 7309, stackpage=a2765000)
Stack: b4db0000 b4db0170 f69b3170 00ea8000 00000080 00000006 00000070
00000000
80286031 a2765caa a2765cac 00810180 b4db0000 00003a02 e338c840
00800070
00ea8000 00000000 00000000 f69b3000 f68a0000 00810180 00000800
08208cbc
Call Trace: [<80286031>] [<80286117>] [<80219ccc>] [<80219d41>]
[<801d209f>]
[<801d26eb>] [<801d2ac7>] [<801d2b2a>] [<801387fd>] [<801d2952>]
[<80139b
1d327e>] [<80137807>] [<80106f27>]
Code: 89 10 c7 01 00 00 00 00 c7 41 04 00 00 00 00 8b 03 89 48 04
>>EIP; 802895f5 <lvm_snapshot_remap_block+a9/f8> <=====
>>ebx; f8ab2118 <[nfstracker].bss.end+490c5/ffe98fa9>
>>ecx; f8a79048 <[nfstracker].bss.end+fff5/ffe98fa9>
>>edx; f8a79030 <[nfstracker].bss.end+ffdd/ffe98fa9>
>>edi; b4db0000 <_end+3499065c/7847565c>
>>esp; a2765c4c <_end+223462a8/7847565c>
Trace; 80286031 <lvm_map+3b9/490>
Trace; 80286117 <lvm_make_request_fn+f/1c>
Trace; 80219ccc <generic_make_request+11c/12c>
Trace; 80219d41 <submit_bh+65/80>
Trace; 801d209f <submit_page+57/74>
Trace; 801d26eb <page_state_convert+40b/4a0>
Trace; 801d2ac7 <linvfs_writepage+3b/e0>
Trace; 801d2b2a <linvfs_writepage+9e/e0>
Trace; 801387fd <write_some_buffers+9d/14c>
Trace; 801d2952 <linvfs_get_block+1e/24>
Code; 802895f5 <lvm_snapshot_remap_block+a9/f8>
00000000 <_EIP>:
Code; 802895f5 <lvm_snapshot_remap_block+a9/f8> <=====
0: 89 10 mov %edx,(%eax) <=====
Code; 802895f7 <lvm_snapshot_remap_block+ab/f8>
2: c7 01 00 00 00 00 movl $0x0,(%ecx)
Code; 802895fd <lvm_snapshot_remap_block+b1/f8>
Trace; 80219d41 <submit_bh+65/80>
Trace; 801d209f <submit_page+57/74>
Trace; 801d26eb <page_state_convert+40b/4a0>
Trace; 801d2ac7 <linvfs_writepage+3b/e0>
Trace; 801d2b2a <linvfs_writepage+9e/e0>
Trace; 801387fd <write_some_buffers+9d/14c>
Trace; 801d2952 <linvfs_get_block+1e/24>
Code; 802895f5 <lvm_snapshot_remap_block+a9/f8>
00000000 <_EIP>:
Code; 802895f5 <lvm_snapshot_remap_block+a9/f8> <=====
0: 89 10 mov %edx,(%eax) <=====
Code; 802895f7 <lvm_snapshot_remap_block+ab/f8>
2: c7 01 00 00 00 00 movl $0x0,(%ecx)
Code; 802895fd <lvm_snapshot_remap_block+b1/f8>
8: c7 41 04 00 00 00 00 movl $0x0,0x4(%ecx)
Code; 80289604 <lvm_snapshot_remap_block+b8/f8>
f: 8b 03 mov (%ebx),%eax
Code; 80289606 <lvm_snapshot_remap_block+ba/f8>
11: 89 48 04 mov %ecx,0x4(%eax)
10 warnings issued. Results may not be reliable.
This was run on a 2.4.21 system using LVM 1.0.7. The system has 2GB of
memory and is configured with HIGHME64G. We are running the latest
2.4.21 XFS and are using xfs_freeze to quiesce the file system. We are
not using the VFS_LOCK patch.
Anyone have any ideas what may be causing this?
Thanks,
Andrew
--
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Andrew Patterson Voice: (970) 898-3261
Hewlett-Packard Company Email: andrew@fc.hp.com
^ permalink raw reply [flat|nested] 9+ messages in thread* Re: [linux-lvm] Oops when running snapshots 2003-07-09 17:27 Andrew Patterson @ 2003-07-10 4:34 ` Heinz J . Mauelshagen 2003-07-10 12:28 ` Andrew Patterson 2003-08-15 17:16 ` Andrew Patterson 0 siblings, 2 replies; 9+ messages in thread From: Heinz J . Mauelshagen @ 2003-07-10 4:34 UTC (permalink / raw) To: linux-lvm On Wed, Jul 09, 2003 at 04:26:36PM -0600, Andrew Patterson wrote: > Ran a load test last night where we created snapshots two snapshots > every 2 hours under heavy file system load. We got the following oops: > > > Unable to handle kernel NULL pointer dereference at virtual address > 00000000 > 802895f5 <SNIP> > 11: 89 48 04 mov %ecx,0x4(%eax) > > > 10 warnings issued. Results may not be reliable. > > This was run on a 2.4.21 system using LVM 1.0.7. The system has 2GB of > memory and is configured with HIGHME64G. We are running the latest > 2.4.21 XFS and are using xfs_freeze to quiesce the file system. We are > not using the VFS_LOCK patch. > > Anyone have any ideas what may be causing this? VM problems ? (we had some reports ITR) Do you have a chance to retest without HIGHMEM to see if it still fails ? > > Thanks, > > Andrew > > -- > =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= > Andrew Patterson Voice: (970) 898-3261 > Hewlett-Packard Company Email: andrew@fc.hp.com > > > > _______________________________________________ > linux-lvm mailing list > linux-lvm@sistina.com > http://lists.sistina.com/mailman/listinfo/linux-lvm > read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/ -- Regards, Heinz -- The LVM Guy -- *** Software bugs are stupid. Nevertheless it needs not so stupid people to solve them *** =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Heinz Mauelshagen Sistina Software Inc. Senior Consultant/Developer Am Sonnenhang 11 56242 Marienrachdorf Germany Mauelshagen@Sistina.com +49 2626 141200 FAX 924446 =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Oops when running snapshots 2003-07-10 4:34 ` Heinz J . Mauelshagen @ 2003-07-10 12:28 ` Andrew Patterson 2003-08-15 17:16 ` Andrew Patterson 1 sibling, 0 replies; 9+ messages in thread From: Andrew Patterson @ 2003-07-10 12:28 UTC (permalink / raw) To: linux-lvm; +Cc: Heinz J . Mauelshagen On Thu, 2003-07-10 at 03:32, Heinz J . Mauelshagen wrote: > On Wed, Jul 09, 2003 at 04:26:36PM -0600, Andrew Patterson wrote: > > Ran a load test last night where we created snapshots two snapshots > > every 2 hours under heavy file system load. We got the following oops: > > > > > > Unable to handle kernel NULL pointer dereference at virtual address > > 00000000 > > 802895f5 > <SNIP> > > 11: 89 48 04 mov %ecx,0x4(%eax) > > > > > > 10 warnings issued. Results may not be reliable. > > > > This was run on a 2.4.21 system using LVM 1.0.7. The system has 2GB of > > memory and is configured with HIGHME64G. We are running the latest > > 2.4.21 XFS and are using xfs_freeze to quiesce the file system. We are > > not using the VFS_LOCK patch. > > > > Anyone have any ideas what may be causing this? > > VM problems ? (we had some reports ITR) > Do you have a chance to retest without HIGHMEM to see if it still fails ? > I looked through mailing list archives about this. Most the of himem problems resulted in vmmalloc.c calls in the oops. I'll try it again without himem to see if this helps. Thanks, Andrew > > > > Thanks, ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [linux-lvm] Oops when running snapshots 2003-07-10 4:34 ` Heinz J . Mauelshagen 2003-07-10 12:28 ` Andrew Patterson @ 2003-08-15 17:16 ` Andrew Patterson 1 sibling, 0 replies; 9+ messages in thread From: Andrew Patterson @ 2003-08-15 17:16 UTC (permalink / raw) To: linux-lvm; +Cc: Heinz J . Mauelshagen On Thu, 2003-07-10 at 03:32, Heinz J . Mauelshagen wrote: > On Wed, Jul 09, 2003 at 04:26:36PM -0600, Andrew Patterson wrote: > > Ran a load test last night where we created snapshots two snapshots > > every 2 hours under heavy file system load. We got the following oops: > > > > > > Unable to handle kernel NULL pointer dereference at virtual address > > 00000000 > > 802895f5 > <SNIP> > > 11: 89 48 04 mov %ecx,0x4(%eax) > > > > > > 10 warnings issued. Results may not be reliable. > > > > This was run on a 2.4.21 system using LVM 1.0.7. The system has 2GB of > > memory and is configured with HIGHME64G. We are running the latest > > 2.4.21 XFS and are using xfs_freeze to quiesce the file system. We are > > not using the VFS_LOCK patch. > > > > Anyone have any ideas what may be causing this? > > VM problems ? (we had some reports ITR) > Do you have a chance to retest without HIGHMEM to see if it still fails ? > I finally had a chance to rerun with HIMEM turned off. I also made one other change. We are now using the linux-2.4.21-VFS-lock.patch I found in the device mapper CVS archives instead of using xfs_freeze (we thought we were running into some race condition with xfs_freeze. Some other details: We are running multiple snapshots on the same volume. The snapshots and data set changes are very large (constantly changing up to 12 GB using random and sequential read/writes). We have 2 GB of RAM on the system (the kernel only uses 1 GB due to no HIMEM). We then get the following oops after creating one or more snapshots: ksymoops 2.4.8 on i686 2.4.21. Options used -v /tmp/vmlinux (specified) -k /proc/ksyms (default) -l /proc/modules (default) -o /lib/modules/2.4.21/ (default) -m /boot/System.map (specified) Warning (compare_maps): ksyms_base symbol acpi_fadt_R__ver_acpi_fadt not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol acpi_gbl_FADT_R__ver_acpi_gbl_FADT not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_get_best_pio_mode_R__ver_ide_get_best_pio_mode not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_pci_register_driver_R__ver_ide_pci_register_driver not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_pci_unregister_driver_R__ver_ide_pci_unregister_driver not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_pio_timings_R__ver_ide_pio_timings not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_set_xfer_rate_R__ver_ide_set_xfer_rate not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_setup_pci_device_R__ver_ide_setup_pci_device not found in vmlinux. Ignoring ksyms_base entry Warning (compare_maps): ksyms_base symbol ide_setup_pci_devices_R__ver_ide_setup_pci_devices not found in vmlinux. Ignoring ksyms_base entry Unable to handle kernel NULL pointer dereference at virtual address 00000000 c0286b55 *pde = 00000000 Oops: 0002 deadman multipath md bonding1 bonding cpqci cpqhealth cpqrom e100 lpfcdd CPU: 0 EIP: 0010:[<c0286b55>] Not tainted Using defaults from ksymoops -t elf32-i386 -a i386 EFLAGS: 00010202 eax: 00000000 ebx: f9083e50 ecx: f8acbbf0 edx: f8aa19b0 esi: 0140e500 edi: f3967000 ebp: 00004400 esp: f7e17e58 ds: 0018 es: 0018 ss: 0018 Process kswapd (pid: 11, stackpage=f7e17000) Stack: f3967000 f3967170 f3e3e770 01d50000 00000080 00000001 00000058 00000000 c0283581 f7e17eb6 f7e17eb8 01400180 f3967000 00003a01 e5187800 013fe3d8 01d50000 00000020 00000200 f3e3e600 f3c4a000 01400180 000013f0 44001000 Call Trace: [<c0283581>] [<c0283667>] [<c021892c>] [<c02189a1>] [<c013a6ac>] [<c013a85f>] [<c0138c70>] [<c012f812>] [<c012fb10>] [<c012fb7c>] [<c012fc81>] [<c012fce6>] [<c012fdfd>] [<c0105684>] Code: 89 10 c7 01 00 00 00 00 c7 41 04 00 00 00 00 8b 03 89 48 04 >>EIP; c0286b55 <lvm_snapshot_remap_block+a9/f8> <===== >>ebx; f9083e50 <[deadman].bss.end+654695/ffed2841> >>ecx; f8acbbf0 <[deadman].bss.end+9c435/ffed2841> >>edx; f8aa19b0 <[deadman].bss.end+721f5/ffed2841> >>edi; f3967000 <_end+33564cdc/38492cdc> >>esp; f7e17e58 <_end+37a15b34/38492cdc> Trace; c0283581 <lvm_map+3b9/490> Trace; c0283667 <lvm_make_request_fn+f/1c> Trace; c021892c <generic_make_request+120/130> Trace; c02189a1 <submit_bh+65/80> Trace; c013a6ac <sync_page_buffers+94/ac> Trace; c013a85f <try_to_free_buffers+19b/1c4> Trace; c0138c70 <try_to_release_page+44/48> Trace; c012f812 <shrink_cache+24e/3cc> Trace; c012fb10 <shrink_caches+78/a8> Trace; c012fb7c <try_to_free_pages_zone+3c/5c> Trace; c012fc81 <kswapd_balance_pgdat+41/8c> Trace; c012fce6 <kswapd_balance+1a/30> Trace; c012fdfd <kswapd+99/b4> Trace; c0105684 <arch_kernel_thread+28/38> Code; c0286b55 <lvm_snapshot_remap_block+a9/f8> 00000000 <_EIP>: Code; c0286b55 <lvm_snapshot_remap_block+a9/f8> <===== 0: 89 10 mov %edx,(%eax) <===== Code; c0286b57 <lvm_snapshot_remap_block+ab/f8> 2: c7 01 00 00 00 00 movl $0x0,(%ecx) Code; c0286b5d <lvm_snapshot_remap_block+b1/f8> 8: c7 41 04 00 00 00 00 movl $0x0,0x4(%ecx) Code; c0286b64 <lvm_snapshot_remap_block+b8/f8> f: 8b 03 mov (%ebx),%eax Code; c0286b66 <lvm_snapshot_remap_block+ba/f8> 11: 89 48 04 mov %ecx,0x4(%eax) 9 warnings issued. Results may not be reliable. It looks like we are trying to shrink the buffer cache to release more memory for the block remap. This then oopses. Anyone have any idea how to fix this? Andrew -- =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Andrew Patterson Voice: (970) 898-3261 Hewlett-Packard Company Email: andrew@fc.hp.com ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2004-01-21 5:35 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2004-01-05 4:36 [linux-lvm] Oops when running snapshots Steve McIntyre 2004-01-12 9:36 ` Steve McIntyre 2004-01-14 8:37 ` Andrew Patterson 2004-01-21 5:35 ` Steve McIntyre -- strict thread matches above, loose matches on Subject: below -- 2004-01-12 10:27 Little, Chris 2003-07-09 17:27 Andrew Patterson 2003-07-10 4:34 ` Heinz J . Mauelshagen 2003-07-10 12:28 ` Andrew Patterson 2003-08-15 17:16 ` Andrew Patterson
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.