All of lore.kernel.org
 help / color / mirror / Atom feed
* [linux-lvm] Oops when running snapshots
@ 2003-07-09 17:27 Andrew Patterson
  2003-07-10  4:34 ` Heinz J . Mauelshagen
  0 siblings, 1 reply; 9+ messages in thread
From: Andrew Patterson @ 2003-07-09 17:27 UTC (permalink / raw)
  To: linux-lvm

Ran a load test last night where we created snapshots two snapshots
every 2 hours under heavy file system load.  We got the following oops:


Unable to handle kernel NULL pointer dereference at virtual address
00000000
802895f5
*pde = 55bd7001
Oops: 0002
CPU:    2
EIP:    0010:[<802895f5>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010206
eax: 00000000   ebx: f8ab2118   ecx: f8a79048   edx: f8a79030
esi: 00810180   edi: b4db0000   ebp: 00000820   esp: a2765c4c
ds: 0018   es: 0018   ss: 0018
Process vsftpd (pid: 7309, stackpage=a2765000)
Stack: b4db0000 b4db0170 f69b3170 00ea8000 00000080 00000006 00000070
00000000
       80286031 a2765caa a2765cac 00810180 b4db0000 00003a02 e338c840
00800070
      00ea8000 00000000 00000000 f69b3000 f68a0000 00810180 00000800
08208cbc
Call Trace: [<80286031>]  [<80286117>]  [<80219ccc>]  [<80219d41>] 
[<801d209f>]
  [<801d26eb>]  [<801d2ac7>]  [<801d2b2a>]  [<801387fd>]  [<801d2952>] 
[<80139b
1d327e>]  [<80137807>]  [<80106f27>]
Code: 89 10 c7 01 00 00 00 00 c7 41 04 00 00 00 00 8b 03 89 48 04


>>EIP; 802895f5 <lvm_snapshot_remap_block+a9/f8>   <=====

>>ebx; f8ab2118 <[nfstracker].bss.end+490c5/ffe98fa9>
>>ecx; f8a79048 <[nfstracker].bss.end+fff5/ffe98fa9>
>>edx; f8a79030 <[nfstracker].bss.end+ffdd/ffe98fa9>
>>edi; b4db0000 <_end+3499065c/7847565c>
>>esp; a2765c4c <_end+223462a8/7847565c>

Trace; 80286031 <lvm_map+3b9/490>
Trace; 80286117 <lvm_make_request_fn+f/1c>
Trace; 80219ccc <generic_make_request+11c/12c>
Trace; 80219d41 <submit_bh+65/80>
Trace; 801d209f <submit_page+57/74>
Trace; 801d26eb <page_state_convert+40b/4a0>
Trace; 801d2ac7 <linvfs_writepage+3b/e0>
Trace; 801d2b2a <linvfs_writepage+9e/e0>
Trace; 801387fd <write_some_buffers+9d/14c>
Trace; 801d2952 <linvfs_get_block+1e/24>

Code;  802895f5 <lvm_snapshot_remap_block+a9/f8>
00000000 <_EIP>:
Code;  802895f5 <lvm_snapshot_remap_block+a9/f8>   <=====
   0:   89 10                     mov    %edx,(%eax)   <=====
Code;  802895f7 <lvm_snapshot_remap_block+ab/f8>
   2:   c7 01 00 00 00 00         movl   $0x0,(%ecx)
Code;  802895fd <lvm_snapshot_remap_block+b1/f8>
Trace; 80219d41 <submit_bh+65/80>
Trace; 801d209f <submit_page+57/74>
Trace; 801d26eb <page_state_convert+40b/4a0>
Trace; 801d2ac7 <linvfs_writepage+3b/e0>
Trace; 801d2b2a <linvfs_writepage+9e/e0>
Trace; 801387fd <write_some_buffers+9d/14c>
Trace; 801d2952 <linvfs_get_block+1e/24>

Code;  802895f5 <lvm_snapshot_remap_block+a9/f8>
00000000 <_EIP>:
Code;  802895f5 <lvm_snapshot_remap_block+a9/f8>   <=====
   0:   89 10                     mov    %edx,(%eax)   <=====
Code;  802895f7 <lvm_snapshot_remap_block+ab/f8>
   2:   c7 01 00 00 00 00         movl   $0x0,(%ecx)
Code;  802895fd <lvm_snapshot_remap_block+b1/f8>
   8:   c7 41 04 00 00 00 00      movl   $0x0,0x4(%ecx)
Code;  80289604 <lvm_snapshot_remap_block+b8/f8>
   f:   8b 03                     mov    (%ebx),%eax
Code;  80289606 <lvm_snapshot_remap_block+ba/f8>
  11:   89 48 04                  mov    %ecx,0x4(%eax)


10 warnings issued.  Results may not be reliable.

This was run on a 2.4.21 system using LVM 1.0.7.  The system has 2GB of
memory and is configured with HIGHME64G.  We are running the latest
2.4.21 XFS and are using xfs_freeze to quiesce the file system.  We are
not using the VFS_LOCK patch.

Anyone have any ideas what may be causing this?

Thanks,

Andrew

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
   Andrew Patterson                          Voice:  (970) 898-3261
   Hewlett-Packard Company                   Email:  andrew@fc.hp.com

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: [linux-lvm] Oops when running snapshots
  2003-07-09 17:27 Andrew Patterson
@ 2003-07-10  4:34 ` Heinz J . Mauelshagen
  2003-07-10 12:28   ` Andrew Patterson
  2003-08-15 17:16   ` Andrew Patterson
  0 siblings, 2 replies; 9+ messages in thread
From: Heinz J . Mauelshagen @ 2003-07-10  4:34 UTC (permalink / raw)
  To: linux-lvm

On Wed, Jul 09, 2003 at 04:26:36PM -0600, Andrew Patterson wrote:
> Ran a load test last night where we created snapshots two snapshots
> every 2 hours under heavy file system load.  We got the following oops:
> 
> 
> Unable to handle kernel NULL pointer dereference at virtual address
> 00000000
> 802895f5
<SNIP>
>   11:   89 48 04                  mov    %ecx,0x4(%eax)
> 
> 
> 10 warnings issued.  Results may not be reliable.
> 
> This was run on a 2.4.21 system using LVM 1.0.7.  The system has 2GB of
> memory and is configured with HIGHME64G.  We are running the latest
> 2.4.21 XFS and are using xfs_freeze to quiesce the file system.  We are
> not using the VFS_LOCK patch.
> 
> Anyone have any ideas what may be causing this?

VM problems ? (we had some reports ITR)
Do you have a chance to retest without HIGHMEM to see if it still fails ?

> 
> Thanks,
> 
> Andrew
> 
> -- 
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>    Andrew Patterson                          Voice:  (970) 898-3261
>    Hewlett-Packard Company                   Email:  andrew@fc.hp.com
> 
> 
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

-- 

Regards,
Heinz    -- The LVM Guy --

*** Software bugs are stupid.
    Nevertheless it needs not so stupid people to solve them ***

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

Heinz Mauelshagen                                 Sistina Software Inc.
Senior Consultant/Developer                       Am Sonnenhang 11
                                                  56242 Marienrachdorf
                                                  Germany
Mauelshagen@Sistina.com                           +49 2626 141200
                                                       FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: [linux-lvm] Oops when running snapshots
  2003-07-10  4:34 ` Heinz J . Mauelshagen
@ 2003-07-10 12:28   ` Andrew Patterson
  2003-08-15 17:16   ` Andrew Patterson
  1 sibling, 0 replies; 9+ messages in thread
From: Andrew Patterson @ 2003-07-10 12:28 UTC (permalink / raw)
  To: linux-lvm; +Cc: Heinz J . Mauelshagen

On Thu, 2003-07-10 at 03:32, Heinz J . Mauelshagen wrote:
> On Wed, Jul 09, 2003 at 04:26:36PM -0600, Andrew Patterson wrote:
> > Ran a load test last night where we created snapshots two snapshots
> > every 2 hours under heavy file system load.  We got the following oops:
> > 
> > 
> > Unable to handle kernel NULL pointer dereference at virtual address
> > 00000000
> > 802895f5
> <SNIP>
> >   11:   89 48 04                  mov    %ecx,0x4(%eax)
> > 
> > 
> > 10 warnings issued.  Results may not be reliable.
> > 
> > This was run on a 2.4.21 system using LVM 1.0.7.  The system has 2GB of
> > memory and is configured with HIGHME64G.  We are running the latest
> > 2.4.21 XFS and are using xfs_freeze to quiesce the file system.  We are
> > not using the VFS_LOCK patch.
> > 
> > Anyone have any ideas what may be causing this?
> 
> VM problems ? (we had some reports ITR)
> Do you have a chance to retest without HIGHMEM to see if it still fails ?
> 

I looked through mailing list archives about this.  Most the of himem
problems resulted in vmmalloc.c calls in the oops.  I'll try it again
without himem to see if this helps.

Thanks,

Andrew

> > 
> > Thanks,

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: [linux-lvm] Oops when running snapshots
  2003-07-10  4:34 ` Heinz J . Mauelshagen
  2003-07-10 12:28   ` Andrew Patterson
@ 2003-08-15 17:16   ` Andrew Patterson
  1 sibling, 0 replies; 9+ messages in thread
From: Andrew Patterson @ 2003-08-15 17:16 UTC (permalink / raw)
  To: linux-lvm; +Cc: Heinz J . Mauelshagen

On Thu, 2003-07-10 at 03:32, Heinz J . Mauelshagen wrote:
> On Wed, Jul 09, 2003 at 04:26:36PM -0600, Andrew Patterson wrote:
> > Ran a load test last night where we created snapshots two snapshots
> > every 2 hours under heavy file system load.  We got the following oops:
> > 
> > 
> > Unable to handle kernel NULL pointer dereference at virtual address
> > 00000000
> > 802895f5
> <SNIP>
> >   11:   89 48 04                  mov    %ecx,0x4(%eax)
> > 
> > 
> > 10 warnings issued.  Results may not be reliable.
> > 
> > This was run on a 2.4.21 system using LVM 1.0.7.  The system has 2GB of
> > memory and is configured with HIGHME64G.  We are running the latest
> > 2.4.21 XFS and are using xfs_freeze to quiesce the file system.  We are
> > not using the VFS_LOCK patch.
> > 
> > Anyone have any ideas what may be causing this?
> 
> VM problems ? (we had some reports ITR)
> Do you have a chance to retest without HIGHMEM to see if it still fails ?
> 

I finally had a chance to rerun with HIMEM turned off.  I also made one
other change.  We are now using the linux-2.4.21-VFS-lock.patch I found
in the device mapper CVS archives instead of using xfs_freeze (we
thought we were running into some race condition with xfs_freeze.  Some
other details:

We are running multiple snapshots on the same volume.  The snapshots and
data set changes are very large (constantly changing up to 12 GB using
random and sequential read/writes). 

We have 2 GB of RAM on the system (the kernel only uses 1 GB due to no
HIMEM).

We then get the following oops after creating one or more snapshots:


ksymoops 2.4.8 on i686 2.4.21.  Options used
     -v /tmp/vmlinux (specified)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.21/ (default)
     -m /boot/System.map (specified)

Warning (compare_maps): ksyms_base symbol acpi_fadt_R__ver_acpi_fadt not found in vmlinux.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol acpi_gbl_FADT_R__ver_acpi_gbl_FADT not found in vmlinux.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol ide_get_best_pio_mode_R__ver_ide_get_best_pio_mode not found in vmlinux.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol ide_pci_register_driver_R__ver_ide_pci_register_driver not found in vmlinux.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol ide_pci_unregister_driver_R__ver_ide_pci_unregister_driver not found in vmlinux.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol ide_pio_timings_R__ver_ide_pio_timings not found in vmlinux.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol ide_set_xfer_rate_R__ver_ide_set_xfer_rate not found in vmlinux.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol ide_setup_pci_device_R__ver_ide_setup_pci_device not found in vmlinux.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol ide_setup_pci_devices_R__ver_ide_setup_pci_devices not found in vmlinux.  Ignoring ksyms_base entry
Unable to handle kernel NULL pointer dereference at virtual address 00000000
c0286b55
*pde = 00000000
Oops: 0002
deadman multipath md bonding1 bonding cpqci cpqhealth cpqrom e100 lpfcdd
CPU:    0
EIP:    0010:[<c0286b55>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EFLAGS: 00010202
eax: 00000000   ebx: f9083e50   ecx: f8acbbf0   edx: f8aa19b0
esi: 0140e500   edi: f3967000   ebp: 00004400   esp: f7e17e58
ds: 0018   es: 0018   ss: 0018
Process kswapd (pid: 11, stackpage=f7e17000)
Stack: f3967000 f3967170 f3e3e770 01d50000 00000080 00000001 00000058 00000000
       c0283581 f7e17eb6 f7e17eb8 01400180 f3967000 00003a01 e5187800 013fe3d8
       01d50000 00000020 00000200 f3e3e600 f3c4a000 01400180 000013f0 44001000
Call Trace: [<c0283581>]  [<c0283667>]  [<c021892c>]  [<c02189a1>]  [<c013a6ac>]  [<c013a85f>]  [<c0138c70>]  [<c012f812>]  [<c012fb10>]  [<c012fb7c>]  [<c012fc81>]  [<c012fce6>]  [<c012fdfd>]  [<c0105684>]
Code: 89 10 c7 01 00 00 00 00 c7 41 04 00 00 00 00 8b 03 89 48 04


>>EIP; c0286b55 <lvm_snapshot_remap_block+a9/f8>   <=====

>>ebx; f9083e50 <[deadman].bss.end+654695/ffed2841>
>>ecx; f8acbbf0 <[deadman].bss.end+9c435/ffed2841>
>>edx; f8aa19b0 <[deadman].bss.end+721f5/ffed2841>
>>edi; f3967000 <_end+33564cdc/38492cdc>
>>esp; f7e17e58 <_end+37a15b34/38492cdc>

Trace; c0283581 <lvm_map+3b9/490>
Trace; c0283667 <lvm_make_request_fn+f/1c>
Trace; c021892c <generic_make_request+120/130>
Trace; c02189a1 <submit_bh+65/80>
Trace; c013a6ac <sync_page_buffers+94/ac>
Trace; c013a85f <try_to_free_buffers+19b/1c4>
Trace; c0138c70 <try_to_release_page+44/48>
Trace; c012f812 <shrink_cache+24e/3cc>
Trace; c012fb10 <shrink_caches+78/a8>
Trace; c012fb7c <try_to_free_pages_zone+3c/5c>
Trace; c012fc81 <kswapd_balance_pgdat+41/8c>
Trace; c012fce6 <kswapd_balance+1a/30>
Trace; c012fdfd <kswapd+99/b4>
Trace; c0105684 <arch_kernel_thread+28/38>

Code;  c0286b55 <lvm_snapshot_remap_block+a9/f8>
00000000 <_EIP>:
Code;  c0286b55 <lvm_snapshot_remap_block+a9/f8>   <=====
   0:   89 10                     mov    %edx,(%eax)   <=====
Code;  c0286b57 <lvm_snapshot_remap_block+ab/f8>
   2:   c7 01 00 00 00 00         movl   $0x0,(%ecx)
Code;  c0286b5d <lvm_snapshot_remap_block+b1/f8>
   8:   c7 41 04 00 00 00 00      movl   $0x0,0x4(%ecx)
Code;  c0286b64 <lvm_snapshot_remap_block+b8/f8>
   f:   8b 03                     mov    (%ebx),%eax
Code;  c0286b66 <lvm_snapshot_remap_block+ba/f8>
  11:   89 48 04                  mov    %ecx,0x4(%eax)


9 warnings issued.  Results may not be reliable. 

It looks like we are trying to shrink the buffer cache to release more memory 
for the block remap.  This then oopses.  Anyone have any idea how to fix this?

Andrew

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
   Andrew Patterson                          Voice:  (970) 898-3261
   Hewlett-Packard Company                   Email:  andrew@fc.hp.com

^ permalink raw reply	[flat|nested] 9+ messages in thread
* [linux-lvm] Oops when running snapshots
@ 2004-01-05  4:36 Steve McIntyre
  2004-01-12  9:36 ` Steve McIntyre
  0 siblings, 1 reply; 9+ messages in thread
From: Steve McIntyre @ 2004-01-05  4:36 UTC (permalink / raw)
  To: linux-lvm

Has there been any progress on the issue reported by Andrew Patterson
in

http://lists.sistina.com/pipermail/linux-lvm/2003-July/014422.html

???

I've got a brand new server that seems to have just shown the same
problem over the Christmas period: twin P4 Xeon, 1GB RAM (configured
for 4GB highmem), kernel 2.4.23. Oops message:

ksymoops 2.4.5 on i686 2.4.23.  Options used
     -V (default)
     -k /proc/ksyms (default)
     -l /proc/modules (default)
     -o /lib/modules/2.4.23/ (default)
     -m /boot/System.map-2.4.23-20031224 (specified)

No modules in ksyms, skipping objects
Warning (read_lsmod): no symbols in lsmod, is /proc/modules a valid lsmod file?
Warning (compare_maps): ksyms_base symbol IO_APIC_get_PCI_irq_vector_R__ver_IO_APIC_get_PCI_irq_vector not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __br_write_lock_R__ver___br_write_lock not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __br_write_unlock_R__ver___br_write_unlock not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __brlock_array_R__ver___brlock_array not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __global_cli_R__ver___global_cli not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __global_restore_flags_R__ver___global_restore_flags not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __global_save_flags_R__ver___global_save_flags not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol __global_sti_R__ver___global_sti not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol atomic_dec_and_lock_R__ver_atomic_dec_and_lock not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol cpu_data_R__ver_cpu_data not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol cpu_online_map_R__ver_cpu_online_map not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol create_bounce_R__ver_create_bounce not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol del_timer_sync_R__ver_del_timer_sync not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol drive_info_R__ver_drive_info not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol flush_tlb_page_R__ver_flush_tlb_page not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol global_irq_holder_R__ver_global_irq_holder not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol highmem_start_page_R__ver_highmem_start_page not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol ide_pci_register_host_proc_R__ver_ide_pci_register_host_proc not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol kernel_flag_cacheline_R__ver_kernel_flag_cacheline not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol kmap_high_R__ver_kmap_high not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol kmap_prot_R__ver_kmap_prot not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol kmap_pte_R__ver_kmap_pte not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol kunmap_high_R__ver_kunmap_high not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol set_cpus_allowed_R__ver_set_cpus_allowed not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol smp_call_function_R__ver_smp_call_function not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol smp_num_cpus_R__ver_smp_num_cpus not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol synchronize_irq_R__ver_synchronize_irq not found in System.map.  Ignoring ksyms_base entry
Warning (compare_maps): ksyms_base symbol tqueue_lock_R__ver_tqueue_lock not found in System.map.  Ignoring ksyms_base entry
Dec 29 03:04:59 cetus kernel: c0287ed5
Dec 29 03:04:59 cetus kernel: Oops: 0002
Dec 29 03:04:59 cetus kernel: CPU:    3
Dec 29 03:04:59 cetus kernel: EIP:    0010:[<c0287ed5>]    Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
Dec 29 03:04:59 cetus kernel: EFLAGS: 00010202
Dec 29 03:04:59 cetus kernel: eax: 00000000   ebx: f91e89a8   ecx: f8b09450   edx: f8b02778
Dec 29 03:04:59 cetus kernel: esi: 01fb8200   edi: d8858800   ebp: 00000831   esp: f6abfc3c
Dec 29 03:04:59 cetus kernel: ds: 0018   es: 0018   ss: 0018
Dec 29 03:04:59 cetus kernel: Process nfsd (pid: 286, stackpage=f6abf000)
Dec 29 03:04:59 cetus kernel: Stack: d8858800 d8858970 f7dee770 05a00000 00000080 00000002 00000070 00000000 
Dec 29 03:04:59 cetus kernel:        c0284951 f6abfc9a f6abfc9c 01fb8180 d8858800 00003a00 e5ddfa00 076800f0 
Dec 29 03:04:59 cetus kernel:        05a00000 dea60780 02dc00e8 f7dee600 f786d000 01fb8180 0003b400 08310000 
Dec 29 03:04:59 cetus kernel: Call Trace:    [<c0284951>] [<c0284a37>] [<c01e3b8c>] [<c01e3bf2>] [<c013d06c>]
Dec 29 03:04:59 cetus kernel:   [<c013d1a7>] [<c013b309>] [<c0130f72>] [<c01312e0>] [<c0131350>] [<c0131e78>]
Dec 29 03:04:59 cetus kernel:   [<c014e7fd>] [<c0132182>] [<c0131e2e>] [<c01294ce>] [<c0129be5>] [<c0129e3c>]
Dec 29 03:04:59 cetus kernel:   [<c012a4df>] [<c012a36c>] [<c0191f84>] [<c019714b>] [<c018e5d3>] [<c02fec67>]
Dec 29 03:04:59 cetus kernel:   [<c018e3bf>] [<c0105694>]
Dec 29 03:04:59 cetus kernel: Code: 89 10 c7 01 00 00 00 00 c7 41 04 00 00 00 00 8b 03 89 48 04 


>>EIP; c0287ed5 <lvm_snapshot_remap_block+a9/f8>   <=====

>>ebx; f91e89a8 <END_OF_CODE+38d5cae8/????>
>>ecx; f8b09450 <END_OF_CODE+3867d590/????>
>>edx; f8b02778 <END_OF_CODE+386768b8/????>
>>esi; 01fb8200 Before first symbol
>>edi; d8858800 <END_OF_CODE+183cc940/????>
>>ebp; 00000831 Before first symbol
>>esp; f6abfc3c <END_OF_CODE+36633d7c/????>

Trace; c0284951 <lvm_map+3b9/490>
Trace; c0284a37 <lvm_make_request_fn+f/1c>
Trace; c01e3b8c <generic_make_request+120/130>
Trace; c01e3bf2 <submit_bh+56/e0>
Trace; c013d06c <sync_page_buffers+94/ac>
Trace; c013d1a7 <try_to_free_buffers+123/148>
Trace; c013b309 <try_to_release_page+49/4c>
Trace; c0130f72 <shrink_cache+21e/3f4>
Trace; c01312e0 <shrink_caches+3c/48>
Trace; c0131350 <try_to_free_pages_zone+64/e4>
Trace; c0131e78 <balance_classzone+48/1c8>
Trace; c014e7fd <iget4+4d/e8>
Trace; c0132182 <__alloc_pages+18a/28c>
Trace; c0131e2e <_alloc_pages+16/18>
Trace; c01294ce <page_cache_read+7a/d0>
Trace; c0129be5 <generic_file_readahead+105/13c>
Trace; c0129e3c <do_generic_file_read+1f0/494>
Trace; c012a4df <generic_file_read+93/190>
Trace; c012a36c <file_read_actor+0/e0>
Trace; c0191f84 <nfsd_read+1bc/260>
Trace; c019714b <nfsd3_proc_read+127/184>
Trace; c018e5d3 <nfsd_dispatch+d3/19a>
Trace; c02fec67 <svc_process+28f/4f0>
Trace; c018e3bf <nfsd+1f3/334>
Trace; c0105694 <arch_kernel_thread+28/38>

Code;  c0287ed5 <lvm_snapshot_remap_block+a9/f8>
00000000 <_EIP>:
Code;  c0287ed5 <lvm_snapshot_remap_block+a9/f8>   <=====
   0:   89 10                     mov    %edx,(%eax)   <=====
Code;  c0287ed7 <lvm_snapshot_remap_block+ab/f8>
   2:   c7 01 00 00 00 00         movl   $0x0,(%ecx)
Code;  c0287edd <lvm_snapshot_remap_block+b1/f8>
   8:   c7 41 04 00 00 00 00      movl   $0x0,0x4(%ecx)
Code;  c0287ee4 <lvm_snapshot_remap_block+b8/f8>
   f:   8b 03                     mov    (%ebx),%eax
Code;  c0287ee6 <lvm_snapshot_remap_block+ba/f8>
  11:   89 48 04                  mov    %ecx,0x4(%eax)

Jan  2 08:50:33 cetus kernel: ACPI: LAPIC_NMI (acpi_id[0x00] polarity[0x1] trigger[0x3] lint[0x1])
Jan  2 08:50:33 cetus kernel: ACPI: LAPIC_NMI (acpi_id[0x01] polarity[0x1] trigger[0x3] lint[0x1])
Jan  2 08:50:33 cetus kernel: cpu: 0, clocks: 1329112, slice: 265822
Jan  2 08:50:33 cetus kernel: cpu: 1, clocks: 1329112, slice: 265822
Jan  2 08:50:33 cetus kernel: cpu: 3, clocks: 1329112, slice: 265822
Jan  2 08:50:33 cetus kernel: cpu: 2, clocks: 1329112, slice: 265822
Jan  2 08:50:33 cetus kernel: e1000: eth0 NIC Link is Up 100 Mbps Full Duplex

29 warnings issued.  Results may not be reliable.


-- 
Steve McIntyre, Plasmon                      smcintyre@software.plasmon.com
		Oh My God! They Killed init! You Bastards!
"Can't keep my eyes from the circling sky,                 
"Tongue-tied & twisted, Just an earth-bound misfit, I..."  

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: [linux-lvm] Oops when running snapshots
  2004-01-05  4:36 [linux-lvm] Oops when running snapshots Steve McIntyre
@ 2004-01-12  9:36 ` Steve McIntyre
  2004-01-14  8:37   ` Andrew Patterson
  0 siblings, 1 reply; 9+ messages in thread
From: Steve McIntyre @ 2004-01-12  9:36 UTC (permalink / raw)
  To: linux-lvm

On Mon, Jan 05, 2004 at 10:35:00AM +0000, Me @ Plasmon wrote:
>Has there been any progress on the issue reported by Andrew Patterson
>in
>
>http://lists.sistina.com/pipermail/linux-lvm/2003-July/014422.html
>
>???

Apparently not. Should I expect a response here, or ask on the main
kernel list?

-- 
Steve McIntyre, Plasmon                      smcintyre@software.plasmon.com
"Can't keep my eyes from the circling sky,                 
"Tongue-tied & twisted, Just an earth-bound misfit, I..."  

^ permalink raw reply	[flat|nested] 9+ messages in thread
* RE: [linux-lvm] Oops when running snapshots
@ 2004-01-12 10:27 Little, Chris
  0 siblings, 0 replies; 9+ messages in thread
From: Little, Chris @ 2004-01-12 10:27 UTC (permalink / raw)
  To: 'linux-lvm@sistina.com'

Personally, I have had no responses concerning snapshot issues.  Perhaps
snapshot is very low priority, or maybe I'm just really far back in the
queue.

> -----Original Message-----
> From: Steve McIntyre [mailto:smcintyre@software.plasmon.com]
> Sent: Monday, January 12, 2004 8:35 AM
> To: linux-lvm@sistina.com
> Subject: Re: [linux-lvm] Oops when running snapshots
> 
> 
> On Mon, Jan 05, 2004 at 10:35:00AM +0000, Me @ Plasmon wrote:
> >Has there been any progress on the issue reported by Andrew Patterson
> >in
> >
> >http://lists.sistina.com/pipermail/linux-lvm/2003-July/014422.html
> >
> >???
> 
> Apparently not. Should I expect a response here, or ask on the main
> kernel list?
> 
> -- 
> Steve McIntyre, Plasmon                      
> smcintyre@software.plasmon.com
> "Can't keep my eyes from the circling sky,                 
> "Tongue-tied & twisted, Just an earth-bound misfit, I..."  
> 
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
> 

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: [linux-lvm] Oops when running snapshots
  2004-01-12  9:36 ` Steve McIntyre
@ 2004-01-14  8:37   ` Andrew Patterson
  2004-01-21  5:35     ` Steve McIntyre
  0 siblings, 1 reply; 9+ messages in thread
From: Andrew Patterson @ 2004-01-14  8:37 UTC (permalink / raw)
  To: linux-lvm; +Cc: smcintyre

[-- Attachment #1: Type: text/plain, Size: 9222 bytes --]

On Mon, 2004-01-12 at 07:35, Steve McIntyre wrote:
> On Mon, Jan 05, 2004 at 10:35:00AM +0000, Me @ Plasmon wrote:
> >Has there been any progress on the issue reported by Andrew Patterson
> >in
> >
> >http://lists.sistina.com/pipermail/linux-lvm/2003-July/014422.html
> >
> >???
> 
> Apparently not. Should I expect a response here, or ask on the main
> kernel list?


We worked over this problem with Heinz on LVM 1.07.  One of our kernel
hackers (along with Heinz) came up with several possible solutions to a
race condition in the snapshot code.  They are presented below.  We
found that option #1 worked, but sometimes it could take a long time to
create multiple snapshots of the same LV under extremely heavy load. 
The first snapshot is fine, but the second or more can take hours. 
Option #2 created a huge performance penalty to writes on the original
LV (90-99%).  I don't remember that option #3 helped any, and we never
tried #4.  We settled with option #1 and solved the long snapshot
creation time by stopping I/O while creating the snapshot.

The LVM snapshot problem is the same one I had with the nfs export hash
table (classic reader/writer problem).  Read and write semaphores are
being
used to allow concurrent read access to the snapshot hash table while
allowing exclusive write access.  However, lvm_snapshot_remap_block ->
lvm_find_exception_table does the hash table optimization (i.e. writing
to the hash table) regardless of whether the read or write semaphore is
held.

The snapshot hash table has 128K entries, so there's
more potential for performance improvement. Here's the call tree for
lvm_find_exception_table:

lvm_find_exception_table
- lvm_snapshot_remap_block
-- __remap_snapshot
-- _remap_snapshot
-- lvm_map

lvm_map, _remap_snapshot, and __remap_snapshot all call
lvm_snapshot_remap_block.  

1) lvm_map 
lvm_map() {
   down_read(&lv->lv_lock)
   ...
   if (lvm_snapshot_remap_block(...))
   ...
   up_read(&lv->lv_lock)
}

This is bad, because lvm_snapshot_remap_block will call
lvm_find_exception_table, which could write to the hash table while only
holding the read semaphore during the optimization step.

2) _remap_snapshot + __remap_snapshot

lvm_map() {
   down_read(&lv->lv_lock)
   ...
   _remap_snapshot() {
     down_read(&lv->lock)  // again!
     r = lvm_snapshot_remap_block()
     up_read(&lv->lock)
     if (!r)
        __remap_snapshot() {
           down_write(&lv->lock)  // good idea, but we already hold
read!
              if (!lvm_snapshot_remap_block()) {}
           up_write(&lv->lock)
        }
   }
   up_read(&lv->lv_lock)
}

This is bad, because down_read(&lv->lock) is called twice unnecessarily.
Also, lvm_snapshot_remap_block will still possibly write to the hash
table
while only holding the read semaphore.  I don't know how
lvm_map->_remap_snapshot->__remap_snapshot works at all.  down_write
should
block until all the readers have released the semaphore.  Since this
kernel
process already holds the read semaphore, it should never get the chance
to
get the write semaphore.

Options to fix (included as attachments and inline):

1) don't do the hash table optimization in lvm_find_exception_table,
which
we tried last night.  No crashes, but slow.  Doesn't fix the "get the
write
semaphore while holding the read semaphore" problem in
__remap_snapshot().

--- lvm-snap.c~ Mon Aug 25 15:28:50 2003
+++ lvm-snap.c-erik     Thu Aug 28 13:33:28 2003
@@ -130,11 +130,13 @@ static inline lv_block_exception_t *lvm_
                exception = list_entry(next, lv_block_exception_t,
hash);
                if (exception->rsector_org == org_start &&
                    exception->rdev_org == org_dev) {
+#if 0
                        if (i) {
                                /* fun, isn't it? :) */
                                list_del(next);
                                list_add(next, hash_table);
                        }
+#endif
                        ret = exception;
                        break;
                }

2) clean up the locking: always get write semaphore before going into
lvm_snapshot_remap_block, realize we already hold the read semaphore:

--- lvm.c~      Fri Aug 29 13:56:26 2003
+++ lvm.c-erik  Fri Aug 29 14:08:30 2003
@@ -1183,6 +1183,7 @@ static void __remap_snapshot(kdev_t rdev
 {

        /* copy a chunk from the origin to a snapshot device */
+       up_read(&lv->lv_lock);
        down_write(&lv->lv_lock);

        /* we must redo lvm_snapshot_remap_block in order to avoid a
@@ -1192,6 +1193,7 @@ static void __remap_snapshot(kdev_t rdev
                lvm_write_COW_table_block(vg, lv);

        up_write(&lv->lv_lock);
+       down_read(&lv->lv_lock);
 }

 static inline void _remap_snapshot(kdev_t rdev, ulong rsector,
@@ -1200,9 +1202,11 @@ static inline void _remap_snapshot(kdev_
        int r;

        /* check to see if this chunk is already in the snapshot */
-       down_read(&lv->lv_lock);
-       r = lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv);
        up_read(&lv->lv_lock);
+       down_write(&lv->lv_lock);
+       r = lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv);
+       up_write(&lv->lv_lock);
+       down_read(&lv->lv_lock);

        if (!r)
                /* we haven't yet copied this block to the snapshot */
@@ -1340,9 +1344,16 @@ static int lvm_map(struct buffer_head *b
                goto out;

        if (lv->lv_access & LV_SNAPSHOT) {      /* remap snapshot */
+               up_read(&lv->lv_lock);
+               down_write(&lv->lv_lock);
                if (lvm_snapshot_remap_block(&rdev_map, &rsector_map,
-                                            pe_start, lv) < 0)
+                                            pe_start, lv) < 0) {
+                       up_write(&lv->lv_lock);
+                       down_read(&lv->lv_lock);
                        goto bad;
+               }
+               up_write(&lv->lv_lock);
+               down_read(&lv->lv_lock);

        } else if (rw == WRITE || rw == WRITEA) {       /* snapshot
origin
*/
                lv_t *snap;

3) clean up the locking: grab the write semaphore in lvm_map and don't
get any other semaphores.  This isn't recommended and I'm sure Heinz
will cry when he sees this.  This defeats much of the purpose of having
separate read and write semaphores, and lvm_map might will hold the
write semaphore for a long time.  This will kill concurrent access to a
volume, and will probably hurt performance.

--- lvm.c~      Fri Aug 29 13:56:26 2003
+++ lvm.c-erik  Fri Aug 29 14:16:01 2003
@@ -1183,7 +1183,6 @@ static void __remap_snapshot(kdev_t rdev
 {

        /* copy a chunk from the origin to a snapshot device */
-       down_write(&lv->lv_lock);

        /* we must redo lvm_snapshot_remap_block in order to avoid a
           race condition in the gap where no lock was held */
@@ -1191,7 +1190,6 @@ static void __remap_snapshot(kdev_t rdev
            !lvm_snapshot_COW(rdev, rsector, pe_start, rsector, vg, lv))
                lvm_write_COW_table_block(vg, lv);

-       up_write(&lv->lv_lock);
 }

 static inline void _remap_snapshot(kdev_t rdev, ulong rsector,
@@ -1200,9 +1198,7 @@ static inline void _remap_snapshot(kdev_
        int r;

        /* check to see if this chunk is already in the snapshot */
-       down_read(&lv->lv_lock);
        r = lvm_snapshot_remap_block(&rdev, &rsector, pe_start, lv);
-       up_read(&lv->lv_lock);

        if (!r)
                /* we haven't yet copied this block to the snapshot */
@@ -1253,7 +1249,7 @@ static int lvm_map(struct buffer_head *b
        lv_t *lv = vg_this->lv[LV_BLK(minor)];


-       down_read(&lv->lv_lock);
+       down_write(&lv->lv_lock);
        if (!(lv->lv_status & LV_ACTIVE)) {
                printk(KERN_ALERT
                       "%s - lvm_map: ll_rw_blk for inactive LV %s\n",
@@ -1327,7 +1323,7 @@ static int lvm_map(struct buffer_head *b
                if (_defer_extent(bh, rw, rdev_map,
                                  rsector_map, vg_this->pe_size)) {

-                       up_read(&lv->lv_lock);
+                       up_write(&lv->lv_lock);
                        return 0;
                }

@@ -1365,13 +1361,13 @@ static int lvm_map(struct buffer_head *b
       out:
        bh->b_rdev = rdev_map;
        bh->b_rsector = rsector_map;
-       up_read(&lv->lv_lock);
+       up_write(&lv->lv_lock);
        return 1;

       bad:
        if (bh->b_end_io)
                buffer_IO_error(bh);
-       up_read(&lv->lv_lock);
+       up_write(&lv->lv_lock);
        return -1;
 }                              /* lvm_map() */

4) in lvm_snapshot_remap_block, somehow determine if we hold the read
semaphore and magically turn it into a write semaphore before doing the
optimization.  I'm not going to attempt this one.


-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Andrew Patterson                Voice:  (970) 898-3261
Hewlett-Packard Company         Email:  andrew@fc.hp.com



[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

^ permalink raw reply	[flat|nested] 9+ messages in thread
* Re: [linux-lvm] Oops when running snapshots
  2004-01-14  8:37   ` Andrew Patterson
@ 2004-01-21  5:35     ` Steve McIntyre
  0 siblings, 0 replies; 9+ messages in thread
From: Steve McIntyre @ 2004-01-21  5:35 UTC (permalink / raw)
  To: andrew; +Cc: linux-lvm

On Tue, Jan 13, 2004 at 11:35:46AM -0700, Andrew Patterson wrote:
>
>We worked over this problem with Heinz on LVM 1.07.  One of our kernel
>hackers (along with Heinz) came up with several possible solutions to a
>race condition in the snapshot code.  They are presented below.  We
>found that option #1 worked, but sometimes it could take a long time to
>create multiple snapshots of the same LV under extremely heavy load. 
>The first snapshot is fine, but the second or more can take hours. 
>Option #2 created a huge performance penalty to writes on the original
>LV (90-99%).  I don't remember that option #3 helped any, and we never
>tried #4.  We settled with option #1 and solved the long snapshot
>creation time by stopping I/O while creating the snapshot.

Ok, thanks. We had another hard lockup overnight caused by this
problem. I've just applied #1; I'll let you know how it goes.

<snip>

>Options to fix (included as attachments and inline):
>
>1) don't do the hash table optimization in lvm_find_exception_table,
>which
>we tried last night.  No crashes, but slow.  Doesn't fix the "get the
>write
>semaphore while holding the read semaphore" problem in
>__remap_snapshot().
>
>--- lvm-snap.c~ Mon Aug 25 15:28:50 2003
>+++ lvm-snap.c-erik     Thu Aug 28 13:33:28 2003
>@@ -130,11 +130,13 @@ static inline lv_block_exception_t *lvm_
>                exception = list_entry(next, lv_block_exception_t,
>hash);
>                if (exception->rsector_org == org_start &&
>                    exception->rdev_org == org_dev) {
>+#if 0
>                        if (i) {
>                                /* fun, isn't it? :) */
>                                list_del(next);
>                                list_add(next, hash_table);
>                        }
>+#endif
>                        ret = exception;
>                        break;
>                }

-- 
Steve McIntyre, Plasmon                      smcintyre@software.plasmon.com
Getting a SCSI chain working is perfectly simple if you remember that there
must be exactly three terminations: one on one end of the cable, one on the
far end, and the goat, terminated over the SCSI chain with a silver-handled
knife whilst burning *black* candles. --- Anthony DeBoer

^ permalink raw reply	[flat|nested] 9+ messages in thread
end of thread, other threads:[~2004-01-21  5:35 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2004-01-05  4:36 [linux-lvm] Oops when running snapshots Steve McIntyre
2004-01-12  9:36 ` Steve McIntyre
2004-01-14  8:37   ` Andrew Patterson
2004-01-21  5:35     ` Steve McIntyre
  -- strict thread matches above, loose matches on Subject: below --
2004-01-12 10:27 Little, Chris
2003-07-09 17:27 Andrew Patterson
2003-07-10  4:34 ` Heinz J . Mauelshagen
2003-07-10 12:28   ` Andrew Patterson
2003-08-15 17:16   ` Andrew Patterson

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.