* [linux-lvm] Problems with snapshots on sparc64
@ 2003-05-28 19:09 Daniel Bayer
2003-06-02 4:50 ` Heinz J . Mauelshagen
0 siblings, 1 reply; 6+ messages in thread
From: Daniel Bayer @ 2003-05-28 19:09 UTC (permalink / raw)
To: linux-lvm
Hello
I have a E250 running Debian/stable with Linux 2.4.20 (vanilla kernel,
except for xfs). Three weeks ago I created a LV with an xfs on it. I
also created a backupscript that uses snapshots. For two weeks
everything worked fine.
Then I got oopses during the backup (mainly from kupdated). After that
the system was rebooted. I had to boot with init=/bin/sh since the
system freezed while mounting the xfs partitions (another story). I
did a vgscan and a vgchange. After that I did a
"lvdisplay /dev/vg1/lv_snapshot" and got the usual output and 3 times
the error "Trying to vfree() nonexistent vm area (addr)". Then I tried
"lvdisplay /dev/vg1/lv" and also got the vfree()-error and a
Segmention-Fault after the line that says that there is a snapshot of
this LV. I removed the snapshot and "lvdisplay /dev/vg1/lv" worked
fine. After that I could bring the whole system in a working state.
I could reproduce this.
The address in the vfree()-error message only changes after a reboot.
I also found older vfree()-errors in the log.
The Backtrace of the oops always ends in
| Trace; 005143e8 <lvm_map+488/580>
| Trace; 005144e8 <lvm_make_request_fn+8/40>
Another related problem is, that "cat /proc/lvm/global" in the most
cases doesn't output the usual data but some random memory. Its really
odd. But if I try often enough I sometimes also get the right output
(and no, I didn't calculated the probability). The other files in
/proc/lvm work fine.
Since I moved the important stuff to another system I can do some
tests and provide further informations.
Daniel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [linux-lvm] Problems with snapshots on sparc64
2003-05-28 19:09 [linux-lvm] Problems with snapshots on sparc64 Daniel Bayer
@ 2003-06-02 4:50 ` Heinz J . Mauelshagen
2003-06-02 8:11 ` Daniel Bayer
0 siblings, 1 reply; 6+ messages in thread
From: Heinz J . Mauelshagen @ 2003-06-02 4:50 UTC (permalink / raw)
To: linux-lvm
Daniel,
which LVM version (kernel and tools) and do you have HIGHMEM configured
(IOW: more than 960M memory) ?
There's flaws with Linux 2.4.20 highmem which show with snapshots :(
On Thu, May 29, 2003 at 02:10:29AM +0200, Daniel Bayer wrote:
> Hello
>
> I have a E250 running Debian/stable with Linux 2.4.20 (vanilla kernel,
> except for xfs). Three weeks ago I created a LV with an xfs on it. I
> also created a backupscript that uses snapshots. For two weeks
> everything worked fine.
>
> Then I got oopses during the backup (mainly from kupdated). After that
> the system was rebooted. I had to boot with init=/bin/sh since the
> system freezed while mounting the xfs partitions (another story). I
> did a vgscan and a vgchange. After that I did a
> "lvdisplay /dev/vg1/lv_snapshot" and got the usual output and 3 times
> the error "Trying to vfree() nonexistent vm area (addr)". Then I tried
> "lvdisplay /dev/vg1/lv" and also got the vfree()-error and a
> Segmention-Fault after the line that says that there is a snapshot of
> this LV. I removed the snapshot and "lvdisplay /dev/vg1/lv" worked
> fine. After that I could bring the whole system in a working state.
> I could reproduce this.
>
> The address in the vfree()-error message only changes after a reboot.
> I also found older vfree()-errors in the log.
>
> The Backtrace of the oops always ends in
> | Trace; 005143e8 <lvm_map+488/580>
> | Trace; 005144e8 <lvm_make_request_fn+8/40>
>
> Another related problem is, that "cat /proc/lvm/global" in the most
> cases doesn't output the usual data but some random memory. Its really
> odd. But if I try often enough I sometimes also get the right output
> (and no, I didn't calculated the probability). The other files in
> /proc/lvm work fine.
>
> Since I moved the important stuff to another system I can do some
> tests and provide further informations.
>
>
> Daniel
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
--
Regards,
Heinz -- The LVM Guy --
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
Mauelshagen@Sistina.com +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [linux-lvm] Problems with snapshots on sparc64
2003-06-02 4:50 ` Heinz J . Mauelshagen
@ 2003-06-02 8:11 ` Daniel Bayer
2003-06-02 10:15 ` Heinz J . Mauelshagen
0 siblings, 1 reply; 6+ messages in thread
From: Daniel Bayer @ 2003-06-02 8:11 UTC (permalink / raw)
To: linux-lvm
Hallo
On Mon, Jun 02, 2003 at 11:39:39AM +0200, Heinz J . Mauelshagen wrote:
> which LVM version (kernel and tools)
kernel: 1.0.5+(22/07/2002)
tools: 1.0.4-4
> and do you have HIGHMEM configured
> (IOW: more than 960M memory) ?
Its a 64Bit architecture. There is no need for things like HIGHMEM.
But yes, the system has 2GB RAM.
> There's flaws with Linux 2.4.20 highmem which show with snapshots :(
But this problem:
> > Another related problem is, that "cat /proc/lvm/global" in the most
> > cases doesn't output the usual data but some random memory. Its really
> > odd. But if I try often enough I sometimes also get the right output
> > (and no, I didn't calculated the probability). The other files in
> > /proc/lvm work fine.
appears also if there are no snapshots at all. (This time I only had
to try 2 times to get the version-string.)
Tomorrow I will have some time to update the kernel and the tools to
the current versions.
Daniel
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [linux-lvm] Problems with snapshots on sparc64
2003-06-02 8:11 ` Daniel Bayer
@ 2003-06-02 10:15 ` Heinz J . Mauelshagen
2003-06-03 14:38 ` Daniel Bayer
0 siblings, 1 reply; 6+ messages in thread
From: Heinz J . Mauelshagen @ 2003-06-02 10:15 UTC (permalink / raw)
To: linux-lvm
On Mon, Jun 02, 2003 at 03:12:32PM +0200, Daniel Bayer wrote:
> Hallo
>
> On Mon, Jun 02, 2003 at 11:39:39AM +0200, Heinz J . Mauelshagen wrote:
> > which LVM version (kernel and tools)
>
> kernel: 1.0.5+(22/07/2002)
> tools: 1.0.4-4
>
> > and do you have HIGHMEM configured
> > (IOW: more than 960M memory) ?
>
> Its a 64Bit architecture. There is no need for things like HIGHMEM.
> But yes, the system has 2GB RAM.
Oh, overlooked that in your email :(
>
> > There's flaws with Linux 2.4.20 highmem which show with snapshots :(
>
> But this problem:
> > > Another related problem is, that "cat /proc/lvm/global" in the most
> > > cases doesn't output the usual data but some random memory. Its really
> > > odd. But if I try often enough I sometimes also get the right output
> > > (and no, I didn't calculated the probability). The other files in
> > > /proc/lvm work fine.
>
> appears also if there are no snapshots at all. (This time I only had
> to try 2 times to get the version-string.)
>
> Tomorrow I will have some time to update the kernel and the tools to
> the current versions.
Yes, my recommendation anyway.
>
>
> Daniel
>
> _______________________________________________
> linux-lvm mailing list
> linux-lvm@sistina.com
> http://lists.sistina.com/mailman/listinfo/linux-lvm
> read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/
--
Regards,
Heinz -- The LVM Guy --
*** Software bugs are stupid.
Nevertheless it needs not so stupid people to solve them ***
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Heinz Mauelshagen Sistina Software Inc.
Senior Consultant/Developer Am Sonnenhang 11
56242 Marienrachdorf
Germany
Mauelshagen@Sistina.com +49 2626 141200
FAX 924446
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: [linux-lvm] Problems with snapshots on sparc64
2003-06-02 10:15 ` Heinz J . Mauelshagen
@ 2003-06-03 14:38 ` Daniel Bayer
2003-06-04 10:53 ` [linux-lvm] LVM + software Raid 5 : freeze when using snapshots Serge Rossi
0 siblings, 1 reply; 6+ messages in thread
From: Daniel Bayer @ 2003-06-03 14:38 UTC (permalink / raw)
To: linux-lvm
On Mon, Jun 02, 2003 at 05:05:07PM +0200, Heinz J . Mauelshagen wrote:
> On Mon, Jun 02, 2003 at 03:12:32PM +0200, Daniel Bayer wrote:
> > But this problem:
> > > > Another related problem is, that "cat /proc/lvm/global" in the most
> > > > cases doesn't output the usual data but some random memory. Its really
> > > > odd. But if I try often enough I sometimes also get the right output
> > > > (and no, I didn't calculated the probability). The other files in
> > > > /proc/lvm work fine.
> >
> > appears also if there are no snapshots at all. (This time I only had
> > to try 2 times to get the version-string.)
I just did the update of the kernel (1.0.7(28/03/2003)) and tools
(1.0.7-3).
The problem with /proc/lvm/global still appears:
| tau:~ # hexdump -C < /proc/lvm/global
| 00000000 00 06 d9 84 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| 00000010 00 06 d9 40 00 00 00 00 00 04 f4 08 00 00 00 00 |...@............|
| 00000020 00 06 e6 fc 00 00 00 00 00 06 da bc 00 00 00 00 |................|
| 00000030 00 06 f1 70 00 00 00 00 00 06 f0 94 00 00 00 00 |...p............|
| 00000040 00 06 db f0 00 00 00 00 00 06 f5 88 00 00 00 00 |................|
| 00000050 00 06 d8 a0 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| 00000060 00 06 e6 a0 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| 00000070 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| 00000080 00 00 00 00 00 00 00 00 00 06 d9 08 00 00 00 00 |................|
| 00000090 00 00 00 00 00 00 00 00 00 06 ea dc 00 00 00 00 |................|
| 000000a0 00 00 00 00 00 00 00 00 00 06 fb cc 00 00 00 00 |................|
| 000000b0 00 06 f9 88 00 00 00 00 00 06 f5 88 00 00 00 00 |................|
| 000000c0 00 07 b0 9c 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| 000000d0 00 06 db 50 00 00 00 00 00 06 c7 d8 00 00 00 00 |...P............|
| 000000e0 00 06 f3 bc 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| 000000f0 00 00 00 00 00 00 00 00 00 06 d9 84 00 00 00 00 |................|
| 00000100 00 00 00 00 00 00 00 00 00 06 f5 70 00 00 00 00 |...........p....|
| 00000110 00 06 d8 68 00 00 00 00 00 06 fc e8 00 00 00 00 |...h............|
| 00000120 00 06 e6 c4 00 00 00 00 00 06 da bc 00 00 00 00 |................|
| 00000130 00 06 f1 70 00 00 00 00 00 06 f0 94 00 00 00 00 |...p............|
| 00000140 00 06 db f0 00 00 00 00 00 06 f5 88 00 00 00 00 |................|
| 00000150 00 00 00 00 00 00 00 00 00 06 c1 e0 00 00 00 00 |................|
| 00000160 00 06 e7 38 00 00 00 00 00 06 d2 a4 00 00 00 00 |...8............|
| 00000170 00 06 d4 1c 00 00 00 00 00 06 c1 5c 00 00 00 00 |...........\....|
| 00000180 00 06 fc 88 00 00 00 00 00 06 d9 08 00 00 00 00 |................|
| 00000190 00 00 00 00 00 00 00 00 00 06 ea dc 00 00 00 00 |................|
| 000001a0 00 00 00 00 00 00 00 00 00 06 f8 94 00 00 00 00 |................|
| 000001b0 00 06 f9 88 00 00 00 00 00 06 f5 88 00 00 00 00 |................|
| 000001c0 00 06 d8 50 00 00 00 00 00 04 f3 00 00 00 00 00 |...P............|
| 000001d0 00 06 db 50 00 00 00 00 00 06 f4 94 00 00 00 00 |...P............|
| 000001e0 00 06 f3 bc 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| 000001f0 00 00 00 00 00 00 00 00 00 06 eb 88 00 00 00 00 |................|
| 00000200 00 00 00 00 00 00 00 00 00 06 e9 60 00 00 00 00 |...........`....|
| 00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
| *
| 000003b0 00 00 00 00 00 00 00 |.......|
| 000003b7
| tau:~ #
At the moment there are four or five outputs that appears often.
Others are more seldom. Sometimes there are strings in the output,
that (at least I believe so) belong to some userspace programm.
If I do a "lvdisplay" on a snapshot I still get the vfree-error:
| Jun 3 20:58:16 tau kernel: Trying to vfree() nonexistent vm area (00000001401a8000)
| Jun 3 20:58:45 tau last message repeated 5 times
As I'm only logged in remotely I wont try to crash the system now.
Daniel
^ permalink raw reply [flat|nested] 6+ messages in thread
* [linux-lvm] LVM + software Raid 5 : freeze when using snapshots.
2003-06-03 14:38 ` Daniel Bayer
@ 2003-06-04 10:53 ` Serge Rossi
0 siblings, 0 replies; 6+ messages in thread
From: Serge Rossi @ 2003-06-04 10:53 UTC (permalink / raw)
To: linux-lvm
I'm using two IBM file servers (Samba, NFS...) running RH 7.2 (with
latest updates) and LVM over software Raid 5.
RH Kernel 2.4.20-13.7 (Including LVM 1.0.3)
lvm tools and lib 1.0.5
Hardware : Dual Xeon MP (HT activated) + 1 GB Ram
Each server have 14 * 73 GB Disks for the datas.
md0 and md1 are RAID 5 units of 7 disks each, both included in a VG.
The VG is subdivded into several 100 GB LV and some free space for
snapshots.
Without snapshots, everything runs perfectly : 2 months without problems
with 1500 Windows PCs (SMB clients) and 500 SGI Workstations (NFS
clients). Load on the servers is between 0.1 and 1.0.
We tried to snapshot 3 LV each night. After 2 or 3 days (6 or 9
snapshots stored in the VG), the load on the servers was between 2.0 and
3.0 but sometimes one of the servers stop responding.
The kernel is still responsive to ping or Sysrq but not on the command
line, even on the console.
No activity at all on the disks.
Sysrq S only syncs the system disk (not in the VG) but did not sync the
LVs. Reboot is the only way out.
Nothing special in messages (no Oops).
We removed the snapshots and the servers are stable again.
I have not tried to test what is the number of snapshots required to
freeze the servers (difficult on production servers !!!).
Did anybody have any idea on this problem ?
Are the servers too weak to handle the load caused by the snapshots ?
--
----------------------------------------------------------------------
Serge Rossi - System & Network Architect Tel : +33 (0)1 34 55 91 14
iDVU - Renault Fax : +33 (0)1 34 55 92 25
----------------------------------------------------------------------
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2003-06-04 10:53 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2003-05-28 19:09 [linux-lvm] Problems with snapshots on sparc64 Daniel Bayer
2003-06-02 4:50 ` Heinz J . Mauelshagen
2003-06-02 8:11 ` Daniel Bayer
2003-06-02 10:15 ` Heinz J . Mauelshagen
2003-06-03 14:38 ` Daniel Bayer
2003-06-04 10:53 ` [linux-lvm] LVM + software Raid 5 : freeze when using snapshots Serge Rossi
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox