3.6.25 - Journal replayed back to 3 weeks ago

3.6.25 - Journal replayed back to 3 weeks ago

[Neil Robinson]

Hi,

this morning when I started up my notebook (running Windows XP) with a
VMware session running Gentoo, the boot sequence claimed that the
reiserfs drives had not been cleanly umounted (not true, I powered down
the usual way on Friday evening -- su to root and then issued the
poweroff command). It then replayed the journals of the two partitions
using reiserfs. When it finished and booted, it was as if my entire
machine had stepped back in time by 3 weeks or so (to around the 23rd of
December). Since then I had installed and built openoffice, emacs, and
numerous other bits and pieces. I also lost all of the email that was
living in the courier-imap server.

I am *very* concerned about this behaviour. I have successfully
restarted, booted, etc. literally dozens of times since mid-December. I
have now just installed a software RAID using RAID 5 on Gentoo and using
reiserfs for a fairly large system (250GB on 8 SCSI U160 drives) with an
available hot spare and a tape backup unit. Losing a few weeks of
relatively insignificant changes is nothing compared with possibly
losing the contents of my company's master file server. Can anyone tell
me why reiserfs rolled back all the way to mid-December in spite of
numerous reboots and how I can avoid a rerun of this scenario *ever*
again. Is ther some way to tell it to commit its changes that I am not
doing and should be aware of?

Ciao, Neil

Re: 3.6.25 - Journal replayed back to 3 weeks ago

[Chris Mason]

On Mon, 2004-01-12 at 13:47, Neil Robinson wrote:
> Hi,
> 
> this morning when I started up my notebook (running Windows XP) with a
> VMware session running Gentoo, the boot sequence claimed that the
> reiserfs drives had not been cleanly umounted (not true, I powered down
> the usual way on Friday evening -- su to root and then issued the
> poweroff command). It then replayed the journals of the two partitions
> using reiserfs. When it finished and booted, it was as if my entire
> machine had stepped back in time by 3 weeks or so (to around the 23rd of
> December). Since then I had installed and built openoffice, emacs, and
> numerous other bits and pieces. I also lost all of the email that was
> living in the courier-imap server.
> 
> I am *very* concerned about this behaviour. I have successfully
> restarted, booted, etc. literally dozens of times since mid-December. I
> have now just installed a software RAID using RAID 5 on Gentoo and using
> reiserfs for a fairly large system (250GB on 8 SCSI U160 drives) with an
> available hot spare and a tape backup unit. Losing a few weeks of
> relatively insignificant changes is nothing compared with possibly
> losing the contents of my company's master file server. Can anyone tell
> me why reiserfs rolled back all the way to mid-December in spite of
> numerous reboots and how I can avoid a rerun of this scenario *ever*
> again. Is ther some way to tell it to commit its changes that I am not
> doing and should be aware of?

That's not supposed to happen.  Lets start with details about which
version of the kernel you were using.

-chris

Re: 3.6.25 - Journal replayed back to 3 weeks ago

[Neil Robinson]

On Mon, 2004-01-12 at 19:04, Chris Mason wrote:
> That's not supposed to happen.  Lets start with details about which
> version of the kernel you were using.

Kernel 2.4.20-gentoo-r9.

Ciao, Neil

Re: 3.6.25 - Journal replayed back to 3 weeks ago

[Neil Robinson]

Chris,

I haven't heard anything since my response, so just in case it wasn't
complete enough:

uname -a results:

2.4.20-gentoo-r9 #1 Sat Dec 6 03:17:43 GMT 2003 i686 Intel(R) Pentium(R)
4 CPU 2.66GHz GenuineIntel GNU/Linux

This is running in a vmware workstation 4.0 Windows XP host. It is set
up with 3 MAX 4GB emulated SCSI drives and 1 emulated MAX 2GB FAT32 IDE
hard drive.

sda1 is ext2 on /boot
sda2 is swap on swap
sda3 is reiserfs on /
sdb1 is reiserfs on /home
sdc1 is reiserfs on /var/tmp/portage
hda1 is vfat on /work

Ciao, Neil

On Mon, 2004-01-12 at 19:04, Chris Mason wrote:
> On Mon, 2004-01-12 at 13:47, Neil Robinson wrote:
> > Hi,
> > 
> > this morning when I started up my notebook (running Windows XP) with a
> > VMware session running Gentoo, the boot sequence claimed that the
> > reiserfs drives had not been cleanly umounted (not true, I powered down
> > the usual way on Friday evening -- su to root and then issued the
> > poweroff command). It then replayed the journals of the two partitions
> > using reiserfs. When it finished and booted, it was as if my entire
> > machine had stepped back in time by 3 weeks or so (to around the 23rd of
> > December). Since then I had installed and built openoffice, emacs, and
> > numerous other bits and pieces. I also lost all of the email that was
> > living in the courier-imap server.
> > 
> > I am *very* concerned about this behaviour. I have successfully
> > restarted, booted, etc. literally dozens of times since mid-December. I
> > have now just installed a software RAID using RAID 5 on Gentoo and using
> > reiserfs for a fairly large system (250GB on 8 SCSI U160 drives) with an
> > available hot spare and a tape backup unit. Losing a few weeks of
> > relatively insignificant changes is nothing compared with possibly
> > losing the contents of my company's master file server. Can anyone tell
> > me why reiserfs rolled back all the way to mid-December in spite of
> > numerous reboots and how I can avoid a rerun of this scenario *ever*
> > again. Is ther some way to tell it to commit its changes that I am not
> > doing and should be aware of?
> 
> That's not supposed to happen.  Lets start with details about which
> version of the kernel you were using.
> 
> -chris
> 
> 
> 

Re: 3.6.25 - Journal replayed back to 3 weeks ago

[Chris Mason]

On Thu, 2004-01-15 at 00:23, Neil Robinson wrote:
> Chris,
> 
> I haven't heard anything since my response, so just in case it wasn't
> complete enough:
> 
> uname -a results:
> 
> 2.4.20-gentoo-r9 #1 Sat Dec 6 03:17:43 GMT 2003 i686 Intel(R) Pentium(R)
> 4 CPU 2.66GHz GenuineIntel GNU/Linux
> 
> This is running in a vmware workstation 4.0 Windows XP host. It is set
> up with 3 MAX 4GB emulated SCSI drives and 1 emulated MAX 2GB FAT32 IDE
> hard drive.
> 
> sda1 is ext2 on /boot
> sda2 is swap on swap
> sda3 is reiserfs on /
> sdb1 is reiserfs on /home
> sdc1 is reiserfs on /var/tmp/portage
> hda1 is vfat on /work

Ok, please tell me more about the vmware setup.  Is are the vmware
drives configured to do copy on write?

-chris

Re: 3.6.25 - Journal replayed back to 3 weeks ago

[Neil Robinson]

On Thu, 2004-01-15 at 14:53, Chris Mason wrote:
> On Thu, 2004-01-15 at 00:23, Neil Robinson wrote:
> > Chris,
> > 
> > I haven't heard anything since my response, so just in case it wasn't
> > complete enough:
> > 
> > uname -a results:
> > 
> > 2.4.20-gentoo-r9 #1 Sat Dec 6 03:17:43 GMT 2003 i686 Intel(R) Pentium(R)
> > 4 CPU 2.66GHz GenuineIntel GNU/Linux
> > 
> > This is running in a vmware workstation 4.0 Windows XP host. It is set
> > up with 3 MAX 4GB emulated SCSI drives and 1 emulated MAX 2GB FAT32 IDE
> > hard drive.
> > 
> > sda1 is ext2 on /boot
> > sda2 is swap on swap
> > sda3 is reiserfs on /
> > sdb1 is reiserfs on /home
> > sdc1 is reiserfs on /var/tmp/portage
> > hda1 is vfat on /work
> 
> Ok, please tell me more about the vmware setup.  Is are the vmware
> drives configured to do copy on write?

According to the documentation, changes made to the virtual drives are
committed to the physical drive immediately. Here is the salient info
from the dmesg:

scsi: ***** BusLogic SCSI Driver Version 2.1.15 of 17 August 1998 *****
scsi: Copyright 1995-1998 by Leonard N. Zubkoff <lnz@dandelion.com>
scsi0: Configuring BusLogic Model BT-958 PCI Wide Ultra SCSI Host
Adapter
scsi0:   Firmware Version: 5.07B, I/O Address: 0x10C0, IRQ Channel:
11/Level
scsi0:   PCI Bus: 0, Device: 16, Address: 0xF8000000, Host Adapter SCSI
ID: 7
scsi0:   Parity Checking: Enabled, Extended Translation: Enabled
scsi0:   Synchronous Negotiation: Ultra, Wide Negotiation: Enabled
scsi0:   Disconnect/Reconnect: Enabled, Tagged Queuing: Enabled
scsi0:   Scatter/Gather Limit: 128 of 8192 segments, Mailboxes: 211
scsi0:   Driver Queue Depth: 211, Host Adapter Queue Depth: 192
scsi0:   Tagged Queue Depth: Automatic, Untagged Queue Depth: 3
scsi0:   Error Recovery Strategy: Default, SCSI Bus Reset: Enabled
scsi0: *** BusLogic BT-958 Initialized Successfully ***
scsi0 : BusLogic BT-958
  Vendor: VMware,   Model: VMware Virtual S  Rev: 1.0 
  Type:   Direct-Access                      ANSI SCSI revision: 02
  Vendor: VMware,   Model: VMware Virtual S  Rev: 1.0 
  Type:   Direct-Access                      ANSI SCSI revision: 02
  Vendor: VMware,   Model: VMware Virtual S  Rev: 1.0 
  Type:   Direct-Access                      ANSI SCSI revision: 02
scsi0: Target 0: Queue Depth 28, Asynchronous
scsi0: Target 1: Queue Depth 28, Asynchronous
scsi0: Target 2: Queue Depth 28, Asynchronous
scsi0: Target 3: Queue Depth 3, Asynchronous
scsi0: Target 4: Queue Depth 3, Asynchronous
scsi0: Target 5: Queue Depth 3, Asynchronous
scsi0: Target 6: Queue Depth 3, Asynchronous
scsi0: Target 8: Queue Depth 3, Asynchronous
scsi0: Target 9: Queue Depth 3, Asynchronous
scsi0: Target 10: Queue Depth 3, Asynchronous
scsi0: Target 11: Queue Depth 3, Asynchronous
scsi0: Target 12: Queue Depth 3, Asynchronous
scsi0: Target 13: Queue Depth 3, Asynchronous
scsi0: Target 14: Queue Depth 3, Asynchronous
scsi0: Target 15: Queue Depth 3, Asynchronous
Attached scsi disk sda at scsi0, channel 0, id 0, lun 0
Attached scsi disk sdb at scsi0, channel 0, id 1, lun 0
Attached scsi disk sdc at scsi0, channel 0, id 2, lun 0
SCSI device sda: 8388608 512-byte hdwr sectors (4295 MB)
 /dev/scsi/host0/bus0/target0/lun0: p1 p2 p3
SCSI device sdb: 8388608 512-byte hdwr sectors (4295 MB)
 /dev/scsi/host0/bus0/target1/lun0: p1
SCSI device sdc: 8388608 512-byte hdwr sectors (4295 MB)
 /dev/scsi/host0/bus0/target2/lun0: p1

Ciao, Neil

Re: 3.6.25 - Journal replayed back to 3 weeks ago

[Chris Mason]

On Thu, 2004-01-15 at 11:33, Neil Robinson wrote:

> > Ok, please tell me more about the vmware setup.  Is are the vmware
> > drives configured to do copy on write?
> 
> According to the documentation, changes made to the virtual drives are
> committed to the physical drive immediately. Here is the salient info
> from the dmesg:

vmware has a mode where the changes are written to a log file instead of
to the main file.  The client doesn't see things are split between two
files, since vmware exports them as a single device.  This allows you to
do fancy things like roll back changes, or share a base file between two
hosts and have changes to into private files.

It's possible that you're in this mode and somehow lost a log file.

Are the virtual drives being exported to linux real block devices in
windows, or are the files on some windows filesystem?

-chris

Re: 3.6.25 - Journal replayed back to 3 weeks ago

[Neil Robinson]

On Thu, 2004-01-15 at 17:56, Chris Mason wrote:
> On Thu, 2004-01-15 at 11:33, Neil Robinson wrote:
> 
> > > Ok, please tell me more about the vmware setup.  Is are the vmware
> > > drives configured to do copy on write?
> > 
> > According to the documentation, changes made to the virtual drives are
> > committed to the physical drive immediately. Here is the salient info
> > from the dmesg:
> 
> vmware has a mode where the changes are written to a log file instead of
> to the main file.  The client doesn't see things are split between two
> files, since vmware exports them as a single device.  This allows you to
> do fancy things like roll back changes, or share a base file between two
> hosts and have changes to into private files.
> 
> It's possible that you're in this mode and somehow lost a log file.
> 
> Are the virtual drives being exported to linux real block devices in
> windows, or are the files on some windows filesystem?

Uh oh, I think I now know what happened. Ouch. I was not clear what the
REDO files were for, since I seemed to have lots of them taking up a
considerable amount of drive space. I deleted them (probably taking all
of my changes with them). On the whole, I am quite relieved. Although I
did it to myself (this is my first experience using vmware), I realize
now what happened and it won't happen again. Even more important, it
seems my problems have nothing to do with the Reiser File System which
is also a relief. I wish to apologize to you and everyone on the list
for taking up your time with soething that, as it turns out, has nothing
really to do with you.

Thanks for all of your help.

Ciao, Neil

Re: 3.6.25 - Journal replayed back to 3 weeks ago

[Chris Mason]

On Thu, 2004-01-15 at 14:19, Neil Robinson wrote:

> Uh oh, I think I now know what happened. Ouch. I was not clear what the
> REDO files were for, since I seemed to have lots of them taking up a
> considerable amount of drive space. I deleted them (probably taking all
> of my changes with them). On the whole, I am quite relieved. Although I
> did it to myself (this is my first experience using vmware), I realize
> now what happened and it won't happen again. Even more important, it
> seems my problems have nothing to do with the Reiser File System which
> is also a relief. I wish to apologize to you and everyone on the list
> for taking up your time with soething that, as it turns out, has nothing
> really to do with you.
> 
> Thanks for all of your help.

Good news, I wasn't looking forward to hunting that as an FS bug ;-) 
Thanks for the details.

-chris