Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[David Greaves]

David Robinson wrote:
> David Greaves wrote:
>> This isn't a regression.
>>
>> I was seeing these problems on 2.6.21 (but 22 was in -rc so I waited 
>> to try it).
>> I tried 2.6.22-rc4 (with Tejun's patches) to see if it had improved - no.
>>
>> Note this is a different (desktop) machine to that involved my recent 
>> bugs.
>>
>> The machine will work for days (continually powered up) without a 
>> problem and then exhibits a filesystem failure within minutes of a 
>> resume.
>>
>> I know xfs/raid are OK with hibernate. Is lvm?
> 
> I have LVM working with hibernate w/o any problems (w/ ext3). If there 
> were a problem it wouldn't be with LVM but with device-mapper, and I 
> doubt there's a problem with either. The stack trace shows that you're 
> within XFS code (but it's likely its hibernate).

Thanks - that's good to know.
The suspicion arises because I have xfs on raid1 as root and have *never* had a 
problem with that filesystem. It's *always* xfs on lvm on raid5. I also have 
another system (previously discussed) that reliably hibernated xfs on raid6.

(Clearly raid5 is in my suspect list)

> You can easily check whether its LVM/device-mapper:
> 
> 1) check "dmsetup table" - it should be the same before hibernating and 
> after resuming.
> 
> 2) read directly from the LV - ie, "dd if=/dev/mapper/video_vg-video_lv 
> of=/dev/null bs=10M count=200".
> 
> If dmsetup shows the same info and you can read directly from the LV I 
> doubt it would be a LVM/device-mapper problem.

OK, that gave me an idea.

Freeze the filesystem
md5sum the lvm
hibernate
resume
md5sum the lvm

so:


haze:~# xfs_freeze -f /scratch/

Without this sync, the next two md5sums differed..
haze:~# sync
haze:~# dd if=/dev/video_vg/video_lv bs=10M count=200 | md5sum
200+0 records in
200+0 records out
2097152000 bytes (2.1 GB) copied, 41.2495 seconds, 50.8 MB/s
f42539366bb4269623fa4db14e8e8be2  -
haze:~# dd if=/dev/video_vg/video_lv bs=10M count=200 | md5sum
200+0 records in
200+0 records out
2097152000 bytes (2.1 GB) copied, 41.8111 seconds, 50.2 MB/s
f42539366bb4269623fa4db14e8e8be2  -


haze:~# echo platform > /sys/power/disk
haze:~# echo disk > /sys/power/state


haze:~# dd if=/dev/video_vg/video_lv bs=10M count=200 | md5sum
200+0 records in
200+0 records out
2097152000 bytes (2.1 GB) copied, 42.0478 seconds, 49.9 MB/s
f42539366bb4269623fa4db14e8e8be2  -
haze:~# xfs_freeze -u /scratch/

So the lvm and below looks OK...

I'll see how it behaves now the filesystem has been frozen/thawed over the 
hibernate...

David

Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[David Greaves]

David Greaves wrote:
> David Robinson wrote:
>> David Greaves wrote:
>>> This isn't a regression.
>>>
>>> I was seeing these problems on 2.6.21 (but 22 was in -rc so I waited 
>>> to try it).
>>> I tried 2.6.22-rc4 (with Tejun's patches) to see if it had improved - 
>>> no.
>>>
>>> Note this is a different (desktop) machine to that involved my recent 
>>> bugs.
>>>
>>> The machine will work for days (continually powered up) without a 
>>> problem and then exhibits a filesystem failure within minutes of a 
>>> resume.

<snip>

> OK, that gave me an idea.
> 
> Freeze the filesystem
> md5sum the lvm
> hibernate
> resume
> md5sum the lvm
<snip>
> So the lvm and below looks OK...
> 
> I'll see how it behaves now the filesystem has been frozen/thawed over 
> the hibernate...


And it appears to behave well. (A few hours compile/clean cycling kernel builds 
on that filesystem were OK).


Historically I've done:
sync
echo platform > /sys/power/disk
echo disk > /sys/power/state
# resume

and had filesystem corruption (only on this machine, my other hibernating xfs 
machines don't have this problem)

So doing:
xfs_freeze -f /scratch
sync
echo platform > /sys/power/disk
echo disk > /sys/power/state
# resume
xfs_freeze -u /scratch

Works (for now - more usage testing tonight)

David

Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[David Chinner]

On Mon, Jun 18, 2007 at 08:49:34AM +0100, David Greaves wrote:
> David Greaves wrote:
> >OK, that gave me an idea.
> >
> >Freeze the filesystem
> >md5sum the lvm
> >hibernate
> >resume
> >md5sum the lvm
> <snip>
> >So the lvm and below looks OK...
> >
> >I'll see how it behaves now the filesystem has been frozen/thawed over 
> >the hibernate...
> 
> 
> And it appears to behave well. (A few hours compile/clean cycling kernel 
> builds on that filesystem were OK).
> 
> 
> Historically I've done:
> sync
> echo platform > /sys/power/disk
> echo disk > /sys/power/state
> # resume
> 
> and had filesystem corruption (only on this machine, my other hibernating 
> xfs machines don't have this problem)
> 
> So doing:
> xfs_freeze -f /scratch
> sync
> echo platform > /sys/power/disk
> echo disk > /sys/power/state
> # resume
> xfs_freeze -u /scratch
>
> Works (for now - more usage testing tonight)

Verrry interesting.

What you were seeing was an XFS shutdown occurring because the free space
btree was corrupted. IOWs, the process of suspend/resume has resulted
in either bad data being written to disk, the correct data not being
written to disk or the cached block being corrupted in memory.

If you run xfs_check on the filesystem after it has shut down after a resume,
can you tell us if it reports on-disk corruption? Note: do not run xfs_repair
to check this - it does not check the free space btrees; instead it simply
rebuilds them from scratch. If xfs_check reports an error, then run xfs_repair
to fix it up.

FWIW, I'm on record stating that "sync" is not sufficient to quiesce an XFS
filesystem for a suspend/resume to work safely and have argued that the only
safe thing to do is freeze the filesystem before suspend and thaw it after
resume. This is why I originally asked you to test that with the other problem
that you reported. Up until this point in time, there's been no evidence to
prove either side of the argument......

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[David Greaves]

OK, just an quick ack

When I resumed tonight (having done a freeze/thaw over the suspend) some libata 
errors threw up during the resume and there was an eventual hard hang. Maybe I 
spoke to soon?

I'm going to have to do some more testing...

David Chinner wrote:
> On Mon, Jun 18, 2007 at 08:49:34AM +0100, David Greaves wrote:
>> David Greaves wrote:
>> So doing:
>> xfs_freeze -f /scratch
>> sync
>> echo platform > /sys/power/disk
>> echo disk > /sys/power/state
>> # resume
>> xfs_freeze -u /scratch
>>
>> Works (for now - more usage testing tonight)
> 
> Verrry interesting.
Good :)


> What you were seeing was an XFS shutdown occurring because the free space
> btree was corrupted. IOWs, the process of suspend/resume has resulted
> in either bad data being written to disk, the correct data not being
> written to disk or the cached block being corrupted in memory.
That's the kind of thing I was suspecting, yes.

> If you run xfs_check on the filesystem after it has shut down after a resume,
> can you tell us if it reports on-disk corruption? Note: do not run xfs_repair
> to check this - it does not check the free space btrees; instead it simply
> rebuilds them from scratch. If xfs_check reports an error, then run xfs_repair
> to fix it up.
OK, I can try this tonight...

> FWIW, I'm on record stating that "sync" is not sufficient to quiesce an XFS
> filesystem for a suspend/resume to work safely and have argued that the only
> safe thing to do is freeze the filesystem before suspend and thaw it after
> resume. This is why I originally asked you to test that with the other problem
> that you reported. Up until this point in time, there's been no evidence to
> prove either side of the argument......
> 
> Cheers,
> 
> Dave.

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[David Greaves]

David Greaves wrote:
> I'm going to have to do some more testing...
done


> David Chinner wrote:
>> On Mon, Jun 18, 2007 at 08:49:34AM +0100, David Greaves wrote:
>>> David Greaves wrote:
>>> So doing:
>>> xfs_freeze -f /scratch
>>> sync
>>> echo platform > /sys/power/disk
>>> echo disk > /sys/power/state
>>> # resume
>>> xfs_freeze -u /scratch
>>>
>>> Works (for now - more usage testing tonight)
>>
>> Verrry interesting.
> Good :)
Now, not so good :)


>> What you were seeing was an XFS shutdown occurring because the free space
>> btree was corrupted. IOWs, the process of suspend/resume has resulted
>> in either bad data being written to disk, the correct data not being
>> written to disk or the cached block being corrupted in memory.
> That's the kind of thing I was suspecting, yes.
> 
>> If you run xfs_check on the filesystem after it has shut down after a 
>> resume,
>> can you tell us if it reports on-disk corruption? Note: do not run 
>> xfs_repair
>> to check this - it does not check the free space btrees; instead it 
>> simply
>> rebuilds them from scratch. If xfs_check reports an error, then run 
>> xfs_repair
>> to fix it up.
> OK, I can try this tonight...


This is on 2.6.22-rc5

So I hibernated last night and resumed this morning.
Before hibernating I froze and sync'ed. After resume I thawed it. (Sorry Dave)

Here are some photos of the screen during resume. This is not 100% reproducable 
- it seems to occur only if the system is shutdown for 30mins or so.

Tejun, I wonder if error handling during resume is problematic? I got the same 
errors in 2.6.21. I have never seen these (or any other libata) errors other 
than during resume.

http://www.dgreaves.com/pub/2.6.22-rc5-resume-failure.jpg
(hard to read, here's one from 2.6.21
http://www.dgreaves.com/pub/2.6.21-resume-failure.jpg

I _think_ I've only seen the xfs problem when a resume shows these errors.


Ok, to try and cause a problem I ran a make and got this back at once:
make: stat: Makefile: Input/output error
make: stat: clean: Input/output error
make: *** No rule to make target `clean'.  Stop.
make: stat: GNUmakefile: Input/output error
make: stat: makefile: Input/output error


I caught the first dmesg this time:

Filesystem "dm-0": XFS internal error xfs_btree_check_sblock at line 334 of file 
fs/xfs/xfs_btree.c.  Caller 0xc01b58e1
  [<c0104f6a>] show_trace_log_lvl+0x1a/0x30
  [<c0105c52>] show_trace+0x12/0x20
  [<c0105d15>] dump_stack+0x15/0x20
  [<c01daddf>] xfs_error_report+0x4f/0x60
  [<c01cd736>] xfs_btree_check_sblock+0x56/0xd0
  [<c01b58e1>] xfs_alloc_lookup+0x181/0x390
  [<c01b5b06>] xfs_alloc_lookup_le+0x16/0x20
  [<c01b30c1>] xfs_free_ag_extent+0x51/0x690
  [<c01b4ea4>] xfs_free_extent+0xa4/0xc0
  [<c01bf739>] xfs_bmap_finish+0x119/0x170
  [<c01e3f4a>] xfs_itruncate_finish+0x23a/0x3a0
  [<c02046a2>] xfs_inactive+0x482/0x500
  [<c0210ad4>] xfs_fs_clear_inode+0x34/0xa0
  [<c017d777>] clear_inode+0x57/0xe0
  [<c017d8e5>] generic_delete_inode+0xe5/0x110
  [<c017da77>] generic_drop_inode+0x167/0x1b0
  [<c017cedf>] iput+0x5f/0x70
  [<c01735cf>] do_unlinkat+0xdf/0x140
  [<c0173640>] sys_unlink+0x10/0x20
  [<c01040a4>] syscall_call+0x7/0xb
  =======================
xfs_force_shutdown(dm-0,0x8) called from line 4258 of file fs/xfs/xfs_bmap.c. 
Return address = 0xc021101e
Filesystem "dm-0": Corruption of in-memory data detected.  Shutting down 
filesystem: dm-0
Please umount the filesystem, and rectify the problem(s)

so I cd'ed out of /scratch and umounted.

I then tried the xfs_check.

haze:~# xfs_check /dev/video_vg/video_lv
ERROR: The filesystem has valuable metadata changes in a log which needs to
be replayed.  Mount the filesystem to replay the log, and unmount it before
re-running xfs_check.  If you are unable to mount the filesystem, then use
the xfs_repair -L option to destroy the log and attempt a repair.
Note that destroying the log may cause corruption -- please attempt a mount
of the filesystem before doing this.
haze:~# mount /scratch/
haze:~# umount /scratch/
haze:~# xfs_check /dev/video_vg/video_lv

Message from syslogd@haze at Tue Jun 19 08:47:30 2007 ...
haze kernel: Bad page state in process 'xfs_db'

Message from syslogd@haze at Tue Jun 19 08:47:30 2007 ...
haze kernel: page:c1767bc0 flags:0x80010008 mapping:00000000 mapcount:-64 count:0

Message from syslogd@haze at Tue Jun 19 08:47:30 2007 ...
haze kernel: Trying to fix it up, but a reboot is needed

Message from syslogd@haze at Tue Jun 19 08:47:30 2007 ...
haze kernel: Backtrace:

Message from syslogd@haze at Tue Jun 19 08:47:30 2007 ...
haze kernel: Bad page state in process 'syslogd'

Message from syslogd@haze at Tue Jun 19 08:47:30 2007 ...
haze kernel: page:c1767cc0 flags:0x80010008 mapping:00000000 mapcount:-64 count:0

Message from syslogd@haze at Tue Jun 19 08:47:30 2007 ...
haze kernel: Trying to fix it up, but a reboot is needed

Message from syslogd@haze at Tue Jun 19 08:47:30 2007 ...
haze kernel: Backtrace:

ugh. Try again
haze:~# xfs_check /dev/video_vg/video_lv
haze:~#

whilst running a top reported this as roughly the peak memory usage:
  8759 root      18   0  479m 474m  876 R  2.0 46.9   0:02.49 xfs_db
so it looks like it didn't run out of memory (machine has 1Gb).

Dave, I ran xfs_check -v... but I got bored when it reached 122M of bz2 
compressed output with no sign of stopping... still got it if it's any use...

lots of:
setting block 0/0 to sb
setting block 0/1 to freelist
setting block 0/2 to freelist
setting block 0/3 to freelist
setting block 0/4 to freelist
setting block 0/75 to btbno
setting block 0/346901 to free1
setting block 0/346903 to free1
setting block 0/346904 to free1
setting block 0/346905 to free1
   and stuff like this
inode 128 mode 040777 fmt extents afmt extents nex 1 anex 0 nblk 1 sz 4096
inode 128 nlink 39 is dir
inode 128 extent [0,7,1,0]

I then rebooted and ran a repair which didn't show any damage.

David

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[Tejun Heo]

Hello,

David Greaves wrote:
>> Good :)
> Now, not so good :)

Oh, crap.  :-)

> So I hibernated last night and resumed this morning.
> Before hibernating I froze and sync'ed. After resume I thawed it. (Sorry
> Dave)
> 
> Here are some photos of the screen during resume. This is not 100%
> reproducable - it seems to occur only if the system is shutdown for
> 30mins or so.
> 
> Tejun, I wonder if error handling during resume is problematic? I got
> the same errors in 2.6.21. I have never seen these (or any other libata)
> errors other than during resume.
> 
> http://www.dgreaves.com/pub/2.6.22-rc5-resume-failure.jpg
> (hard to read, here's one from 2.6.21
> http://www.dgreaves.com/pub/2.6.21-resume-failure.jpg

Your controller is repeatedly reporting PHY readiness changed exception.
 Are you reading the system image from the device attached to the first
SATA port?

> I _think_ I've only seen the xfs problem when a resume shows these errors.

The error handling itself tries very hard to ensure that there is no
data corruption in case of errors.  All commands which experience
exceptions are retried but if the drive itself is doing something
stupid, there's only so much the driver can do.

How reproducible is the problem?  Does the problem go away or occur more
often if you change the drive you write the memory image to?

-- 
tejun

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[David Greaves]

Tejun Heo wrote:
> Hello,
again...

> David Greaves wrote:
>>> Good :)
>> Now, not so good :)
> 
> Oh, crap.  :-)
<grin>

>> So I hibernated last night and resumed this morning.
>> Before hibernating I froze and sync'ed. After resume I thawed it. (Sorry
>> Dave)
>>
>> Here are some photos of the screen during resume. This is not 100%
>> reproducable - it seems to occur only if the system is shutdown for
>> 30mins or so.
>>
>> Tejun, I wonder if error handling during resume is problematic? I got
>> the same errors in 2.6.21. I have never seen these (or any other libata)
>> errors other than during resume.
>>
>> http://www.dgreaves.com/pub/2.6.22-rc5-resume-failure.jpg
>> (hard to read, here's one from 2.6.21
>> http://www.dgreaves.com/pub/2.6.21-resume-failure.jpg
> 
> Your controller is repeatedly reporting PHY readiness changed exception.
>  Are you reading the system image from the device attached to the first
> SATA port?

Yes if you mean 1st as in the one after the zero-th ...

resume=/dev/sdb4
haze:~# swapon -s
Filename                                Type            Size    Used    Priority
/dev/sdb4                               partition       1004020 0       -1

dmesg snippet below...

sda is part of the /scratch xfs array though. SMART doesn't show any problems 
and of course all is well other than during a resume.

sda/b are on sata_sil (a cheap plugin pci card)

> 
>> I _think_ I've only seen the xfs problem when a resume shows these errors.
> 
> The error handling itself tries very hard to ensure that there is no
> data corruption in case of errors.  All commands which experience
> exceptions are retried but if the drive itself is doing something
> stupid, there's only so much the driver can do.
> 
> How reproducible is the problem?  Does the problem go away or occur more
> often if you change the drive you write the memory image to?

I don't think there should be activity on the sda drive during resume itself.

[I broke my / md mirror and am using some of that for swap/resume for now]

I did change the swap/resume device to sdd2 (different controller, onboard 
sata_via) and there was no EH during resume. The system seemed OK, wrote a few 
Gb of video and did a kernel compile.
I repeated this test, no EH during resume, no problems.
I even ran xfs_fsr, the defragment utility, to stress the fs.

I retain this configuration and try again tonight but it looks like there _may_ 
be a link between EH during resume and my problems...

Of course, I don't understand why it *should* EH during resume, it doesn't 
during boot or normal operation...

Any more tests you'd like me to try?

David


dmesg snippet...

sata_sil 0000:00:0a.0: version 2.2
ACPI: PCI Interrupt 0000:00:0a.0[A] -> GSI 16 (level, low) -> IRQ 18
scsi0 : sata_sil
PM: Adding info for No Bus:host0
scsi1 : sata_sil
PM: Adding info for No Bus:host1
ata1: SATA max UDMA/100 cmd 0xf881e080 ctl 0xf881e08a bmdma 0xf881e000 irq 0
ata2: SATA max UDMA/100 cmd 0xf881e0c0 ctl 0xf881e0ca bmdma 0xf881e008 irq 0
ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata1.00: ATA-7: Maxtor 6B200M0, BANC1980, max UDMA/100
ata1.00: 390721968 sectors, multi 0: LBA48
ata1.00: configured for UDMA/100
ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ata2.00: ata_hpa_resize 1: sectors = 312581808, hpa_sectors = 312581808
ata2.00: ATA-6: ST3160023AS, 3.18, max UDMA/133
ata2.00: 312581808 sectors, multi 0: LBA48
ata2.00: ata_hpa_resize 1: sectors = 312581808, hpa_sectors = 312581808
ata2.00: configured for UDMA/100
PM: Adding info for No Bus:target0:0:0
scsi 0:0:0:0: Direct-Access     ATA      Maxtor 6B200M0   BANC PQ: 0 ANSI: 5
PM: Adding info for scsi:0:0:0:0
sd 0:0:0:0: [sda] 390721968 512-byte hardware sectors (200050 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO 
or FUA
sd 0:0:0:0: [sda] 390721968 512-byte hardware sectors (200050 MB)
sd 0:0:0:0: [sda] Write Protect is off
sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00
sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO 
or FUA
  sda: sda1
sd 0:0:0:0: [sda] Attached SCSI disk
sd 0:0:0:0: Attached scsi generic sg0 type 0
PM: Adding info for No Bus:target1:0:0
scsi 1:0:0:0: Direct-Access     ATA      ST3160023AS      3.18 PQ: 0 ANSI: 5
PM: Adding info for scsi:1:0:0:0
sd 1:0:0:0: [sdb] 312581808 512-byte hardware sectors (160042 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO 
or FUA
sd 1:0:0:0: [sdb] 312581808 512-byte hardware sectors (160042 MB)
sd 1:0:0:0: [sdb] Write Protect is off
sd 1:0:0:0: [sdb] Mode Sense: 00 3a 00 00
sd 1:0:0:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO 
or FUA
  sdb: sdb1 sdb2 sdb3 sdb4
sd 1:0:0:0: [sdb] Attached SCSI disk
sd 1:0:0:0: Attached scsi generic sg1 type 0

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[Tejun Heo]

David Greaves wrote:
> Tejun Heo wrote:
>> Your controller is repeatedly reporting PHY readiness changed exception.
>>  Are you reading the system image from the device attached to the first
>> SATA port?
> 
> Yes if you mean 1st as in the one after the zero-th ...

I meant the first first (0th).

>> How reproducible is the problem?  Does the problem go away or occur more
>> often if you change the drive you write the memory image to?
> 
> I don't think there should be activity on the sda drive during resume
> itself.
> 
> [I broke my / md mirror and am using some of that for swap/resume for now]
> 
> I did change the swap/resume device to sdd2 (different controller,
> onboard sata_via) and there was no EH during resume. The system seemed
> OK, wrote a few Gb of video and did a kernel compile.
> I repeated this test, no EH during resume, no problems.
> I even ran xfs_fsr, the defragment utility, to stress the fs.
> 
> I retain this configuration and try again tonight but it looks like
> there _may_ be a link between EH during resume and my problems...
> 
> Of course, I don't understand why it *should* EH during resume, it
> doesn't during boot or normal operation...

EH occurs during boot, suspend and resume all the time.  It just runs in
quiet mode to avoid disturbing the users too much.  In your case, EH is
kicking in due to actual exception conditions so it's being verbose to
give clue about what's going on.

It's really weird tho.  The PHY RDY status changed events are coming
from the device which is NOT used while resuming and it's before any
actual PM events are triggered.  Your kernel just boots, swsusp realizes
it's resuming and tries to read memory image from the swap device.
While reading, the disk controller raises consecutive PHY readiness
changed interrupts.  EH recovers them alright but the end result seems
to indicate that the loaded image is corrupt.

So, there's no device suspend/resume code involved at all.  The kernel
just booted and is trying to read data from the drive.  Please try with
only the first drive attached and see what happens.

Thanks.

-- 
tejun

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[David Greaves]

been away, back now...

Tejun Heo wrote:
> David Greaves wrote:
>> Tejun Heo wrote:
>>> How reproducible is the problem?  Does the problem go away or occur more
>>> often if you change the drive you write the memory image to?
>> I don't think there should be activity on the sda drive during resume
>> itself.
>>
>> [I broke my / md mirror and am using some of that for swap/resume for now]
>>
>> I did change the swap/resume device to sdd2 (different controller,
>> onboard sata_via) and there was no EH during resume. The system seemed
>> OK, wrote a few Gb of video and did a kernel compile.
>> I repeated this test, no EH during resume, no problems.
>> I even ran xfs_fsr, the defragment utility, to stress the fs.
>>
>> I retain this configuration and try again tonight but it looks like
>> there _may_ be a link between EH during resume and my problems...
Having retained this new configuration for a couple of days now I haven't had 
any problems.
This is good but not really ideal since / isn't mirrored anymore :(

>> Of course, I don't understand why it *should* EH during resume, it
>> doesn't during boot or normal operation...
> 
> EH occurs during boot, suspend and resume all the time.  It just runs in
> quiet mode to avoid disturbing the users too much.  In your case, EH is
> kicking in due to actual exception conditions so it's being verbose to
> give clue about what's going on.
I was trying to say that I don't actually see any errors being handled in normal 
operation.
I'm not sure if you are saying that these PHY RDY events are normally handled 
quietly (which would explain it).


> It's really weird tho.  The PHY RDY status changed events are coming
> from the device which is NOT used while resuming
yes - but the erroring device which is not being used is on the same controller 
as the device with the in-use resume partition.

> and it's before any
> actual PM events are triggered.  Your kernel just boots, swsusp realizes
> it's resuming and tries to read memory image from the swap device.
yes

> While reading, the disk controller raises consecutive PHY readiness
> changed interrupts.  EH recovers them alright but the end result seems
> to indicate that the loaded image is corrupt.
Yes, that's consistent with what I'm seeing.

When I move the swap/resume partition to a different controller (ie when I broke 
the / mirror and used the freed space) the problem seems to go away.

I am seeing messages in dmesg though:
ATA: abnormal status 0xD0 on port 0xf881e0c7
ATA: abnormal status 0xD0 on port 0xf881e0c7
ATA: abnormal status 0xD0 on port 0xf881e0c7
ATA: abnormal status 0xD0 on port 0xf881e0c7
ATA: abnormal status 0xD0 on port 0xf881e0c7
ata1.00: configured for UDMA/100
ata2.00: revalidation failed (errno=-2)
ata2: failed to recover some devices, retrying in 5 secs
sd 0:0:0:0: [sda] 390721968 512-byte hardware sectors (200050 MB)

sd 0:0:0:0: resuming
sd 0:0:0:0: [sda] Starting disk
ATA: abnormal status 0x7F on port 0x00019807
ATA: abnormal status 0x7F on port 0x00019007
ATA: abnormal status 0x7F on port 0x00019007
ATA: abnormal status 0x7F on port 0x00019807

ata2: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
ATA: abnormal status 0xD0 on port 0xf881e0c7
ATA: abnormal status 0xD0 on port 0xf881e0c7
ATA: abnormal status 0xD0 on port 0xf881e0c7
ATA: abnormal status 0xD0 on port 0xf881e0c7
ATA: abnormal status 0xD0 on port 0xf881e0c7
ata1.00: configured for UDMA/100
ata2.00: revalidation failed (errno=-2)
ata2: failed to recover some devices, retrying in 5 secs


> So, there's no device suspend/resume code involved at all.  The kernel
> just booted and is trying to read data from the drive.  Please try with
> only the first drive attached and see what happens.
That's kinda hard; swap and root are on different drives...

Does it help that although the errors above appear, the system seems OK when I 
just use the other controller?

I have to be cautious what I do with this machine as it's the wife's active 
desktop box <grin>.

David

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[David Greaves]

David Greaves wrote:
> been away, back now...
again...

David Greaves wrote:
> When I move the swap/resume partition to a different controller (ie when 
> I broke the / mirror and used the freed space) the problem seems to go 
> away.
No, it's not gone away - but it's taking longer to show up.
I can try and put together a test loop that does work, hibernates, resumes and 
repeats but since I know it crashes at some point there doesn't seem much point 
unless I'm looking for something.
There's not much in the logs - is there any other instrumentation that people 
could suggest?
DaveC, given this is happening without (obvious) libata errors do you think it 
may be something in the XFS/md/hibernate area?

If there's anything to be tried then I'll also move to 2.6.22-rc6.


 > Tejun Heo wrote:
 >> It's really weird tho.  The PHY RDY status changed events are coming
 >> from the device which is NOT used while resuming

There is an obvious problem there though Tejun (the errors even when sda isn't 
involved in the OS boot) - can I start another thread about that issue/bug 
later? I need to reshuffle partitions so I'd rather get the hibernate working 
first and then go back to it if that's OK?

David

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[Tejun Heo]

David Greaves wrote:
>> Tejun Heo wrote:
>>> It's really weird tho.  The PHY RDY status changed events are coming
>>> from the device which is NOT used while resuming
> 
> There is an obvious problem there though Tejun (the errors even when sda
> isn't involved in the OS boot) - can I start another thread about that
> issue/bug later? I need to reshuffle partitions so I'd rather get the
> hibernate working first and then go back to it if that's OK?

Yeah, sure.  The problem is that we don't know whether or how those two
are related.  It would be great if there's a way to verify memory image
read from hibernation is intact.  Rafael, any ideas?

Thanks.

-- 
tejun

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[Rafael J. Wysocki]

On Monday, 2 July 2007 12:56, Tejun Heo wrote:
> David Greaves wrote:
> >> Tejun Heo wrote:
> >>> It's really weird tho.  The PHY RDY status changed events are coming
> >>> from the device which is NOT used while resuming
> > 
> > There is an obvious problem there though Tejun (the errors even when sda
> > isn't involved in the OS boot) - can I start another thread about that
> > issue/bug later? I need to reshuffle partitions so I'd rather get the
> > hibernate working first and then go back to it if that's OK?
> 
> Yeah, sure.  The problem is that we don't know whether or how those two
> are related.  It would be great if there's a way to verify memory image
> read from hibernation is intact.  Rafael, any ideas?

Well, s2disk has an option to compute an MD5 checksum of the image during
the hibernation and verify it while reading the image.  Still, s2disk/resume
aren't very easy to install  and configure ...

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[David Greaves]

Rafael J. Wysocki wrote:
> On Monday, 2 July 2007 12:56, Tejun Heo wrote:
>> David Greaves wrote:
>>>> Tejun Heo wrote:
>>>>> It's really weird tho.  The PHY RDY status changed events are coming
>>>>> from the device which is NOT used while resuming
>>> There is an obvious problem there though Tejun (the errors even when sda
>>> isn't involved in the OS boot) - can I start another thread about that
>>> issue/bug later? I need to reshuffle partitions so I'd rather get the
>>> hibernate working first and then go back to it if that's OK?
>> Yeah, sure.  The problem is that we don't know whether or how those two
>> are related.  It would be great if there's a way to verify memory image
>> read from hibernation is intact.  Rafael, any ideas?
> 
> Well, s2disk has an option to compute an MD5 checksum of the image during
> the hibernation and verify it while reading the image.
(Assuming you mean the mainline version)

Sounds like a good think to try next...
Couldn't see anything on this in ../Documentation/power/*
How do I enable it?


>  Still, s2disk/resume
> aren't very easy to install  and configure ...

I have it working fine on 2 other machines now so that doesn't appear to be a 
problem.

David

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[Rafael J. Wysocki]

On Monday, 2 July 2007 16:32, David Greaves wrote:
> Rafael J. Wysocki wrote:
> > On Monday, 2 July 2007 12:56, Tejun Heo wrote:
> >> David Greaves wrote:
> >>>> Tejun Heo wrote:
> >>>>> It's really weird tho.  The PHY RDY status changed events are coming
> >>>>> from the device which is NOT used while resuming
> >>> There is an obvious problem there though Tejun (the errors even when sda
> >>> isn't involved in the OS boot) - can I start another thread about that
> >>> issue/bug later? I need to reshuffle partitions so I'd rather get the
> >>> hibernate working first and then go back to it if that's OK?
> >> Yeah, sure.  The problem is that we don't know whether or how those two
> >> are related.  It would be great if there's a way to verify memory image
> >> read from hibernation is intact.  Rafael, any ideas?
> > 
> > Well, s2disk has an option to compute an MD5 checksum of the image during
> > the hibernation and verify it while reading the image.
> (Assuming you mean the mainline version)
> 
> Sounds like a good think to try next...
> Couldn't see anything on this in ../Documentation/power/*
> How do I enable it?

Add 'compute checksum = y' to the s2disk's configuration file.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[David Greaves]

Rafael J. Wysocki wrote:
> On Monday, 2 July 2007 16:32, David Greaves wrote:
>> Rafael J. Wysocki wrote:
>>> On Monday, 2 July 2007 12:56, Tejun Heo wrote:
>>>> David Greaves wrote:
>>>>>> Tejun Heo wrote:
>>>>>>> It's really weird tho.  The PHY RDY status changed events are coming
>>>>>>> from the device which is NOT used while resuming
>>>>> There is an obvious problem there though Tejun (the errors even when sda
>>>>> isn't involved in the OS boot) - can I start another thread about that
>>>>> issue/bug later? I need to reshuffle partitions so I'd rather get the
>>>>> hibernate working first and then go back to it if that's OK?
>>>> Yeah, sure.  The problem is that we don't know whether or how those two
>>>> are related.  It would be great if there's a way to verify memory image
>>>> read from hibernation is intact.  Rafael, any ideas?
>>> Well, s2disk has an option to compute an MD5 checksum of the image during
>>> the hibernation and verify it while reading the image.
>> (Assuming you mean the mainline version)
>>
>> Sounds like a good think to try next...
>> Couldn't see anything on this in ../Documentation/power/*
>> How do I enable it?
> 
> Add 'compute checksum = y' to the s2disk's configuration file.

Ah, right - that's uswsusp isn't it? Which isn't what I'm having problems with 
AFAIK?

My suspend procedure is:

xfs_freeze -f /scratch
sync
echo platform > /sys/power/disk
echo disk > /sys/power/state
xfs_freeze -u /scratch

Which should work (actually it should work without the sync/xfs_freeze too).

So to debug the problem I'd like to minimally extend this process rather than 
replace it with another approach.

I take it there isn't an 'echo y > /sys/power/do_image_checksum'?

David

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[Rafael J. Wysocki]

On Monday, 2 July 2007 18:36, David Greaves wrote:
> Rafael J. Wysocki wrote:
> > On Monday, 2 July 2007 16:32, David Greaves wrote:
> >> Rafael J. Wysocki wrote:
> >>> On Monday, 2 July 2007 12:56, Tejun Heo wrote:
> >>>> David Greaves wrote:
> >>>>>> Tejun Heo wrote:
> >>>>>>> It's really weird tho.  The PHY RDY status changed events are coming
> >>>>>>> from the device which is NOT used while resuming
> >>>>> There is an obvious problem there though Tejun (the errors even when sda
> >>>>> isn't involved in the OS boot) - can I start another thread about that
> >>>>> issue/bug later? I need to reshuffle partitions so I'd rather get the
> >>>>> hibernate working first and then go back to it if that's OK?
> >>>> Yeah, sure.  The problem is that we don't know whether or how those two
> >>>> are related.  It would be great if there's a way to verify memory image
> >>>> read from hibernation is intact.  Rafael, any ideas?
> >>> Well, s2disk has an option to compute an MD5 checksum of the image during
> >>> the hibernation and verify it while reading the image.
> >> (Assuming you mean the mainline version)
> >>
> >> Sounds like a good think to try next...
> >> Couldn't see anything on this in ../Documentation/power/*
> >> How do I enable it?
> > 
> > Add 'compute checksum = y' to the s2disk's configuration file.
> 
> Ah, right - that's uswsusp isn't it? Which isn't what I'm having problems with 
> AFAIK?
> 
> My suspend procedure is:
> 
> xfs_freeze -f /scratch
> sync
> echo platform > /sys/power/disk
> echo disk > /sys/power/state
> xfs_freeze -u /scratch
> 
> Which should work (actually it should work without the sync/xfs_freeze too).
> 
> So to debug the problem I'd like to minimally extend this process rather than 
> replace it with another approach.

Well, this is not entirely "another approach".  Only the saving of the image is
done differently, the rest is the same.

> I take it there isn't an 'echo y > /sys/power/do_image_checksum'?

No, there is not anything like that.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[Rafael J. Wysocki]

On Tuesday, 19 June 2007 11:24, David Greaves wrote:
> David Greaves wrote:
> > I'm going to have to do some more testing...
> done
> 
> 
> > David Chinner wrote:
> >> On Mon, Jun 18, 2007 at 08:49:34AM +0100, David Greaves wrote:
> >>> David Greaves wrote:
> >>> So doing:
> >>> xfs_freeze -f /scratch
> >>> sync
> >>> echo platform > /sys/power/disk
> >>> echo disk > /sys/power/state
> >>> # resume
> >>> xfs_freeze -u /scratch
> >>>
> >>> Works (for now - more usage testing tonight)
> >>
> >> Verrry interesting.
> > Good :)
> Now, not so good :)
> 
> 
> >> What you were seeing was an XFS shutdown occurring because the free space
> >> btree was corrupted. IOWs, the process of suspend/resume has resulted
> >> in either bad data being written to disk, the correct data not being
> >> written to disk or the cached block being corrupted in memory.
> > That's the kind of thing I was suspecting, yes.
> > 
> >> If you run xfs_check on the filesystem after it has shut down after a 
> >> resume,
> >> can you tell us if it reports on-disk corruption? Note: do not run 
> >> xfs_repair
> >> to check this - it does not check the free space btrees; instead it 
> >> simply
> >> rebuilds them from scratch. If xfs_check reports an error, then run 
> >> xfs_repair
> >> to fix it up.
> > OK, I can try this tonight...
> 
> 
> This is on 2.6.22-rc5

Is the Tejun's patch

http://www.sisk.pl/kernel/hibernation_and_suspend/2.6.22-rc5/patches/30-block-always-requeue-nonfs-requests-at-the-front.patch

applied on top of that?

Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[David Greaves]

Rafael J. Wysocki wrote:
>> This is on 2.6.22-rc5
> 
> Is the Tejun's patch
> 
> http://www.sisk.pl/kernel/hibernation_and_suspend/2.6.22-rc5/patches/30-block-always-requeue-nonfs-requests-at-the-front.patch
> 
> applied on top of that?

2.6.22-rc5 includes it.

(but, when I was testing rc4, I did apply this patch)

David

Re: [linux-lvm] 2.6.22-rc5 XFS fails after hibernate/resume

[David Chinner]

On Tue, Jun 19, 2007 at 10:24:23AM +0100, David Greaves wrote:
> David Greaves wrote:
> so I cd'ed out of /scratch and umounted.
> 
> I then tried the xfs_check.
> 
> haze:~# xfs_check /dev/video_vg/video_lv
> ERROR: The filesystem has valuable metadata changes in a log which needs to
> be replayed.  Mount the filesystem to replay the log, and unmount it before
> re-running xfs_check.  If you are unable to mount the filesystem, then use
> the xfs_repair -L option to destroy the log and attempt a repair.
> Note that destroying the log may cause corruption -- please attempt a mount
> of the filesystem before doing this.
> haze:~# mount /scratch/
> haze:~# umount /scratch/
> haze:~# xfs_check /dev/video_vg/video_lv
> 
> Message from syslogd@haze at Tue Jun 19 08:47:30 2007 ...
> haze kernel: Bad page state in process 'xfs_db'

I think we can safely say that your system is hosed at this point ;)

> ugh. Try again
> haze:~# xfs_check /dev/video_vg/video_lv
> haze:~#

zero output means no on-disk corruption was found. Everything is
consistent on disk, so that seems to indicate something in memory has been
crispy fried by the suspend/resume....

> Dave, I ran xfs_check -v... but I got bored when it reached 122M of bz2 
> compressed output with no sign of stopping... still got it if it's any 
> use...

No, not useful. It's a log of every operation it does and so is really
only useful for debugging xfs-check problems ;)

> I then rebooted and ran a repair which didn't show any damage.

Not surprising as your first check showed no damage.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[Pavel Machek]

Hi!

> FWIW, I'm on record stating that "sync" is not sufficient to quiesce an XFS
> filesystem for a suspend/resume to work safely and have argued that the only

Hmm, so XFS writes to disk even when its threads are frozen?

> safe thing to do is freeze the filesystem before suspend and thaw it after
> resume. This is why I originally asked you to test that with the other problem

Could you add that to the XFS threads if it is really required? They
do know that they are being frozen for suspend.

							Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[Rafael J. Wysocki]

On Wednesday, 27 June 2007 22:49, Pavel Machek wrote:
> Hi!
> 
> > FWIW, I'm on record stating that "sync" is not sufficient to quiesce an XFS
> > filesystem for a suspend/resume to work safely and have argued that the only
> 
> Hmm, so XFS writes to disk even when its threads are frozen?
> 
> > safe thing to do is freeze the filesystem before suspend and thaw it after
> > resume. This is why I originally asked you to test that with the other problem
> 
> Could you add that to the XFS threads if it is really required? They
> do know that they are being frozen for suspend.

Well, do you remember the workqueues?  They are still nonfreezable.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [linux-pm] Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[Pavel Machek]

On Thu 2007-06-28 17:27:34, Rafael J. Wysocki wrote:
> On Wednesday, 27 June 2007 22:49, Pavel Machek wrote:
> > Hi!
> > 
> > > FWIW, I'm on record stating that "sync" is not sufficient to quiesce an XFS
> > > filesystem for a suspend/resume to work safely and have argued that the only
> > 
> > Hmm, so XFS writes to disk even when its threads are frozen?
> > 
> > > safe thing to do is freeze the filesystem before suspend and thaw it after
> > > resume. This is why I originally asked you to test that with the other problem
> > 
> > Could you add that to the XFS threads if it is really required? They
> > do know that they are being frozen for suspend.
> 
> Well, do you remember the workqueues?  They are still nonfreezable.

Oops, that would explain it :-(. Can we make XFS stop using them?
									Pavel
-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: [linux-pm] Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[Rafael J. Wysocki]

On Friday, 29 June 2007 00:00, Pavel Machek wrote:
> On Thu 2007-06-28 17:27:34, Rafael J. Wysocki wrote:
> > On Wednesday, 27 June 2007 22:49, Pavel Machek wrote:
> > > Hi!
> > > 
> > > > FWIW, I'm on record stating that "sync" is not sufficient to quiesce an XFS
> > > > filesystem for a suspend/resume to work safely and have argued that the only
> > > 
> > > Hmm, so XFS writes to disk even when its threads are frozen?
> > > 
> > > > safe thing to do is freeze the filesystem before suspend and thaw it after
> > > > resume. This is why I originally asked you to test that with the other problem
> > > 
> > > Could you add that to the XFS threads if it is really required? They
> > > do know that they are being frozen for suspend.
> > 
> > Well, do you remember the workqueues?  They are still nonfreezable.
> 
> Oops, that would explain it :-(. Can we make XFS stop using them?

I'm afraid that we can't.

There are two solutions possible, IMO.  One would be to make these workqueues
freezable, which is possible, but hacky and Oleg didn't like that very much.
The second would be to freeze XFS from within the hibernation code path,
using freeze_bdev().

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [linux-pm] Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[David Chinner]

On Fri, Jun 29, 2007 at 12:16:44AM +0200, Rafael J. Wysocki wrote:
> There are two solutions possible, IMO.  One would be to make these workqueues
> freezable, which is possible, but hacky and Oleg didn't like that very much.
> The second would be to freeze XFS from within the hibernation code path,
> using freeze_bdev().

The second is much more likely to work reliably. If freezing the
filesystem leaves something in an inconsistent state, then it's
something I can reproduce and debug without needing to
suspend/resume.

FWIW, don't forget you need to thaw the filesystem on resume.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: [linux-pm] Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[David Greaves]

David Chinner wrote:
> On Fri, Jun 29, 2007 at 12:16:44AM +0200, Rafael J. Wysocki wrote:
>> There are two solutions possible, IMO.  One would be to make these workqueues
>> freezable, which is possible, but hacky and Oleg didn't like that very much.
>> The second would be to freeze XFS from within the hibernation code path,
>> using freeze_bdev().
> 
> The second is much more likely to work reliably. If freezing the
> filesystem leaves something in an inconsistent state, then it's
> something I can reproduce and debug without needing to
> suspend/resume.
> 
> FWIW, don't forget you need to thaw the filesystem on resume.

I've been a little distracted recently - sorry. I'll re-read the thread and see 
if there are any test actions I need to complete.

I do know that the corruption problems I've been having:
a) only happen after hibernate/resume
b) only ever happen on one of 2 XFS filesystems
c) happen even when the script does xfs_freeze;sync;hibernate;xfs_thaw

What happens if a filesystem is frozen and I hibernate?
Will it be thawed when I resume?

David

Re: [linux-pm] Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[David Chinner]

On Fri, Jun 29, 2007 at 08:40:00AM +0100, David Greaves wrote:
> What happens if a filesystem is frozen and I hibernate?
> Will it be thawed when I resume?

If you froze it yourself, then you'll have to thaw it yourself.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group

Re: [linux-pm] Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[David Greaves]

David Chinner wrote:
> On Fri, Jun 29, 2007 at 08:40:00AM +0100, David Greaves wrote:
>> What happens if a filesystem is frozen and I hibernate?
>> Will it be thawed when I resume?
> 
> If you froze it yourself, then you'll have to thaw it yourself.

So hibernate will not attempt to re-freeze a frozen fs and, during resume, it 
will only thaw filesystems that were frozen by the suspend?

David

Re: [linux-pm] Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[Rafael J. Wysocki]

On Friday, 29 June 2007 09:54, David Greaves wrote:
> David Chinner wrote:
> > On Fri, Jun 29, 2007 at 08:40:00AM +0100, David Greaves wrote:
> >> What happens if a filesystem is frozen and I hibernate?
> >> Will it be thawed when I resume?
> > 
> > If you froze it yourself, then you'll have to thaw it yourself.
> 
> So hibernate will not attempt to re-freeze a frozen fs and, during resume, it 
> will only thaw filesystems that were frozen by the suspend?

Right now it doesn't freeze (or thaw) any filesystems.  It just sync()s them
before creating the hibernation image.

However, the fact that you've seen corruption with the XFS filesystems frozen
before the hibernation indicates that the problem occurs on a lower level.

Greetings,
Rafael


-- 
"Premature optimization is the root of all evil." - Donald Knuth

Re: [linux-pm] Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[David Greaves]

Rafael J. Wysocki wrote:
> On Friday, 29 June 2007 09:54, David Greaves wrote:
>> David Chinner wrote:
>>> On Fri, Jun 29, 2007 at 08:40:00AM +0100, David Greaves wrote:
>>>> What happens if a filesystem is frozen and I hibernate?
>>>> Will it be thawed when I resume?
>>> If you froze it yourself, then you'll have to thaw it yourself.
>> So hibernate will not attempt to re-freeze a frozen fs and, during resume, it 
>> will only thaw filesystems that were frozen by the suspend?
> 
> Right now it doesn't freeze (or thaw) any filesystems.  It just sync()s them
> before creating the hibernation image.
Thanks. Yes I realise that :)
I wasn't clear, I should have said:
So hibernate should not attempt to re-freeze a frozen fs and, during resume, it
should only thaw filesystems that were frozen by the suspend.


> However, the fact that you've seen corruption with the XFS filesystems frozen
> before the hibernation indicates that the problem occurs on a lower level.
And that was why I chimed in - I don't think freezing fixes the problem (though 
it may make sense for other reasons).

David

Re: [linux-lvm] 2.6.22-rc4 XFS fails after hibernate/resume

[David Chinner]

On Wed, Jun 27, 2007 at 08:49:24PM +0000, Pavel Machek wrote:
> Hi!
> 
> > FWIW, I'm on record stating that "sync" is not sufficient to quiesce an XFS
> > filesystem for a suspend/resume to work safely and have argued that the only
> 
> Hmm, so XFS writes to disk even when its threads are frozen?

They issue async I/O before they sleep and expects
processing to be done on I/O completion via workqueues.

> > safe thing to do is freeze the filesystem before suspend and thaw it after
> > resume. This is why I originally asked you to test that with the other problem
> 
> Could you add that to the XFS threads if it is really required? They
> do know that they are being frozen for suspend.

We don't suspend the threads on a filesystem freeze - they continue
run. A filesystem freeze guarantees the filesystem clean and that
the in memory state matches what is on disk. It is not possible for
the filesytem to issue I/O or have outstanding I/O when it is in the
frozen state, so the state of the threads and/or workqueues does not
matter because they will be idle.

Cheers,

Dave.
-- 
Dave Chinner
Principal Engineer
SGI Australian Software Group