fsck in background?

fsck in background?

[myciel]

Hi All

Yes, I know this sounds strange, but if one have 1T filesystem full of 
small files,
reiserfsck estimates duration to be of order of months,
so I'm curious if maybe fscking in background is planned? - idea is not new
and bsd guys say that they have it  for freebsd 5.0 in ffs.


rafal mycielski

Re: fsck in background?

[Oleg Drokin]

Hello!

On Mon, Mar 31, 2003 at 12:30:37PM +0200, myciel wrote:

> Yes, I know this sounds strange, but if one have 1T filesystem full of 
> small files,
> reiserfsck estimates duration to be of order of months,

We have a patch that significantly speeds things up.
In the similar case 28+ years estimated for pass 0 turned into 208 hours total
operations after speedup patch was applied.
We plan to include it into reiserfsprogs 3.6.6-pre1, but if you need that urgently,
tell us.

> so I'm curious if maybe fscking in background is planned? - idea is not new
> and bsd guys say that they have it  for freebsd 5.0 in ffs.

This is not possible for reiserfs, I think.
reiserfs have non-constant metadata location , so while you can certainly
check fs consistency of some snapshot, you cannot fix it because all the real data
might have ben shifted to other blocks, old blocks might have been already freed and so on.

Bye,
    Oleg

Re: fsck in background?

[Hans Reiser]

Oleg Drokin wrote:

>Hello!
>
>On Mon, Mar 31, 2003 at 12:30:37PM +0200, myciel wrote:
>
>  
>
>>Yes, I know this sounds strange, but if one have 1T filesystem full of 
>>small files,
>>reiserfsck estimates duration to be of order of months,
>>    
>>
>
>We have a patch that significantly speeds things up.
>In the similar case 28+ years estimated for pass 0 turned into 208 hours total
>

As I said before, 208 hours is unacceptable and Vitaly needs to do more 
work on it. If Vitaly doesn't know how to fix it, then I need to review 
his algorithms in detail.

>operations after speedup patch was applied.
>We plan to include it into reiserfsprogs 3.6.6-pre1, but if you need that urgently,
>tell us.
>
>  
>
>>so I'm curious if maybe fscking in background is planned? - idea is not new
>>and bsd guys say that they have it  for freebsd 5.0 in ffs.
>>    
>>
>
>This is not possible for reiserfs, I think.
>reiserfs have non-constant metadata location , so while you can certainly
>check fs consistency of some snapshot, you cannot fix it because all the real data
>might have ben shifted to other blocks, old blocks might have been already freed and so on.
>
It can be done, but I don't have the funding/staff for it.

>
>Bye,
>    Oleg
>
>
>  
>


-- 
Hans

Re: fsck in background?

[Oleg Drokin]

Hello!

On Mon, Mar 31, 2003 at 09:14:07PM +0400, Hans Reiser wrote:
> >>Yes, I know this sounds strange, but if one have 1T filesystem full of 
> >>small files,
> >>reiserfsck estimates duration to be of order of months,
> >We have a patch that significantly speeds things up.
> >In the similar case 28+ years estimated for pass 0 turned into 208 hours 
> >total
> As I said before, 208 hours is unacceptable and Vitaly needs to do more 
> work on it. If Vitaly doesn't know how to fix it, then I need to review 
> his algorithms in detail.

Well, we have identified some other bottlenecks, and Vitaly will work on those,
when we will fix current problems we are dealing with.

Bye,
    Oleg

Re: fsck in background?

[Vitaly Fertman]

On Monday 31 March 2003 21:14, Hans Reiser wrote:
> Oleg Drokin wrote:
> >Hello!
> >
> >On Mon, Mar 31, 2003 at 12:30:37PM +0200, myciel wrote:
> >>Yes, I know this sounds strange, but if one have 1T filesystem full of
> >>small files,
> >>reiserfsck estimates duration to be of order of months,
> >
> >We have a patch that significantly speeds things up.
> >In the similar case 28+ years estimated for pass 0 turned into 208 hours
> > total
>
> As I said before, 208 hours is unacceptable and Vitaly needs to do more
> work on it. If Vitaly doesn't know how to fix it, then I need to review
> his algorithms in detail.

As I wrote you the objectid handling is not the problem anymore, and I 
described also that the performance problem for now is nlink calculation 
on semantic rebuild pass which needs many writes on disk what is not easy
to optimize in the case with shared objectids. Although it could be solved 
with another pass which would solve all such shared objectid problems just 
before semantic rebuild pass. 

-- 

Thanks,
Vitaly Fertman

Re: fsck in background?

[Hans Reiser]

Vitaly Fertman wrote:

>On Monday 31 March 2003 21:14, Hans Reiser wrote:
>  
>
>>Oleg Drokin wrote:
>>    
>>
>>>Hello!
>>>
>>>On Mon, Mar 31, 2003 at 12:30:37PM +0200, myciel wrote:
>>>      
>>>
>>>>Yes, I know this sounds strange, but if one have 1T filesystem full of
>>>>small files,
>>>>reiserfsck estimates duration to be of order of months,
>>>>        
>>>>
>>>We have a patch that significantly speeds things up.
>>>In the similar case 28+ years estimated for pass 0 turned into 208 hours
>>>total
>>>      
>>>
>>As I said before, 208 hours is unacceptable and Vitaly needs to do more
>>work on it. If Vitaly doesn't know how to fix it, then I need to review
>>his algorithms in detail.
>>    
>>
>
>As I wrote you the objectid handling is not the problem anymore, and I 
>described also that the performance problem for now is nlink calculation 
>on semantic rebuild pass which needs many writes on disk what is not easy
>to optimize in the case with shared objectids.
>
by shared objectids you mean when more than one file is discovered by 
fsck to have the same oid?  what is the problem?

> Although it could be solved 
>with another pass which would solve all such shared objectid problems just 
>before semantic rebuild pass. 
>
>  
>


-- 
Hans

Re: fsck in background?

[Andreas Dilger]

On Mar 31, 2003  21:14 +0400, Hans Reiser wrote:
> Oleg Drokin wrote:
> 
> >On Mon, Mar 31, 2003 at 12:30:37PM +0200, myciel wrote:
> >>so I'm curious if maybe fscking in background is planned? - idea is not new
> >>and bsd guys say that they have it  for freebsd 5.0 in ffs.
> >>    
> >>
> >
> >This is not possible for reiserfs, I think.
> >reiserfs have non-constant metadata location , so while you can certainly
> >check fs consistency of some snapshot, you cannot fix it because all the
> >real data might have ben shifted to other blocks, old blocks might have
> >been already freed and so on.
>
> It can be done, but I don't have the funding/staff for it.

I wrote a script to do this once, using LVM snapshots.  Attached here.
I haven't used it in a long time, and didn't do much other than write
it and test it out a bit, but it likely works OK.

It obviously doesn't actually fix any problems that it detects, but
for systems that run a long time it avoids the need to do a shutdown
to verify large filesystems are intact.  For ext2/3 it also resets the
"last checked" count/time so that ext3 will not do gratuitous full fscks
if the system has been up over 6 months (assuming you run the script
periodically while the system is running).

Cheers, Andreas
============================= lvm-fsck =============================
#!/bin/sh
# Automatically checks ext2/ext3 filesystems that are currently mounted
# and also residing on LVM logical volumes, so that we can snapshot them.
# You need to have the LVM VFS locking patch applied for this to work.
#
# (C) Andreas Dilger, 2001
#
# Licensed under the GNU General Public License, version 2 or later

#set -vx

# Use the PATH to find any installed fsck programs
PATH=$PATH:/sbin:/usr/sbin:/usr/local/sbin
PROG=`basename $0`

FSCKLV=lvfsck
lvscan | grep ACTIVE | while read LVSCAN DASH ACTIVE LV LV2 SIZE; do
	[ "$LV" = "Snapshot" ] && continue
	if [ "$LV" = "Original" ]; then
		LV=`echo $LV2 | tr -d \"`
	else
		LV=`echo $LV | tr -d \"`
	fi
	VG="`dirname $LV`"
	LVS="$VG/$FSCKLV"
	# This could be smarter (i.e. removing old snapshot after checking if
	# another lvm-fsck is running, but we don't want to do concurrent fscks
	# if it takes a really long time to run, or if the script has problems.
	# Sadly, lvdisplay does not return an error code if $LVS doesn't exist
	if [ "`lvdisplay $LVS 2> /dev/null`" ]; then
		echo "$LVS exists!  Unable to check $LV"
		continue
	fi

	# Check if this LV is mounted and has a fsck-able filesystem on it
	AWKLV="`echo $LV | tr / .`"
	FSTYPE=`mount | awk "/^$AWKLV/ { print \\$5 }"`
	FSCK="`which fsck.$FSTYPE 2> /dev/null`"
	[ "$FSCK" ] || continue

	echo "$PROG: running read-only $FSCK on $LV"
	# Just a guess at how much snapshot space we need
	SIZE="`df -P $LV | tail +2 | awk '{ print $2 / 500 }'`"
	lvcreate -s -L ${SIZE}k -n $FSCKLV $LV > /dev/null
	rc=$?
	if [ $rc -ne 0 ]; then
		echo "Creating snapshot of $LV at $LVS failed with rc=$rc" 1>&2
		continue
	fi
	case $FSTYPE in
	ext2|ext3) $FSCK -f -n $LVS; rc=$?
		if [ $rc -eq 0 ]; then
			tune2fs -C 0 $LV > /dev/null
			tune2fs -T now $LV > /dev/null 2>&1
			;;
	reiserfs) echo Yes | $FSCK --check $LVS; rc=$? ;;
	*) echo "Don't know how to check $FSTYPE filesystems passively"; rc=$1;;
	esac
	[ $rc -ne 0 ] && echo "$FSCK of $LV failed with rc=$rc" 1>&2
	lvremove -f $LVS > /dev/null
	rc=$?
	[ $rc -ne 0 ] && echo "lvremove of $LVS failed with rc=$rc" 1>&2
done
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://www-mddsp.enel.ucalgary.ca/People/adilger/

Re: fsck in background?

[Hans Reiser]

Andreas Dilger wrote:

>On Mar 31, 2003  21:14 +0400, Hans Reiser wrote:
>  
>
>>Oleg Drokin wrote:
>>
>>    
>>
>>>On Mon, Mar 31, 2003 at 12:30:37PM +0200, myciel wrote:
>>>      
>>>
>>>>so I'm curious if maybe fscking in background is planned? - idea is not new
>>>>and bsd guys say that they have it  for freebsd 5.0 in ffs.
>>>>   
>>>>
>>>>        
>>>>
>>>This is not possible for reiserfs, I think.
>>>reiserfs have non-constant metadata location , so while you can certainly
>>>check fs consistency of some snapshot, you cannot fix it because all the
>>>real data might have ben shifted to other blocks, old blocks might have
>>>been already freed and so on.
>>>      
>>>
>>It can be done, but I don't have the funding/staff for it.
>>    
>>
>
>I wrote a script to do this once, using LVM snapshots.  Attached here.
>I haven't used it in a long time, and didn't do much other than write
>it and test it out a bit, but it likely works OK.
>
>It obviously doesn't actually fix any problems that it detects, but
>for systems that run a long time it avoids the need to do a shutdown
>to verify large filesystems are intact.  For ext2/3 it also resets the
>"last checked" count/time so that ext3 will not do gratuitous full fscks
>if the system has been up over 6 months (assuming you run the script
>periodically while the system is running).
>
>Cheers, Andreas
>============================= lvm-fsck =============================
>#!/bin/sh
># Automatically checks ext2/ext3 filesystems that are currently mounted
># and also residing on LVM logical volumes, so that we can snapshot them.
># You need to have the LVM VFS locking patch applied for this to work.
>#
># (C) Andreas Dilger, 2001
>#
># Licensed under the GNU General Public License, version 2 or later
>
>#set -vx
>
># Use the PATH to find any installed fsck programs
>PATH=$PATH:/sbin:/usr/sbin:/usr/local/sbin
>PROG=`basename $0`
>
>FSCKLV=lvfsck
>lvscan | grep ACTIVE | while read LVSCAN DASH ACTIVE LV LV2 SIZE; do
>	[ "$LV" = "Snapshot" ] && continue
>	if [ "$LV" = "Original" ]; then
>		LV=`echo $LV2 | tr -d \"`
>	else
>		LV=`echo $LV | tr -d \"`
>	fi
>	VG="`dirname $LV`"
>	LVS="$VG/$FSCKLV"
>	# This could be smarter (i.e. removing old snapshot after checking if
>	# another lvm-fsck is running, but we don't want to do concurrent fscks
>	# if it takes a really long time to run, or if the script has problems.
>	# Sadly, lvdisplay does not return an error code if $LVS doesn't exist
>	if [ "`lvdisplay $LVS 2> /dev/null`" ]; then
>		echo "$LVS exists!  Unable to check $LV"
>		continue
>	fi
>
>	# Check if this LV is mounted and has a fsck-able filesystem on it
>	AWKLV="`echo $LV | tr / .`"
>	FSTYPE=`mount | awk "/^$AWKLV/ { print \\$5 }"`
>	FSCK="`which fsck.$FSTYPE 2> /dev/null`"
>	[ "$FSCK" ] || continue
>
>	echo "$PROG: running read-only $FSCK on $LV"
>	# Just a guess at how much snapshot space we need
>	SIZE="`df -P $LV | tail +2 | awk '{ print $2 / 500 }'`"
>	lvcreate -s -L ${SIZE}k -n $FSCKLV $LV > /dev/null
>	rc=$?
>	if [ $rc -ne 0 ]; then
>		echo "Creating snapshot of $LV at $LVS failed with rc=$rc" 1>&2
>		continue
>	fi
>	case $FSTYPE in
>	ext2|ext3) $FSCK -f -n $LVS; rc=$?
>		if [ $rc -eq 0 ]; then
>			tune2fs -C 0 $LV > /dev/null
>			tune2fs -T now $LV > /dev/null 2>&1
>			;;
>	reiserfs) echo Yes | $FSCK --check $LVS; rc=$? ;;
>	*) echo "Don't know how to check $FSTYPE filesystems passively"; rc=$1;;
>	esac
>	[ $rc -ne 0 ] && echo "$FSCK of $LV failed with rc=$rc" 1>&2
>	lvremove -f $LVS > /dev/null
>	rc=$?
>	[ $rc -ne 0 ] && echo "lvremove of $LVS failed with rc=$rc" 1>&2
>done
>--
>Andreas Dilger
>http://sourceforge.net/projects/ext2resize/
>http://www-mddsp.enel.ucalgary.ca/People/adilger/
>
>
>
>  
>
Oleg, consider putting a link to this on our web site.....

-- 
Hans

Re: fsck in background?

[myciel]

Oleg Drokin wrote:

>We have a patch that significantly speeds things up.
>In the similar case 28+ years estimated for pass 0 turned into 208 hours total
>operations after speedup patch was applied.
>We plan to include it into reiserfsprogs 3.6.6-pre1, but if you need that urgently,
>tell us.
>  
>

thanks for answer, so please let me know where I can get the patch.
And I have another question - again maybe this is not good idea, but:
if we can fsck of snapshot (without error fixing), so maybe
it would be possible to store output data from readonly fscking
and just feed reiserfsck in rw mode to avoid scanning whole fs during 
repairing,
and fix only corrupted places found during readonly scan?

rafal

Re: fsck in background?

[Oleg Drokin]

Hello!

On Tue, Apr 08, 2003 at 11:40:54AM +0200, myciel wrote:
> >We have a patch that significantly speeds things up.
> >In the similar case 28+ years estimated for pass 0 turned into 208 hours 
> >total
> >operations after speedup patch was applied.
> >We plan to include it into reiserfsprogs 3.6.6-pre1, but if you need that 
> >urgently,
> >tell us.
> thanks for answer, so please let me know where I can get the patch.

It is now included into reiserfsprogs 3.6.6-pre1, you can get it from
ftp://ftp.namesys.com/pub/reiserfsprogs/pre/reiserfsprogs-3.6.6-pre1.tar.gz

> And I have another question - again maybe this is not good idea, but:
> if we can fsck of snapshot (without error fixing), so maybe
> it would be possible to store output data from readonly fscking
> and just feed reiserfsck in rw mode to avoid scanning whole fs during 
> repairing,
> and fix only corrupted places found during readonly scan?

Well, this is still pretty complicated.
Nodes might have been shifted around compared to the snapshot we did the checking on.

Bye,
    Oleg

Re: fsck in background?

[Robin H. Johnson]

[-- Attachment #1: Type: text/plain, Size: 1072 bytes --]

On Mon, Mar 31, 2003 at 12:30:37PM +0200, myciel wrote:
> Yes, I know this sounds strange, but if one have 1T filesystem full of
> small files, reiserfsck estimates duration to be of order of months,
While this would be nice, it is possible that duration is a little off
;-). 
Our 1.2Tb RAID3 (single ReiserFS partition) takes ~3 hours to run fsck.
Again, mostly small files, but only at 50% capacity so far.

We've got another 1Tb RAID3 array arriving in about 5 weeks, and I'll
test it out to see how long reiserfsck actually takes when it's full of
small files. The joys of a research sysadmin :-).

> so I'm curious if maybe fscking in background is planned? - idea is not new
> and bsd guys say that they have it  for freebsd 5.0 in ffs.
I dunno if this is planned for Reiser4 or not, but it certainly would be
nice :-).

-- 
Robin Hugh Johnson
E-Mail     : robbat2@orbis-terrarum.net
Home Page  : http://www.orbis-terrarum.net/?l=people.robbat2
ICQ#       : 30269588 or 41961639
GnuPG FP   : 11AC BA4F 4778 E3F6 E4ED  F38E B27B 944E 3488 4E85

[-- Attachment #2: Type: application/pgp-signature, Size: 232 bytes --]