* XFS_Repair PRoblem
@ 2007-12-06 16:10 Kingghost
2007-12-06 21:06 ` David Chinner
0 siblings, 1 reply; 12+ messages in thread
From: Kingghost @ 2007-12-06 16:10 UTC (permalink / raw)
To: xfs
Hello,
So I was seeing my serial link go up and down so I rebooted and everything
was working fine, except my vg0 wouldnt mount. So I tried to xfs_repair it
and this is the output I recieved.
slutb0x:/# xfs_repair -o assume_xfs /dev/vg0/media
Phase 1 - find and verify superblock...
sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with
calculated value 128
resetting superblock root inode pointer to 128
sb realtime bitmap inode 18446744073709551615 (NULLFSINO) inconsistent with
calculated value 129
resetting superblock realtime bitmap ino pointer to 129
sb realtime summary inode 18446744073709551615 (NULLFSINO) inconsistent with
calculated value 130
resetting superblock realtime summary ino pointer to 130
Phase 2 - using internal log
- zero log...
missing inbetween
rebuilding directory inode 1225702584
bad hash table for directory inode 1342177425 (no data entry): rebuilding
rebuilding directory inode 1342177425
bad hash table for directory inode 1344688512 (no data entry): rebuilding
rebuilding directory inode 1344688512
bad hash table for directory inode 1344689011 (no data entry): rebuilding
rebuilding directory inode 1344689011
bad hash table for directory inode 1347027107 (no data entry): rebuilding
rebuilding directory inode 1347027107
bad hash table for directory inode 1347297139 (no data entry): rebuilding
rebuilding directory inode 1347297139
bad hash table for directory inode 1348463898 (no data entry): rebuilding
rebuilding directory inode 1348463898
bad hash table for directory inode 1386426400 (no data entry): rebuilding
rebuilding directory inode 1386426400
bad hash table for directory inode 1407214284 (no data entry): rebuilding
rebuilding directory inode 1407214284
rebuilding directory inode 1409980307
bad hash table for directory inode 1476395171 (no data entry): rebuilding
rebuilding directory inode 1476395171
bad hash table for directory inode 1482430980 (no data entry): rebuilding
rebuilding directory inode 1482430980
bad hash table for directory inode 1483051225 (no data entry): rebuilding
rebuilding directory inode 1483051225
bad hash table for directory inode 1485162665 (no data entry): rebuilding
rebuilding directory inode 1485162665
bad hash table for directory inode 1490611323 (no data entry): rebuilding
rebuilding directory inode 1490611323
rebuilding directory inode 1545482625
bad hash table for directory inode 1546105138 (no data entry): rebuilding
rebuilding directory inode 1546105138
bad hash table for directory inode 1546563350 (no data entry): rebuilding
rebuilding directory inode 1546563350
bad hash table for directory inode 1576719133 (no data entry): rebuilding
rebuilding directory inode 1576719133
bad hash table for directory inode 1610612892 (no data entry): rebuilding
rebuilding directory inode 1610612892
bad hash table for directory inode 1618357762 (no data entry): rebuilding
rebuilding directory inode 1618357762
bad hash table for directory inode 1619798333 (no data entry): rebuilding
rebuilding directory inode 1619798333
bad hash table for directory inode 1669673937 (no data entry): rebuilding
rebuilding directory inode 1669673937
bad hash table for directory inode 1754694080 (no data entry): rebuilding
rebuilding directory inode 1754694080
bad hash table for directory inode 1756293956 (no data entry): rebuilding
rebuilding directory inode 1756293956
bad hash table for directory inode 1793985071 (no data entry): rebuilding
rebuilding directory inode 1793985071
bad hash table for directory inode 1879048358 (no data entry): rebuilding
rebuilding directory inode 1879048358
bad hash table for directory inode 1879480988 (no data entry): rebuilding
rebuilding directory inode 1879480988
bad hash table for directory inode 1884893322 (no data entry): rebuilding
rebuilding directory inode 1884893322
bad hash table for directory inode 1885854162 (no data entry): rebuilding
rebuilding directory inode 1885854162
bad hash table for directory inode 1888390172 (no data entry): rebuilding
rebuilding directory inode 1888390172
bad hash table for directory inode 1892569223 (no data entry): rebuilding
rebuilding directory inode 1892569223
bad hash table for directory inode 2015614187 (no data entry): rebuilding
rebuilding directory inode 2015614187
bad hash table for directory inode 2019037629 (no data entry): rebuilding
rebuilding directory inode 2019037629
bad hash table for directory inode 2027745957 (no data entry): rebuilding
rebuilding directory inode 2027745957
bad hash table for directory inode 2027745991 (no data entry): rebuilding
rebuilding directory inode 2027745991
bad hash table for directory inode 2041763187 (no data entry): rebuilding
rebuilding directory inode 2041763187
- traversal finished ...
- moving disconnected inodes to lost+found ...
disconnected dir inode 134217856, moving to lost+found
disconnected dir inode 159404583, moving to lost+found
disconnected dir inode 188928448, moving to lost+found
disconnected dir inode 188935323, moving to lost+found
disconnected inode 209057354, moving to lost+found
disconnected inode 209057355, moving to lost+found
disconnected inode 209057356, moving to lost+found
disconnected dir inode 209513188, moving to lost+found
disconnected dir inode 209513245, moving to lost+found
disconnected dir inode 209513344, moving to lost+found
disconnected dir inode 209513395, moving to lost+found
disconnected dir inode 209513490, moving to lost+found
disconnected dir inode 209513584, moving to lost+found
disconnected dir inode 209677569, moving to lost+found
disconnected dir inode 213712676, moving to lost+found
disconnected inode 262129013, moving to lost+found
disconnected inode 262129014, moving to lost+found
disconnected inode 262129016, moving to lost+found
disconnected dir inode 262132666, moving to lost+found
disconnected dir inode 262141188, moving to lost+found
disconnected inode 279860753, moving to lost+found
disconnected inode 279860754, moving to lost+found
disconnected dir inode 279860755, moving to lost+found
disconnected inode 279860757, moving to lost+found
disconnected inode 279860758, moving to lost+found
disconnected inode 279860759, moving to lost+found
disconnected inode 279860760, moving to lost+found
disconnected inode 279860761, moving to lost+found
disconnected inode 279860762, moving to lost+found
disconnected inode 279860763, moving to lost+found
disconnected inode 279860764, moving to lost+found
disconnected inode 279860766, moving to lost+found
disconnected inode 279860767, moving to lost+found
disconnected inode 279860770, moving to lost+found
disconnected inode 279860771, moving to lost+found
disconnected inode 279860772, moving to lost+found
disconnected inode 279860773, moving to lost+found
disconnected inode 279860774, moving to lost+found
disconnected inode 279860776, moving to lost+found
disconnected dir inode 279874575, moving to lost+found
disconnected inode 292512566, moving to lost+found
disconnected inode 292512567, moving to lost+found
disconnected inode 292512568, moving to lost+found
disconnected inode 292512569, moving to lost+found
disconnected dir inode 292519637, moving to lost+found
disconnected inode 292519660, moving to lost+found
disconnected inode 292774410, moving to lost+found
disconnected inode 292774412, moving to lost+found
disconnected inode 292774413, moving to lost+found
disconnected dir inode 307451136, moving to lost+found
disconnected inode 342075439, moving to lost+found
disconnected dir inode 342357112, moving to lost+found
disconnected inode 345352743, moving to lost+found
disconnected inode 345352744, moving to lost+found
disconnected inode 345352745, moving to lost+found
disconnected inode 345352746, moving to lost+found
disconnected dir inode 350201220, moving to lost+found
disconnected dir inode 404630767, moving to lost+found
disconnected inode 404635207, moving to lost+found
disconnected dir inode 467834646, moving to lost+found
disconnected dir inode 468060626, moving to lost+found
disconnected inode 497816685, moving to lost+found
disconnected inode 497816686, moving to lost+found
disconnected inode 497816687, moving to lost+found
disconnected inode 497816688, moving to lost+found
disconnected inode 497816689, moving to lost+found
disconnected inode 497816690, moving to lost+found
disconnected inode 497816691, moving to lost+found
disconnected inode 497816692, moving to lost+found
disconnected inode 497816702, moving to lost+found
disconnected dir inode 500814219, moving to lost+found
disconnected dir inode 500815365, moving to lost+found
disconnected dir inode 527158122, moving to lost+found
disconnected dir inode 536871084, moving to lost+found
disconnected dir inode 538001664, moving to lost+found
disconnected dir inode 541074254, moving to lost+found
disconnected dir inode 541664864, moving to lost+found
disconnected dir inode 541770349, moving to lost+found
disconnected dir inode 551347552, moving to lost+found
disconnected dir inode 677133324, moving to lost+found
disconnected dir inode 681949599, moving to lost+found
disconnected dir inode 683059743, moving to lost+found
disconnected inode 683118797, moving to lost+found
disconnected inode 683118798, moving to lost+found
disconnected inode 683118799, moving to lost+found
disconnected dir inode 683118800, moving to lost+found
disconnected inode 730956960, moving to lost+found
disconnected inode 730956961, moving to lost+found
disconnected inode 730956962, moving to lost+found
disconnected inode 730956963, moving to lost+found
disconnected inode 730956964, moving to lost+found
disconnected inode 730956965, moving to lost+found
disconnected inode 730956966, moving to lost+found
disconnected inode 730956967, moving to lost+found
disconnected inode 730956968, moving to lost+found
disconnected inode 730956969, moving to lost+found
disconnected inode 730956970, moving to lost+found
disconnected inode 730956971, moving to lost+found
disconnected inode 730956972, moving to lost+found
disconnected inode 730956973, moving to lost+found
disconnected inode 730956974, moving to lost+found
disconnected inode 730956975, moving to lost+found
disconnected inode 730956976, moving to lost+found
disconnected inode 730956977, moving to lost+found
disconnected inode 730956978, moving to lost+found
disconnected inode 730956979, moving to lost+found
disconnected inode 730956980, moving to lost+found
disconnected inode 730956981, moving to lost+found
disconnected inode 730956982, moving to lost+found
disconnected inode 730956983, moving to lost+found
disconnected inode 730956984, moving to lost+found
disconnected inode 730956985, moving to lost+found
disconnected inode 730956986, moving to lost+found
disconnected inode 730956987, moving to lost+found
disconnected inode 730956988, moving to lost+found
disconnected inode 730956989, moving to lost+found
disconnected inode 730956990, moving to lost+found
disconnected inode 730956991, moving to lost+found
disconnected inode 730956992, moving to lost+found
disconnected inode 730956993, moving to lost+found
disconnected inode 730956994, moving to lost+found
disconnected inode 730956995, moving to lost+found
disconnected inode 730956996, moving to lost+found
disconnected inode 730956997, moving to lost+found
disconnected inode 730956998, moving to lost+found
disconnected inode 730956999, moving to lost+found
disconnected inode 730957000, moving to lost+found
disconnected inode 730957001, moving to lost+found
disconnected inode 730957002, moving to lost+found
disconnected inode 730957003, moving to lost+found
disconnected inode 730957004, moving to lost+found
disconnected inode 730957005, moving to lost+found
disconnected inode 730957006, moving to lost+found
disconnected inode 730957007, moving to lost+found
disconnected inode 730957008, moving to lost+found
disconnected inode 730957009, moving to lost+found
disconnected inode 730957010, moving to lost+found
disconnected inode 730957011, moving to lost+found
disconnected inode 730957012, moving to lost+found
disconnected inode 730957013, moving to lost+found
disconnected inode 730957014, moving to lost+found
disconnected inode 730957015, moving to lost+found
disconnected inode 730957016, moving to lost+found
disconnected inode 730957017, moving to lost+found
disconnected inode 730957018, moving to lost+found
disconnected inode 730957019, moving to lost+found
disconnected inode 730957020, moving to lost+found
disconnected inode 730957021, moving to lost+found
disconnected inode 730957022, moving to lost+found
disconnected dir inode 730957023, moving to lost+found
disconnected dir inode 730975064, moving to lost+found
disconnected dir inode 731107961, moving to lost+found
disconnected dir inode 731173866, moving to lost+found
disconnected dir inode 731229206, moving to lost+found
disconnected dir inode 731317750, moving to lost+found
disconnected dir inode 752117594, moving to lost+found
disconnected dir inode 752117670, moving to lost+found
disconnected inode 809433019, moving to lost+found
disconnected dir inode 823780059, moving to lost+found
disconnected dir inode 823780089, moving to lost+found
fatal error -- creation of .. entry failed (117), filesystem may be out of
space
The thing is I have free space, its weird.
--
View this message in context: http://www.nabble.com/XFS_Repair-PRoblem-tf4956832.html#a14194915
Sent from the Xfs - General mailing list archive at Nabble.com.
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS_Repair PRoblem
2007-12-06 16:10 XFS_Repair PRoblem Kingghost
@ 2007-12-06 21:06 ` David Chinner
2007-12-07 0:51 ` Kingghost
0 siblings, 1 reply; 12+ messages in thread
From: David Chinner @ 2007-12-06 21:06 UTC (permalink / raw)
To: Kingghost; +Cc: xfs
On Thu, Dec 06, 2007 at 08:10:21AM -0800, Kingghost wrote:
>
> Hello,
>
> So I was seeing my serial link go up and down so I rebooted and everything
> was working fine, except my vg0 wouldnt mount. So I tried to xfs_repair it
> and this is the output I recieved.
>
> slutb0x:/# xfs_repair -o assume_xfs /dev/vg0/media
> Phase 1 - find and verify superblock...
> sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with
> calculated value 128
NULLFSINO? Something has overwritten your superblock with a bunch of -1
values?
> resetting superblock root inode pointer to 128
> sb realtime bitmap inode 18446744073709551615 (NULLFSINO) inconsistent with
> calculated value 129
> resetting superblock realtime bitmap ino pointer to 129
> sb realtime summary inode 18446744073709551615 (NULLFSINO) inconsistent with
> calculated value 130
> resetting superblock realtime summary ino pointer to 130
> Phase 2 - using internal log
> - zero log...
Uh-oh - you ran a xfs_repair -L, didn't you?
>
> missing inbetween
>
> rebuilding directory inode 1225702584
> bad hash table for directory inode 1342177425 (no data entry): rebuilding
And a bunch of trashed directory structures...
> disconnected dir inode 752117670, moving to lost+found
> disconnected inode 809433019, moving to lost+found
> disconnected dir inode 823780059, moving to lost+found
> disconnected dir inode 823780089, moving to lost+found
>
> fatal error -- creation of .. entry failed (117), filesystem may be out of
> space
117 = EUCLEAN - corrupted filesystem. Sounds like there's more corruption
there than was discovered or the underlying disk is still corruption blocks.
What version of repair are you running ?
This is a dying disk you're trying to repair right?
Cheers,
Dave.
--
Dave Chinner
Principal Engineer
SGI Australian Software Group
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: XFS_Repair PRoblem
2007-12-06 21:06 ` David Chinner
@ 2007-12-07 0:51 ` Kingghost
0 siblings, 0 replies; 12+ messages in thread
From: Kingghost @ 2007-12-07 0:51 UTC (permalink / raw)
To: xfs
Hello,
Ya it said if I couldnt mount it to use -L so I did. Apparently this was a
mistake.
I dd_rescued all the data to a new drive, so drive isnt the issue anymore.
David Chinner wrote:
>
> On Thu, Dec 06, 2007 at 08:10:21AM -0800, Kingghost wrote:
>>
>> Hello,
>>
>> So I was seeing my serial link go up and down so I rebooted and
>> everything
>> was working fine, except my vg0 wouldnt mount. So I tried to xfs_repair
>> it
>> and this is the output I recieved.
>>
>> slutb0x:/# xfs_repair -o assume_xfs /dev/vg0/media
>> Phase 1 - find and verify superblock...
>> sb root inode value 18446744073709551615 (NULLFSINO) inconsistent with
>> calculated value 128
>
> NULLFSINO? Something has overwritten your superblock with a bunch of -1
> values?
>
>> resetting superblock root inode pointer to 128
>> sb realtime bitmap inode 18446744073709551615 (NULLFSINO) inconsistent
>> with
>> calculated value 129
>> resetting superblock realtime bitmap ino pointer to 129
>> sb realtime summary inode 18446744073709551615 (NULLFSINO) inconsistent
>> with
>> calculated value 130
>> resetting superblock realtime summary ino pointer to 130
>> Phase 2 - using internal log
>> - zero log...
>
> Uh-oh - you ran a xfs_repair -L, didn't you?
>>
>> missing inbetween
>>
>> rebuilding directory inode 1225702584
>> bad hash table for directory inode 1342177425 (no data entry): rebuilding
>
> And a bunch of trashed directory structures...
>
>> disconnected dir inode 752117670, moving to lost+found
>> disconnected inode 809433019, moving to lost+found
>> disconnected dir inode 823780059, moving to lost+found
>> disconnected dir inode 823780089, moving to lost+found
>>
>> fatal error -- creation of .. entry failed (117), filesystem may be out
>> of
>> space
>
> 117 = EUCLEAN - corrupted filesystem. Sounds like there's more corruption
> there than was discovered or the underlying disk is still corruption
> blocks.
> What version of repair are you running ?
>
> This is a dying disk you're trying to repair right?
>
> Cheers,
>
> Dave.
> --
> Dave Chinner
> Principal Engineer
> SGI Australian Software Group
>
>
>
>
--
View this message in context: http://www.nabble.com/XFS_Repair-PRoblem-tf4956832.html#a14204640
Sent from the Xfs - General mailing list archive at Nabble.com.
^ permalink raw reply [flat|nested] 12+ messages in thread
* xfs_repair problem.
@ 2008-12-21 15:03 David Bernick
2008-12-21 15:55 ` Eric Sandeen
0 siblings, 1 reply; 12+ messages in thread
From: David Bernick @ 2008-12-21 15:03 UTC (permalink / raw)
To: xfs
I have a filesystem where an xfs_growfs went bad. We were adding
storage to a pre-existing infrastructure and when growing, we received
an error. SInce then, we've been unable to mount.
It's a large filesystem (see below) and the addition of the extra data
has made it larger. We tried an xfs_repair but it died, as the machine
only has 4GB RAM. We're going to put 32 GB in RAM and see if that
helps. The original FS size is about 13T and the addition brought it
to 29T.
Since we've been unable to write or mount, we've "backed off" the
addition and are left with our original, which we'd like to mount.
We try to mount and get an error about the root inode not being
readable. Makes sense as the root inode is null (according to xfs_db).
So before we run another big xfs_repair:
1. What is the math of filesystems size, number of files and how much
RAM is needed for such a task? Is 32 GB enough for 1/2 Billion files
and 13 TB?
2. Any way I can just find my rootino,rbmino,rsumino and put them in the DB?
Any other advice?
/proc/partitions:
253 0 13084291072 dm-0
--- Logical volume ---
LV Name /dev/docs/v5Docs
VG Name docs
LV UUID G85Zi9-s63C-yWrU-yyf0-STP6-YOhJ-6Ne3pS
LV Write Access read/write
LV Status available
# open 1
LV Size 12.19 TB
Current LE 3194407
Segments 2
Allocation inherit
Read ahead sectors 0
Block device 253:0
DB:
magicnum = 0x58465342
blocksize = 4096
dblocks = 3064987648
rblocks = 0
rextents = 0
uuid = f086bb71-d67b-4cc1-b622-1f10349e6a49
logstart = 1073741828
rootino = null
rbmino = null
rsumino = null
rextsize = 1
agblocks = 67108864
agcount = 46
rbmblocks = 0
logblocks = 32768
versionnum = 0x3084
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 26
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 0
ifree = 0
fdblocks = 0
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0
[[HTML alternate version deleted]]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: xfs_repair problem.
2008-12-21 15:03 xfs_repair problem David Bernick
@ 2008-12-21 15:55 ` Eric Sandeen
2008-12-21 16:52 ` David Bernick
0 siblings, 1 reply; 12+ messages in thread
From: Eric Sandeen @ 2008-12-21 15:55 UTC (permalink / raw)
To: David Bernick; +Cc: xfs
David Bernick wrote:
> I have a filesystem where an xfs_growfs went bad. We were adding
> storage to a pre-existing infrastructure and when growing, we received
> an error.
The error you encountered would be worth mentioning in detail...
> SInce then, we've been unable to mount.
>
> It's a large filesystem (see below) and the addition of the extra data
> has made it larger. We tried an xfs_repair but it died, as the machine
> only has 4GB RAM.
Do you have the latest version of repair? (xfs_repair -V; 2.10.2 is latest)
> We're going to put 32 GB in RAM and see if that
> helps. The original FS size is about 13T and the addition brought it
> to 29T.
on a 64-bit box I hope? what kernel version?
> Since we've been unable to write or mount, we've "backed off" the
> addition and are left with our original, which we'd like to mount.
How did you back it off? either the fs grew or it didn't; and you can't
shrink... so I guess it did not grow...
> We try to mount and get an error about the root inode not being
> readable. Makes sense as the root inode is null (according to xfs_db).
>
> So before we run another big xfs_repair:
>
> 1. What is the math of filesystems size, number of files and how much
> RAM is needed for such a task? Is 32 GB enough for 1/2 Billion files
> and 13 TB?
>
> 2. Any way I can just find my rootino,rbmino,rsumino and put them in the DB?
I looked at the db output below & re-made a similar sparse fs:
[root tmp]# bc
3064987648*4096
12554189406208
quit
[root tmp]# mkfs.xfs -dfile,name=testfile,size=12554189406208
meta-data=testfile isize=256 agcount=12,
agsize=268435455 blks
= sectsz=512 attr=2
data = bsize=4096 blocks=3064987648, imaxpct=5
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096
log =internal log bsize=4096 blocks=32768, version=2
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
ls -lh [root tmp]# ls -l testfile
-rw-r--r-- 1 root root 12554189406208 Dec 21 09:51 testfile
[root tmp]# xfs_db testfile
xfs_db> sb 0
xfs_db> p
magicnum = 0x58465342
blocksize = 4096
dblocks = 3064987648
rblocks = 0
rextents = 0
uuid = 4b0451c8-5be4-452f-b161-a3ada3ec1a20
logstart = 1610612740
rootino = 128
rbmino = 129
rsumino = 130
so that's most likely what it should be.
> Any other advice?
post more details of how things actually fail and what happened...
> /proc/partitions:
> 253 0 13084291072 dm-0
>
>
> DB:
what db command? printing sb 0 I assume but it' worth being explicit.
-Eric
> magicnum = 0x58465342
> blocksize = 4096
> dblocks = 3064987648
> rblocks = 0
> rextents = 0
> uuid = f086bb71-d67b-4cc1-b622-1f10349e6a49
> logstart = 1073741828
> rootino = null
> rbmino = null
> rsumino = null
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: xfs_repair problem.
2008-12-21 15:55 ` Eric Sandeen
@ 2008-12-21 16:52 ` David Bernick
2008-12-21 17:01 ` Eric Sandeen
2008-12-21 17:12 ` Eric Sandeen
0 siblings, 2 replies; 12+ messages in thread
From: David Bernick @ 2008-12-21 16:52 UTC (permalink / raw)
To: Eric Sandeen; +Cc: xfs
Thanks for the help so far:
It my output was from "sb 0". Thanks for reminding me to be explicit.
The system is a 64-bit system with 32-GB of RAM. It's going through the FS
right now with XFS repair.
Output of xfs_repair says, "arno=3" and about 81.6% of RAM is used by the
process. Think 32 G will be enough to handle this task?
I actually don't KNOW the original error, unfortunately, when growing. I
came into this late.
We're using repair 2.9.4. Worth getting a more recent version?
Kernel is - 2.6.18-92.1.1.el5
I "backed off" by vgsplit-ing the new physical device from the original
vgroup, so I was left with my original partition. I am hoping to mount the
original device since the "expanded" fs didn't work. I am hoping xfs_repair
helps that.
Why do you say its a "sparse" fs?
How do you go about writing to these inodes with these values:
rootino = 128
rbmino = 129
rsumino = 130
without affecting the data?
Thanks for any help!
On Sun, Dec 21, 2008 at 10:55 AM, Eric Sandeen <sandeen@sandeen.net> wrote:
> David Bernick wrote:
> > I have a filesystem where an xfs_growfs went bad. We were adding
> > storage to a pre-existing infrastructure and when growing, we received
> > an error.
>
> The error you encountered would be worth mentioning in detail...
>
> > SInce then, we've been unable to mount.
> >
> > It's a large filesystem (see below) and the addition of the extra data
> > has made it larger. We tried an xfs_repair but it died, as the machine
> > only has 4GB RAM.
>
> Do you have the latest version of repair? (xfs_repair -V; 2.10.2 is
> latest)
>
> > We're going to put 32 GB in RAM and see if that
> > helps. The original FS size is about 13T and the addition brought it
> > to 29T.
>
> on a 64-bit box I hope? what kernel version?
>
> > Since we've been unable to write or mount, we've "backed off" the
> > addition and are left with our original, which we'd like to mount.
>
> How did you back it off? either the fs grew or it didn't; and you can't
> shrink... so I guess it did not grow...
>
> > We try to mount and get an error about the root inode not being
> > readable. Makes sense as the root inode is null (according to xfs_db).
> >
> > So before we run another big xfs_repair:
> >
> > 1. What is the math of filesystems size, number of files and how much
> > RAM is needed for such a task? Is 32 GB enough for 1/2 Billion files
> > and 13 TB?
> >
> > 2. Any way I can just find my rootino,rbmino,rsumino and put them in the
> DB?
>
> I looked at the db output below & re-made a similar sparse fs:
>
> [root tmp]# bc
> 3064987648*4096
> 12554189406208
> quit
> [root tmp]# mkfs.xfs -dfile,name=testfile,size=12554189406208
> meta-data=testfile isize=256 agcount=12,
> agsize=268435455 blks
> = sectsz=512 attr=2
> data = bsize=4096 blocks=3064987648, imaxpct=5
> = sunit=0 swidth=0 blks
> naming =version 2 bsize=4096
> log =internal log bsize=4096 blocks=32768, version=2
> = sectsz=512 sunit=0 blks, lazy-count=0
> realtime =none extsz=4096 blocks=0, rtextents=0
> ls -lh [root tmp]# ls -l testfile
> -rw-r--r-- 1 root root 12554189406208 Dec 21 09:51 testfile
> [root tmp]# xfs_db testfile
> xfs_db> sb 0
> xfs_db> p
> magicnum = 0x58465342
> blocksize = 4096
> dblocks = 3064987648
> rblocks = 0
> rextents = 0
> uuid = 4b0451c8-5be4-452f-b161-a3ada3ec1a20
> logstart = 1610612740
> rootino = 128
> rbmino = 129
> rsumino = 130
>
> so that's most likely what it should be.
>
> > Any other advice?
>
> post more details of how things actually fail and what happened...
>
>
> > /proc/partitions:
> > 253 0 13084291072 dm-0
> >
>
> >
> > DB:
>
> what db command? printing sb 0 I assume but it' worth being explicit.
>
> -Eric
>
>
> > magicnum = 0x58465342
> > blocksize = 4096
> > dblocks = 3064987648
> > rblocks = 0
> > rextents = 0
> > uuid = f086bb71-d67b-4cc1-b622-1f10349e6a49
> > logstart = 1073741828
> > rootino = null
> > rbmino = null
> > rsumino = null
>
>
[[HTML alternate version deleted]]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: xfs_repair problem.
2008-12-21 16:52 ` David Bernick
@ 2008-12-21 17:01 ` Eric Sandeen
2008-12-21 17:07 ` David Bernick
2008-12-21 22:08 ` David Bernick
2008-12-21 17:12 ` Eric Sandeen
1 sibling, 2 replies; 12+ messages in thread
From: Eric Sandeen @ 2008-12-21 17:01 UTC (permalink / raw)
To: David Bernick; +Cc: xfs
David Bernick wrote:
> Thanks for the help so far:
>
> It my output was from "sb 0". Thanks for reminding me to be explicit.
>
> The system is a 64-bit system with 32-GB of RAM. It's going through the
> FS right now with XFS repair.
> Output of xfs_repair says, "arno=3" and about 81.6% of RAM is used by
> the process. Think 32 G will be enough to handle this task?
> I actually don't KNOW the original error, unfortunately, when growing. I
> came into this late.
>
> We're using repair 2.9.4. Worth getting a more recent version?
2.9.8 had some memory usage improvements (reductions) for repair IIRC
> Kernel is - 2.6.18-92.1.1.el5
heh; RHEL5 does not support xfs ;)
You probably hit:
TAKE 959978 - growing an XFS filesystem by more than 2TB is broken
http://oss.sgi.com/archives/xfs/2007-01/msg00053.html
I'd see if you can get centos to backport that fix (I assume you're
using centos or at least their kernel module; if not you can backport it
yourself...)
> I "backed off" by vgsplit-ing the new physical device from the original
> vgroup, so I was left with my original partition. I am hoping to mount
> the original device since the "expanded" fs didn't work. I am hoping
> xfs_repair helps that.
well, you don't want to take out part of the device if the fs thinks it
owns it now, but from the db output I think you still have the smaller size.
I'd read through:
http://oss.sgi.com/archives/xfs/2008-01/msg00085.html
and see if it helps you recover.
-Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: xfs_repair problem.
2008-12-21 17:01 ` Eric Sandeen
@ 2008-12-21 17:07 ` David Bernick
2008-12-21 22:08 ` David Bernick
1 sibling, 0 replies; 12+ messages in thread
From: David Bernick @ 2008-12-21 17:07 UTC (permalink / raw)
To: Eric Sandeen; +Cc: xfs
The xfs_repair is running. seems to have stabilized at about 91% RAM usage.
It's at a part where its "clearing inode number in entry at offset...", so
hopefully its just progressing.
I'll grab the newest version of xfs_repair and try that if this all fails.
WHere can I grab the latest?
>
> > Kernel is - 2.6.18-92.1.1.el5
>
> heh; RHEL5 does not support xfs ;)
>
shhh. i know that :)
"well, you don't want to take out part of the device if the fs thinks it
owns it now, but from the db output I think you still have the smaller
size."
That's what I thought from looking at everything, too.
"I'd read through:
http://oss.sgi.com/archives/xfs/2008-01/msg00085.html"
I've been pouring through that thread since 3am EST :)
[[HTML alternate version deleted]]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: xfs_repair problem.
2008-12-21 17:01 ` Eric Sandeen
2008-12-21 17:07 ` David Bernick
@ 2008-12-21 22:08 ` David Bernick
2008-12-21 22:14 ` David Bernick
2008-12-21 22:20 ` Eric Sandeen
1 sibling, 2 replies; 12+ messages in thread
From: David Bernick @ 2008-12-21 22:08 UTC (permalink / raw)
To: Eric Sandeen; +Cc: xfs
So I ran an xfs_repair -v on my filesystem. While the FS was originally 12T
and 95% full. I ran xfs_repair and it throw many "out of space errors" when
it was running. That makes some sense.
I expanded it with a new device with xfs_grow. It seems to work, because the
disk is now bigger when I mount it.
When I ran xfs_repair on that (latest version), it reverts back to the
original size. Is there anything I need to do to make the xfs_growfs
permanent?
If I can make the XFS partition bigger, I can likely make it work because
then it won't run out of space! But the partition, despite saying its
"bigger" here, doesn't seem to "take" after the xfs_repair. Any ideas?
Below is the xfs_db and some other useful things.
8 17 12695312483 sdb1
8 33 8803844062 sdc1
8 34 4280452000 sdc2
--- Logical volume ---
LV Name /dev/docs/v5Docs
VG Name docs
LV UUID G85Zi9-s63C-yWrU-yyf0-STP6-YOhJ-6Ne3pS
LV Write Access read/write
LV Status available
# open 1
LV Size 24.01 TB
Current LE 6293848
Segments 3
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:5
xfs_db> sb 0
xfs_db> print
magicnum = 0x58465342
blocksize = 4096
dblocks = 6444900352
rblocks = 0
rextents = 0
uuid = f086bb71-d67b-4cc1-b622-1f10349e6a49
logstart = 1073741828
rootino = 128
rbmino = 129
rsumino = 130
rextsize = 1
agblocks = 67108864
agcount = 97
rbmblocks = 0
logblocks = 32768
versionnum = 0x3084
sectsize = 512
inodesize = 256
inopblock = 16
fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
blocklog = 12
sectlog = 9
inodelog = 8
inopblog = 4
agblklog = 26
rextslog = 0
inprogress = 0
imax_pct = 25
icount = 149545792
ifree = 274
fdblocks = 4275133930
frextents = 0
uquotino = 0
gquotino = 0
qflags = 0
flags = 0
shared_vn = 0
inoalignmt = 2
unit = 0
width = 0
dirblklog = 0
logsectlog = 0
logsectsize = 0
logsunit = 0
features2 = 0
On Sun, Dec 21, 2008 at 12:01 PM, Eric Sandeen <sandeen@sandeen.net> wrote:
> David Bernick wrote:
> > Thanks for the help so far:
> >
> > It my output was from "sb 0". Thanks for reminding me to be explicit.
> >
> > The system is a 64-bit system with 32-GB of RAM. It's going through the
> > FS right now with XFS repair.
> > Output of xfs_repair says, "arno=3" and about 81.6% of RAM is used by
> > the process. Think 32 G will be enough to handle this task?
> > I actually don't KNOW the original error, unfortunately, when growing. I
> > came into this late.
> >
> > We're using repair 2.9.4. Worth getting a more recent version?
>
> 2.9.8 had some memory usage improvements (reductions) for repair IIRC
>
> > Kernel is - 2.6.18-92.1.1.el5
>
> heh; RHEL5 does not support xfs ;)
>
> You probably hit:
>
> TAKE 959978 - growing an XFS filesystem by more than 2TB is broken
> http://oss.sgi.com/archives/xfs/2007-01/msg00053.html
>
> I'd see if you can get centos to backport that fix (I assume you're
> using centos or at least their kernel module; if not you can backport it
> yourself...)
>
> > I "backed off" by vgsplit-ing the new physical device from the original
> > vgroup, so I was left with my original partition. I am hoping to mount
> > the original device since the "expanded" fs didn't work. I am hoping
> > xfs_repair helps that.
>
> well, you don't want to take out part of the device if the fs thinks it
> owns it now, but from the db output I think you still have the smaller
> size.
>
> I'd read through:
>
> http://oss.sgi.com/archives/xfs/2008-01/msg00085.html
>
> and see if it helps you recover.
>
> -Eric
>
[[HTML alternate version deleted]]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: xfs_repair problem.
2008-12-21 22:08 ` David Bernick
@ 2008-12-21 22:14 ` David Bernick
2008-12-21 22:20 ` Eric Sandeen
1 sibling, 0 replies; 12+ messages in thread
From: David Bernick @ 2008-12-21 22:14 UTC (permalink / raw)
To: Eric Sandeen; +Cc: xfs
and: are you available for professional services to help us on this problem?
Oh, and:
[root@luceneindex1-nap ~]# xfs_info /var/mnt/v5Docs/
meta-data=/dev/docs/v5Docs isize=256 agcount=97, agsize=67108864
blks
= sectsz=512 attr=0
data = bsize=4096 blocks=6444900352, imaxpct=25
= sunit=0 swidth=0 blks, unwritten=1
naming =version 2 bsize=4096
log =internal bsize=4096 blocks=32768, version=1
= sectsz=512 sunit=0 blks, lazy-count=0
realtime =none extsz=4096 blocks=0, rtextents=0
On Sun, Dec 21, 2008 at 5:08 PM, David Bernick <dbernick@gmail.com> wrote:
> So I ran an xfs_repair -v on my filesystem. While the FS was originally 12T
> and 95% full. I ran xfs_repair and it throw many "out of space errors" when
> it was running. That makes some sense.
>
> I expanded it with a new device with xfs_grow. It seems to work, because
> the disk is now bigger when I mount it.
>
> When I ran xfs_repair on that (latest version), it reverts back to the
> original size. Is there anything I need to do to make the xfs_growfs
> permanent?
>
> If I can make the XFS partition bigger, I can likely make it work because
> then it won't run out of space! But the partition, despite saying its
> "bigger" here, doesn't seem to "take" after the xfs_repair. Any ideas?
>
> Below is the xfs_db and some other useful things.
> 8 17 12695312483 sdb1
> 8 33 8803844062 sdc1
> 8 34 4280452000 sdc2
>
> --- Logical volume ---
> LV Name /dev/docs/v5Docs
> VG Name docs
> LV UUID G85Zi9-s63C-yWrU-yyf0-STP6-YOhJ-6Ne3pS
> LV Write Access read/write
> LV Status available
> # open 1
> LV Size 24.01 TB
> Current LE 6293848
> Segments 3
> Allocation inherit
> Read ahead sectors auto
> - currently set to 256
> Block device 253:5
> xfs_db> sb 0
> xfs_db> print
> magicnum = 0x58465342
> blocksize = 4096
> dblocks = 6444900352
> rblocks = 0
> rextents = 0
> uuid = f086bb71-d67b-4cc1-b622-1f10349e6a49
> logstart = 1073741828
> rootino = 128
> rbmino = 129
> rsumino = 130
> rextsize = 1
> agblocks = 67108864
> agcount = 97
> rbmblocks = 0
> logblocks = 32768
> versionnum = 0x3084
> sectsize = 512
> inodesize = 256
> inopblock = 16
> fname = "\000\000\000\000\000\000\000\000\000\000\000\000"
> blocklog = 12
> sectlog = 9
> inodelog = 8
> inopblog = 4
> agblklog = 26
> rextslog = 0
> inprogress = 0
> imax_pct = 25
> icount = 149545792
> ifree = 274
> fdblocks = 4275133930
> frextents = 0
> uquotino = 0
> gquotino = 0
> qflags = 0
> flags = 0
> shared_vn = 0
> inoalignmt = 2
> unit = 0
> width = 0
> dirblklog = 0
> logsectlog = 0
> logsectsize = 0
> logsunit = 0
> features2 = 0
>
>
> On Sun, Dec 21, 2008 at 12:01 PM, Eric Sandeen <sandeen@sandeen.net>wrote:
>
>> David Bernick wrote:
>> > Thanks for the help so far:
>> >
>> > It my output was from "sb 0". Thanks for reminding me to be explicit.
>> >
>> > The system is a 64-bit system with 32-GB of RAM. It's going through the
>> > FS right now with XFS repair.
>> > Output of xfs_repair says, "arno=3" and about 81.6% of RAM is used by
>> > the process. Think 32 G will be enough to handle this task?
>> > I actually don't KNOW the original error, unfortunately, when growing. I
>> > came into this late.
>> >
>> > We're using repair 2.9.4. Worth getting a more recent version?
>>
>> 2.9.8 had some memory usage improvements (reductions) for repair IIRC
>>
>> > Kernel is - 2.6.18-92.1.1.el5
>>
>> heh; RHEL5 does not support xfs ;)
>>
>> You probably hit:
>>
>> TAKE 959978 - growing an XFS filesystem by more than 2TB is broken
>> http://oss.sgi.com/archives/xfs/2007-01/msg00053.html
>>
>> I'd see if you can get centos to backport that fix (I assume you're
>> using centos or at least their kernel module; if not you can backport it
>> yourself...)
>>
>> > I "backed off" by vgsplit-ing the new physical device from the original
>> > vgroup, so I was left with my original partition. I am hoping to mount
>> > the original device since the "expanded" fs didn't work. I am hoping
>> > xfs_repair helps that.
>>
>> well, you don't want to take out part of the device if the fs thinks it
>> owns it now, but from the db output I think you still have the smaller
>> size.
>>
>> I'd read through:
>>
>> http://oss.sgi.com/archives/xfs/2008-01/msg00085.html
>>
>> and see if it helps you recover.
>>
>> -Eric
>>
>
>
[[HTML alternate version deleted]]
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 12+ messages in thread* Re: xfs_repair problem.
2008-12-21 22:08 ` David Bernick
2008-12-21 22:14 ` David Bernick
@ 2008-12-21 22:20 ` Eric Sandeen
1 sibling, 0 replies; 12+ messages in thread
From: Eric Sandeen @ 2008-12-21 22:20 UTC (permalink / raw)
To: David Bernick; +Cc: xfs
David Bernick wrote:
> So I ran an xfs_repair -v on my filesystem. While the FS was originally
> 12T and 95% full. I ran xfs_repair and it throw many "out of space
> errors" when it was running. That makes some sense.
>
> I expanded it with a new device with xfs_grow. It seems to work, because
> the disk is now bigger when I mount it.
I should have pointed out; growing less than 2T at a time is safe, even
with that bug ...
> When I ran xfs_repair on that (latest version), it reverts back to the
> original size. Is there anything I need to do to make the xfs_growfs
> permanent?
No, it should just work. But 2T at a time on that older code w/ teh bug.
> If I can make the XFS partition bigger, I can likely make it work
> because then it won't run out of space! But the partition, despite
> saying its "bigger" here, doesn't seem to "take" after the xfs_repair.
> Any ideas?
Add a terabyte at a time :)
(or backport that patch...)
-Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: xfs_repair problem.
2008-12-21 16:52 ` David Bernick
2008-12-21 17:01 ` Eric Sandeen
@ 2008-12-21 17:12 ` Eric Sandeen
1 sibling, 0 replies; 12+ messages in thread
From: Eric Sandeen @ 2008-12-21 17:12 UTC (permalink / raw)
To: David Bernick; +Cc: xfs
David Bernick wrote:
> How do you go about writing to these inodes with these values:
> rootino = 128
> rbmino = 129
> rsumino = 130
> without affecting the data?
forgot this answer...
start xfs_db with -x, then:
xfs_db> sb 0
xfs_db> write rsumino 130
rsumino = 130
etc
-Eric
_______________________________________________
xfs mailing list
xfs@oss.sgi.com
http://oss.sgi.com/mailman/listinfo/xfs
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2008-12-21 22:20 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2007-12-06 16:10 XFS_Repair PRoblem Kingghost
2007-12-06 21:06 ` David Chinner
2007-12-07 0:51 ` Kingghost
-- strict thread matches above, loose matches on Subject: below --
2008-12-21 15:03 xfs_repair problem David Bernick
2008-12-21 15:55 ` Eric Sandeen
2008-12-21 16:52 ` David Bernick
2008-12-21 17:01 ` Eric Sandeen
2008-12-21 17:07 ` David Bernick
2008-12-21 22:08 ` David Bernick
2008-12-21 22:14 ` David Bernick
2008-12-21 22:20 ` Eric Sandeen
2008-12-21 17:12 ` Eric Sandeen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox