* e2fsck repeatedly asks to clear the same entry?
@ 2012-11-23 18:26 Mark Casey
2012-11-23 19:18 ` Andreas Dilger
0 siblings, 1 reply; 6+ messages in thread
From: Mark Casey @ 2012-11-23 18:26 UTC (permalink / raw)
To: linux-ext4
Hello list,
I'm in a bit of a loop trying to fix my ext4 filesystem; it always goes
like this even after several passes.
> root@host:/home/luser# /root/latest/sbin/e2fsck -f /dev/vgdalr6/lv1
> e2fsck 1.42.5 (29-Jul-2012)
> Pass 1: Checking inodes, blocks, and sizes
> Pass 2: Checking directory structure
> Entry 'A5 11-3' in /share/path/09/Brett/Pines/Flynt's Side Drive - Complete Archive Copy/SA Version Pines/Chris Pics 11-2-10 (268533857) has deleted/unused inode 15115. Clear<y>? yes
> Pass 3: Checking directory connectivity
> Pass 4: Checking reference counts
> Pass 5: Checking group summary information
>
> /dev/vgdalr6/lv1: ***** FILE SYSTEM WAS MODIFIED *****
> /dev/vgdalr6/lv1: 8993801/268697600 files (0.8% non-contiguous), 2152234742/4299161600 blocks
> root@host:/home/luser#
Would anyone have any suggestions how to proceed?
The cause of this is that I did an unsupported resize (a shrink) by
commenting out one of resize2fs' checks...as described here: (note I'm
not claiming this as "permission"; I knew it might not work)
http://article.gmane.org/gmane.comp.file-systems.ext4/35375
resize2fs gave no indication of trouble but the check that followed
didn't go perfectly. 6 or 7 directory entries needed to be cleared and
I've restored ~10gb from backup, but otherwise this current issue with
the directory 'A5 11-3' is the only symptom presenting. I have most of
the e2fsck log that followed the resize in case that would be of use.
Thank you,
Mark
^ permalink raw reply [flat|nested] 6+ messages in thread* Re: e2fsck repeatedly asks to clear the same entry? 2012-11-23 18:26 e2fsck repeatedly asks to clear the same entry? Mark Casey @ 2012-11-23 19:18 ` Andreas Dilger 2012-11-24 6:27 ` Mark Casey 2012-11-24 17:34 ` Eric Sandeen 0 siblings, 2 replies; 6+ messages in thread From: Andreas Dilger @ 2012-11-23 19:18 UTC (permalink / raw) To: Mark Casey; +Cc: linux-ext4@vger.kernel.org On 2012-11-23, at 11:26, Mark Casey <markc@unifiedgroup.com> wrote: > I'm in a bit of a loop trying to fix my ext4 filesystem; it always goes like this even after several passes. > >> root@host:/home/luser# /root/latest/sbin/e2fsck -f /dev/vgdalr6/lv1 >> e2fsck 1.42.5 (29-Jul-2012) >> Pass 1: Checking inodes, blocks, and sizes >> Pass 2: Checking directory structure >> Entry 'A5 11-3' in /share/path/09/Brett/Pines/Flynt's Side Drive - Complete Archive Copy/SA Version Pines/Chris Pics 11-2-10 (268533857) has deleted/unused inode 15115. Clear<y>? yes >> Pass 3: Checking directory connectivity >> Pass 4: Checking reference counts >> Pass 5: Checking group summary information >> >> /dev/vgdalr6/lv1: ***** FILE SYSTEM WAS MODIFIED ***** >> /dev/vgdalr6/lv1: 8993801/268697600 files (0.8% non-contiguous), 2152234742/4299161600 blocks >> root@host:/home/luser# > > > Would anyone have any suggestions how to proceed? > > The cause of this is that I did an unsupported resize (a shrink) by commenting out one of resize2fs' checks...as described here: (note I'm not claiming this as "permission"; I knew it might not work) E2fsck should be able to fix (i.e. get into some consistent state) anything regardless of how it got into that state. > http://article.gmane.org/gmane.comp.file-systems.ext4/35375 > > this current issue with the directory 'A5 11-3' is the only symptom presenting. I have most of the e2fsck log that followed the resize in case that would be of use. It would probably be useful to get information from debugfs for this directory and inode (stats, ls -l <268533857>, stat <15115>, and checki 15115). Normally I'd say that getting a e2image of the filesystem would be useful for debugging and to create a test case, but since the filesystem is 16TB in size that won't be practical. In the worst case, it should be possible to fix this manually on debugfs either by marking the inode in use in the bitmap (seti 15115) or clear the inode number in the directory entry (on my phone right now and can't check the command for this). It would be nice to get a test case first, so that e2fsck could be fixed, so if this isn't causing you grief it would be nice to keep this around until there is a chance to understand the problem. Cheers, Andreas ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: e2fsck repeatedly asks to clear the same entry? 2012-11-23 19:18 ` Andreas Dilger @ 2012-11-24 6:27 ` Mark Casey 2012-11-24 19:17 ` Andreas Dilger 2012-11-24 17:34 ` Eric Sandeen 1 sibling, 1 reply; 6+ messages in thread From: Mark Casey @ 2012-11-24 6:27 UTC (permalink / raw) To: linux-ext4 On 11/23/2012 1:18 PM, Andreas Dilger wrote: > On 2012-11-23, at 11:26, Mark Casey <markc@unifiedgroup.com> wrote: >> I'm in a bit of a loop trying to fix my ext4 filesystem; it always goes like this even after several passes. >> >>> root@host:/home/luser# /root/latest/sbin/e2fsck -f /dev/vgdalr6/lv1 >>> e2fsck 1.42.5 (29-Jul-2012) >>> Pass 1: Checking inodes, blocks, and sizes >>> Pass 2: Checking directory structure >>> Entry 'A5 11-3' in /share/path/09/Brett/Pines/Flynt's Side Drive - Complete Archive Copy/SA Version Pines/Chris Pics 11-2-10 (268533857) has deleted/unused inode 15115. Clear<y>? yes >>> Pass 3: Checking directory connectivity >>> Pass 4: Checking reference counts >>> Pass 5: Checking group summary information >>> >>> /dev/vgdalr6/lv1: ***** FILE SYSTEM WAS MODIFIED ***** >>> /dev/vgdalr6/lv1: 8993801/268697600 files (0.8% non-contiguous), 2152234742/4299161600 blocks >>> root@host:/home/luser# >> >> >> Would anyone have any suggestions how to proceed? >> >> The cause of this is that I did an unsupported resize (a shrink) by commenting out one of resize2fs' checks...as described here: (note I'm not claiming this as "permission"; I knew it might not work) > > E2fsck should be able to fix (i.e. get into some consistent state) anything regardless of how it got into that state. > >> http://article.gmane.org/gmane.comp.file-systems.ext4/35375 >> >> this current issue with the directory 'A5 11-3' is the only symptom presenting. I have most of the e2fsck log that followed the resize in case that would be of use. > > It would probably be useful to get information from debugfs for this directory and inode (stats, ls -l <268533857>, stat <15115>, and checki 15115). I've gathered that info as best I can. Last one didn't want to work. Let me know if there's more: * stats * Filesystem volume name: <none> Last mounted on: /home Filesystem UUID: 3652885c-e8c6-4f4d-86a0-a4c1d1784557 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 268697600 Block count: 4299161600 Reserved block count: 10747903 Free blocks: 2146933332 Free inodes: 259703799 First block: 0 Block size: 4096 Fragment size: 4096 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 2048 Inode blocks per group: 128 RAID stride: 64 RAID stripe width: 576 Flex block group size: 16 Filesystem created: Sun Sep 9 18:40:39 2012 Last mount time: Fri Nov 23 11:12:16 2012 Last write time: Fri Nov 23 12:27:56 2012 Mount count: 0 Maximum mount count: -1 Last checked: Fri Nov 23 12:27:56 2012 Check interval: 0 (<none>) Lifetime writes: 12 TB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 28 Desired extra isize: 28 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: 94884f6d-8b2e-4830-a33b-02652aee727c Journal backup: inode blocks Directories: 258340 * ls -l * 268533857 42757 (2) 1000 1000 4096 22-Nov-2012 12:38 . 268304391 42757 (2) 1000 1000 4096 11-Jan-2011 18:06 .. 15111 42757 (2) 1000 1000 20480 11-Jan-2011 17:44 Group 1 15112 42757 (2) 1000 1000 4096 11-Jan-2011 17:44 Group 2 15113 42757 (2) 1000 1000 16384 11-Jan-2011 17:45 Group 3 15114 42757 (2) 1000 1000 12288 11-Jan-2011 17:46 Group 4 11-2 15115 42757 (2) 1000 1000 36864 11-Jan-2011 17:48 Group 5 11-3 15116 42757 (2) 1000 1000 40960 11-Jan-2011 17:51 Group 6 11-4 * stat * User: 1000 Group: 1000 Size: 36864 File ACL: 0 Directory ACL: 0 Links: 0 Blockcount: 80 Fragment: Address: 0 Number: 0 Size: 0 ctime: 0x50ae708d:cb6e6828 -- Thu Nov 22 12:35:57 2012 atime: 0x504eda66:cb6e6828 -- Tue Sep 11 01:29:58 2012 mtime: 0x4d2cec69:00000000 -- Tue Jan 11 17:48:57 2011 crtime: 0x504ed9f5:dab0e61c -- Tue Sep 11 01:28:05 2012 dtime: 0x50af000c -- Thu Nov 22 22:48:12 2012 Size of extra inode fields: 28 > > Normally I'd say that getting a e2image of the filesystem would be useful for debugging and to create a test case, but since the filesystem is 16TB in size that won't be practical. > > In the worst case, it should be possible to fix this manually on debugfs either by marking the inode in use in the bitmap (seti 15115) or clear the inode number in the directory entry (on my phone right now and can't check the command for this). > > It would be nice to get a test case first, so that e2fsck could be fixed, so if this isn't causing you grief it would be nice to keep this around until there is a chance to understand the problem. I'm assuming that with the rest of the fsck coming up clean it is safe to use the filesystem in production come Monday (Samba)? These files are older so I wouldn't mind setting the permissions so that no one can get to them for a bit. What would I need to do to get a test case going? > > Cheers, Andreas-- > To unsubscribe from this list: send the line "unsubscribe linux-ext4" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Thank you, Mark ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: e2fsck repeatedly asks to clear the same entry? 2012-11-24 6:27 ` Mark Casey @ 2012-11-24 19:17 ` Andreas Dilger 2012-11-26 3:52 ` Mark Casey 0 siblings, 1 reply; 6+ messages in thread From: Andreas Dilger @ 2012-11-24 19:17 UTC (permalink / raw) To: Mark Casey; +Cc: linux-ext4 On 2012-11-23, at 23:27, Mark Casey <markc@unifiedgroup.com> wrote: > On 11/23/2012 1:18 PM, Andreas Dilger wrote: >> On 2012-11-23, at 11:26, Mark Casey <markc@unifiedgroup.com> wrote: >>> I'm in a bit of a loop trying to fix my ext4 filesystem; it always goes like this even after several passes. >>> >>>> root@host:/home/luser# /root/latest/sbin/e2fsck -f /dev/vgdalr6/lv1 >>>> e2fsck 1.42.5 (29-Jul-2012) >>>> Pass 1: Checking inodes, blocks, and sizes >>>> Pass 2: Checking directory structure >>>> Entry 'A5 11-3' in /share/path/09/Brett/Pines/Flynt's Side Drive - Complete Archive Copy/SA Version Pines/Chris Pics 11-2-10 (268533857) has deleted/unused inode 15115. Clear<y>? yes >>>> Pass 3: Checking directory connectivity >>>> Pass 4: Checking reference counts >>>> Pass 5: Checking group summary information >>>> >>>> /dev/vgdalr6/lv1: ***** FILE SYSTEM WAS MODIFIED ***** >>>> /dev/vgdalr6/lv1: 8993801/268697600 files (0.8% non-contiguous), 2152234742/4299161600 blocks >>>> root@host:/home/luser# >>> >>> >>> Would anyone have any suggestions how to proceed? >>> >>> The cause of this is that I did an unsupported resize (a shrink) by commenting out one of resize2fs' checks...as described here: (note I'm not claiming this as "permission"; I knew it might not work) >> >> E2fsck should be able to fix (i.e. get into some consistent state) anything regardless of how it got into that state. >> >>> http://article.gmane.org/gmane.comp.file-systems.ext4/35375 >>> >>> this current issue with the directory 'A5 11-3' is the only symptom presenting. I have most of the e2fsck log that followed the resize in case that would be of use. >> >> It would probably be useful to get information from debugfs for this directory and inode (stats, ls -l <268533857>, stat <15115>, and checki 15115). > > I've gathered that info as best I can. Last one didn't want to work. Let me know if there's more: > > * stats * > Filesystem volume name: <none> > Last mounted on: /home > Filesystem UUID: 3652885c-e8c6-4f4d-86a0-a4c1d1784557 > Filesystem magic number: 0xEF53 > Filesystem revision #: 1 (dynamic) > Filesystem features: has_journal ext_attr dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file uninit_bg dir_nlink extra_isize > Filesystem flags: signed_directory_hash > Default mount options: user_xattr acl > Filesystem state: clean > Errors behavior: Continue > Filesystem OS type: Linux > Inode count: 268697600 > Block count: 4299161600 > Reserved block count: 10747903 > Free blocks: 2146933332 > Free inodes: 259703799 > First block: 0 > Block size: 4096 > Fragment size: 4096 > Blocks per group: 32768 > Fragments per group: 32768 > Inodes per group: 2048 > Inode blocks per group: 128 > RAID stride: 64 > RAID stripe width: 576 > Flex block group size: 16 > Filesystem created: Sun Sep 9 18:40:39 2012 > Last mount time: Fri Nov 23 11:12:16 2012 > Last write time: Fri Nov 23 12:27:56 2012 > Mount count: 0 > Maximum mount count: -1 > Last checked: Fri Nov 23 12:27:56 2012 > Check interval: 0 (<none>) > Lifetime writes: 12 TB > Reserved blocks uid: 0 (user root) > Reserved blocks gid: 0 (group root) > First inode: 11 > Inode size: 256 > Required extra isize: 28 > Desired extra isize: 28 > Journal inode: 8 > Default directory hash: half_md4 > Directory Hash Seed: 94884f6d-8b2e-4830-a33b-02652aee727c > Journal backup: inode blocks > Directories: 258340 > > * ls -l * > 268533857 42757 (2) 1000 1000 4096 22-Nov-2012 12:38 . > 268304391 42757 (2) 1000 1000 4096 11-Jan-2011 18:06 .. > 15111 42757 (2) 1000 1000 20480 11-Jan-2011 17:44 Group 1 > 15112 42757 (2) 1000 1000 4096 11-Jan-2011 17:44 Group 2 > 15113 42757 (2) 1000 1000 16384 11-Jan-2011 17:45 Group 3 > 15114 42757 (2) 1000 1000 12288 11-Jan-2011 17:46 Group 4 11-2 > 15115 42757 (2) 1000 1000 36864 11-Jan-2011 17:48 Group 5 11-3 So this is the problematic entry. The directory entry looks ok, though it doesn't have the same name as e2fsck reports. It claims the entry is "A5 11-3", which is a bit bizarre. > 15116 42757 (2) 1000 1000 40960 11-Jan-2011 17:51 Group 6 11-4 > > * stat * > User: 1000 Group: 1000 Size: 36864 > File ACL: 0 Directory ACL: 0 > Links: 0 Blockcount: 80 > Fragment: Address: 0 Number: 0 Size: 0 > ctime: 0x50ae708d:cb6e6828 -- Thu Nov 22 12:35:57 2012 > atime: 0x504eda66:cb6e6828 -- Tue Sep 11 01:29:58 2012 > mtime: 0x4d2cec69:00000000 -- Tue Jan 11 17:48:57 2011 > crtime: 0x504ed9f5:dab0e61c -- Tue Sep 11 01:28:05 2012 > dtime: 0x50af000c -- Thu Nov 22 22:48:12 2012 > Size of extra inode fields: 28 Both the nlinks here and the dtime show that this inode is deleted, so e2fsck is right in reporting that the directory entry is wrong. To fix this manually, you would need to set the inode number for this entry to zero, but I don't recall off the top of my head how to do this. >> Normally I'd say that getting a e2image of the filesystem would be useful for debugging and to create a test case, but since the filesystem is 16TB in size that won't be practical. >> >> In the worst case, it should be possible to fix this manually on debugfs either by marking the inode in use in the bitmap (seti 15115) or clear the inode number in the directory entry (on my phone right now and can't check the command for this). >> >> It would be nice to get a test case first, so that e2fsck could be fixed, so if this isn't causing you grief it would be nice to keep this around until there is a chance to understand the problem. > > I'm assuming that with the rest of the fsck coming up clean it is safe to use the filesystem in production come Monday (Samba)? Yes, though if this directory is accessed it might turn the filesystem read-only. > These files are older so I wouldn't mind setting the permissions so that no one can get to them for a bit. What would I need to do to get a test case going? Just mark the parent directory inaccessible: # chmod 000 "/share/path/09/Brett/Pines/Flynt's Side Drive - Complete Archive Copy/SA Version Pines/Chris Pics 11-2-10/Group 5 11-3" Cheers, Andreas ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: e2fsck repeatedly asks to clear the same entry? 2012-11-24 19:17 ` Andreas Dilger @ 2012-11-26 3:52 ` Mark Casey 0 siblings, 0 replies; 6+ messages in thread From: Mark Casey @ 2012-11-26 3:52 UTC (permalink / raw) To: linux-ext4 On 11/24/2012 1:17 PM, Andreas Dilger wrote: > > So this is the problematic entry. The directory entry looks ok, though it doesn't have the same name as e2fsck reports. It claims the entry is "A5 11-3", which is a bit bizarre. My fault. I was initially inconsistent in deciding what parts of our real tree I wanted online. So there is nothing named "A5 11-3". Should not be an issue again; sorry to add complication. > > ... > > Yes, though if this directory is accessed it might turn the filesystem read-only. Yep, it does. > >> These files are older so I wouldn't mind setting the permissions so that no one can get to them for a bit. What would I need to do to get a test case going? > > Just mark the parent directory inaccessible: > > # chmod 000 "/share/path/09/Brett/Pines/Flynt's Side Drive - Complete Archive Copy/SA Version Pines/Chris Pics 11-2-10/Group 5 11-3" > No problem. Once I've got that done I'll see what I can do with e2image. I really appreciate the input so I'd like to do whatever I can if you still think it might lead to some sort of bugfix. So far the only changes made were to restore the other files that the post-resize fsck had to remove/free. Final total was a couple gigs across 7 dirs. I've looked for any stat differences between the current file tree and the one from the backup just before the resize. The only issue found is that there appear to be ~200 directories that were not removed by fsck but appear to have had their modtimes reset by it instead. The actual files contained were untouched. If that is also no big concern then I think later tonight, after/if I can get an e2image done, I'll just restore their modtimes from the backup to make things pretty again. Thank you, Mark ^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: e2fsck repeatedly asks to clear the same entry? 2012-11-23 19:18 ` Andreas Dilger 2012-11-24 6:27 ` Mark Casey @ 2012-11-24 17:34 ` Eric Sandeen 1 sibling, 0 replies; 6+ messages in thread From: Eric Sandeen @ 2012-11-24 17:34 UTC (permalink / raw) To: Andreas Dilger; +Cc: Mark Casey, linux-ext4@vger.kernel.org On 11/23/12 1:18 PM, Andreas Dilger wrote: > On 2012-11-23, at 11:26, Mark Casey <markc@unifiedgroup.com> wrote: >> I'm in a bit of a loop trying to fix my ext4 filesystem; it always >> goes like this even after several passes. ... > Normally I'd say that getting a e2image of the filesystem would be > useful for debugging and to create a test case, but since the > filesystem is 16TB in size that won't be practical. It might not be that bad. You could make a raw e2image, mount it, remove some of the non-affected dir trees, (maybe make another e2image -r of that modified image), and it might compress pretty well. Just a thought, if the email debugging doesn't yield results. -Eric ^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2012-11-26 3:52 UTC | newest] Thread overview: 6+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2012-11-23 18:26 e2fsck repeatedly asks to clear the same entry? Mark Casey 2012-11-23 19:18 ` Andreas Dilger 2012-11-24 6:27 ` Mark Casey 2012-11-24 19:17 ` Andreas Dilger 2012-11-26 3:52 ` Mark Casey 2012-11-24 17:34 ` Eric Sandeen
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).