Re: same ext4 file system corruption on different machines

linux-ext4.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* Re: same ext4 file system corruption on different machines
       [not found] ` <1391000743.2601.26.camel@snow.station>
@ 2014-01-29 17:38   ` Theodore Ts'o
  2014-01-30  7:59     ` Luca Ognibene
  2014-01-30 11:02     ` Luca Ognibene
  0 siblings, 2 replies; 5+ messages in thread
From: Theodore Ts'o @ 2014-01-29 17:38 UTC (permalink / raw)
  To: Luca Ognibene; +Cc: linux-kernel, linux-ext4

On Wed, Jan 29, 2014 at 02:05:43PM +0100, Luca Ognibene wrote:
> I say "same ext4 file system corruption" because e2fsck reports errors
> on inodes around 127233 on all file systems.a I'm not sure about the
> syslog errors because i have syslog logs for only the latest faulty
> partition.

The e2fsck output shows that all of the inodes in a tight sequential
range around 127233 are getting corrupted.  That implies that a
specific block is getting corrupted.  You can see which block by using
the imap command in debugfs:

# debugfs -R "imap <12345>" /dev/sda3
debugfs 1.42.9 (28-Dec-2013)
Inode 12345 is part of block group 1
      located at block 1828, offset 0x0800

The fact that the corruption is so consistenth is highly suspicious.
It tends to rule out hardware errrors, but it tends to also rule out
most kernel bugs.  If it's caused by some race condition, or wild
pointer dereference, it's highly unlikely it would result in the same
block getting overwritten with garbage.

It might be worthwhile to try using the block_dump command, but that's
not in the 1.42 version of e2fsprogs.  You'd have to upgrade to a
newer version of e2fsprogs, or find some other block editor that
understands 4k block numbers.  For example:

502# debugfs /dev/sda3
debugfs 1.42.9 (28-Dec-2013)
debugfs:  imap <11>
Inode 11 is part of block group 0
      located at block 1057, offset 0x0a00
debugfs:  bd 1057
0000  0000 0000 0000 0000 3650 6951 3650 6951  ........6PiQ6PiQ
0020  3650 6951 0000 0000 0000 0000 0000 0000  6PiQ............
0040  0000 0000 0000 0000 0000 0000 0000 0000  ................
*
0400  ed41 0000 0010 0000 76f5 e852 2797 e252  .A......v..R'..R
0420  2797 e252 0000 0000 0000 2400 0800 0000  '..R......$.....
0440  0000 0800 4201 0000 0af3 0100 0400 0000  ....B...........
0460  0000 0000 0000 0000 0100 0000 2124 0000  ............!$..
0500  0000 0000 0000 0000 0000 0000 0000 0000  ................
*
0600  1c00 0000 d08b 1ed0 d08b 1ed0 f426 f411  .............&..
0620  3650 6951 0000 0000 0000 0000 0000 02ea  6PiQ............
0640  0706 4400 0000 0000 1c00 0000 0000 0000  ..D.............
0660  7365 6c69 6e75 7800 0000 0000 0000 0000  selinux.........
0700  0000 0000 0000 0000 0000 0000 0000 0000  ................
*
0740  0000 0000 7379 7374 656d 5f75 3a6f 626a  ....system_u:obj
0760  6563 745f 723a 726f 6f74 5f74 3a73 3000  ect_r:root_t:s0.
1000  0000 0000 0000 0000 0000 0000 0000 0000  ................
 ...

Do this *before* you allow e2fsck to fix the file system.  It may be
that you'll see something that will identify the source of where the
data which is corrupting the inode table.

Cheers,

					- Ted

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: same ext4 file system corruption on different machines
  2014-01-29 17:38   ` same ext4 file system corruption on different machines Theodore Ts'o
@ 2014-01-30  7:59     ` Luca Ognibene
  2014-01-31  3:00       ` Theodore Ts'o
  2014-01-30 11:02     ` Luca Ognibene
  1 sibling, 1 reply; 5+ messages in thread
From: Luca Ognibene @ 2014-01-30  7:59 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: linux-kernel, linux-ext4

Il giorno mer, 29/01/2014 alle 12.38 -0500, Theodore Ts'o ha scritto:
> On Wed, Jan 29, 2014 at 02:05:43PM +0100, Luca Ognibene wrote:
> > I say "same ext4 file system corruption" because e2fsck reports errors
> > on inodes around 127233 on all file systems.a I'm not sure about the
> > syslog errors because i have syslog logs for only the latest faulty
> > partition.
> 
> The e2fsck output shows that all of the inodes in a tight sequential
> range around 127233 are getting corrupted.  That implies that a
> specific block is getting corrupted.  You can see which block by using
> the imap command in debugfs:
> 
> # debugfs -R "imap <12345>" /dev/sda3
> debugfs 1.42.9 (28-Dec-2013)
> Inode 12345 is part of block group 1
>       located at block 1828, offset 0x0800

These are debugfs(1.42.9) outputs. BROKEN1/BROKEN2 are two partitions
with the problem, CORRECT1 is a system without the problem made from the
same root image.

BROKEN1

debugfs:  imap <127233>
Inode 127233 is part of block group 16
        located at block 524320, offset 0x0000

debugfs:  bd 524320
0000  ffff ffff ffff ffff ffff ffff ffff ffff  ................
*

debugfs:  ex <127233>
debugfs:  

debugfs: stats
Filesystem volume name:   <none>
Last mounted on:          /root/dom/a
Filesystem UUID:          6ee8fc7f-9229-44c2-8173-921b9d625436
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index
filetype extent flex_bg sparse_super large_file huge_file uninit_bg
dir_nlink extra_isize
Filesystem flags:         signed_directory_hash 
Default mount options:    user_xattr acl
Filesystem state:         clean with errors
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              230608
Block count:              921600
Reserved block count:     9216
Free blocks:              298621
Free inodes:              128483
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      224
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         7952
Inode blocks per group:   497
Flex block group size:    16
Filesystem created:       Wed May 15 12:44:06 2013
Last mount time:          Tue Jan 28 17:45:16 2014
Last write time:          Thu Jan 30 07:31:43 2014
Mount count:              37
Maximum mount count:      -1
Last checked:             Fri Dec 13 16:56:50 2013
Check interval:           15552000 (6 months)
Next check after:         Wed Jun 11 17:56:50 2014
Lifetime writes:          2692 MB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      43472f4b-ba7f-4204-97d8-9addd838f9a6
Journal backup:           inode blocks
FS Error count:           13
First error time:         Wed Jan 15 08:46:12 2014
First error function:     ext4_iget
First error line #:       3888
First error inode #:      127233
First error block #:      0
Last error time:          Tue Jan 28 17:46:07 2014
Last error function:      ext4_iget
Last error line #:        3888
Last error inode #:       127233
Last error block #:       0
Directories:              11616
 Group  0: block bitmap at 226, inode bitmap at 242, inode table at 258
           17418 free blocks, 0 free inodes, 1013 used directories, 0
unused inodes
           [Checksum 0x2793]
 Group  1: block bitmap at 227, inode bitmap at 243, inode table at 755
           1 free block, 0 free inodes, 741 used directories, 0 unused
inodes
           [Checksum 0xb365]
 Group  2: block bitmap at 228, inode bitmap at 244, inode table at 1252
           114 free blocks, 0 free inodes, 1175 used directories, 0
unused inodes
           [Checksum 0xed5f]
 Group  3: block bitmap at 229, inode bitmap at 245, inode table at 1749
           10815 free blocks, 0 free inodes, 868 used directories, 0
unused inodes
           [Checksum 0x3127]
 Group  4: block bitmap at 230, inode bitmap at 246, inode table at 2246
           0 free blocks, 0 free inodes, 861 used directories, 0 unused
inodes
           [Checksum 0xfd99]
 Group  5: block bitmap at 231, inode bitmap at 247, inode table at 2743
           0 free blocks, 1 free inode, 741 used directories, 0 unused
inodes
           [Checksum 0x1b9e]
 Group  6: block bitmap at 232, inode bitmap at 248, inode table at 3240
           0 free blocks, 364 free inodes, 487 used directories, 0
unused inodes
           [Checksum 0x0b53]
 Group  7: block bitmap at 233, inode bitmap at 249, inode table at 3737
           78 free blocks, 3977 free inodes, 348 used directories, 3874
unused inodes
           [Checksum 0x8ffb]
 Group  8: block bitmap at 234, inode bitmap at 250, inode table at 4234
           0 free blocks, 7922 free inodes, 30 used directories, 7695
unused inodes
           [Checksum 0x0c36]
 Group  9: block bitmap at 235, inode bitmap at 251, inode table at 4731
           0 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0xefe1]
 Group 10: block bitmap at 236, inode bitmap at 252, inode table at 5228
           3259 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0x58db]
 Group 11: block bitmap at 237, inode bitmap at 253, inode table at 5725
           24703 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0x3373]
 Group 12: block bitmap at 238, inode bitmap at 254, inode table at 6222
           15305 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0xfb7a]
 Group 13: block bitmap at 239, inode bitmap at 255, inode table at 6719
           0 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0x2b8a]
 Group 14: block bitmap at 240, inode bitmap at 256, inode table at 7216
           19607 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0xfe1d]
 Group 15: block bitmap at 241, inode bitmap at 257, inode table at 7713
           21398 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0x556e]
 Group 16: block bitmap at 524288, inode bitmap at 524304, inode table
at 524320
           20533 free blocks, 996 free inodes, 762 used directories, 0
unused inodes
           [Checksum 0x7cef]
 Group 17: block bitmap at 524289, inode bitmap at 524305, inode table
at 524817
           2746 free blocks, 0 free inodes, 1828 used directories, 0
unused inodes
           [Checksum 0xda0d]
 Group 18: block bitmap at 524290, inode bitmap at 524306, inode table
at 525314
           1 free block, 0 free inodes, 537 used directories, 0 unused
inodes
           [Checksum 0xf843]
 Group 19: block bitmap at 524291, inode bitmap at 524307, inode table
at 525811
           0 free blocks, 0 free inodes, 626 used directories, 0 unused
inodes
           [Checksum 0xbbbf]
 Group 20: block bitmap at 524292, inode bitmap at 524308, inode table
at 526308
           1737 free blocks, 0 free inodes, 1188 used directories, 0
unused inodes
           [Checksum 0xed82]
 Group 21: block bitmap at 524293, inode bitmap at 524309, inode table
at 526805
           4641 free blocks, 3897 free inodes, 409 used directories,
3896 unused inodes
           [Checksum 0xb072]
 Group 22: block bitmap at 524294, inode bitmap at 524310, inode table
at 527302
           6752 free blocks, 7950 free inodes, 2 used directories, 7950
unused inodes
           [Checksum 0xfb30]
 Group 23: block bitmap at 524295, inode bitmap at 524311, inode table
at 527799
           14797 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0x8e2e]
 Group 24: block bitmap at 524296, inode bitmap at 524312, inode table
at 528296
           32768 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0x1c83]
 Group 25: block bitmap at 524297, inode bitmap at 524313, inode table
at 528793
           32542 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0xd472]
 Group 26: block bitmap at 524298, inode bitmap at 524314, inode table
at 529290
           32768 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Block not init, Checksum 0xc0d5]
 Group 27: block bitmap at 524299, inode bitmap at 524315, inode table
at 529787
           32542 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0x4a8a]
 Group 28: block bitmap at 524300, inode bitmap at 524316, inode table
at 530284
           4096 free blocks, 7952 free inodes, 0 used directories, 7952
unused inodes
           [Inode not init, Checksum 0x0c1e]

BROKEN2
root@ay5:~/dom# ../debugfs /dev/mapper/loop2p2 
debugfs 1.42.9 (28-Dec-2013)
debugfs:  imap <127233>
Inode 127233 is part of block group 16
        located at block 524320, offset 0x0000
debugfs:  bd 524320
0000  ffff ffff ffff ffff ffff ffff ffff ffff  ................
*


CORRECT1
debugfs:  imap <127233> (it's the /lib/x86_64-linux-gnu/ directory)
Inode 127233 is part of block group 16
        located at block 524320, offset 0x0000

debugfs:  bd 524320
0000  ed41 0000 0010 0000 9e36 6551 0d8f 9d52  .A.......6eQ...R
0020  0d8f 9d52 0000 0000 0000 0300 0800 0000  ...R............
0040  0000 0800 9100 0000 0af3 0100 0400 0000  ................
0060  0000 0000 0000 0000 0100 0000 0d00 0800  ................
0100  0000 0000 0000 0000 0000 0000 0000 0000  ................
*
0140  0000 0000 8dbb 0181 0000 0000 0000 0000  ................
0160  0000 0000 0000 0000 0000 0000 0000 0000  ................
0200  1c00 0000 685b 8833 685b 8833 e0fc bf03  ....h[.3h[.3....
0220  0f67 9351 5418 bfb6 0000 0000 0000 0000  .g.QT...........
0240  0000 0000 0000 0000 0000 0000 0000 0000  ................
*
0400  a481 0000 50dc 0100 575c 5d52 0f67 9351  ....P...W\]R.g.Q
0420  f7fd 544f 0000 0000 0000 0100 f000 0000  ..TO............
0440  0000 0800 0100 0000 0af3 0100 0400 0000  ................
0460  0000 0000 0000 0000 1e00 0000 0080 0800  ................
0500  0000 0000 0000 0000 0000 0000 0000 0000  ................
*
0540  0000 0000 8ebb 0181 0000 0000 0000 0000  ................
0560  0000 0000 0000 0000 0000 0000 0000 0000  ................
0600  1c00 0000 5418 bfb6 0000 0000 8499 039f  ....T...........
0620  0f67 9351 5418 bfb6 0000 0000 0000 0000  .g.QT...........
0640  0000 0000 0000 0000 0000 0000 0000 0000  ................
...........

debugfs:  ex <127233>
Level Entries       Logical        Physical Length Flags
 0/ 0   1/  1     0 -     0 524301 - 524301      1 

> The fact that the corruption is so consistenth is highly suspicious.
> It tends to rule out hardware errrors, but it tends to also rule out
> most kernel bugs.  If it's caused by some race condition, or wild
> pointer dereference, it's highly unlikely it would result in the same
> block getting overwritten with garbage.

Yes it's indeed very strange.. i tend to rule out application errors
because i don't write directly to the device so i don't think i can
break a filesystem from userspace. I've checked previous and next blocks
and they seem ok, only the block 524320 is getting corrupted. Any idea
on what should i look for now?

ciao
Luca

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: same ext4 file system corruption on different machines
  2014-01-30  7:59     ` Luca Ognibene
@ 2014-01-31  3:00       ` Theodore Ts'o
  2014-02-04  9:05         ` Luca Ognibene
  0 siblings, 1 reply; 5+ messages in thread
From: Theodore Ts'o @ 2014-01-31  3:00 UTC (permalink / raw)
  To: Luca Ognibene; +Cc: linux-kernel, linux-ext4

On Thu, Jan 30, 2014 at 08:59:09AM +0100, Luca Ognibene wrote:
> Yes it's indeed very strange.. i tend to rule out application errors
> because i don't write directly to the device so i don't think i can
> break a filesystem from userspace. I've checked previous and next blocks
> and they seem ok, only the block 524320 is getting corrupted. Any idea
> on what should i look for now?

Are you willing to try 3.12.9 or 3.13.1 upstream kernel?  Let's see if
changing the kernel makes any difference.  I don't recall any ext4
problems like this, but maybe it's device driver problem.

The other thing I'd ask is whether you can swap out the hard drive
interface --- can you use a USB 3.0 attached drive, or something like
that?

One final thing that you could try doing, depending on how
easily/quickly you can reproduce the problem, is to use blktrace and
see if you can catch who or what is writing to that specific block
which is getting corrupted.

						- Ted

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: same ext4 file system corruption on different machines
  2014-01-31  3:00       ` Theodore Ts'o
@ 2014-02-04  9:05         ` Luca Ognibene
  0 siblings, 0 replies; 5+ messages in thread
From: Luca Ognibene @ 2014-02-04  9:05 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: linux-kernel, linux-ext4

Il giorno gio, 30/01/2014 alle 22.00 -0500, Theodore Ts'o ha scritto:
> On Thu, Jan 30, 2014 at 08:59:09AM +0100, Luca Ognibene wrote:
> > Yes it's indeed very strange.. i tend to rule out application errors
> > because i don't write directly to the device so i don't think i can
> > break a filesystem from userspace. I've checked previous and next blocks
> > and they seem ok, only the block 524320 is getting corrupted. Any idea
> > on what should i look for now?
> 
> Are you willing to try 3.12.9 or 3.13.1 upstream kernel?  Let's see if
> changing the kernel makes any difference.  I don't recall any ext4
> problems like this, but maybe it's device driver problem.

I'll try but my problem is that i can't replicate it.. i have to wait
weeks to see it happen on clients machines.

> The other thing I'd ask is whether you can swap out the hard drive
> interface --- can you use a USB 3.0 attached drive, or something like
> that?

I can't do that, sorry.

ciao
Luca


^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: same ext4 file system corruption on different machines
  2014-01-29 17:38   ` same ext4 file system corruption on different machines Theodore Ts'o
  2014-01-30  7:59     ` Luca Ognibene
@ 2014-01-30 11:02     ` Luca Ognibene
  1 sibling, 0 replies; 5+ messages in thread
From: Luca Ognibene @ 2014-01-30 11:02 UTC (permalink / raw)
  To: Theodore Ts'o; +Cc: linux-kernel, linux-ext4

Il giorno mer, 29/01/2014 alle 12.38 -0500, Theodore Ts'o ha scritto:
> On Wed, Jan 29, 2014 at 02:05:43PM +0100, Luca Ognibene wrote:
> > I say "same ext4 file system corruption" because e2fsck reports errors
> > on inodes around 127233 on all file systems.a I'm not sure about the
> > syslog errors because i have syslog logs for only the latest faulty
> > partition.
> 

Sorry i made a mistake in the first email. These dom drives are IDE, not
SATA.

ciao
Luca

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2014-02-04  9:05 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
     [not found] <1390999955.2601.22.camel@snow.station>
     [not found] ` <1391000743.2601.26.camel@snow.station>
2014-01-29 17:38   ` same ext4 file system corruption on different machines Theodore Ts'o
2014-01-30  7:59     ` Luca Ognibene
2014-01-31  3:00       ` Theodore Ts'o
2014-02-04  9:05         ` Luca Ognibene
2014-01-30 11:02     ` Luca Ognibene

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).