Re: 2.6.28.9: EXT3/NFS inodes corruption

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Jan Kara]

On Tue 28-07-09 13:27:15, Sylvain Rochet wrote:
> On Mon, Jul 27, 2009 at 05:42:53PM +0200, Jan Kara wrote:
> > On Sat 25-07-09 17:17:52, Sylvain Rochet wrote:
> > > > 
> > > > Can you still see the corruption with 2.6.30 kernel?
> > > 
> > > Not upgraded yet, we'll give a try.
> 
> Done, now featuring 2.6.30.3 ;)
  OK, drop me an email if you will see corruption also with this kernel.

> > > > If you can still see this problem, could you run: debugfs /dev/md10
> > > > and send output of the command:
> > > > stat <40420228>
> > > > (or whatever the corrupted inode number will be)
> > > > and also:
> > > > dump <40420228> /tmp/corrupted_dir
> > > 
> > > One inode get corrupted recently, here is the output:
> > > 
> > > root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# ls -lai
> > > total 64
> > > 88539836 drwxr-sr-x  2 18804 23084  4096 2009-07-25 07:53 .
> > > 88539821 drwxr-sr-x 20 18804 23084  4096 2008-08-20 10:14 ..
> > > 88541578 -rw-rw-rw-  1 18804 23084   471 2009-07-25 04:55 -inc_forum-10-wa.3cb1921f
> > > 88541465 -rw-rw-rw-  1 18804 23084  6693 2009-07-25 07:53 -inc_rss_item-32-wa.23d91cc2
> > > 88541471 -rw-rw-rw-  1 18804 23084  1625 2009-07-25 07:53 -inc_rubriques-17-wa.f2f152f0
> > > 88541549 -rw-rw-rw-  1 18804 23084  2813 2009-07-25 03:04 INDEX-.edfac52c
> > > 88541366 -rw-rw-rw-  1 18804 23084     0 2008-08-17 20:44 .ok
> > >        ? ?---------  ? ?     ?         ?                ? spip%3Farticle19.f8740dca
> > > 88541671 -rw-rw-rw-  1 18804 23084  5619 2009-07-24 21:07 spip%3Fauteur1.c64f7f7e
> > > 88541460 -rw-rw-rw-  1 18804 23084  5636 2009-07-24 19:30 spip%3Fmot5.f3e9adda
> > > 88540284 -rw-rw-rw-  1 18804 23084  3802 2009-07-25 16:10 spip%3Fpage%3Dforum-30.63b2c1b1
> > > 88541539 -rw-rw-rw-  1 18804 23084 12972 2009-07-25 11:14 spip%3Fpage%3Djquery.cce608b6.gz
> >   OK, so we couldn't stat a directory...
> > 
> > > root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cat spip%3Farticle19.f8740dca
> > > cat: spip%3Farticle19.f8740dca: Stale NFS file handle
> >   This is probably the misleading output from ext3_iget(). It should give
> > you EIO in the latest kernel.
> 
> root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cat spip%3Farticle19.f8740dca 
> cat: spip%3Farticle19.f8740dca: Input/output error
> 
> It has much more sense now. We thought the problem was around NFS due 
> the the previous error message, actually this is probably not the best 
> looking path.
  Yes, EIO makes more sence. I think the problem is NFS connected anyway
though :). But I don't have a clue how it can happen yet. Maybe I can try
adding some low-cost debugging checks if you'd be willing to run such
kernel...
  I'm adding to CC linux-nfs just in case someone has an idea.

> > > root@bazooka:~# debugfs /dev/md10
> > > debugfs 1.40-WIP (14-Nov-2006)
> > > 
> > >     debugfs:  stat <88539836>
> > > 
> > > Inode: 88539836   Type: directory    Mode:  0755   Flags: 0x0   Generation: 791796957
> > > User: 18804   Group: 23084   Size: 4096
> > > File ACL: 0    Directory ACL: 0
> > > Links: 2   Blockcount: 8
> > > Fragment:  Address: 0    Number: 0    Size: 0
> > > ctime: 0x4a6a9dd5 -- Sat Jul 25 07:53:25 2009
> > > atime: 0x4a0de585 -- Fri May 15 23:58:29 2009
> > > mtime: 0x4a6a9dd5 -- Sat Jul 25 07:53:25 2009
> > > Size of extra inode fields: 4
> > > BLOCKS:
> > > (0):177096928
> > > TOTAL: 1
> > > 
> > > 
> > >     debugfs:  ls <88539836>
> > > 
> > >  88539836  (12) .    88539821  (32) ..    88541366  (12) .ok
> > >  88541465  (56) -inc_rss_item-32-wa.23d91cc2
> > >  88541539  (40) spip%3Fpage%3Djquery.cce608b6.gz
> > >  88540284  (40) spip%3Fpage%3Dforum-30.63b2c1b1
> > >  88541460  (28) spip%3Fmot5.f3e9adda
> > >  88541471  (160) -inc_rubriques-17-wa.f2f152f0
> > >  88541549  (24) INDEX-.edfac52c    88541578  (284) -inc_forum-10-wa.3cb1921f
> > >  88541562  (36) spip%3Farticle19.f8740dca
> > >  88541671  (3372) spip%3Fauteur1.c64f7f7e
> >   The directory itself looks fine...
> > 
> > >     debugfs:  stat <88541562>
> > > 
> > > Inode: 88541562   Type: regular    Mode:  0666   Flags: 0x0   Generation: 860068541
> > > User: 18804   Group: 23084   Size: 0
> > > File ACL: 0    Directory ACL: 0
> > > Links: 0   Blockcount: 0
> > > Fragment:  Address: 0    Number: 0    Size: 0
> > > ctime: 0x4a6a8fac -- Sat Jul 25 06:53:00 2009
> > > atime: 0x4a6a612f -- Sat Jul 25 03:34:39 2009
> > > mtime: 0x4a6a8fac -- Sat Jul 25 06:53:00 2009
> > > dtime: 0x4a6a8fac -- Sat Jul 25 06:53:00 2009
> > > Size of extra inode fields: 4
> > > BLOCKS:
> > 
> >   Ah, OK, here's the problem. The directory points to a file which is
> > obviously deleted (note the "Links: 0"). All the content of the inode seems
> > to indicate that the file was correctly deleted (you might check that the
> > corresponding bit in the bitmap is cleared via: "icheck 88541562").
> 
> root@bazooka:~# debugfs /dev/md10
> debugfs 1.40-WIP (14-Nov-2006)
> debugfs:  icheck 88541562
> Block   Inode number
> 88541562        <block not found>
  Ah, wrong debugfs command. I should have written:
testi <88541562>

> >   The question is how it could happen the directory still points to the
> > inode. Really strange. It looks as if we've lost a write to the directory
> > but I don't see how. Are there any suspitious kernel messages in this case?
> 
> There were nothing for a while, but since the reboot there are some 
> about this inode: 
> 
> EXT3-fs error (device md10): ext3_lookup: deleted inode referenced: 88541562
  Yes, that's to be expected given the corruption any NFS error messages?

> > > We'll try.
> > 
> >   It probably won't help. This particular directory had just one block so
> > DIR_INDEX had no effect on it.
> 
> Let's keep dir_index for now, then.
  OK.

> >   OK, so it's probably not a storage device problem. Good to know.
> 
> We also thought about motherboard, CPU, or chassis issues, but 
> everything has been replaced.
> 
> 
> The check of the MD raid6 array always ends happily:
> 
> Jul  5 01:06:01 bazooka kernel: md: data-check of RAID array md10
> Jul  5 01:06:01 bazooka kernel: md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
> Jul  5 01:06:01 bazooka kernel: md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
> Jul  5 01:06:01 bazooka kernel: md: using 128k window, over a total of 143373888 blocks.
> Jul  5 04:28:28 bazooka kernel: md: md10: data-check done.
> 
> 
> We never saw modification to the data of files themselves, maybe it 
> happened, but we never saw any evidence of that. Of course, due to the 
> modification of the filesystem structure, we saw files replaced by other 
> files ;)

								Honza

-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Sylvain Rochet]

[-- Attachment #1: Type: text/plain, Size: 3815 bytes --]

Hi,


On Tue, Jul 28, 2009 at 03:52:26PM +0200, Jan Kara wrote:
> On Tue 28-07-09 13:27:15, Sylvain Rochet wrote:
> > On Mon, Jul 27, 2009 at 05:42:53PM +0200, Jan Kara wrote:
> > > On Sat 25-07-09 17:17:52, Sylvain Rochet wrote:
> > > > > 
> > > > > Can you still see the corruption with 2.6.30 kernel?
> > > > 
> > > > Not upgraded yet, we'll give a try.
> > 
> > Done, now featuring 2.6.30.3 ;)
> 
> OK, drop me an email if you will see corruption also with this kernel.

Lets move out the corrupted directory ;)

root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# rm -- * .ok 
rm: cannot remove `spip%3Farticle19.f8740dca': Input/output error
root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cd ..
root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache# mv e/ /data/lost+found/wooops


> > > This is probably the misleading output from ext3_iget(). It should give
> > > you EIO in the latest kernel.
> > 
> > root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cat spip%3Farticle19.f8740dca 
> > cat: spip%3Farticle19.f8740dca: Input/output error
> > 
> > It has much more sense now. We thought the problem was around NFS due 
> > the the previous error message, actually this is probably not the best 
> > looking path.
> 
> Yes, EIO makes more sence. I think the problem is NFS connected anyway
> though :). But I don't have a clue how it can happen yet. Maybe I can try
> adding some low-cost debugging checks if you'd be willing to run such
> kernel...

Without any problem, we have 24/7/365 physical access and we don't need 
to provide high-availability services.

Anyway, the data hosted aren't that important, there is little or even 
no need for strict confidentiality, so we will be happy to provide ssh 
access to whom would like to look deeper into this issue.


> I'm adding to CC linux-nfs just in case someone has an idea.
> 
> > >   Ah, OK, here's the problem. The directory points to a file which is
> > > obviously deleted (note the "Links: 0"). All the content of the inode seems
> > > to indicate that the file was correctly deleted (you might check that the
> > > corresponding bit in the bitmap is cleared via: "icheck 88541562").
> > 
> > root@bazooka:~# debugfs /dev/md10
> > debugfs 1.40-WIP (14-Nov-2006)
> > debugfs:  icheck 88541562
> > Block   Inode number
> > 88541562        <block not found>
> 
> Ah, wrong debugfs command. I should have written:
> testi <88541562>

debugfs:  testi <88541562>
Inode 88541562 is not in use


> > >   The question is how it could happen the directory still points to the
> > > inode. Really strange. It looks as if we've lost a write to the directory
> > > but I don't see how. Are there any suspitious kernel messages in this case?
> > 
> > There were nothing for a while, but since the reboot there are some 
> > about this inode: 
> > 
> > EXT3-fs error (device md10): ext3_lookup: deleted inode referenced: 88541562
> 
> Yes, that's to be expected given the corruption any NFS error messages?

There are some error messages on NFS clients, however they are quite old.

Apr 19 15:38:21 gin kernel: NFS: Buggy server - nlink == 0!
May  3 20:00:52 gin kernel: NFS: Buggy server - nlink == 0!
May  3 23:24:03 gin kernel: NFS: Buggy server - nlink == 0!
May  7 11:40:57 gin kernel: NFS: Buggy server - nlink == 0!
May  7 14:41:02 gin kernel: NFS: Buggy server - nlink == 0!
May 26 11:10:42 cognac kernel: NFS: Buggy server - nlink == 0!
May 26 11:13:28 cognac kernel: NFS: Buggy server - nlink == 0!
May 26 12:34:39 cognac kernel: NFS: Buggy server - nlink == 0!
May 26 12:39:43 cognac kernel: NFS: Buggy server - nlink == 0!

This is obviously related to the corruption.



Sylvain

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.28.9: EXT3/NFS inodes corruption

[J. Bruce Fields]

On Tue, Jul 28, 2009 at 06:41:42PM +0200, Sylvain Rochet wrote:
> Hi,
> 
> 
> On Tue, Jul 28, 2009 at 03:52:26PM +0200, Jan Kara wrote:
> > On Tue 28-07-09 13:27:15, Sylvain Rochet wrote:
> > > On Mon, Jul 27, 2009 at 05:42:53PM +0200, Jan Kara wrote:
> > > > On Sat 25-07-09 17:17:52, Sylvain Rochet wrote:
> > > > > > 
> > > > > > Can you still see the corruption with 2.6.30 kernel?
> > > > > 
> > > > > Not upgraded yet, we'll give a try.
> > > 
> > > Done, now featuring 2.6.30.3 ;)
> > 
> > OK, drop me an email if you will see corruption also with this kernel.
> 
> Lets move out the corrupted directory ;)
> 
> root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# rm -- * .ok 
> rm: cannot remove `spip%3Farticle19.f8740dca': Input/output error
> root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cd ..
> root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache# mv e/ /data/lost+found/wooops
> 
> 
> > > > This is probably the misleading output from ext3_iget(). It should give
> > > > you EIO in the latest kernel.
> > > 
> > > root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cat spip%3Farticle19.f8740dca 
> > > cat: spip%3Farticle19.f8740dca: Input/output error
> > > 
> > > It has much more sense now. We thought the problem was around NFS due 
> > > the the previous error message, actually this is probably not the best 
> > > looking path.
> > 
> > Yes, EIO makes more sence. I think the problem is NFS connected anyway
> > though :). But I don't have a clue how it can happen yet. Maybe I can try
> > adding some low-cost debugging checks if you'd be willing to run such
> > kernel...
> 
> Without any problem, we have 24/7/365 physical access and we don't need 
> to provide high-availability services.
> 
> Anyway, the data hosted aren't that important, there is little or even 
> no need for strict confidentiality, so we will be happy to provide ssh 
> access to whom would like to look deeper into this issue.
> 
> 
> > I'm adding to CC linux-nfs just in case someone has an idea.
> > 
> > > >   Ah, OK, here's the problem. The directory points to a file which is
> > > > obviously deleted (note the "Links: 0"). All the content of the inode seems
> > > > to indicate that the file was correctly deleted (you might check that the
> > > > corresponding bit in the bitmap is cleared via: "icheck 88541562").
> > > 
> > > root@bazooka:~# debugfs /dev/md10
> > > debugfs 1.40-WIP (14-Nov-2006)
> > > debugfs:  icheck 88541562
> > > Block   Inode number
> > > 88541562        <block not found>
> > 
> > Ah, wrong debugfs command. I should have written:
> > testi <88541562>
> 
> debugfs:  testi <88541562>
> Inode 88541562 is not in use
> 
> 
> > > >   The question is how it could happen the directory still points to the
> > > > inode. Really strange. It looks as if we've lost a write to the directory
> > > > but I don't see how. Are there any suspitious kernel messages in this case?
> > > 
> > > There were nothing for a while, but since the reboot there are some 
> > > about this inode: 
> > > 
> > > EXT3-fs error (device md10): ext3_lookup: deleted inode referenced: 88541562
> > 
> > Yes, that's to be expected given the corruption any NFS error messages?
> 
> There are some error messages on NFS clients, however they are quite old.
> 
> Apr 19 15:38:21 gin kernel: NFS: Buggy server - nlink == 0!
> May  3 20:00:52 gin kernel: NFS: Buggy server - nlink == 0!
> May  3 23:24:03 gin kernel: NFS: Buggy server - nlink == 0!
> May  7 11:40:57 gin kernel: NFS: Buggy server - nlink == 0!
> May  7 14:41:02 gin kernel: NFS: Buggy server - nlink == 0!
> May 26 11:10:42 cognac kernel: NFS: Buggy server - nlink == 0!
> May 26 11:13:28 cognac kernel: NFS: Buggy server - nlink == 0!
> May 26 12:34:39 cognac kernel: NFS: Buggy server - nlink == 0!
> May 26 12:39:43 cognac kernel: NFS: Buggy server - nlink == 0!
> 
> This is obviously related to the corruption.

It might be interesting to know whether the file that we returned to the
client with nlink 0 was the same that you later saw corruption on; maybe
adding a printk of the inode number there would help.

Googling around on that error message, a previous thread:

	http://marc.info/?t=107429333300004&r=1&w=4

seems to conclude it's a bug, but doesn't followup with a fix.  And I
don't see any mention of possible filesystem corruption.

Is NFSv4 involved here?  I wonder if something that might otherwise be
only a problem for the client could become a problem for the server if
it attempts to do further operations with an unlinked inode in a
compound operation that follows a lookup.

--b.

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Sylvain Rochet]

[-- Attachment #1: Type: text/plain, Size: 1107 bytes --]

Hi,

On Tue, Jul 28, 2009 at 05:12:15PM -0400, J. Bruce Fields wrote:
> On Tue, Jul 28, 2009 at 06:41:42PM +0200, Sylvain Rochet wrote:
> > 
> > [...]
> > May 26 12:34:39 cognac kernel: NFS: Buggy server - nlink == 0!
> > May 26 12:39:43 cognac kernel: NFS: Buggy server - nlink == 0!
> > 
> > This is obviously related to the corruption.
> 
> It might be interesting to know whether the file that we returned to the
> client with nlink 0 was the same that you later saw corruption on; maybe
> adding a printk of the inode number there would help.
> 
> Googling around on that error message, a previous thread:
> 
> 	http://marc.info/?t=107429333300004&r=1&w=4
> 
> seems to conclude it's a bug, but doesn't followup with a fix.  And I
> don't see any mention of possible filesystem corruption.
> 
> Is NFSv4 involved here?  I wonder if something that might otherwise be
> only a problem for the client could become a problem for the server if
> it attempts to do further operations with an unlinked inode in a
> compound operation that follows a lookup.

NFSv3 here.

Sylvain

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Jan Kara]

On Tue 28-07-09 18:41:42, Sylvain Rochet wrote:
> Hi,
> 
> 
> On Tue, Jul 28, 2009 at 03:52:26PM +0200, Jan Kara wrote:
> > On Tue 28-07-09 13:27:15, Sylvain Rochet wrote:
> > > On Mon, Jul 27, 2009 at 05:42:53PM +0200, Jan Kara wrote:
> > > > On Sat 25-07-09 17:17:52, Sylvain Rochet wrote:
> > > > > > 
> > > > > > Can you still see the corruption with 2.6.30 kernel?
> > > > > 
> > > > > Not upgraded yet, we'll give a try.
> > > 
> > > Done, now featuring 2.6.30.3 ;)
> > 
> > OK, drop me an email if you will see corruption also with this kernel.
> 
> Lets move out the corrupted directory ;)
> 
> root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# rm -- * .ok 
> rm: cannot remove `spip%3Farticle19.f8740dca': Input/output error
> root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cd ..
> root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache# mv e/ /data/lost+found/wooops
  Actually, leaving that file in the filesystem can potentially lead to
strange effects because eventually the inode "spip%3Farticle19.f8740dca"
points to gets reallocated and then you can get e.g. a hardlinked
directory. On the other hand having it lost+found should be safe enough.

> > > > This is probably the misleading output from ext3_iget(). It should give
> > > > you EIO in the latest kernel.
> > > 
> > > root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cat spip%3Farticle19.f8740dca 
> > > cat: spip%3Farticle19.f8740dca: Input/output error
> > > 
> > > It has much more sense now. We thought the problem was around NFS due 
> > > the the previous error message, actually this is probably not the best 
> > > looking path.
> > 
> > Yes, EIO makes more sence. I think the problem is NFS connected anyway
> > though :). But I don't have a clue how it can happen yet. Maybe I can try
> > adding some low-cost debugging checks if you'd be willing to run such
> > kernel...
> 
> Without any problem, we have 24/7/365 physical access and we don't need 
> to provide high-availability services.
  Cool, I'll try to cook up something then.

> Anyway, the data hosted aren't that important, there is little or even 
> no need for strict confidentiality, so we will be happy to provide ssh 
> access to whom would like to look deeper into this issue.
  I don't need to go that far (at least for now) but thanks for the offer.

> > I'm adding to CC linux-nfs just in case someone has an idea.
> > 
> > > >   Ah, OK, here's the problem. The directory points to a file which is
> > > > obviously deleted (note the "Links: 0"). All the content of the inode seems
> > > > to indicate that the file was correctly deleted (you might check that the
> > > > corresponding bit in the bitmap is cleared via: "icheck 88541562").
> > > 
> > > root@bazooka:~# debugfs /dev/md10
> > > debugfs 1.40-WIP (14-Nov-2006)
> > > debugfs:  icheck 88541562
> > > Block   Inode number
> > > 88541562        <block not found>
> > 
> > Ah, wrong debugfs command. I should have written:
> > testi <88541562>
> 
> debugfs:  testi <88541562>
> Inode 88541562 is not in use
  Yes, again this confirms that the inode was just correctly deleted. But
somehow a pointer to it remained in the directory.

> > > >   The question is how it could happen the directory still points to the
> > > > inode. Really strange. It looks as if we've lost a write to the directory
> > > > but I don't see how. Are there any suspitious kernel messages in this case?
> > > 
> > > There were nothing for a while, but since the reboot there are some 
> > > about this inode: 
> > > 
> > > EXT3-fs error (device md10): ext3_lookup: deleted inode referenced: 88541562
> > 
> > Yes, that's to be expected given the corruption any NFS error messages?
> 
> There are some error messages on NFS clients, however they are quite old.
> 
> Apr 19 15:38:21 gin kernel: NFS: Buggy server - nlink == 0!
> May  3 20:00:52 gin kernel: NFS: Buggy server - nlink == 0!
> May  3 23:24:03 gin kernel: NFS: Buggy server - nlink == 0!
> May  7 11:40:57 gin kernel: NFS: Buggy server - nlink == 0!
> May  7 14:41:02 gin kernel: NFS: Buggy server - nlink == 0!
> May 26 11:10:42 cognac kernel: NFS: Buggy server - nlink == 0!
> May 26 11:13:28 cognac kernel: NFS: Buggy server - nlink == 0!
> May 26 12:34:39 cognac kernel: NFS: Buggy server - nlink == 0!
> May 26 12:39:43 cognac kernel: NFS: Buggy server - nlink == 0!
> 
> This is obviously related to the corruption.
  Yes, this is a consequence of the bug - somebody deleted an inode because
i_nlink dropped down to 0 but the inode was in fact still referenced.

									Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Sylvain Rochet]

[-- Attachment #1: Type: text/plain, Size: 1127 bytes --]

Hi,


On Wed, Jul 29, 2009 at 02:58:12PM +0200, Jan Kara wrote:
> On Tue 28-07-09 18:41:42, Sylvain Rochet wrote:
> > 
> > Lets move out the corrupted directory ;)
> > 
> > root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# rm -- * .ok 
> > rm: cannot remove `spip%3Farticle19.f8740dca': Input/output error
> > root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cd ..
> > root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache# mv e/ /data/lost+found/wooops
> 
> Actually, leaving that file in the filesystem can potentially lead to
> strange effects because eventually the inode "spip%3Farticle19.f8740dca"
> points to gets reallocated and then you can get e.g. a hardlinked
> directory. On the other hand having it lost+found should be safe enough.

This happened a few times in the past, we saw corrupted dentries reappearing with 
a new file. New files with reference count set to 1 (but obviously should be 2 in 
this case). So the rule is "do not delete corrupted dentries anyway, keep them 
safe in lost+found and do not touch it" ;).


Sylvain

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Jan Kara]

[-- Attachment #1: Type: text/plain, Size: 3322 bytes --]

  Hi,

On Tue 28-07-09 18:41:42, Sylvain Rochet wrote:
> On Tue, Jul 28, 2009 at 03:52:26PM +0200, Jan Kara wrote:
> > On Tue 28-07-09 13:27:15, Sylvain Rochet wrote:
> > > On Mon, Jul 27, 2009 at 05:42:53PM +0200, Jan Kara wrote:
> > > > On Sat 25-07-09 17:17:52, Sylvain Rochet wrote:
> > > > > > 
> > > > > > Can you still see the corruption with 2.6.30 kernel?
> > > > > 
> > > > > Not upgraded yet, we'll give a try.
> > > 
> > > Done, now featuring 2.6.30.3 ;)
> > 
> > OK, drop me an email if you will see corruption also with this kernel.
> 
> Lets move out the corrupted directory ;)
> 
> root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# rm -- * .ok 
> rm: cannot remove `spip%3Farticle19.f8740dca': Input/output error
> root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cd ..
> root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache# mv e/ /data/lost+found/wooops
> 
> > > > This is probably the misleading output from ext3_iget(). It should give
> > > > you EIO in the latest kernel.
> > > 
> > > root@bazooka:/data/web/ed/90/48/walotux.walon.org/htdocs/tmp/cache/e# cat spip%3Farticle19.f8740dca 
> > > cat: spip%3Farticle19.f8740dca: Input/output error
> > > 
> > > It has much more sense now. We thought the problem was around NFS due 
> > > the the previous error message, actually this is probably not the best 
> > > looking path.
> > 
> > Yes, EIO makes more sence. I think the problem is NFS connected anyway
> > though :). But I don't have a clue how it can happen yet. Maybe I can try
> > adding some low-cost debugging checks if you'd be willing to run such
> > kernel...
> 
> Without any problem, we have 24/7/365 physical access and we don't need 
> to provide high-availability services.
> 
> Anyway, the data hosted aren't that important, there is little or even 
> no need for strict confidentiality, so we will be happy to provide ssh 
> access to whom would like to look deeper into this issue.
> 
> 
> > I'm adding to CC linux-nfs just in case someone has an idea.
> > 
> > > >   Ah, OK, here's the problem. The directory points to a file which is
> > > > obviously deleted (note the "Links: 0"). All the content of the inode seems
> > > > to indicate that the file was correctly deleted (you might check that the
> > > > corresponding bit in the bitmap is cleared via: "icheck 88541562").
> > > 
> > > root@bazooka:~# debugfs /dev/md10
> > > debugfs 1.40-WIP (14-Nov-2006)
> > > debugfs:  icheck 88541562
> > > Block   Inode number
> > > 88541562        <block not found>
> > 
> > Ah, wrong debugfs command. I should have written:
> > testi <88541562>
> 
> debugfs:  testi <88541562>
> Inode 88541562 is not in use
  OK, I've found some time and written the debugging patch. Hopefully it
will tell us more. It should output messages to the kernel log if it
finds something suspicious - like:
No dentry for unlinked inode...
Dentry ... for unlinked inode ... has no parent
Found directory entry ... for unlinked inode

  When you see such messages in the log, send them to me please. Also
attach the System.map file so that I can translate the address where
i_nlink was dropped - for that ext3 should be compiled into the kernel
(should not be a module). Thanks a lot for testing.

								Honza

-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

[-- Attachment #2: 0001-ext3-Debug-unlinking-of-inodes.patch --]
[-- Type: text/x-patch, Size: 3524 bytes --]

>From b32511dbd58c8d9111001a33d253a283943bbf7a Mon Sep 17 00:00:00 2001
From: Jan Kara <jack@suse.cz>
Date: Tue, 4 Aug 2009 00:17:35 +0200
Subject: [PATCH] ext3: Debug unlinking of inodes

Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/ext3/inode.c    |   28 ++++++++++++++++++++++++++++
 fs/ext3/namei.c    |    2 +-
 include/linux/fs.h |    6 ++++++
 3 files changed, 35 insertions(+), 1 deletions(-)

diff --git a/fs/ext3/inode.c b/fs/ext3/inode.c
index b49908a..dca30a2 100644
--- a/fs/ext3/inode.c
+++ b/fs/ext3/inode.c
@@ -2876,6 +2876,9 @@ bad_inode:
 	return ERR_PTR(ret);
 }
 
+struct buffer_head *ext3_find_entry(struct inode *dir,
+                                        struct qstr *entry,
+                                        struct ext3_dir_entry_2 **res_dir);
 /*
  * Post the struct inode info into an on-disk inode location in the
  * buffer-cache.  This gobbles the caller's reference to the
@@ -2892,6 +2895,31 @@ static int ext3_do_update_inode(handle_t *handle,
 	struct buffer_head *bh = iloc->bh;
 	int err = 0, rc, block;
 
+	if (!inode->i_nlink && !inode->i_checked_drop) {
+		struct dentry *dentry;
+		struct ext3_dir_entry_2 *de;
+		struct buffer_head *bh;
+
+		inode->i_checked_drop = 1;
+		if (list_empty(&inode->i_dentry)) {
+			printk("No dentry for unlinked inode %lu\nNlink dropped at 0x%lx\n", inode->i_ino, inode->i_dropped);
+			dump_stack();
+			goto next;
+		}
+		dentry = list_entry(inode->i_dentry.next, struct dentry, d_alias);
+		if (!dentry->d_parent) {
+			printk("Dentry %s for unlinked inode %lu has no parent\nNlink dropped at 0x%lx\n", dentry->d_name.name, inode->i_ino, inode->i_dropped);
+			dump_stack();
+			goto next;
+		}
+		bh = ext3_find_entry(dentry->d_parent->d_inode, &dentry->d_name, &de);
+		if (bh && le32_to_cpu(de->inode) == inode->i_ino) {
+			printk("Found directory entry %s for unlinked inode %lu\nNlink dropped at 0x%lx\n", dentry->d_name.name, inode->i_ino, inode->i_dropped);
+			brelse(bh);
+			dump_stack();
+		}
+	}
+next:
 	/* For fields not not tracking in the in-memory inode,
 	 * initialise them to zero for new inodes. */
 	if (ei->i_state & EXT3_STATE_NEW)
diff --git a/fs/ext3/namei.c b/fs/ext3/namei.c
index 6ff7b97..e66b6c0 100644
--- a/fs/ext3/namei.c
+++ b/fs/ext3/namei.c
@@ -850,7 +850,7 @@ static inline int search_dirblock(struct buffer_head * bh,
  * The returned buffer_head has ->b_count elevated.  The caller is expected
  * to brelse() it when appropriate.
  */
-static struct buffer_head *ext3_find_entry(struct inode *dir,
+struct buffer_head *ext3_find_entry(struct inode *dir,
 					struct qstr *entry,
 					struct ext3_dir_entry_2 **res_dir)
 {
diff --git a/include/linux/fs.h b/include/linux/fs.h
index a36ffa5..271c51c 100644
--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -780,6 +780,8 @@ struct inode {
 	struct posix_acl	*i_acl;
 	struct posix_acl	*i_default_acl;
 #endif
+	unsigned long		i_dropped;
+	int			i_checked_drop;
 	void			*i_private; /* fs or device private pointer */
 };
 
@@ -1693,6 +1695,8 @@ static inline void inode_inc_link_count(struct inode *inode)
 static inline void drop_nlink(struct inode *inode)
 {
 	inode->i_nlink--;
+	inode->i_dropped = _THIS_IP_;
+	inode->i_checked_drop = 0;
 }
 
 /**
@@ -1706,6 +1710,8 @@ static inline void drop_nlink(struct inode *inode)
 static inline void clear_nlink(struct inode *inode)
 {
 	inode->i_nlink = 0;
+	inode->i_dropped = _THIS_IP_;
+	inode->i_checked_drop = 0;
 }
 
 static inline void inode_dec_link_count(struct inode *inode)
-- 
1.6.0.2

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Sylvain Rochet]

[-- Attachment #1: Type: text/plain, Size: 915 bytes --]

Hi,


On Tue, Aug 04, 2009 at 12:29:01AM +0200, Jan Kara wrote:
> 
>   OK, I've found some time and written the debugging patch. Hopefully it
> will tell us more. It should output messages to the kernel log if it
> finds something suspicious - like:
> No dentry for unlinked inode...
> Dentry ... for unlinked inode ... has no parent
> Found directory entry ... for unlinked inode
> 
>   When you see such messages in the log, send them to me please. Also
> attach the System.map file so that I can translate the address where
> i_nlink was dropped - for that ext3 should be compiled into the kernel
> (should not be a module). Thanks a lot for testing.

Patch applied.

And there is already a lot of output.

http://edony.tuxfamily.org/~grad/bazooka/System.map-2.6.30.4
http://edony.tuxfamily.org/~grad/bazooka/config-2.6.30.4
http://edony.tuxfamily.org/~grad/bazooka/kern.log


Sylvain

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Jan Kara]

[-- Attachment #1: Type: text/plain, Size: 1703 bytes --]

  Hi,

On Tue 04-08-09 13:15:05, Sylvain Rochet wrote:
> On Tue, Aug 04, 2009 at 12:29:01AM +0200, Jan Kara wrote:
> > 
> >   OK, I've found some time and written the debugging patch. Hopefully it
> > will tell us more. It should output messages to the kernel log if it
> > finds something suspicious - like:
> > No dentry for unlinked inode...
> > Dentry ... for unlinked inode ... has no parent
> > Found directory entry ... for unlinked inode
> > 
> >   When you see such messages in the log, send them to me please. Also
> > attach the System.map file so that I can translate the address where
> > i_nlink was dropped - for that ext3 should be compiled into the kernel
> > (should not be a module). Thanks a lot for testing.
> 
> Patch applied.
> 
> And there is already a lot of output.
> 
> http://edony.tuxfamily.org/~grad/bazooka/System.map-2.6.30.4
> http://edony.tuxfamily.org/~grad/bazooka/config-2.6.30.4
> http://edony.tuxfamily.org/~grad/bazooka/kern.log
  Thanks for testing. So you seem to be really stressting the path where
creation of new files / directories fails (probably due to group quota).  I
have one idea what could cause your filesystem corruption, although it's a
wild guess... Please try attached oneliner.
  Also your corruption reminded me that Al Viro has been fixing problems
where we could cache one inode twice when a filesystem was mounted over NFS
and that could also lead to a filesystem corruption. So I'm adding him to
CC just in case he has some idea. BTW Al, what do you think about the
problem I describe in the attached patch? I'm not sure if it can cause some
real problems but in theory it could...

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

[-- Attachment #2: 0001-fs-Make-sure-data-stored-into-inode-is-properly-see.patch --]
[-- Type: text/x-patch, Size: 985 bytes --]

>From 78513d3a5628fda0f8d685d732b7bc73bd4c9222 Mon Sep 17 00:00:00 2001
From: Jan Kara <jack@suse.cz>
Date: Wed, 5 Aug 2009 00:42:21 +0200
Subject: [PATCH] fs: Make sure data stored into inode is properly seen before unlocking new inode

In theory it could happen that on one CPU we initialize a new inode but clearing
of I_NEW | I_LOCK gets reordered before some of the initialization. Thus on
another CPU we return not fully uptodate inode from iget_locked().

Signed-off-by: Jan Kara <jack@suse.cz>
---
 fs/inode.c |    1 +
 1 files changed, 1 insertions(+), 0 deletions(-)

diff --git a/fs/inode.c b/fs/inode.c
index 901bad1..e9a8e77 100644
--- a/fs/inode.c
+++ b/fs/inode.c
@@ -696,6 +696,7 @@ void unlock_new_inode(struct inode *inode)
 	 * just created it (so there can be no old holders
 	 * that haven't tested I_LOCK).
 	 */
+	smp_mb();
 	WARN_ON((inode->i_state & (I_LOCK|I_NEW)) != (I_LOCK|I_NEW));
 	inode->i_state &= ~(I_LOCK|I_NEW);
 	wake_up_inode(inode);
-- 
1.6.0.2

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Sylvain Rochet]

[-- Attachment #1: Type: text/plain, Size: 1063 bytes --]

Hi,


On Wed, Aug 05, 2009 at 12:56:19AM +0200, Jan Kara wrote:
> 
> Thanks for testing. So you seem to be really stressting the path where
> creation of new files / directories fails (probably due to group quota).

Yes, there are 29 groups over quota on a total of 4499. Those are mainly 
spammed websites and therefore quite stressed due to the amount of tries 
to add new "data".


> I have one idea what could cause your filesystem corruption, although 
> it's a wild guess... Please try attached oneliner.

Running since yesterday.


> Also your corruption reminded me that Al Viro has been fixing problems
> where we could cache one inode twice when a filesystem was mounted over NFS
> and that could also lead to a filesystem corruption. So I'm adding him to
> CC just in case he has some idea. BTW Al, what do you think about the
> problem I describe in the attached patch? I'm not sure if it can cause some
> real problems but in theory it could...

Should we upgrade NFS clients as well ?  (now running 2.6.28.9)


Sylvain

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.28.9: EXT3/NFS inodes corruption

[J. Bruce Fields]

On Thu, Aug 06, 2009 at 03:15:56PM +0200, Sylvain Rochet wrote:
> Hi,
> 
> 
> On Wed, Aug 05, 2009 at 12:56:19AM +0200, Jan Kara wrote:
> > 
> > Thanks for testing. So you seem to be really stressting the path where
> > creation of new files / directories fails (probably due to group quota).
> 
> Yes, there are 29 groups over quota on a total of 4499. Those are mainly 
> spammed websites and therefore quite stressed due to the amount of tries 
> to add new "data".
> 
> 
> > I have one idea what could cause your filesystem corruption, although 
> > it's a wild guess... Please try attached oneliner.
> 
> Running since yesterday.
> 
> 
> > Also your corruption reminded me that Al Viro has been fixing problems
> > where we could cache one inode twice when a filesystem was mounted over NFS
> > and that could also lead to a filesystem corruption. So I'm adding him to
> > CC just in case he has some idea. BTW Al, what do you think about the
> > problem I describe in the attached patch? I'm not sure if it can cause some
> > real problems but in theory it could...
> 
> Should we upgrade NFS clients as well ?  (now running 2.6.28.9)

The client version shouldn't matter.

--b.

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Jan Kara]

  Hello,

On Thu 06-08-09 15:15:56, Sylvain Rochet wrote:
> > I have one idea what could cause your filesystem corruption, although 
> > it's a wild guess... Please try attached oneliner.
> 
> Running since yesterday.
  Any news after a week of running? How often did the corruption happen
previously?

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Sylvain Rochet]

[-- Attachment #1: Type: text/plain, Size: 829 bytes --]

Hi!,

On Thu, Aug 13, 2009 at 12:34:53AM +0200, Jan Kara wrote:
>   Hello,
> 
> On Thu 06-08-09 15:15:56, Sylvain Rochet wrote:
> > > I have one idea what could cause your filesystem corruption, although 
> > > it's a wild guess... Please try attached oneliner.
> > 
> > Running since yesterday.
> 
> Any news after a week of running? How often did the corruption happen
> previously?

Sorry for the late answer, I was lurking at HAR ;-)

So, everything is fine, but the problem happened only one time on this 
server, so we cannot conclude anything after a few weeks. However, 
I now have physical access back, so we will switch back to the former 
server where the problem happened quite frequently, then we will see!

By the way, syslogd is happy, eating about 350 MiB of kernel logs a day ;)

Sylvain

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Simon Kirby]

On Thu, Aug 20, 2009 at 07:19:53PM +0200, Sylvain Rochet wrote:

> So, everything is fine, but the problem happened only one time on this 
> server, so we cannot conclude anything after a few weeks. However, 
> I now have physical access back, so we will switch back to the former 
> server where the problem happened quite frequently, then we will see!

Not to derail the thread, but you were definitely seeing the same issues
with stock 2.6.30.4, right?  We had all sorts of corruption happening for
files served via NFS with 2.6.28 and 2.6.29, but everything was magically
fixed on 2.6.30 (though we needed a lot of fscking).  I never did track
down what change fixed it, since it took a while to reproduce.

Hmm.  I just noticed what seems to be a new occurrence of "deleted inode
referenced" on a box with 2.6.30.  We saw many when we first upgraded to
2.6.30 due to the corruption caused by 2.6.29, but those all occurred
within a day or so and were fsck'd.  I would have thought the backup
sweeps would have tripped over that inode way before now...

Just wondering if you can confirm that the errors you saw with 2.6.30.4
were not leftover from older kernels.

Cheers,

Simon-

Re: 2.6.28.9: EXT3/NFS inodes corruption

[Sylvain Rochet]

[-- Attachment #1: Type: text/plain, Size: 2631 bytes --]

Hi,


On Thu, Aug 20, 2009 at 05:00:35PM -0700, Simon Kirby wrote:
> On Thu, Aug 20, 2009 at 07:19:53PM +0200, Sylvain Rochet wrote:
> 
> > So, everything is fine, but the problem happened only one time on this 
> > server, so we cannot conclude anything after a few weeks. However, 
> > I now have physical access back, so we will switch back to the former 
> > server where the problem happened quite frequently, then we will see!
> 
> Not to derail the thread, but you were definitely seeing the same issues
> with stock 2.6.30.4, right?

Nope, the last issue we had came from 2.6.28.9.

We upgraded to 2.6.30.3 on the advice of Jan, then we "upgraded" to 
2.6.30.3 with the first Jan's patch to add some debug output 
(0001-ext3-Debug-unlinking-of-inodes.patch). Finally we upgraded to 
2.6.30.4 with the first and the second Jan's patch 
(0001-fs-Make-sure-data-stored-into-inode-is-properly-see.patch) to add 
a smp_mb() in the unlock_new_inode() function.


> We had all sorts of corruption happening for files served via NFS with 
> 2.6.28 and 2.6.29, but everything was magically fixed on 2.6.30 
> (though we needed a lot of fscking).  I never did track down what 
> change fixed it, since it took a while to reproduce.

Same here, everything is fine since 2.6.30. We will switch back to the 
quad-core server where the corruption happen(ed) in a few days. We are 
now using a bi-opteron server because we suspected hardware issues on 
the quad-core, the corruption happened only one time on the bi-opteron 
(which is IMHO a sufficient evidence to discard hardware issue). I guess 
the issue was(or is) kinda SMP related.

And yep, we also had long times playing with fsck ;-) Luckily that the 
corruption only occurs on new files, and new files are mostly caches, 
sessions, logs, and such, so fsck used its chainsaw on quite 
not-really-important files.


> Hmm.  I just noticed what seems to be a new occurrence of "deleted inode
> referenced" on a box with 2.6.30.  We saw many when we first upgraded to
> 2.6.30 due to the corruption caused by 2.6.29, but those all occurred
> within a day or so and were fsck'd.  I would have thought the backup
> sweeps would have tripped over that inode way before now...
> 
> Just wondering if you can confirm that the errors you saw with 2.6.30.4
> were not leftover from older kernels.

The few garbaged inodes from 2.6.28.9 (and previous) were pushed to 
lost+found to prevent future use of them. We do a fsck when we moved to 
2.6.30.4 that fixed everything. We never had corruption yet with the 
2.6.30.4.


Sylvain

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]