NFS: server error: fileid changed

NFS: server error: fileid changed

[Jan Engelhardt]

Hi,


Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really 
unreliable, having clients start to emit messages like

(nfsv3)
[42306.521225] NFS: server nova error: fileid changed
[42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0

On nfsv4, it is even worse, as updates on the server are not immediately 
reflected on the client like it was the case on nfsv3.
Unfortunately, that is all the information I currently have.
(Server is on 2.6.36-rc8, client on 2.6.37-rc1.)

thanks,
Jan

Re: NFS: server error: fileid changed

[J. Bruce Fields]

On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really 
> unreliable, having clients start to emit messages like
> 
> (nfsv3)
> [42306.521225] NFS: server nova error: fileid changed
> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
> 
> On nfsv4, it is even worse, as updates on the server are not immediately 
> reflected on the client like it was the case on nfsv3.
> Unfortunately, that is all the information I currently have.
> (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)

Have you really being changing only the server, or have you been
changing the client at the same time?

See e.g.

	http://marc.info/?l=linux-nfs&m=129088225122788&w=2

--b.

Re: NFS: server error: fileid changed

[Jan Engelhardt]

On Sunday 2010-11-28 17:05, J. Bruce Fields wrote:
>On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
>> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really 
>> unreliable, having clients start to emit messages like
>> 
>> (nfsv3)
>> [42306.521225] NFS: server nova error: fileid changed
>> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
>> 
>> On nfsv4, it is even worse, as updates on the server are not immediately 
>> reflected on the client like it was the case on nfsv3.
>> Unfortunately, that is all the information I currently have.
>> (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
>
>Have you really being changing only the server, or have you been
>changing the client at the same time?
>
>See e.g.
>	http://marc.info/?l=linux-nfs&m=129088225122788&w=2

For my definition of not touching the server yes. (There may be
atime updates, but that's about it.)

Re: NFS: server error: fileid changed

[J. Bruce Fields]

On Sun, Nov 28, 2010 at 06:56:37PM +0100, Jan Engelhardt wrote:
> 
> On Sunday 2010-11-28 17:05, J. Bruce Fields wrote:
> >On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
> >> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really 
> >> unreliable, having clients start to emit messages like
> >> 
> >> (nfsv3)
> >> [42306.521225] NFS: server nova error: fileid changed
> >> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
> >> 
> >> On nfsv4, it is even worse, as updates on the server are not immediately 
> >> reflected on the client like it was the case on nfsv3.
> >> Unfortunately, that is all the information I currently have.
> >> (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
> >
> >Have you really being changing only the server, or have you been
> >changing the client at the same time?
> >
> >See e.g.
> >	http://marc.info/?l=linux-nfs&m=129088225122788&w=2
> 
> For my definition of not touching the server yes. (There may be
> atime updates, but that's about it.)

I was asking about kernel versions; so:

	client 2.6.37-rc1, server 2.6.33

worked, but

	client 2.6.37-rc1, server 2.6.36-rc8

didn't?

It would be worth trying the most recent client patches anyway, as your
symptoms look suspiciously similar to those reported recently, and we'd
like to rule out the known 2.6.37-rc1 client bugs.

What filesystem are you exporting?

--b.

Re: NFS: server error: fileid changed

[Jan Engelhardt]

On Sunday 2010-11-28 19:19, J. Bruce Fields wrote:
>> >On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
>> >> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really 
>> >> unreliable, having clients start to emit messages like
>> >> 
>> >> (nfsv3)
>> >> [42306.521225] NFS: server nova error: fileid changed
>> >> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
>> >> 
>> >> On nfsv4, it is even worse, as updates on the server are not immediately 
>> >> reflected on the client like it was the case on nfsv3.
>> >> Unfortunately, that is all the information I currently have.
>> >> (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
>> >
>> >Have you really being changing only the server, or have you been
>> >changing the client at the same time?
>> >
>> >See e.g.
>> >	http://marc.info/?l=linux-nfs&m=129088225122788&w=2
>> 
>> For my definition of not touching the server yes. (There may be
>> atime updates, but that's about it.)
>
>I was asking about kernel versions; so:
>
>	client 2.6.37-rc1, server 2.6.33
>
>worked, but
>
>	client 2.6.37-rc1, server 2.6.36-rc8
>
>didn't?
>
>It would be worth trying the most recent client patches anyway, as your
>symptoms look suspiciously similar to those reported recently, and we'd
>like to rule out the known 2.6.37-rc1 client bugs.

So me doing what exactly?

>What filesystem are you exporting?

ext4.

Re: NFS: server error: fileid changed

[J. Bruce Fields]

On Sun, Dec 05, 2010 at 03:41:54PM +0100, Jan Engelhardt wrote:
> 
> On Sunday 2010-11-28 19:19, J. Bruce Fields wrote:
> >> >On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
> >> >> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really 
> >> >> unreliable, having clients start to emit messages like
> >> >> 
> >> >> (nfsv3)
> >> >> [42306.521225] NFS: server nova error: fileid changed
> >> >> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
> >> >> 
> >> >> On nfsv4, it is even worse, as updates on the server are not immediately 
> >> >> reflected on the client like it was the case on nfsv3.
> >> >> Unfortunately, that is all the information I currently have.
> >> >> (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
> >> >
> >> >Have you really being changing only the server, or have you been
> >> >changing the client at the same time?
> >> >
> >> >See e.g.
> >> >	http://marc.info/?l=linux-nfs&m=129088225122788&w=2
> >> 
> >> For my definition of not touching the server yes. (There may be
> >> atime updates, but that's about it.)
> >
> >I was asking about kernel versions; so:
> >
> >	client 2.6.37-rc1, server 2.6.33
> >
> >worked, but
> >
> >	client 2.6.37-rc1, server 2.6.36-rc8
> >
> >didn't?
> >
> >It would be worth trying the most recent client patches anyway, as your
> >symptoms look suspiciously similar to those reported recently, and we'd
> >like to rule out the known 2.6.37-rc1 client bugs.
> 
> So me doing what exactly?

I don't understand the question.  You may actually want to wait till the
discussion of the client changes dies down and then try the next -rc.

(But it would still be helpful to have an answer to my question about
kernel versions above.)

> >What filesystem are you exporting?
> 
> ext4.

OK, thanks.--b.

Re: NFS: server error: fileid changed

[Jan Engelhardt]

On Sunday 2010-11-28 19:19, J. Bruce Fields wrote:
>>>On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
>>>> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really 
>>>> unreliable, having clients start to emit messages like
>>>> 
>>>> (nfsv3)
>>>> [42306.521225] NFS: server nova error: fileid changed
>>>> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
>>>> 
>>>> On nfsv4, it is even worse, as updates on the server are not
>>>> immediately reflected on the client like it was the case on
>>>> nfsv3. Unfortunately, that is all the information I currently
>>>> have. (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
>>>
>>>Have you really being changing only the server, or have you been
>>>changing the client at the same time?

Fast forward. I have now a 2.6.38-rc1+ client with a 2.6.37 server, 
using nfs4 with ext4. The "fileid changes" messages seem to be gone.

I change a file on the server, compile it there, and then would expect 
the client to pick up the change, but some awkward internal caching 
mechanism pins the old copy and I don't get, say, new debug messages I 
have added to my program that was just recompiled.

Re: NFS: server error: fileid changed

[J. Bruce Fields]

On Sun, Jan 30, 2011 at 02:27:34PM +0100, Jan Engelhardt wrote:
> 
> On Sunday 2010-11-28 19:19, J. Bruce Fields wrote:
> >>>On Sun, Nov 28, 2010 at 03:23:23PM +0100, Jan Engelhardt wrote:
> >>>> Sometime between 2.6.33 and 2.6.36 it seems, knfsd has become really 
> >>>> unreliable, having clients start to emit messages like
> >>>> 
> >>>> (nfsv3)
> >>>> [42306.521225] NFS: server nova error: fileid changed
> >>>> [42306.521226] fsid 0:10: expected fileid 0x1b007be, got 0x1b006b0
> >>>> 
> >>>> On nfsv4, it is even worse, as updates on the server are not
> >>>> immediately reflected on the client like it was the case on
> >>>> nfsv3. Unfortunately, that is all the information I currently
> >>>> have. (Server is on 2.6.36-rc8, client on 2.6.37-rc1.)
> >>>
> >>>Have you really being changing only the server, or have you been
> >>>changing the client at the same time?
> 
> Fast forward. I have now a 2.6.38-rc1+ client with a 2.6.37 server, 
> using nfs4 with ext4. The "fileid changes" messages seem to be gone.

That leaves in the dark about that particular problem.

> I change a file on the server, compile it there, and then would expect 
> the client to pick up the change, but some awkward internal caching 
> mechanism pins the old copy and I don't get, say, new debug messages I 
> have added to my program that was just recompiled.

Remind me what the backend filesystem is?  Also, does

	echo 0 >/proc/sys/fs/leases-enable

on startup fix the problem?

--b.