Null characters in files on NFS mounted volume....

Null characters in files on NFS mounted volume....

[Dan Yocum]

Hi all,

I've got a "New and Exciting" <tm> bug using 2.4.18 with Trond's client
patches (sans the rpc_tweaks dif), but I'm not sure it's a client side
problem.

This is an intermitent problem:  When attempting to read a file (any file)
on an NFS mounted volume, there appear to be "holes" in the file filled with
ASCII NULL characters.  Reading the file on the local machines shows no such
holes, and other NFS clients don't see the holes, either.  A simple
umount/remount solves the problem.

Oh, I should add that these are autofs mounted volumes.

We thought these might be network driver problems, but we're seeing it on
natsemi, acenic and eepro100 drivers, and I doubt they could all be flaky.

Any ideas?

Thanks,
Dan


-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: bandwidth@sourceforge.net
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Null characters in files on NFS mounted volume....

[Trond Myklebust]

>>>>> " " == Dan Yocum <yocum@fnal.gov> writes:

     > Hi all, I've got a "New and Exciting" <tm> bug using 2.4.18
     > with Trond's client patches (sans the rpc_tweaks dif), but I'm
     > not sure it's a client side problem.

     > This is an intermitent problem: When attempting to read a file
     > (any file) on an NFS mounted volume, there appear to be "holes"
     > in the file filled with ASCII NULL characters.  Reading the
     > file on the local machines shows no such holes, and other NFS
     > clients don't see the holes, either.  A simple umount/remount
     > solves the problem.

Any details?

Are these for instance ordinary files? Are you perhaps engaging in
some multiple NFS client read/writes to the same file? ...  Is it tied
to the client patches, or does it occur with stock 2.4.18 fresh off
the l-k presses?

Cheers,
  Trond

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: bandwidth@sourceforge.net
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Null characters in files on NFS mounted volume....

[Dan Yocum]

Trond,

Nothing special in /var/log/messages on either the client or server that
would help in debugging the problem, unfortunately.

I'm seeing the problem on these 2 client systems:

vanilla 2.4.18 + linux-2.4.18-NFS_ALL.dif - linux-2.4.18-rpc_tweaks.dif +
xfs-1.1-2.4.18-all.patch.

("-", above, meaning 'patch -R')

And:

kernel-2.4.9-13SGI_XFS_1.0.2, which is a Red Hat 2.4.9 kernel with XFS
v1.0.2 patches (built by SGI folks).  I can get you the appropriate patches
from their patch set,  if you want.

The server system is the 1st kernel described above.

The files are FITS files (text header tables, followed by binary data).  The
data was written once to disk and is read by many clients.

If I could reproduce the problem at will, I'd try a generic kernel, but it's
intermittent at best, and we need the patches to access the IRIX servers. 
:-(

When it happens again, I'll do some more debugging on the problem.

Thanks,
Dan



Trond Myklebust wrote:
> 
> >>>>> " " == Dan Yocum <yocum@fnal.gov> writes:
> 
>      > Hi all, I've got a "New and Exciting" <tm> bug using 2.4.18
>      > with Trond's client patches (sans the rpc_tweaks dif), but I'm
>      > not sure it's a client side problem.
> 
>      > This is an intermitent problem: When attempting to read a file
>      > (any file) on an NFS mounted volume, there appear to be "holes"
>      > in the file filled with ASCII NULL characters.  Reading the
>      > file on the local machines shows no such holes, and other NFS
>      > clients don't see the holes, either.  A simple umount/remount
>      > solves the problem.
> 
> Any details?
> 
> Are these for instance ordinary files? Are you perhaps engaging in
> some multiple NFS client read/writes to the same file? ...  Is it tied
> to the client patches, or does it occur with stock 2.4.18 fresh off
> the l-k presses?
> 
> Cheers,
>   Trond


-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: bandwidth@sourceforge.net
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Null characters in files on NFS mounted volume....

[Trond Myklebust]

M=E5ndag 13. mai 2002 18:54 skreiv Dan Yocum:
> Trond,
>
> Nothing special in /var/log/messages on either the client or server tha=
t
> would help in debugging the problem, unfortunately.
>
> I'm seeing the problem on these 2 client systems:
>
> vanilla 2.4.18 + linux-2.4.18-NFS_ALL.dif - linux-2.4.18-rpc_tweaks.dif=
 +
> xfs-1.1-2.4.18-all.patch.
>
> ("-", above, meaning 'patch -R')
>
> And:
>
> kernel-2.4.9-13SGI_XFS_1.0.2, which is a Red Hat 2.4.9 kernel with XFS
> v1.0.2 patches (built by SGI folks).  I can get you the appropriate pat=
ches
> from their patch set,  if you want.
>
> The server system is the 1st kernel described above.

Are you able to reproduce it on a non-XFS server?

Cheers,
  Trond

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: bandwidth@sourceforge.net
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Null characters in files on NFS mounted volume....

[Dan Yocum]

Trond Myklebust wrote:

> 
> Are you able to reproduce it on a non-XFS server?


Heh.  I don't have any non-XFS machines.  But, I'll post the problem to the
XFS lists - maybe someone there has seen it.


Cheers,
Dan


-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: bandwidth@sourceforge.net
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Null characters in files on NFS mounted volume....

[Neil Brown]

On Monday May 13, yocum@fnal.gov wrote:
> Hi all,
> 
> I've got a "New and Exciting" <tm> bug using 2.4.18 with Trond's client
> patches (sans the rpc_tweaks dif), but I'm not sure it's a client side
> problem.
> 
> This is an intermitent problem:  When attempting to read a file (any file)
> on an NFS mounted volume, there appear to be "holes" in the file filled with
> ASCII NULL characters.  Reading the file on the local machines shows no such
> holes, and other NFS clients don't see the holes, either.  A simple
> umount/remount solves the problem.

I remembers long ago some referred to NFS as
   Nulls Frequently Substituted
because they were experiencing exactly this sort of thing, and it
wasn't on Linux at all.

You wouldn't be using 'soft' mounts would you?

NeilBrown

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: bandwidth@sourceforge.net
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Null characters in files on NFS mounted volume....

[Dan Yocum]

Neil Brown wrote:
> 
> On Monday May 13, yocum@fnal.gov wrote:
> > Hi all,
> >
> > I've got a "New and Exciting" <tm> bug using 2.4.18 with Trond's client
> > patches (sans the rpc_tweaks dif), but I'm not sure it's a client side
> > problem.
> >
> > This is an intermitent problem:  When attempting to read a file (any file)
> > on an NFS mounted volume, there appear to be "holes" in the file filled with
> > ASCII NULL characters.  Reading the file on the local machines shows no such
> > holes, and other NFS clients don't see the holes, either.  A simple
> > umount/remount solves the problem.
> 
> I remembers long ago some referred to NFS as
>    Nulls Frequently Substituted
> because they were experiencing exactly this sort of thing, and it
> wasn't on Linux at all.

Ha!  :-)

I did find a similar occurence while googling
(http://www.geocrawler.com/archives/3/4/1998/11/0/59164/), reported back in
'98 on LinuxPPC, that produced similar results, but since this happened 3
times on the same day to 3 different machines, I thought I'd report it.


> 
> You wouldn't be using 'soft' mounts would you?


yup, automounted with these options:  

-rw,grpid,soft,intr,quota,nodev,nosuid,timeo=10,retrans=3


I suppose they could be hard....  I'll think about that.

Cheers,
Dan



-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: bandwidth@sourceforge.net
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Null characters in files on NFS mounted volume....

[Ion Badulescu]

On Tue, 14 May 2002 09:41:39 -0500, Dan Yocum <yocum@fnal.gov> wrote:
>> > This is an intermitent problem:  When attempting to read a file (any file)
>> > on an NFS mounted volume, there appear to be "holes" in the file filled with
>> > ASCII NULL characters.  Reading the file on the local machines shows no such
>> > holes, and other NFS clients don't see the holes, either.  A simple
>> > umount/remount solves the problem.
>> 
> Neil Brown wrote:
>> You wouldn't be using 'soft' mounts would you?
> 
> yup, automounted with these options:  
> 
> -rw,grpid,soft,intr,quota,nodev,nosuid,timeo=10,retrans=3

In that case let me ask you this: is your application checking the return
code of the read()? or are you using mmap() instead of read()?

If read() returns zero-filled buffers without signalling an errors, it's
a problem in the NFS client. That is, assumming it's not some bad interaction
between XFS and the NFS server -- so try to clear up that possibility
by mounting with the hard option.

If you're using mmap(), then I don't even know how the system should tell
you about the problem... I suppose it would be sending a SIGSEGV or SIGBUS
signal, since it's ultimately a failing page fault, but I haven't tested it.

> I suppose they could be hard....  I'll think about that.

Yes, they _should_ be hard if you care about your data integrity.

But that shouldn't stop us from fixing bugs in the 'soft' model, if they
exist... which is why I'm asking you all these details. :-)

Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: bandwidth@sourceforge.net
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Null characters in files on NFS mounted volume....

[Dan Yocum]

Ion,

The apps are apache and plain old 'less,' so I bet they just use read().

And the problem hasn't cropped up again (yet) and hadn't cropped up in ~1
year with older kernels (specifically 2.4.5).  I just upgraded to 2.4.18
last Thursday when it did start.

I'll change the options to hard and see if that makes any changes.

Thanks,
Dan

Ion Badulescu wrote:
> 
> On Tue, 14 May 2002 09:41:39 -0500, Dan Yocum <yocum@fnal.gov> wrote:
> >> > This is an intermitent problem:  When attempting to read a file (any file)
> >> > on an NFS mounted volume, there appear to be "holes" in the file filled with
> >> > ASCII NULL characters.  Reading the file on the local machines shows no such
> >> > holes, and other NFS clients don't see the holes, either.  A simple
> >> > umount/remount solves the problem.
> >>
> > Neil Brown wrote:
> >> You wouldn't be using 'soft' mounts would you?
> >
> > yup, automounted with these options:
> >
> > -rw,grpid,soft,intr,quota,nodev,nosuid,timeo=10,retrans=3
> 
> In that case let me ask you this: is your application checking the return
> code of the read()? or are you using mmap() instead of read()?
> 
> If read() returns zero-filled buffers without signalling an errors, it's
> a problem in the NFS client. That is, assumming it's not some bad interaction
> between XFS and the NFS server -- so try to clear up that possibility
> by mounting with the hard option.
> 
> If you're using mmap(), then I don't even know how the system should tell
> you about the problem... I suppose it would be sending a SIGSEGV or SIGBUS
> signal, since it's ultimately a failing page fault, but I haven't tested it.
> 
> > I suppose they could be hard....  I'll think about that.
> 
> Yes, they _should_ be hard if you care about your data integrity.
> 
> But that shouldn't stop us from fixing bugs in the 'soft' model, if they
> exist... which is why I'm asking you all these details. :-)
> 
> Ion
> 
> --
>   It is better to keep your mouth shut and be thought a fool,
>             than to open it and remove all doubt.




-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply
the hardware. You get the recognition. Email Us: bandwidth@sourceforge.net
_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Null characters in files on NFS mounted volume....

[Dan Yocum]

Just an update - the null characters problem hasn't cropped up again, and I
haven't done anything to "fix" the problem, either - the mounts are still
'soft', etc.

I'm about ready to declare it a non-reproducable network error and nothing
to do with NFS, but if it turns up again, I'll let you know.

Thanks,
Dan

Ion Badulescu wrote:
> 
> On Tue, 14 May 2002 09:41:39 -0500, Dan Yocum <yocum@fnal.gov> wrote:
> >> > This is an intermitent problem:  When attempting to read a file (any file)
> >> > on an NFS mounted volume, there appear to be "holes" in the file filled with
> >> > ASCII NULL characters.  Reading the file on the local machines shows no such
> >> > holes, and other NFS clients don't see the holes, either.  A simple
> >> > umount/remount solves the problem.
> >>
> > Neil Brown wrote:
> >> You wouldn't be using 'soft' mounts would you?
> >
> > yup, automounted with these options:
> >
> > -rw,grpid,soft,intr,quota,nodev,nosuid,timeo=10,retrans=3
> 
> In that case let me ask you this: is your application checking the return
> code of the read()? or are you using mmap() instead of read()?
> 
> If read() returns zero-filled buffers without signalling an errors, it's
> a problem in the NFS client. That is, assumming it's not some bad interaction
> between XFS and the NFS server -- so try to clear up that possibility
> by mounting with the hard option.
> 
> If you're using mmap(), then I don't even know how the system should tell
> you about the problem... I suppose it would be sending a SIGSEGV or SIGBUS
> signal, since it's ultimately a failing page fault, but I haven't tested it.
> 
> > I suppose they could be hard....  I'll think about that.
> 
> Yes, they _should_ be hard if you care about your data integrity.
> 
> But that shouldn't stop us from fixing bugs in the 'soft' model, if they
> exist... which is why I'm asking you all these details. :-)
> 
> Ion
> 
> --
>   It is better to keep your mouth shut and be thought a fool,
>             than to open it and remove all doubt.





-- 
Dan Yocum
Sloan Digital Sky Survey, Fermilab  630.840.6509
yocum@fnal.gov, http://www.sdss.org
SDSS.  Mapping the Universe.

_______________________________________________________________

Don't miss the 2002 Sprint PCS Application Developer's Conference
August 25-28 in Las Vegas -- http://devcon.sprintpcs.com/adp/index.cfm

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs

Re: Null characters in files on NFS mounted volume....

[Ion Badulescu]

On Fri, 24 May 2002 11:44:27 -0500, Dan Yocum <yocum@fnal.gov> wrote:
> Just an update - the null characters problem hasn't cropped up again, and I
> haven't done anything to "fix" the problem, either - the mounts are still
> 'soft', etc.
> 
> I'm about ready to declare it a non-reproducable network error and nothing
> to do with NFS, but if it turns up again, I'll let you know.

With 'soft' mounts, it's not 'if', but 'when'...

Ion

-- 
  It is better to keep your mouth shut and be thought a fool,
            than to open it and remove all doubt.

_______________________________________________________________

Don't miss the 2002 Sprint PCS Application Developer's Conference
August 25-28 in Las Vegas -- http://devcon.sprintpcs.com/adp/index.cfm

_______________________________________________
NFS maillist  -  NFS@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfs