Reading libs fails through NFS

Reading libs fails through NFS

[J.A. Magallon]

Hi all...

Anybody has any idea about why this fails:

    fd = open("/lib/libnss_files.so.2", O_RDONLY);
    res = read(fd,buf,512);

/lib is NFS mounted:

192.168.0.1:/lib on /lib type nfs (ro,noatime,nfsvers=3,nolock,addr=192.168.0.1)

and the read fails.
The original code does a getprotobyname("tcp") (netpipe), that fails when it
tries to read the same lib.

The node boots via PXE, with a version of libnss_files.so.2 on the /lib present
in the initrd, which is replaced by the mounted one.

TIA

-- 
J.A. Magallon <jamagallon()able!es>     \                 Software is like sex:
werewolf!able!es                         \           It's better when it's free
Mandrake Linux release 10.0 (Cooker) for i586
Linux 2.4.23-rc1-jam2 (gcc 3.3.1 (Mandrake Linux 9.2 3.3.1-4mdk))

Re: Reading libs fails through NFS

[Trond Myklebust]

>>>>> " " == J A Magallon <J.A.> writes:

     > Hi all...  Anybody has any idea about why this fails:

     >     fd = open("/lib/libnss_files.so.2", O_RDONLY); res =
     >     read(fd,buf,512);

No. Nobody else will be able to tell you either until you tell us what
setup you are using.

Cheers,
  Trond

Re: Reading libs fails through NFS

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 945 bytes --]

On Mon, 17 Nov 2003 01:45:39 +0100, "J.A. Magallon" said:

>     fd = open("/lib/libnss_files.so.2", O_RDONLY);

> The node boots via PXE, with a version of libnss_files.so.2 on the /lib present
> in the initrd, which is replaced by the mounted one.

Just a shot in the dark, but could there be a bug in the NFS code where it's
getting upset that there's cached pages of the file in memory, but the file
that the cached page is from isn't from the file that NFS can see? (note that
this is possibly *different* than a stale NFS handle when a file is unlinked
and then recreated - here chasing the origin of the page doesn't point at
the NFS mount, but at the initrd mount.

Yes, when you mount over a directory, the previous contents are supposed to
become invisible.  I wonder if there's a bug with that if a file is read and
pages cached 'sufficiently early' in the boot process (i.e. before the
real root gets mounted over the initrd root..)

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Re: [BProc] Re: Reading libs fails through NFS

[J.A. Magallon]

On 11.17, Trond Myklebust wrote:
> >>>>> " " == J A Magallon <J.A.> writes:
> 
>      > Hi all...  Anybody has any idea about why this fails:
> 
>      >     fd = open("/lib/libnss_files.so.2", O_RDONLY); res =
>      >     read(fd,buf,512);
> 
> No. Nobody else will be able to tell you either until you tell us what
> setup you are using.
> 

I run a small bproc cluster. Nodes are diskless, boot with a custom initrd,
and execute a simple linuxrc (comments and echo's stripped):

PATH=/bin:/sbin:/usr/bin:/usr/sbin
mount -n -o remount,rw /
mount -t proc none /proc
mount -t devpts -omode=0620 none /dev/pts
mount -t tmpfs none /dev/shm
mount -t tmpfs none /tmp
ifconfig lo up 127.0.0.1
modprobe eth0
ifconfig eth0 up
dhcpcd -H -D -R -N eth0
portmap
mount /lib
mount /bin
mount /sbin
mount /usr
mount /opt
mount /home
mount /work/shared
modprobe eth1
ifconfig eth1 up
dhcpcd -R -N eth1
ntpdate -v 192.168.0.1
modprobe bproc
bpslave -v -d -r 192.168.1.1

fstab for nodes (in /etc in initrd) is:

rootfs / rootfs defaults 0 0
none /proc proc defaults 0 0
none /dev/pts devpts mode=0620 0 0
none /dev/shm tmpfs defaults 0 0
none /tmp tmpfs defaults 0 0
192.168.0.1:/lib    /lib            nfs nfsvers=3,ro,noac,suid
192.168.0.1:/bin    /bin            nfs nfsvers=3,ro,noac,suid
192.168.0.1:/sbin   /sbin           nfs nfsvers=3,ro,noac,suid
192.168.0.1:/usr    /usr            nfs nfsvers=3,ro,noac,suid
192.168.0.1:/opt    /opt            nfs nfsvers=3,ro,noac,suid
192.168.0.1:/home   /home           nfs nfsvers=3,rw
192.168.0.1:/work   /work/shared    nfs nfsvers=3,rw

For example, /opt is just the mount point in initrd, so it is empty. It is
ro, just soft to use:

annwn:/opt> pwd
/opt
annwn:/opt> ls
aleph/  coin/  intel/  mpich/
annwn:/opt> bpsh 0 pwd
/opt
annwn:/opt> bpsh 0 ls 
ls: reading directory .: Invalid argument
annwn:~> bpsh 0 ls /opt/*
ls: reading directory /opt/aleph: Invalid argument
/opt/aleph:
ls: reading directory /opt/coin: Invalid argument

/opt/coin:
ls: reading directory /opt/intel: Invalid argument

/opt/intel:
ls: 
/opt/mpich:
reading directory /opt/mpich: Invalid argument

annwn:~> bpsh 0 strace ls /opt
...
stat64("/opt", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
open("/opt", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
getdents64(3, 0x8060360, 8192)          = -1 EINVAL (Invalid argument)
close(3)                                = 0
...
annwn:/opt> bpsh 0 strace ls
...
open(".", O_RDONLY|O_NONBLOCK|O_LARGEFILE|O_DIRECTORY) = 3
fstat64(3, {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
getdents64(3, 0x8060350, 8192)          = -1 EINVAL (Invalid argument)
close(3)                                = 0
...

It looks like readdir fails (getdents).
Uh ? 
I use a custom kernel, still have to try with plain -rc3 + bproc. But
do you have any ideas about what is going/what am I doing wrong ? This
all worked some time ago with my hacked -jam kernels.

TIA for you attention.

-- 
J.A. Magallon <jamagallon()able!es>     \                 Software is like sex:
werewolf!able!es                         \           It's better when it's free
Mandrake Linux release 10.0 (Cooker) for i586
Linux 2.4.23-rc3-jam1 (gcc 3.3.1 (Mandrake Linux 9.2 3.3.1-4mdk))