Re: NFS problem on Microblaze LE

[Michal Simek <monstr@monstr.eu>]

Chuck Lever wrote:
> On Mar 3, 2011, at 4:29 AM, Michal Simek wrote:
> 
>> J. Bruce Fields wrote:
>>> On Wed, Mar 02, 2011 at 07:20:10PM +0100, Michal Simek wrote:
>>>> J. Bruce Fields wrote:
>>>>> On Wed, Mar 02, 2011 at 05:11:53PM +0100, Michal Simek wrote:
>>>>>> J. Bruce Fields wrote:
>>>>>>> On Wed, Mar 02, 2011 at 02:04:18PM +0100, Michal Simek wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am getting some troubles to get nfs work on new Microblaze
>>>>>>>> little-endian platform and I would like to ask you for some
>>>>>>>> recommendations how to debug it.
>>>>>>>>
>>>>>>>> First of all I need to write that Microblaze big-endian platforms have no problem.
>>>>>>>> The problem only happen if I use mount without -o nolock option
>>>>>>>> (mount -t nfs 192.168.0.101:/tftpboot/nfs /mnt)
>>>>>>>> If I use -o nolock option I have no problem to use nfs.
>>>>>>>>
>>>>>>>> I use xilinx emaclite and axi emac(it is not in the mainline now)
>>>>>>>> driver and I have no problem to use dhcp, ftp, http, telnet and
>>>>>>>> other internet protocols.
>>>>>>>>
>>>>>>>> I compared debug logs on big and little endian platform(rootfs has
>>>>>>>> the same setting) I found that little-endian got packet which is
>>>>>>>> shorter than on big endian which I have added to the log below.
>>>>>>>> The second thing, which I think is connected to the previous point,
>>>>>>>> is that I am getting BADCRED in rpc_verify_headers.
>>>>>>>>
>>>>>>>> Is there any option/macro/recommended debug technique how to see
>>>>>>>> packets? I need to get some clue how to see packet and then how they
>>>>>>>> are passed to rpc_verify_header function.
>>>>>>> A good first step would be to look at the network traffic with
>>>>>>> wireshark.
>>>>>> Yes, I am looking at it all the time but I can't see anything weird.
>>>>>> Look at attachment. 192.168.0.101 - host, 192.168.0.103 target.
>>>>>>
>>>>>> There are two NULL calls and two reply calls.
>>>>> Yes, looks normal.  I wonder why everything exept portmap is using udp,
>>>>> but your debugging traces refer to tcp?
>>>>>
>>>>> Oh, wait, it's talking about portmap map/unmap calls: could try try
>>>>> running wireshark on the loopback interface?  (run with -ilo).
>>>>>
>>>> It is captured by tcpdump (tcpdump -i lo -e -S -n -vvv -x -w nfs)
>>>> If you want to use different setting please let me know. (I have
>>>> also verbose node but saving to file should be enough for you).
>>> A little odd; -s0 to get the whole packet might help.
>> I can't use -s0 because I use older tcpdump but that shouldn't be a problem.
>> Packet dumps for LE and BE are attached.
>>
>>> You may also want to take a look at it yourself in wireshark.  Probably
>>> you'll see the BADCRED error in one of the replies once you manage to
>>> capture the right stuff.
>> I have looked at it and I see two things.
>> 1. TCP checksum is incorrect but BE has the same behavior that's why I think it is fine.
>> 2. Packet #9 (V2 UNSET Reply (Call In 8)) contains Reply state: denied and AUTH_ERROR
>> bad credential (seal broken) that's the confirmation what I saw from the kernel debug logs.
>>
>> What does it caused this rejection?
>>
>> I am looking for it in the kernel.
> 
> Which kernel release is this?  (uname -a)

I can see this behavior on several kernels 2.6.35-37. Currently I use 2.6.37.2.

> 
> Which distribution is this?  

It is petalinux distribution which is embedded distribution for Microblaze and PPC440.

> In user space, does it use portmap with glibc RPC, or rpcbind with libtirpc?

I use the latest portmap from git://neil.brown.name/portmap that's why I suspect with glibc RPC.

Is there any endian changes in glibc implementation? We have done this port (not personally me) but 
if is I think that we miss this change.

Thanks,
Michal

-- 
Michal Simek, Ing. (M.Eng)
w: www.monstr.eu p: +42-0-721842854
Maintainer of Linux kernel 2.6 Microblaze Linux - http://www.monstr.eu/fdt/
Microblaze U-BOOT custodian