Re: cifs: ls of mount point gives input/output error (probably related to CIFS: getdents() broken for large dirs)

[Konstantinos Skarlatos <k.skarlatos-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]

[-- Attachment #1: Type: text/plain, Size: 5813 bytes --]

On 31/12/2011 12:44 πμ, Konstantinos Skarlatos wrote:
> On 31/12/2011 12:18 πμ, Shirish Pargaonkar wrote:
>> On Fri, Dec 30, 2011 at 3:46 PM, Konstantinos Skarlatos
>> <k.skarlatos-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>  wrote:
>>> On 30/12/2011 8:00 μμ, Konstantinos Skarlatos wrote:
>>>> On 30/12/2011 3:11 μμ, Jeff Layton wrote:
>>>>> On Fri, 30 Dec 2011 11:04:59 +0200
>>>>> Konstantinos Skarlatos<k.skarlatos-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>    wrote:
>>>>>
>>>>>> On 29/12/2011 3:54 μμ, Konstantinos Skarlatos wrote:
>>>>>>> On Πέμπτη, 29 Δεκέμβριος 2011 3:39:30 μμ, Jeff Layton wrote:
>>>>>>>> On Thu, 29 Dec 2011 12:30:18 +0200
>>>>>>>> Konstantinos Skarlatos<k.skarlatos-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>     wrote:
>>>>>>>>
>>>>>>>>> On 29/12/2011 4:04 πμ, Jeff Layton wrote:
>>>>>>>>>> On Thu, 29 Dec 2011 02:08:57 +0200
>>>>>>>>>> Konstantinos Skarlatos<k.skarlatos-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>      wrote:
>>>>>>>>>>
>>>>>>>>>>> I mount via cifs a windows XP share, df gives me correct sizes,
>>>>>>>>>>> but when
>>>>>>>>>>> I ls the mount point i get input/output error.
>>>>>>>>>>> strace: http://pastebin.com/WXf8M1nu
>>>>>>>>>>>
>>>>>>>>>>> mount --verbose -t cifs -o 
>>>>>>>>>>> username=administrator,password=blahblah
>>>>>>>>>>> //192.168.0.11/jobs /mnt/backups/montaz/jobs
>>>>>>>>>>> mount.cifs kernel mount options:
>>>>>>>>>>>
>>>>>>>>>>> ip=192.168.0.11,unc=\\192.168.0.11\jobs,,ver=1,user=administrator,pass=******** 
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> df
>>>>>>>>>>> //192.168.0.11/jobs                                       
>>>>>>>>>>> 114464
>>>>>>>>>>> 105196      9268  92% /mnt/backups/montaz/jobs
>>>>>>>>>>>
>>>>>>>>>>> ls /mnt/backups/montaz/jobs/
>>>>>>>>>>> ls: reading directory /mnt/backups/montaz/jobs/: 
>>>>>>>>>>> Input/output error
>>>>>>>>>>> total 0
>>>>>>>>>>>
>>>>>>>>>>> the fun thing is that i can cd to a lower level directory, 
>>>>>>>>>>> and ls
>>>>>>>>>>> works
>>>>>>>>>>> fine there! only the mount point has the problem
>>>>>>>>>>>
>>>>>>>>>>> ls /mnt/backups/montaz/jobs/test
>>>>>>>>>>> total 44K
>>>>>>>>>>> drwxr-xr-x 1 root root    0 Apr 30  2010 blah blah/
>>>>>>>>>>> ......
>>>>>>>>>>>
>>>>>>>>>>> kernel version 3.2rc7
>>>>>>>>>>>
>>>>>>>>>>> this seems to be related to :
>>>>>>>>>>> https://lkml.org/lkml/2011/8/1/427
>>>>>>>>>>> Re: [3.0.0+][Regression][Bisected] CIFS: getdents() broken for
>>>>>>>>>>> large dirs
>>>>>>>>>>>
>>>>>>>>>> Hmmm, maybe. What makes you think that it's related? What 
>>>>>>>>>> sort of
>>>>>>>>>> server are you seeing this against?
>>>>>>>>> Windows XP service pack 2 (greek)
>>>>>>>>
>>>>>>>> How many files are in the directory?
>>>>>>>>
>>>>>>> 140 folders and 20 files
>>>>>>>
>>>>>> Attached is a tcp dump of my session.
>>>>> I tried reproducing this here, but wasn't able to. Testing against my
>>>>> xp box worked fine.
>>>>>
>>>>> Most likely, the FIND_FILE responses are falling afoul of the code in
>>>>> coalesce_t2 or check2ndT2. Unfortunately that code is pretty
>>>>> complicated and I'm not certain what the problem actually is...
>>>>>
>>>>> One thing that's interesting is that the total data being sent in the
>>>>> request is rather large (16336 bytes). I think that's legit, but 
>>>>> maybe
>>>>> it's exceeding the end of the buffer once we try to coalesce it.
>>>>>
>>>>> Would it be possible to get the cFYI output from this test?
>>>> I did not get a cFYI output from that test, but i redid a 
>>>> mount-ls-umount
>>>> and am attaching the tcpdump
>>>> Also here http://pastebin.com/J20uC6kU you can find the cifsFYI and 
>>>> the
i forgot to mention that since i got this pastebin and the attached 
tcpdump the contents of jobs have not changed at all, so you can compare 
with these if you like
>>>> contents of /proc/fs/cifs/DebugData form the same test
>>>>>
>>>>> Is this a regression? Did it work with earlier kernels and only
>>>>> recently start failing?
>>>>>
>>>> I do not know, and i am a bit afraid to downgrade this machine 
>>>> below 3.0
>>>> due to some changes arch linux has introduced recently. I can 
>>>> always set up
>>>> a few virtual machines though, and i can even request permission 
>>>> from my
>>>> company to give you shell access if you like. Which kernel versions 
>>>> would
>>>> you like me to test?
>>> I just tested  3.1.5-1-ARCH on a virtual machine and it works, so it is
>>> probably a regression... On the same virtual machine 3.2-rc7 produces
>>> input/output error. The virtual machine is a fresh install of arch 
>>> linux.
>>> here is the relevant pastebin http://pastebin.com/BwX2DqJC
>>> and attached is the tcpdump
>>> As a am a noob to all this, what should I do next in order to help you?
>>> maybe compile a 3.1 kernel from official sources to make sure no 
>>> patches
>>> from arch linux interfere?
>>>
>> It is the same request from the cifs client in both the successful and
>> unsuccessful cases. trans2, findfirst2, infolevel of 261 and search 
>> count
>> of 150.
>> So nothing has changed from what is emanating from cifs client.
>>
>> In the error case, Windows XP responds with search count of 142 whereas
>> in successful case, Windows XP responds with search count of 
>> significantly
>> less number, e.g. 36.
>>
>> Are files/directories in  jobs  directory less now i.e. has anything 
>> changed
>> in this directory on the server 192.168.0.11 i.e. have files been 
>> deleted,
>> directories removed etc. since the failure was noticed the very first 
>> time?
> there are some changes in the jobs directory since the failure was 
> noticed, but they are quite minor. at most one folder was deleted and 
> one was added, and the pdf files in the root dir (1_B_MAYRO.pdf 
> 1XR.pdf ....7_B_MAYRO.pdf  ) were overwritten, +- one or two files.
>


[-- Attachment #2: cifs-traffic2.pcap --]
[-- Type: application/octet-stream, Size: 23203 bytes --]