Re: Kernel 3.0: Instant kernel crash when mounting CIFS

Re: Kernel 3.0: Instant kernel crash when mounting CIFS

[Jesper Juhl]

Adding some more CIFS people to Cc.


On Mon, 15 Aug 2011, Justin Piszcz wrote:

> Hi,
> 
> Is this normal/is there a limitation on CIFS mounts? I do not have the full
> crash output, I would need to take a picture of the screen, but its completely
> reproducible, crashes every time..  thoughts?
> 
> Aug 14 05:16:41 box kernel: [518217.158195] df              D 0000000000000000
> 0  5465  11246 0x00000004
> Aug 14 05:16:41 box kernel: [518217.158200]  ffff88061f19c090 0000000000000082
> 0000000000000002 ffff880600000000
> Aug 14 05:16:41 box kernel: [518217.158206]  ffff8806275accf0 0000000000004000
> ffff88058513ffd8 ffff88058513ffd8
> Aug 14 05:16:41 box kernel: [518217.158211]  ffff88058513fdf8 ffff880627076240
> 000000000000002f 0000000000000000
> Aug 14 05:16:41 box kernel: [518217.158217] Call Trace:
> Aug 14 05:16:41 box kernel: [518217.158226]  [<ffffffff810ca086>] ?
> do_lookup+0x46/0x310
> Aug 14 05:16:41 box kernel: [518217.158231]  [<ffffffff810c7e9d>] ?
> acl_permission_check+0x4d/0xb0
> Aug 14 05:16:41 box kernel: [518217.158239]  [<ffffffff815b2448>] ?
> __mutex_lock_slowpath+0xc8/0x140
> Aug 14 05:16:41 box kernel: [518217.158243]  [<ffffffff815b1fea>] ?
> mutex_lock+0x1a/0x40
> Aug 14 05:16:41 box kernel: [518217.158250]  [<ffffffff811ec969>] ?
> cifs_reconnect_tcon+0x1a9/0x320
> Aug 14 05:16:41 box kernel: [518217.158254]  [<ffffffff811ecba3>] ?
> smb_init+0x33/0x90
> Aug 14 05:16:41 box kernel: [518217.158258]  [<ffffffff811f30e0>] ?
> CIFSSMBQFSInfo+0x60/0x210
> Aug 14 05:16:41 box kernel: [518217.158262]  [<ffffffff811eb86e>] ?
> cifs_statfs+0xce/0x170
> Aug 14 05:16:41 box kernel: [518217.158268]  [<ffffffff810e5797>] ?
> statfs_by_dentry+0x67/0x100
> Aug 14 05:16:41 box kernel: [518217.158272]  [<ffffffff810e5845>] ?
> vfs_statfs+0x15/0x90
> Aug 14 05:16:41 box kernel: [518217.158275]  [<ffffffff810e58f0>] ?
> user_statfs+0x30/0x50
> Aug 14 05:16:41 box kernel: [518217.158279]  [<ffffffff810e5972>] ?
> sys_statfs+0x12/0x30
> Aug 14 05:16:41 box kernel: [518217.158284]  [<ffffffff815b3b7b>] ?
> system_call_fastpath+0x16/0x1b
> 
> Then it hangs up, on my machine it actually panics and in both cases all I/O
> stops and the machine is unusable.  This is from simply mounting a CIFS share.
> The share sizes are 18T and 28T, is there a limitation on the size that would
> cause the kernel to crash like that?
> 
> mount.cifs //l1/drve /mnt_pnt -o user=myusername,pass=mypassword, then
> df -h or ls /mnt_pnt instant kernel panic. :(
> 
> Remote host is Windows 7 64-bit.
> 
> Justin.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
> 

-- 
Jesper Juhl <jj-IYz4IdjRLj0sV2N9l4h3zg@public.gmane.org>       http://www.chaosbits.net/
Don't top-post http://www.catb.org/jargon/html/T/top-post.html
Plain text mails only, please.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS

[Jeff Layton]

On Mon, 15 Aug 2011 11:18:38 +0200 (CEST)
Jesper Juhl <jj-IYz4IdjRLj0sV2N9l4h3zg@public.gmane.org> wrote:

> 
> Adding some more CIFS people to Cc.
> 
> 
> On Mon, 15 Aug 2011, Justin Piszcz wrote:
> 
> > Hi,
> > 
> > Is this normal/is there a limitation on CIFS mounts? I do not have the full
> > crash output, I would need to take a picture of the screen, but its completely
> > reproducible, crashes every time..  thoughts?
> > 
> > Aug 14 05:16:41 box kernel: [518217.158195] df              D 0000000000000000
> > 0  5465  11246 0x00000004
> > Aug 14 05:16:41 box kernel: [518217.158200]  ffff88061f19c090 0000000000000082
> > 0000000000000002 ffff880600000000
> > Aug 14 05:16:41 box kernel: [518217.158206]  ffff8806275accf0 0000000000004000
> > ffff88058513ffd8 ffff88058513ffd8
> > Aug 14 05:16:41 box kernel: [518217.158211]  ffff88058513fdf8 ffff880627076240
> > 000000000000002f 0000000000000000
> > Aug 14 05:16:41 box kernel: [518217.158217] Call Trace:
> > Aug 14 05:16:41 box kernel: [518217.158226]  [<ffffffff810ca086>] ?
> > do_lookup+0x46/0x310
> > Aug 14 05:16:41 box kernel: [518217.158231]  [<ffffffff810c7e9d>] ?
> > acl_permission_check+0x4d/0xb0
> > Aug 14 05:16:41 box kernel: [518217.158239]  [<ffffffff815b2448>] ?
> > __mutex_lock_slowpath+0xc8/0x140
> > Aug 14 05:16:41 box kernel: [518217.158243]  [<ffffffff815b1fea>] ?
> > mutex_lock+0x1a/0x40
> > Aug 14 05:16:41 box kernel: [518217.158250]  [<ffffffff811ec969>] ?
> > cifs_reconnect_tcon+0x1a9/0x320
> > Aug 14 05:16:41 box kernel: [518217.158254]  [<ffffffff811ecba3>] ?
> > smb_init+0x33/0x90
> > Aug 14 05:16:41 box kernel: [518217.158258]  [<ffffffff811f30e0>] ?
> > CIFSSMBQFSInfo+0x60/0x210
> > Aug 14 05:16:41 box kernel: [518217.158262]  [<ffffffff811eb86e>] ?
> > cifs_statfs+0xce/0x170
> > Aug 14 05:16:41 box kernel: [518217.158268]  [<ffffffff810e5797>] ?
> > statfs_by_dentry+0x67/0x100
> > Aug 14 05:16:41 box kernel: [518217.158272]  [<ffffffff810e5845>] ?
> > vfs_statfs+0x15/0x90
> > Aug 14 05:16:41 box kernel: [518217.158275]  [<ffffffff810e58f0>] ?
> > user_statfs+0x30/0x50
> > Aug 14 05:16:41 box kernel: [518217.158279]  [<ffffffff810e5972>] ?
> > sys_statfs+0x12/0x30
> > Aug 14 05:16:41 box kernel: [518217.158284]  [<ffffffff815b3b7b>] ?
> > system_call_fastpath+0x16/0x1b
> > 
> > Then it hangs up, on my machine it actually panics and in both cases all I/O
> > stops and the machine is unusable.  This is from simply mounting a CIFS share.
> > The share sizes are 18T and 28T, is there a limitation on the size that would
> > cause the kernel to crash like that?
> > 
> > mount.cifs //l1/drve /mnt_pnt -o user=myusername,pass=mypassword, then
> > df -h or ls /mnt_pnt instant kernel panic. :(
> > 
> > Remote host is Windows 7 64-bit.
> > 

There was at least one known oops at mount time with 3.0:

    https://bugzilla.redhat.com/show_bug.cgi?id=727927

...but I can't tell whether you hit that w/o seeing the oops messasge.

It should be fixed in mainline and the patches for that and another
mount time problem will be in the next set of stable kernels. You may
want to pull these two commits into your kernel and try again:

80975d21aae2136ccae1ce914a1602dc1d8b0795
f9e8c45002cacad536b338dfa9e910e341a49c31

Cheers,
-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS

[Justin Piszcz]

On Mon, 15 Aug 2011, Jeff Layton wrote:

> On Mon, 15 Aug 2011 11:18:38 +0200 (CEST)
> Jesper Juhl <jj-IYz4IdjRLj0sV2N9l4h3zg@public.gmane.org> wrote:
>
> There was at least one known oops at mount time with 3.0:
>
>    https://bugzilla.redhat.com/show_bug.cgi?id=727927
>
> ...but I can't tell whether you hit that w/o seeing the oops messasge.
>
> It should be fixed in mainline and the patches for that and another
> mount time problem will be in the next set of stable kernels. You may
> want to pull these two commits into your kernel and try again:
>
> 80975d21aae2136ccae1ce914a1602dc1d8b0795
> f9e8c45002cacad536b338dfa9e910e341a49c31
>

Thanks,

Patches applied to 3.0.1 and will try to mount the CIFS shares later today:

# patch -p1 < /home/war/cifs1
patching file fs/cifs/cifsfs.c
# patch -p1 < /home/war/cifs2
patching file fs/cifs/inode.c

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS

[Justin Piszcz]

On Mon, 15 Aug 2011, Justin Piszcz wrote:

>
>
> On Mon, 15 Aug 2011, Jeff Layton wrote:
>
>> On Mon, 15 Aug 2011 11:18:38 +0200 (CEST)
>> Jesper Juhl <jj-IYz4IdjRLj0sV2N9l4h3zg@public.gmane.org> wrote:
>> 
>> There was at least one known oops at mount time with 3.0:
>>
>>    https://bugzilla.redhat.com/show_bug.cgi?id=727927
>> 
>> ...but I can't tell whether you hit that w/o seeing the oops messasge.
>> 
>> It should be fixed in mainline and the patches for that and another
>> mount time problem will be in the next set of stable kernels. You may
>> want to pull these two commits into your kernel and try again:
>> 
>> 80975d21aae2136ccae1ce914a1602dc1d8b0795
>> f9e8c45002cacad536b338dfa9e910e341a49c31
>> 
>
> Thanks,
>
> Patches applied to 3.0.1 and will try to mount the CIFS shares later today:
>
> # patch -p1 < /home/war/cifs1
> patching file fs/cifs/cifsfs.c
> # patch -p1 < /home/war/cifs2
> patching file fs/cifs/inode.c
>
> Justin.
>
>

Hello,

It crashed again with the patches, this time I got a picture:
http://home.comcast.net/~jpiszcz/20110815/26030-kernel-crash-when-cifs-mount.jpg

The machine is up no issues until you try and mount a CIFS share, then
the problem occurs, every time (and with the patches as well).

3.0.1+80975d21aae2136ccae1ce914a1602dc1d8b0795
      +f9e8c45002cacad536b338dfa9e910e341a49c31

Same result.

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS

[Steve French]

With the more current cifs (ie  patches applied) does it fail on the
mount? or after the mount on the df -h that you mentioned in your
earlier note?

Not immediately obvious to me if the oops has much to do with cifs
code given the call stack, but we may be able to determine a little
more by looking at a wireshark trace of the last network request
and/or response (presumably parsing a cifs QFSInfo response from
Windows 7).

I also run Windows 7 64 server in our test environment, will rebuild
on laster 3.0.1-rc2 and retry but I would be surprised if I can
reproduce this.

On Mon, Aug 15, 2011 at 1:01 PM, Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>
>
> On Mon, 15 Aug 2011, Justin Piszcz wrote:
>
>>
>>
>> On Mon, 15 Aug 2011, Jeff Layton wrote:
>>
>>> On Mon, 15 Aug 2011 11:18:38 +0200 (CEST)
>>> Jesper Juhl <jj-IYz4IdjRLj0sV2N9l4h3zg@public.gmane.org> wrote:
>>>
>>> There was at least one known oops at mount time with 3.0:
>>>
>>>   https://bugzilla.redhat.com/show_bug.cgi?id=727927
>>>
>>> ...but I can't tell whether you hit that w/o seeing the oops messasge.
>>>
>>> It should be fixed in mainline and the patches for that and another
>>> mount time problem will be in the next set of stable kernels. You may
>>> want to pull these two commits into your kernel and try again:
>>>
>>> 80975d21aae2136ccae1ce914a1602dc1d8b0795
>>> f9e8c45002cacad536b338dfa9e910e341a49c31
>>>
>>
>> Thanks,
>>
>> Patches applied to 3.0.1 and will try to mount the CIFS shares later
>> today:
>>
>> # patch -p1 < /home/war/cifs1
>> patching file fs/cifs/cifsfs.c
>> # patch -p1 < /home/war/cifs2
>> patching file fs/cifs/inode.c
>>
>> Justin.
>>
>>
>
> Hello,
>
> It crashed again with the patches, this time I got a picture:
> http://home.comcast.net/~jpiszcz/20110815/26030-kernel-crash-when-cifs-mount.jpg
>
> The machine is up no issues until you try and mount a CIFS share, then
> the problem occurs, every time (and with the patches as well).
>
> 3.0.1+80975d21aae2136ccae1ce914a1602dc1d8b0795
>     +f9e8c45002cacad536b338dfa9e910e341a49c31
>
> Same result.
>
> Justin.
>
>



-- 
Thanks,

Steve

Re: Kernel 3.0: Instant kernel crash when mounting CIFS

[Dave Jones]

On Mon, Aug 15, 2011 at 02:01:12PM -0400, Justin Piszcz wrote:
 > It crashed again with the patches, this time I got a picture:
 > http://home.comcast.net/~jpiszcz/20110815/26030-kernel-crash-when-cifs-mount.jpg
 > 
 > The machine is up no issues until you try and mount a CIFS share, then
 > the problem occurs, every time (and with the patches as well).

This looks like you oopsed in cache_alloc_refill, and then after that
the watchdog kicked in, and spewed traces, scrolling the backtrace for
the actual oops off the screen.

Might be worth trying to reproduce it with CONFIG_LOCKUP_DETECTOR disabled.

We frequently see this sort of thing reported in Fedora. It would be
nice if we had a means of either reading scrollback when we lockup,
or buffering the output so we pause between dumping traces.

	Dave

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Mon, 15 Aug 2011, Justin Piszcz wrote:

>
>
> On Mon, 15 Aug 2011, Justin Piszcz wrote:
>
>> On Mon, 15 Aug 2011, Jeff Layton wrote:
>>

Hi,

Tried with linux-3.1-rc2 and:

You don't even need to get a successful mount:

Does anyone know if any kernel supports CIFS w/out crashing? I'd like to 
backup some CIFS shares, thanks.


mount -t cifs //w2/x /mnt -o user=user,pass=pass

[  881.388836] CIFS VFS: cifs_mount failed w/return code = -22
Message from syslogd@acerlw at Aug 17 15:14:11 ...
  kernel:[  903.639362] ------------[ cut here ]------------

Message from syslogd@acerlw at Aug 17 15:14:11 ...
  kernel:[  903.639380] invalid opcode: 0000 [#1] SMP

Message from syslogd@acerlw at Aug 17 15:14:11 ...
  kernel:[  903.639540] Stack:

Message from syslogd@acerlw at Aug 17 15:14:11 ...
  kernel:[  903.639573] Call Trace:

Message from syslogd@acerlw at Aug 17 15:14:11 ...
  kernel:[  903.639649] Code: 00 e9 d2 00 00 00 49 8b 07 49 39 c7 75 15 49 8b 47 20 41 c7 47 60 01 00 00 00 4c 39 d0 0f 84 ad 00 00 00 8b 53 18 39 50 20 72 2f <0f> 0b 44 8b 40 24 8b 53 0c ff c6 41 8b 3c 24 89 70 20 41 0f af

Message from syslogd@acerlw at Aug 17 15:14:20 ...
  kernel:[  912.901194] Oops: 0002 [#2] SMP

Message from syslogd@acerlw at Aug 17 15:14:20 ...
  kernel:[  912.901343] Stack:

Message from syslogd@acerlw at Aug 17 15:14:20 ...
  kernel:[  912.901375] Call Trace:

Message from syslogd@acerlw at Aug 17 15:14:20 ...
  kernel:[  912.901516] Code: e8 3b fd ff ff 48 8b 43 58 2a 43 50 88 43 4e eb e9 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 48 89 f8 89 d1 c1 e9 03 83 e2 07 <f3> 48 a5 89 d1 f3 a4 c3 20 48 83 ea 20 4c 8b 06 4c 8b 4e 08 4c

Message from syslogd@acerlw at Aug 17 15:14:20 ...
  kernel:[  912.901590] CR2: 0000000000000200

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  906.705093] Stack:

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  906.705093] Call Trace:

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  906.705093] Code: f0 66 0f c1 17 38 f2 74 06 f3 90 8a 17 eb f6 c3 0f 1f 84 00 00 00 00 00 fa b8 00 01 00 00 f0 66 0f c1 07 38 e0 74 06 f3 90 8a 07 <eb> f6 c3 66 2e 0f 1f 84 00 00 00 00 00 fe 07 56 9d f7 c6 00 02

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  905.287886] Stack:

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  905.287886] Call Trace:

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  905.287886] Code: 94 c2 0f b6 c2 85 c0 0f 95 c0 0f b6 c0 c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 b8 00 01 00 00 f0 66 0f c1 07 38 e0 74 06 <f3> 90 8a 07 eb f6 c3 66 66 2e 0f 1f 84 00 00 00 00 00 9c 58 fa

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  965.289061] Stack:

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  965.289061] Call Trace:

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  965.289061] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 65 48 8b 04 25 08 b6 00 00 83 a0 3c e0 ff ff fb 0f ae f0 48 8b 80 38 e0 ff ff a8 08 75 13 fb f4

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  965.289082] Stack:

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  965.289082] Call Trace:

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  965.289082]  <IRQ>

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  965.289082]  <EOI>

Message from syslogd@acerlw at Aug 17 15:15:12 ...
  kernel:[  965.289082] Code: 00 00 00 48 ff c8 75 fb 48 ff c8 c3 0f 1f 80 00 00 00 00 48 8b 05 71 c3 96 00 ff e0 0f 1f 80 00 00 00 00 48 8d 04 bd 00 00 00 00


Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Wed, 17 Aug 2011, Justin Piszcz wrote:

>
>
> On Wed, 17 Aug 2011, Justin Piszcz wrote:
>
>> 
>> 
>> On Mon, 15 Aug 2011, Justin Piszcz wrote:
>
>
> Better output of crash: (again)
>
> [  214.557063] CIFS VFS: cifs_mount failed w/return code = -22
> [  216.607556] CIFS VFS: cifs_mount failed w/return code = -22
> [  222.637424] CIFS VFS: cifs_mount failed w/return code = -22
> [  228.988734] ------------[ cut here ]------------
> [  228.988752] kernel BUG at mm/slab.c:3111!
> [  228.988759] invalid opcode: 0000 [#1] SMP [  228.988769] CPU 1 [ 
> 228.988774] Modules linked in: netconsole rfcomm bnep bluetooth speedstep_lib 
> cryptd aes_x86_64 aes_generic configfs ath9k mac80211 uvcvideo ath9k_common 
> ath9k_hw ath ohci_hcd ssb videodev mmc_core edac_core v4l2_compat_ioctl32 
> i2c_piix4 edac_mce_amd k10temp battery ac cfg80211 pcmcia shpchp video rfkill 
> pci_hotplug wmi pcmcia_core
> [  228.988873] [  228.988880] Pid: 2869, comm: mount Not tainted 3.1.0-rc2 #2 
> Acer            Aspire 7551                    /Aspire 7551 [  228.988901] 
> RIP: 0010:[<ffffffff81646526>]  [<ffffffff81646526>] 
> cache_alloc_refill+0x111/0x4a6
> [  228.988922] RSP: 0018:ffff8801322e3cf8  EFLAGS: 00010046
> [  228.988929] RAX: ffff8801399b9000 RBX: ffff88013f000080 RCX: 
> 0000000000000007
> [  228.988936] RDX: 0000000000000070 RSI: dead000000200200 RDI: 
> 0000000000000035
> [  228.988944] RBP: ffff8801322e3d58 R08: 0000000000000033 R09: 
> ffff88013f004450
> [  228.988950] R10: ffff88013f004460 R11: ffff8801322e3d60 R12: 
> 00000000000000d0
> [  228.988956] R13: ffff88013f0c1400 R14: 00000000000000d0 R15: 
> ffff88013f004440
> [  228.988964] FS:  00007f421e4957e0(0000) GS:ffff88013fc80000(0000) 
> knlGS:0000000000000000
> [  228.988972] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  228.988979] CR2: 000000000130d000 CR3: 0000000132205000 CR4: 
> 00000000000006e0
> [  228.988986] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
> [  228.988993] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
> 0000000000000400
> [  228.989001] Process mount (pid: 2869, threadinfo ffff8801322e2000, task 
> ffff88013f278620)
> [  228.989005] Stack:
> [  228.989005]  ffff8801322e3d28 000000000000003c 000000000000001c 
> 0000000000000000
> [  228.989005]  000000d00000001c 0000001000000000 ffff8801322e3dc8 
> 0000000000000010
> [  228.989005]  0000000000000202 ffff88013f000080 00000000000000d0 
> ffff880139a94940
> [  228.989005] Call Trace:
> [  228.989005]  [<ffffffff810ae053>] __kmalloc+0xb3/0xe0
> [  228.989005]  [<ffffffff810911e5>] kstrdup+0x35/0x60
> [  228.989005]  [<ffffffff810d1e51>] alloc_vfsmnt+0xa1/0x190
> [  228.989005]  [<ffffffff810d21dd>] vfs_kern_mount+0x2d/0xa0
> [  228.989005]  [<ffffffff810d260f>] do_kern_mount+0x4f/0x100
> [  228.989005]  [<ffffffff810d4002>] do_mount+0x532/0x830
> [  228.989005]  [<ffffffff810d3955>] ? copy_mount_options+0x35/0x170
> [  228.989005]  [<ffffffff810d46c3>] sys_mount+0x93/0xe0
> [  228.989005]  [<ffffffff8164df3b>] system_call_fastpath+0x16/0x1b
> [  228.989005] Code: 00 e9 d2 00 00 00 49 8b 07 49 39 c7 75 15 49 8b 47 20 41 
> c7 47 60 01 00 00 00 4c 39 d0 0f 84 ad 00 00 00 8b 53 18 39 50 20 72 2f <0f> 
> 0b 44 8b 40 24 8b 53 0c ff c6 41 8b 7d 00 89 70 20 41 0f af [  228.989005] 
> RIP  [<ffffffff81646526>] cache_alloc_refill+0x111/0x4a6
> [  228.989005]  RSP <ffff8801322e3cf8>
> [  228.989005] ---[ end trace 8859f1f50ceed0f6 ]---
>
> Justin.
>
>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Jeff Layton]

On Wed, 17 Aug 2011 15:47:23 -0400 (EDT)
Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:

> 
> 
> On Wed, 17 Aug 2011, Justin Piszcz wrote:
> 
> >
> >
> > On Wed, 17 Aug 2011, Justin Piszcz wrote:
> >
> >> 
> >> 
> >> On Mon, 15 Aug 2011, Justin Piszcz wrote:
> >
> >
> > Better output of crash: (again)
> >
> > [  214.557063] CIFS VFS: cifs_mount failed w/return code = -22
> > [  216.607556] CIFS VFS: cifs_mount failed w/return code = -22
> > [  222.637424] CIFS VFS: cifs_mount failed w/return code = -22
> > [  228.988734] ------------[ cut here ]------------
> > [  228.988752] kernel BUG at mm/slab.c:3111!
> > [  228.988759] invalid opcode: 0000 [#1] SMP [  228.988769] CPU 1 [ 
> > 228.988774] Modules linked in: netconsole rfcomm bnep bluetooth speedstep_lib 
> > cryptd aes_x86_64 aes_generic configfs ath9k mac80211 uvcvideo ath9k_common 
> > ath9k_hw ath ohci_hcd ssb videodev mmc_core edac_core v4l2_compat_ioctl32 
> > i2c_piix4 edac_mce_amd k10temp battery ac cfg80211 pcmcia shpchp video rfkill 
> > pci_hotplug wmi pcmcia_core
> > [  228.988873] [  228.988880] Pid: 2869, comm: mount Not tainted 3.1.0-rc2 #2 
> > Acer            Aspire 7551                    /Aspire 7551 [  228.988901] 
> > RIP: 0010:[<ffffffff81646526>]  [<ffffffff81646526>] 
> > cache_alloc_refill+0x111/0x4a6
> > [  228.988922] RSP: 0018:ffff8801322e3cf8  EFLAGS: 00010046
> > [  228.988929] RAX: ffff8801399b9000 RBX: ffff88013f000080 RCX: 
> > 0000000000000007
> > [  228.988936] RDX: 0000000000000070 RSI: dead000000200200 RDI: 
> > 0000000000000035
> > [  228.988944] RBP: ffff8801322e3d58 R08: 0000000000000033 R09: 
> > ffff88013f004450
> > [  228.988950] R10: ffff88013f004460 R11: ffff8801322e3d60 R12: 
> > 00000000000000d0
> > [  228.988956] R13: ffff88013f0c1400 R14: 00000000000000d0 R15: 
> > ffff88013f004440
> > [  228.988964] FS:  00007f421e4957e0(0000) GS:ffff88013fc80000(0000) 
> > knlGS:0000000000000000
> > [  228.988972] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> > [  228.988979] CR2: 000000000130d000 CR3: 0000000132205000 CR4: 
> > 00000000000006e0
> > [  228.988986] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> > 0000000000000000
> > [  228.988993] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
> > 0000000000000400
> > [  228.989001] Process mount (pid: 2869, threadinfo ffff8801322e2000, task 
> > ffff88013f278620)
> > [  228.989005] Stack:
> > [  228.989005]  ffff8801322e3d28 000000000000003c 000000000000001c 
> > 0000000000000000
> > [  228.989005]  000000d00000001c 0000001000000000 ffff8801322e3dc8 
> > 0000000000000010
> > [  228.989005]  0000000000000202 ffff88013f000080 00000000000000d0 
> > ffff880139a94940
> > [  228.989005] Call Trace:
> > [  228.989005]  [<ffffffff810ae053>] __kmalloc+0xb3/0xe0
> > [  228.989005]  [<ffffffff810911e5>] kstrdup+0x35/0x60
> > [  228.989005]  [<ffffffff810d1e51>] alloc_vfsmnt+0xa1/0x190
> > [  228.989005]  [<ffffffff810d21dd>] vfs_kern_mount+0x2d/0xa0
> > [  228.989005]  [<ffffffff810d260f>] do_kern_mount+0x4f/0x100
> > [  228.989005]  [<ffffffff810d4002>] do_mount+0x532/0x830
> > [  228.989005]  [<ffffffff810d3955>] ? copy_mount_options+0x35/0x170
> > [  228.989005]  [<ffffffff810d46c3>] sys_mount+0x93/0xe0
> > [  228.989005]  [<ffffffff8164df3b>] system_call_fastpath+0x16/0x1b
> > [  228.989005] Code: 00 e9 d2 00 00 00 49 8b 07 49 39 c7 75 15 49 8b 47 20 41 
> > c7 47 60 01 00 00 00 4c 39 d0 0f 84 ad 00 00 00 8b 53 18 39 50 20 72 2f <0f> 
> > 0b 44 8b 40 24 8b 53 0c ff c6 41 8b 7d 00 89 70 20 41 0f af [  228.989005] 
> > RIP  [<ffffffff81646526>] cache_alloc_refill+0x111/0x4a6
> > [  228.989005]  RSP <ffff8801322e3cf8>
> > [  228.989005] ---[ end trace 8859f1f50ceed0f6 ]---
> >
> > Justin.
> >

The crash is happening in the bowels of the slab allocator.
Specifically, it looks like it's hitting this:

                /*
                 * The slab was either on partial or free list so
                 * there must be at least one object available for
                 * allocation.
                 */
                BUG_ON(slabp->inuse >= cachep->num);

...which looks like maybe the accounting of in-use objects is off. This
really sounds like some sort of memory corruption. I've not been able
to reproduce this so far, but I also had someone report panic here that
might be related:

    https://bugzilla.redhat.com/show_bug.cgi?id=731278

One thing that might be helpful is turning on page poisoning and
redoing this test, that might make it crash sooner and point out the
source of the corruption.

Even better would be a bisect to track down the cause...

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Wed, 17 Aug 2011, Jeff Layton wrote:

> The crash is happening in the bowels of the slab allocator.
> Specifically, it looks like it's hitting this:
>
>                /*
>                 * The slab was either on partial or free list so
>                 * there must be at least one object available for
>                 * allocation.
>                 */
>                BUG_ON(slabp->inuse >= cachep->num);
>
> ...which looks like maybe the accounting of in-use objects is off. This
> really sounds like some sort of memory corruption. I've not been able
> to reproduce this so far, but I also had someone report panic here that
> might be related:
>
>    https://bugzilla.redhat.com/show_bug.cgi?id=731278
>
> One thing that might be helpful is turning on page poisoning and
> redoing this test, that might make it crash sooner and point out the
> source of the corruption.
>
> Even better would be a bisect to track down the cause...


Hi Jeff,

root@acerlw:/usr/src/linux# grep CONFIG_PAGE_POISONING .config
root@acerlw:/usr/src/linux# ls -l ../linux
lrwxrwxrwx 1 root root 13 Aug 17 14:41 ../linux -> linux-3.1-rc2/
root@acerlw:/usr/src/linux#

In what kernel is that feature available, or, how do I enable it?

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Arnaud Lacombe]

Hi,

On Wed, Aug 17, 2011 at 4:45 PM, Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>
>
> On Wed, 17 Aug 2011, Jeff Layton wrote:
>
>> The crash is happening in the bowels of the slab allocator.
>> Specifically, it looks like it's hitting this:
>>
>>               /*
>>                * The slab was either on partial or free list so
>>                * there must be at least one object available for
>>                * allocation.
>>                */
>>               BUG_ON(slabp->inuse >= cachep->num);
>>
>> ...which looks like maybe the accounting of in-use objects is off. This
>> really sounds like some sort of memory corruption. I've not been able
>> to reproduce this so far, but I also had someone report panic here that
>> might be related:
>>
>>   https://bugzilla.redhat.com/show_bug.cgi?id=731278
>>
>> One thing that might be helpful is turning on page poisoning and
>> redoing this test, that might make it crash sooner and point out the
>> source of the corruption.
>>
>> Even better would be a bisect to track down the cause...
>
>
> Hi Jeff,
>
> root@acerlw:/usr/src/linux# grep CONFIG_PAGE_POISONING .config
> root@acerlw:/usr/src/linux# ls -l ../linux
> lrwxrwxrwx 1 root root 13 Aug 17 14:41 ../linux -> linux-3.1-rc2/
> root@acerlw:/usr/src/linux#
>
> In what kernel is that feature available, or, how do I enable it?
>
It is selected by "Kernel hacking" -> "Debug page memory allocations",
provided your arch support pagealloc debug.

 - Arnaud

> Justin.
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/
>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 13767 bytes --]



On Wed, 17 Aug 2011, Arnaud Lacombe wrote:

> Hi,
>
> On Wed, Aug 17, 2011 at 4:45 PM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>>
>>
>> On Wed, 17 Aug 2011, Jeff Layton wrote:
>>
>>> The crash is happening in the bowels of the slab allocator.
>>> Specifically, it looks like it's hitting this:
>>>
>>>               /*
>>>                * The slab was either on partial or free list so
>>>                * there must be at least one object available for
>>>                * allocation.
>>>                */
>>>               BUG_ON(slabp->inuse >= cachep->num);
>>>
>>> ...which looks like maybe the accounting of in-use objects is off. This
>>> really sounds like some sort of memory corruption. I've not been able
>>> to reproduce this so far, but I also had someone report panic here that
>>> might be related:
>>>
>>>   https://bugzilla.redhat.com/show_bug.cgi?id=731278
>>>
>>> One thing that might be helpful is turning on page poisoning and
>>> redoing this test, that might make it crash sooner and point out the
>>> source of the corruption.
>>>
>>> Even better would be a bisect to track down the cause...
>>
>>
>> Hi Jeff,
>>
>> root@acerlw:/usr/src/linux# grep CONFIG_PAGE_POISONING .config
>> root@acerlw:/usr/src/linux# ls -l ../linux
>> lrwxrwxrwx 1 root root 13 Aug 17 14:41 ../linux -> linux-3.1-rc2/
>> root@acerlw:/usr/src/linux#
>>
>> In what kernel is that feature available, or, how do I enable it?
>>
> It is selected by "Kernel hacking" -> "Debug page memory allocations",
> provided your arch support pagealloc debug.
>
> - Arnaud

Hi,

Thanks, a larger dump below with that option enabled:

[  478.103032] general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC
[  478.103049] CPU 1 
[  478.103052] Modules linked in: bnep rfcomm bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ohci_hcd ssb ath9k mac80211 uvcvideo ath9k_common ath9k_hw ath videodev mmc_core video edac_core k10temp edac_mce_amd v4l2_compat_ioctl32 i2c_piix4 battery cfg80211 ac pcmcia shpchp pci_hotplug wmi pcmcia_core rfkill
[  478.103107] 
[  478.103113] Pid: 3978, comm: echo Not tainted 3.1.0-rc2 #3 Acer            Aspire 7551                    /Aspire 7551 
[  478.103126] RIP: 0010:[<ffffffff8134e839>]  [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[  478.103144] RSP: 0018:ffff88012e0f1e88  EFLAGS: 00010282
[  478.103150] RAX: ffff88013b65d740 RBX: 000000000000002a RCX: ffff88012e0f1f48
[  478.103155] RDX: ffffffff8199c18c RSI: ffff88013b7da490 RDI: 9440ffff88013273
[  478.103161] RBP: ffff88012e0f1e88 R08: 00007fcc4da01700 R09: ffff88013b7da490
[  478.103166] R10: 0000000000000000 R11: 0000000000000246 R12: 9440ffff88013273
[  478.103172] R13: 00007fcc4da0e000 R14: ffff8801388e7bc0 R15: ffff8801388e7bc0
[  478.103179] FS:  00007fcc4da01700(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
[  478.103185] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  478.103190] CR2: 00007fcc4d538380 CR3: 000000013277c000 CR4: 00000000000006e0
[  478.103195] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  478.103201] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  478.103207] Process echo (pid: 3978, threadinfo ffff88012e0f0000, task ffff88012e15a150)
[  478.103212] Stack:
[  478.103215]  ffff88012e0f1ef8 ffffffff8134f17b 0000000000000022 00000007fcc4da0e
[  478.103227]  ffff88012e0f1f18 ffffffff8109f2c7 000000002308a472 ffff88013a9df800
[  478.103237]  0000000000000003 000000000000002a 00007fcc4da0e000 ffff88012e0f1f48
[  478.103246] Call Trace:
[  478.103255]  [<ffffffff8134f17b>] tty_write+0x3b/0x290
[  478.103266]  [<ffffffff8109f2c7>] ? do_mmap_pgoff+0x357/0x370
[  478.103274]  [<ffffffff810b6e6a>] vfs_write+0xaa/0x160
[  478.103280]  [<ffffffff810b7155>] sys_write+0x45/0x90
[  478.103290]  [<ffffffff8164debb>] system_call_fastpath+0x16/0x1b
[  478.103295] Code: 00 00 00 00 00 48 89 df e8 c5 f0 d5 ff 48 8b 5d f0 4c 8b 65 f8 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 85 ff 48 89 e5 74 0c 
[  478.103336]  3f 01 54 00 00 75 2b 31 c0 5d c3 8b 76 44 48 89 d1 48 c7 c7 
[  478.103357] RIP  [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[  478.103366]  RSP <ffff88012e0f1e88>
[  478.103372] ---[ end trace df8e9f10dc5e941d ]---
[  478.103700] general protection fault: 0000 [#2] SMP DEBUG_PAGEALLOC
[  478.103711] CPU 0 
[  478.103715] Modules linked in: bnep rfcomm bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ohci_hcd ssb ath9k mac80211 uvcvideo ath9k_common ath9k_hw ath videodev mmc_core video edac_core k10temp edac_mce_amd v4l2_compat_ioctl32 i2c_piix4 battery cfg80211 ac pcmcia shpchp pci_hotplug wmi pcmcia_core rfkill
[  478.103766] 
[  478.103772] Pid: 3933, comm: atd Tainted: G      D     3.1.0-rc2 #3 Acer            Aspire 7551                    /Aspire 7551 
[  478.103785] RIP: 0010:[<ffffffff8134e839>]  [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[  478.103803] RSP: 0018:ffff880139749e88  EFLAGS: 00010282
[  478.103808] RAX: ffff88013b65d740 RBX: 0000000000000013 RCX: ffff880139749f48
[  478.103814] RDX: ffffffff8199c18c RSI: ffff88013b7da490 RDI: 9440ffff88013273
[  478.103820] RBP: ffff880139749e88 R08: 0000000000000000 R09: ffff88013b7da490
[  478.103825] R10: 0000000000000000 R11: 0000000000000246 R12: 9440ffff88013273
[  478.103831] R13: 00007fff04275cb0 R14: ffff8801388e7bc0 R15: ffff8801388e7bc0
[  478.103838] FS:  00007fd613e52700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[  478.103844] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  478.103849] CR2: 00007fd613a051d5 CR3: 000000013a034000 CR4: 00000000000006f0
[  478.103854] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  478.103859] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  478.103866] Process atd (pid: 3933, threadinfo ffff880139748000, task ffff88013f2c3910)
[  478.103870] Stack:
[  478.103873]  ffff880139749ef8 ffffffff8134f17b 0000000000000f8a 0000000000000000
[  478.103885]  ffffffff810349fd 00007fff04275cec 0000000000000004 0000000000000000
[  478.103894]  0000000000000000 0000000000000013 00007fff04275cb0 ffff880139749f48[  478.103903] Call Trace:
[  478.103913]  [<ffffffff8134f17b>] tty_write+0x3b/0x290
[  478.103924]  [<ffffffff810349fd>] ? do_fork+0x13d/0x210
[  478.103932]  [<ffffffff810b6e6a>] vfs_write+0xaa/0x160
[  478.103938]  [<ffffffff810b7155>] sys_write+0x45/0x90
[  478.103948]  [<ffffffff8164debb>] system_call_fastpath+0x16/0x1b
[  478.103953] Code: 00 00 00 00 00 48 89 df e8 c5 f0 d5 ff 48 8b 5d f0 4c 8b 65 f8 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 85 ff 48 89 e5 74 0c 
[  478.103995]  3f 01 54 00 00 75 2b 31 c0 5d c3 8b 76 44 48 89 d1 48 c7 c7 
[  478.104016] RIP  [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[  478.104025]  RSP <ffff880139749e88>
[  478.104072] ---[ end trace df8e9f10dc5e941e ]---
[  478.104333] general protection fault: 0000 [#3] SMP DEBUG_PAGEALLOC
[  478.104352] CPU 0 
[  478.104357] Modules linked in: bnep rfcomm bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ohci_hcd ssb ath9k mac80211 uvcvideo ath9k_common ath9k_hw ath videodev mmc_core video edac_core k10temp edac_mce_amd v4l2_compat_ioctl32 i2c_piix4 battery cfg80211 ac pcmcia shpchp pci_hotplug wmi pcmcia_core rfkill
[  478.104405] 
[  478.104410] Pid: 3933, comm: atd Tainted: G      D     3.1.0-rc2 #3 Acer            Aspire 7551                    /Aspire 7551 
[  478.104422] RIP: 0010:[<ffffffff8134e839>]  [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[  478.104434] RSP: 0018:ffff880139749b28  EFLAGS: 00010282
[  478.104439] RAX: ffff88013f344382 RBX: 9440ffff88013273 RCX: 0000000000000000
[  478.104444] RDX: ffffffff8199c20d RSI: ffff88013b7da490 RDI: 9440ffff88013273
[  478.104450] RBP: ffff880139749b28 R08: 0000000000000000 R09: 0000000000000000
[  478.104455] R10: ffff8801388e7bd0 R11: 0000000000000001 R12: 0000000000000008
[  478.104460] R13: ffff8801388e7bc0 R14: ffff88013b65d740 R15: ffff88013b7da490
[  478.104467] FS:  00007fd613e52700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[  478.104473] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  478.104478] CR2: 00007fd613a051d5 CR3: 0000000001c1d000 CR4: 00000000000006f0
[  478.104483] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  478.104488] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  478.104494] Process atd (pid: 3933, threadinfo ffff880139748000, task ffff88013f2c3910)
[  478.104498] Stack:
[  478.104501]  ffff880139749bd8 ffffffff8134fe31 ffff880139749b48 ffffffff810d259a
[  478.104512]  ffff880139749b98 ffffffff810b85df ffff88012e08b288 ffff88013a154390
[  478.104520]  0000000000000000 ffff8801388e7bd0 ffff88013b7da490 0000000800000001
[  478.104529] Call Trace:
[  478.104538]  [<ffffffff8134fe31>] tty_release+0x41/0x550
[  478.104546]  [<ffffffff810d259a>] ? mntput+0x1a/0x30
[  478.104554]  [<ffffffff810b85df>] ? fput+0x15f/0x200
[  478.104561]  [<ffffffff810b8552>] fput+0xd2/0x200
[  478.104570]  [<ffffffff810b50c1>] filp_close+0x61/0x90
[  478.104578]  [<ffffffff810384bf>] put_files_struct+0x7f/0xe0
[  478.104585]  [<ffffffff810385c4>] exit_files+0x44/0x50
[  478.104591]  [<ffffffff81038bc4>] do_exit+0x5f4/0x790
[  478.104600]  [<ffffffff81036e94>] ? kmsg_dump+0x44/0xe0
[  478.104609]  [<ffffffff81004f25>] oops_end+0x75/0xa0
[  478.104615]  [<ffffffff81005093>] die+0x53/0x80
[  478.104623]  [<ffffffff81002854>] do_general_protection+0x154/0x160
[  478.104631]  [<ffffffff8164da7f>] general_protection+0x1f/0x30
[  478.104641]  [<ffffffff8134e839>] ? tty_paranoia_check+0x9/0x70
[  478.104649]  [<ffffffff8134f17b>] tty_write+0x3b/0x290
[  478.104657]  [<ffffffff810349fd>] ? do_fork+0x13d/0x210
[  478.104664]  [<ffffffff810b6e6a>] vfs_write+0xaa/0x160
[  478.104670]  [<ffffffff810b7155>] sys_write+0x45/0x90
[  478.104679]  [<ffffffff8164debb>] system_call_fastpath+0x16/0x1b
[  478.104683] Code: 00 00 00 00 00 48 89 df e8 c5 f0 d5 ff 48 8b 5d f0 4c 8b 65 f8 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 85 ff 48 89 e5 74 0c 
[  478.104724]  3f 01 54 00 00 75 2b 31 c0 5d c3 8b 76 44 48 89 d1 48 c7 c7 
[  478.104744] RIP  [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[  478.104753]  RSP <ffff880139749b28>
[  478.104757] ---[ end trace df8e9f10dc5e941f ]---
[  478.104761] Fixing recursive fault but reboot is needed!
[  478.152105] general protection fault: 0000 [#4] SMP DEBUG_PAGEALLOC
[  478.152117] CPU 1 
[  478.152120] Modules linked in: bnep rfcomm bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ohci_hcd ssb ath9k mac80211 uvcvideo ath9k_common ath9k_hw ath videodev mmc_core video edac_core k10temp edac_mce_amd v4l2_compat_ioctl32 i2c_piix4 battery cfg80211 ac pcmcia shpchp pci_hotplug wmi pcmcia_core rfkill
[  478.152171] [  478.152177] Pid: 3936, comm: danted Tainted: G      D     3.1.0-rc2 #3 Acer            Aspire 7551                    /Aspire 7551 
[  478.152190] RIP: 0010:[<ffffffff8134e839>]  [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[  478.152208] RSP: 0018:ffff880138abdd18  EFLAGS: 00010286
[  478.152213] RAX: ffff88013f344300 RBX: 88012e1400003000 RCX: 0000000000000000
[  478.152219] RDX: ffffffff8199c20d RSI: ffff88013b5732f0 RDI: 88012e1400003000
[  478.152224] RBP: ffff880138abdd18 R08: 0000000000000000 R09: 0000000000000000
[  478.152229] R10: ffff8801388e7150 R11: 0000000000000001 R12: 0000000000000008
[  478.152235] R13: ffff8801388e7140 R14: ffff88012fa19d80 R15: ffff88013b5732f0
[  478.152241] FS:  00007fe3e8099700(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
[  478.152247] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  478.152253] CR2: 00007fe3e7bad520 CR3: 0000000001c1d000 CR4: 00000000000006e0
[  478.152258] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  478.152263] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  478.152269] Process danted (pid: 3936, threadinfo ffff880138abc000, task ffff88012e0c1810)
[  478.152274] Stack:
[  478.152277]  ffff880138abddc8 ffffffff8134fe31 ffff880138abdd38 ffffffff810d259a
[  478.152288]  ffff880138abdd88 ffffffff810b85df ffff8801326ea3d8 ffff8801325a5250
[  478.152297]  0000000000000000 ffff8801388e7150 ffff88013b5732f0 0000000800000001
[  478.152307] Call Trace:
[  478.152317]  [<ffffffff8134fe31>] tty_release+0x41/0x550
[  478.152326]  [<ffffffff810d259a>] ? mntput+0x1a/0x30
[  478.152334]  [<ffffffff810b85df>] ? fput+0x15f/0x200
[  478.152341]  [<ffffffff810b8552>] fput+0xd2/0x200
[  478.152350]  [<ffffffff810b50c1>] filp_close+0x61/0x90
[  478.152358]  [<ffffffff810384bf>] put_files_struct+0x7f/0xe0
[  478.152365]  [<ffffffff810385c4>] exit_files+0x44/0x50
[  478.152372]  [<ffffffff81038bc4>] do_exit+0x5f4/0x790
[  478.152380]  [<ffffffff810baf46>] ? vfs_stat+0x16/0x20
[  478.152387]  [<ffffffff810bb285>] ? sys_newstat+0x15/0x30
[  478.152394]  [<ffffffff810b7040>] ? vfs_read+0x120/0x160
[  478.152402]  [<ffffffff8103908f>] do_group_exit+0x3f/0xa0
[  478.152409]  [<ffffffff81039102>] sys_exit_group+0x12/0x20
[  478.152418]  [<ffffffff8164debb>] system_call_fastpath+0x16/0x1b
[  478.152423] Code: 00 00 00 00 00 48 89 df e8 c5 f0 d5 ff 48 8b 5d f0 4c 8b 65 f8 c9 c3 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 85 ff 48 89 e5 74 0c 
[  478.152464]  3f 01 54 00 00 75 2b 31 c0 5d c3 8b 76 44 48 89 d1 48 c7 c7 
[  478.152485] RIP  [<ffffffff8134e839>] tty_paranoia_check+0x9/0x70
[  478.152495]  RSP <ffff880138abdd18>
[  478.152501] ---[ end trace df8e9f10dc5e9420 ]---
[  478.152505] Fixing recursive fault but reboot is needed!

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 4775 bytes --]



On Wed, 17 Aug 2011, Justin Piszcz wrote:

>
>
> On Wed, 17 Aug 2011, Arnaud Lacombe wrote:
>
>> Hi,
>> 
>> On Wed, Aug 17, 2011 at 4:45 PM, Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> 
>> wrote:
>>> 
>>> 
>>> On Wed, 17 Aug 2011, Jeff Layton wrote:
>>> 
>>>> The crash is happening in the bowels of the slab allocator.
>>>> Specifically, it looks like it's hitting this:
>>>> 
>>>>               /*
>>>>                * The slab was either on partial or free list so
>>>>                * there must be at least one object available for
>>>>                * allocation.
>>>>                */
>>>>               BUG_ON(slabp->inuse >= cachep->num);
>>>> 
>>>> ...which looks like maybe the accounting of in-use objects is off. This
>>>> really sounds like some sort of memory corruption. I've not been able
>>>> to reproduce this so far, but I also had someone report panic here that
>>>> might be related:
>>>> 
>>>>   https://bugzilla.redhat.com/show_bug.cgi?id=731278

Hi,

Got a better one here:

[   98.386992] CIFS VFS: cifs_mount failed w/return code = -22
[  562.565161] CIFS VFS: cifs_mount failed w/return code = -22
[  596.277441] ------------[ cut here ]------------
[  596.277450] kernel BUG at mm/slab.c:3111!
[  596.277456] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
[  596.277463] CPU 2 
[  596.277466] Modules linked in: rfcomm bnep bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ath9k mac80211 ath9k_common ath9k_hw ohci_hcd ssb ath mmc_core cfg80211 shpchp uvcvideo i2c_piix4 videodev v4l2_compat_ioctl32 pci_hotplug wmi pcmcia rfkill pcmcia_core edac_core k10temp edac_mce_amd video battery ac
[  596.277517] 
[  596.277523] Pid: 4157, comm: ps Not tainted 3.1.0-rc2 #3 Acer            Aspire 7551                    /Aspire 7551 
[  596.277536] RIP: 0010:[<ffffffff816464a6>]  [<ffffffff816464a6>] cache_alloc_refill+0x111/0x4a6
[  596.277554] RSP: 0018:ffff88012e231b88  EFLAGS: 00010046
[  596.277559] RAX: ffff8801394d5000 RBX: ffff88013f000080 RCX: 0000000000000033
[  596.277565] RDX: 0000000000000070 RSI: dead000000200200 RDI: 0000000000000009
[  596.277570] RBP: ffff88012e231be8 R08: 000000000000005f R09: ffff88013f004450
[  596.277576] R10: ffff88013f004460 R11: ffff88012e231d80 R12: 00000000000000d0
[  596.277581] R13: ffff88013f0d1400 R14: 00000000000000d0 R15: ffff88013f004440
[  596.277588] FS:  00007f8bf016c700(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000
[  596.277594] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  596.277599] CR2: 00007f8befd44328 CR3: 000000012e27b000 CR4: 00000000000006e0
[  596.277605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  596.277610] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  596.277616] Process ps (pid: 4157, threadinfo ffff88012e230000, task ffff88013f3f78d0)
[  596.277621] Stack:
[  596.277624]  ffff88013f045c00 ffff88010000003c ffff88012e231bb8 ffff88012f491088
[  596.277635]  000000d02e231bc8 0000001000000000 ffff88012f491118 ffff880132266a40
[  596.277645]  00000000000000d0 0000000000000202 ffff88013f000080 ffff880132266a40
[  596.277654] Call Trace:
[  596.277666]  [<ffffffff810ae0e6>] kmem_cache_alloc+0x76/0xa0
[  596.277675]  [<ffffffff8110bb80>] ? meminfo_proc_open+0x30/0x30
[  596.277684]  [<ffffffff810d58e2>] single_open+0x32/0xa0
[  596.277694]  [<ffffffff8110a095>] ? proc_lookup_de+0xa5/0x100
[  596.277701]  [<ffffffff8110bb65>] meminfo_proc_open+0x15/0x30
[  596.277709]  [<ffffffff811044e8>] proc_reg_open+0x88/0x150
[  596.277717]  [<ffffffff810d4c50>] ? seq_release_private+0x50/0x50
[  596.277726]  [<ffffffff81104460>] ? proc_alloc_inode+0xa0/0xa0
[  596.277735]  [<ffffffff810b5339>] __dentry_open.isra.17+0xf9/0x2d0
[  596.277744]  [<ffffffff810b625e>] nameidata_to_filp+0x4e/0x60
[  596.277753]  [<ffffffff810c4804>] do_last.isra.48+0x204/0x830
[  596.277760]  [<ffffffff810c50a6>] path_openat+0xc6/0x370
[  596.277769]  [<ffffffff8109a965>] ? handle_mm_fault+0x165/0x300
[  596.277776]  [<ffffffff810c53ad>] do_filp_open+0x3d/0xa0
[  596.277786]  [<ffffffff810d0697>] ? alloc_fd+0x47/0x130
[  596.277795]  [<ffffffff810b6362>] do_sys_open+0xf2/0x1d0
[  596.277803]  [<ffffffff810b645b>] sys_open+0x1b/0x20
[  596.277812]  [<ffffffff8164debb>] system_call_fastpath+0x16/0x1b
[  596.277817] Code: 00 e9 d2 00 00 00 49 8b 07 49 39 c7 75 15 49 8b 47 20 41 c7 47 60 01 00 00 00 4c 39 d0 0f 84 ad 00 00 00 8b 53 18 39 50 20 72 2f <0f> 0b 44 8b 40 24 8b 53 0c ff c6 41 8b 7d 00 89 70 20 41 0f af 
[  596.277879] RIP  [<ffffffff816464a6>] cache_alloc_refill+0x111/0x4a6
[  596.277888]  RSP <ffff88012e231b88>
[  596.277894] ---[ end trace 01e175dd97a8992b ]---


Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 17618 bytes --]



On Wed, 17 Aug 2011, Justin Piszcz wrote:

>
>
> On Wed, 17 Aug 2011, Justin Piszcz wrote:
>
>> 
>> 
>> On Wed, 17 Aug 2011, Arnaud Lacombe wrote:
>> 
>>> Hi,
>>> 
>>> On Wed, Aug 17, 2011 at 4:45 PM, Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> 
>>> wrote:
>>>> 
>>>> 
>>>> On Wed, 17 Aug 2011, Jeff Layton wrote:
>>>> 
>>>>> The crash is happening in the bowels of the slab allocator.
>>>>> Specifically, it looks like it's hitting this:
>>>>> 
>>>>>               /*
>>>>>                * The slab was either on partial or free list so
>>>>>                * there must be at least one object available for
>>>>>                * allocation.
>>>>>                */
>>>>>               BUG_ON(slabp->inuse >= cachep->num);
>>>>> 
>>>>> ...which looks like maybe the accounting of in-use objects is off. This
>>>>> really sounds like some sort of memory corruption. I've not been able
>>>>> to reproduce this so far, but I also had someone report panic here that
>>>>> might be related:
>>>>> 
>>>>>   https://bugzilla.redhat.com/show_bug.cgi?id=731278
>
> Hi,
>
> Got a better one here:
>
> [   98.386992] CIFS VFS: cifs_mount failed w/return code = -22
> [  562.565161] CIFS VFS: cifs_mount failed w/return code = -22
> [  596.277441] ------------[ cut here ]------------
> [  596.277450] kernel BUG at mm/slab.c:3111!
> [  596.277456] invalid opcode: 0000 [#1] SMP DEBUG_PAGEALLOC
> [  596.277463] CPU 2 [  596.277466] Modules linked in: rfcomm bnep bluetooth 
> speedstep_lib cryptd aes_x86_64 aes_generic configfs ath9k mac80211 
> ath9k_common ath9k_hw ohci_hcd ssb ath mmc_core cfg80211 shpchp uvcvideo 
> i2c_piix4 videodev v4l2_compat_ioctl32 pci_hotplug wmi pcmcia rfkill 
> pcmcia_core edac_core k10temp edac_mce_amd video battery ac
> [  596.277517] [  596.277523] Pid: 4157, comm: ps Not tainted 3.1.0-rc2 #3 
> Acer            Aspire 7551                    /Aspire 7551 [  596.277536] 
> RIP: 0010:[<ffffffff816464a6>]  [<ffffffff816464a6>] 
> cache_alloc_refill+0x111/0x4a6
> [  596.277554] RSP: 0018:ffff88012e231b88  EFLAGS: 00010046
> [  596.277559] RAX: ffff8801394d5000 RBX: ffff88013f000080 RCX: 
> 0000000000000033
> [  596.277565] RDX: 0000000000000070 RSI: dead000000200200 RDI: 
> 0000000000000009
> [  596.277570] RBP: ffff88012e231be8 R08: 000000000000005f R09: 
> ffff88013f004450
> [  596.277576] R10: ffff88013f004460 R11: ffff88012e231d80 R12: 
> 00000000000000d0
> [  596.277581] R13: ffff88013f0d1400 R14: 00000000000000d0 R15: 
> ffff88013f004440
> [  596.277588] FS:  00007f8bf016c700(0000) GS:ffff88013fd00000(0000) 
> knlGS:0000000000000000
> [  596.277594] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  596.277599] CR2: 00007f8befd44328 CR3: 000000012e27b000 CR4: 
> 00000000000006e0
> [  596.277605] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 
> 0000000000000000
> [  596.277610] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 
> 0000000000000400
> [  596.277616] Process ps (pid: 4157, threadinfo ffff88012e230000, task 
> ffff88013f3f78d0)
> [  596.277621] Stack:
> [  596.277624]  ffff88013f045c00 ffff88010000003c ffff88012e231bb8 
> ffff88012f491088
> [  596.277635]  000000d02e231bc8 0000001000000000 ffff88012f491118 
> ffff880132266a40
> [  596.277645]  00000000000000d0 0000000000000202 ffff88013f000080 
> ffff880132266a40
> [  596.277654] Call Trace:
> [  596.277666]  [<ffffffff810ae0e6>] kmem_cache_alloc+0x76/0xa0
> [  596.277675]  [<ffffffff8110bb80>] ? meminfo_proc_open+0x30/0x30
> [  596.277684]  [<ffffffff810d58e2>] single_open+0x32/0xa0
> [  596.277694]  [<ffffffff8110a095>] ? proc_lookup_de+0xa5/0x100
> [  596.277701]  [<ffffffff8110bb65>] meminfo_proc_open+0x15/0x30
> [  596.277709]  [<ffffffff811044e8>] proc_reg_open+0x88/0x150
> [  596.277717]  [<ffffffff810d4c50>] ? seq_release_private+0x50/0x50
> [  596.277726]  [<ffffffff81104460>] ? proc_alloc_inode+0xa0/0xa0
> [  596.277735]  [<ffffffff810b5339>] __dentry_open.isra.17+0xf9/0x2d0
> [  596.277744]  [<ffffffff810b625e>] nameidata_to_filp+0x4e/0x60
> [  596.277753]  [<ffffffff810c4804>] do_last.isra.48+0x204/0x830
> [  596.277760]  [<ffffffff810c50a6>] path_openat+0xc6/0x370
> [  596.277769]  [<ffffffff8109a965>] ? handle_mm_fault+0x165/0x300
> [  596.277776]  [<ffffffff810c53ad>] do_filp_open+0x3d/0xa0
> [  596.277786]  [<ffffffff810d0697>] ? alloc_fd+0x47/0x130
> [  596.277795]  [<ffffffff810b6362>] do_sys_open+0xf2/0x1d0
> [  596.277803]  [<ffffffff810b645b>] sys_open+0x1b/0x20
> [  596.277812]  [<ffffffff8164debb>] system_call_fastpath+0x16/0x1b
> [  596.277817] Code: 00 e9 d2 00 00 00 49 8b 07 49 39 c7 75 15 49 8b 47 20 41 
> c7 47 60 01 00 00 00 4c 39 d0 0f 84 ad 00 00 00 8b 53 18 39 50 20 72 2f <0f> 
> 0b 44 8b 40 24 8b 53 0c ff c6 41 8b 7d 00 89 70 20 41 0f af [  596.277879] 
> RIP  [<ffffffff816464a6>] cache_alloc_refill+0x111/0x4a6
> [  596.277888]  RSP <ffff88012e231b88>
> [  596.277894] ---[ end trace 01e175dd97a8992b ]---

(it is spewing new errors below)

[  598.897157] 
[  598.897157] Pid: 1097, comm: kworker/2:2 Tainted: G      D W   3.1.0-rc2 #3 Acer            Aspire 7551                    /Aspire 7551 
[  598.897157] RIP: 0010:[<ffffffff8164d491>]  [<ffffffff8164d491>] _raw_spin_lock_irq+0x11/0x20
[  598.897157] RSP: 0018:ffff88013947dd90  EFLAGS: 00000097
[  598.897157] RAX: 0000000000003332 RBX: ffff88013f0d1400 RCX: 0000000000000000
[  598.897157] RDX: ffff88013f0d1400 RSI: ffff88013f004440 RDI: ffff88013f004480
[  598.897157] RBP: ffff88013947dd90 R08: 0000000000000000 R09: ffff88013aa0b440
[  598.897157] R10: ffff88013aa0b440 R11: ffff880139dec7c0 R12: ffff88013f004440
[  598.897157] R13: ffff88013f000080 R14: 0000000000000000 R15: ffffffff810adc80
[  598.897157] FS:  00007f5e69cc4700(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000
[  598.897157] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  598.897157] CR2: 00007f5e693b02c0 CR3: 0000000001c1d000 CR4: 00000000000006e0
[  598.897157] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  598.897157] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  598.897157] Process kworker/2:2 (pid: 1097, threadinfo ffff88013947c000, task ffff88013f25e250)
[  598.897157] Stack:
[  598.897157]  ffff88013947dde0 ffffffff810adc09 ffff880100000000 ffffffff00000000
[  598.897157]  ffff88013947dde0 ffff88013f000080 ffff88013f004440 ffff88013fd0d720
[  598.897157]  0000000000000000 ffffffff810adc80 ffff88013947de10 ffffffff810add08
[  598.897157] Call Trace:
[  598.897157]  [<ffffffff810adc09>] drain_array+0x69/0xe0
[  598.897157]  [<ffffffff810adc80>] ? drain_array+0xe0/0xe0
[  598.897157]  [<ffffffff810add08>] cache_reap+0x88/0x120
[  598.897157]  [<ffffffff8104ae61>] process_one_work+0x101/0x380
[  598.897157]  [<ffffffff8104b68d>] worker_thread+0x15d/0x330
[  598.897157]  [<ffffffff8104b530>] ? manage_workers.isra.32+0x210/0x210
[  598.897157]  [<ffffffff8104f6a7>] kthread+0x87/0x90
[  598.897157]  [<ffffffff8164f074>] kernel_thread_helper+0x4/0x10
[  598.897157]  [<ffffffff8104f620>] ? kthread_worker_fn+0x140/0x140
[  598.897157]  [<ffffffff8164f070>] ? gs_change+0xb/0xb
[  598.897157] Code: fa ba 00 01 00 00 f0 66 0f c1 17 38 f2 74 06 f3 90 8a 17 eb f6 5d c3 0f 1f 00 55 48 89 e5 fa b8 00 01 00 00 f0 66 0f c1 07 38 e0 
[  598.897157]  06 f3 90 8a 07 eb f6 5d c3 0f 1f 44 00 00 55 48 89 e5 fe 07 
[  598.897157] Call Trace:
[  598.897157]  [<ffffffff810adc09>] drain_array+0x69/0xe0
[  598.897157]  [<ffffffff810adc80>] ? drain_array+0xe0/0xe0
[  598.897157]  [<ffffffff810add08>] cache_reap+0x88/0x120
[  598.897157]  [<ffffffff8104ae61>] process_one_work+0x101/0x380
[  598.897157]  [<ffffffff8104b68d>] worker_thread+0x15d/0x330
[  598.897157]  [<ffffffff8104b530>] ? manage_workers.isra.32+0x210/0x210
[  598.897157]  [<ffffffff8104f6a7>] kthread+0x87/0x90
[  598.897157]  [<ffffffff8164f074>] kernel_thread_helper+0x4/0x10
[  598.897157]  [<ffffffff8104f620>] ? kthread_worker_fn+0x140/0x140
[  598.897157]  [<ffffffff8164f070>] ? gs_change+0xb/0xb
[  726.242532] NMI backtrace for cpu 1
[  726.242541] CPU 1 
[  726.242545] Modules linked in: rfcomm bnep bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ath9k mac80211 ath9k_common ath9k_hw ohci_hcd ssb ath mmc_core cfg80211 shpchp uvcvideo i2c_piix4 videodev v4l2_compat_ioctl32 pci_hotplug wmi pcmcia rfkill pcmcia_core edac_core k10temp edac_mce_amd video battery ac
[  726.242601] 
[  726.242608] Pid: 0, comm: kworker/0:0 Tainted: G      D W   3.1.0-rc2 #3 Acer            Aspire 7551                    /Aspire 7551 
[  726.242622] RIP: 0010:[<ffffffff810090b4>]  [<ffffffff810090b4>] default_idle+0x24/0x40
[  726.242639] RSP: 0018:ffff88013f0e3ed8  EFLAGS: 00000246
[  726.242644] RAX: 0000000000000000 RBX: ffff88013f0e3ef4 RCX: 0000000000000001
[  726.242650] RDX: 0000000000000909 RSI: 0000000000000086 RDI: ffffffff81d58aac
[  726.242656] RBP: ffff88013f0e3ed8 R08: 0000000000000000 R09: 0000000000000000
[  726.242661] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81c75e70
[  726.242666] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  726.242673] FS:  00007f30e9aa3700(0000) GS:ffff88013fc80000(0000) knlGS:0000000000000000
[  726.242679] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  726.242684] CR2: 00007f5e692fce02 CR3: 0000000139f11000 CR4: 00000000000006e0
[  726.242689] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  726.242694] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  726.242700] Process kworker/0:0 (pid: 0, threadinfo ffff88013f0e2000, task ffff88013f0dd240)
[  726.242705] Stack:[  726.242708]  ffff88013f0e3f08 ffffffff810091c4 ffff88013f0e3ef8 0000000181054bd5
[  726.242719]  ffff88013f0e3fd8 ffff88013f0e3fd8 ffff88013f0e3f28 ffffffff81000818
[  726.242729]  0000000000000001 0000000000000000 ffff88013f0e3f48 ffffffff81cc6199
[  726.242738] Call Trace:
[  726.242748]  [<ffffffff810091c4>] amd_e400_idle+0x54/0x100
[  726.242756]  [<ffffffff81000818>] cpu_idle+0x78/0xc0
[  726.242765]  [<ffffffff81cc6199>] start_secondary+0x19a/0x19e
[  726.242771] Code: 1f 84 00 00 00 00 00 55 65 48 8b 04 25 08 b6 00 00 83 a0 3c e0 ff ff fb 48 89 e5 0f ae f0 48 8b 80 38 e0 ff ff a8 08 75 14 fb f4 
[  726.242812]  48 8b 04 25 08 b6 00 00 83 88 3c e0 ff ff 04 5d c3 fb eb eb 
[  726.242834] Call Trace:
[  726.242841]  [<ffffffff810091c4>] amd_e400_idle+0x54/0x100
[  726.242848]  [<ffffffff81000818>] cpu_idle+0x78/0xc0
[  726.242855]  [<ffffffff81cc6199>] start_secondary+0x19a/0x19e
[  726.242866] NMI backtrace for cpu 3
[  726.242875] CPU 3 
[  726.242879] Modules linked in: rfcomm bnep bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ath9k mac80211 ath9k_common ath9k_hw ohci_hcd ssb ath mmc_core cfg80211 shpchp uvcvideo i2c_piix4 videodev v4l2_compat_ioctl32 pci_hotplug wmi pcmcia rfkill pcmcia_core edac_core k10temp edac_mce_amd video battery ac
[  726.242936] 
[  726.242942] Pid: 0, comm: kworker/0:1 Tainted: G      D W   3.1.0-rc2 #3 Acer            Aspire 7551                    /Aspire 7551 
[  726.242956] RIP: 0010:[<ffffffff810090b4>]  [<ffffffff810090b4>] default_idle+0x24/0x40
[  726.242974] RSP: 0018:ffff88013f129ed8  EFLAGS: 00000246
[  726.242979] RAX: 0000000000000000 RBX: ffff88013f129ef4 RCX: 0000000000000020
[  726.242985] RDX: 0000000000000000 RSI: 0000000000000086 RDI: ffffffff81d58aac
[  726.242990] RBP: ffff88013f129ed8 R08: ffffffff81c2a3c0 R09: 0000000000000000
[  726.242996] R10: 0000000000000000 R11: 0000000000000000 R12: ffffffff81c75e70
[  726.243001] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[  726.243007] FS:  00007f0e9311c700(0000) GS:ffff88013fd80000(0000) knlGS:0000000000000000
[  726.243013] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  726.243018] CR2: ffffffffff600400 CR3: 0000000139d87000 CR4: 00000000000006e0
[  726.243024] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  726.243029] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[  726.243035] Process kworker/0:1 (pid: 0, threadinfo ffff88013f128000, task ffff88013f1252c0)
[  726.243040] Stack:
[  726.243043]  ffff88013f129f08 ffffffff810091c4 ffff88013f129ef8 0000000381054bd5
[  726.243054]  ffff88013f129fd8 ffff88013f129fd8 ffff88013f129f28 ffffffff81000818
[  726.243061]  0000000000000003 0000000000000000 ffff88013f129f48 ffffffff81cc6199
[  726.243061] Call Trace:
[  726.243061]  [<ffffffff810091c4>] amd_e400_idle+0x54/0x100
[  726.243061]  [<ffffffff81000818>] cpu_idle+0x78/0xc0
[  726.243061]  [<ffffffff81cc6199>] start_secondary+0x19a/0x19e
[  726.243061] Code: 1f 84 00 00 00 00 00 55 65 48 8b 04 25 08 b6 00 00 83 a0 3c e0 ff ff fb 48 89 e5 0f ae f0 48 8b 80 38 e0 ff ff a8 08 75 14 fb f4 
[  726.243061]  48 8b 04 25 08 b6 00 00 83 88 3c e0 ff ff 04 5d c3 fb eb eb 
[  726.243061] Call Trace:
[  726.243061]  [<ffffffff810091c4>] amd_e400_idle+0x54/0x100
[  726.243061]  [<ffffffff81000818>] cpu_idle+0x78/0xc0
[  726.243061]  [<ffffffff81cc6199>] start_secondary+0x19a/0x19e
[  726.243089] NMI backtrace for cpu 0
[  726.243089] CPU 0 
[  726.243089] Modules linked in: rfcomm bnep bluetooth speedstep_lib cryptd aes_x86_64 aes_generic configfs ath9k mac80211 ath9k_common ath9k_hw ohci_hcd ssb ath mmc_core cfg80211 shpchp uvcvideo i2c_piix4 videodev v4l2_compat_ioctl32 pci_hotplug wmi pcmcia rfkill pcmcia_core edac_core k10temp edac_mce_amd video battery ac
[  726.243089] 
[  726.243089] Pid: 0, comm: swapper Tainted: G      D W   3.1.0-rc2 #3 Acer            Aspire 7551                    /Aspire 7551 
[  726.243089] RIP: 0010:[<ffffffff812d4e30>]  [<ffffffff812d4e30>] __delay+0x10/0x10
[  726.243089] RSP: 0018:ffff88013fc03e00  EFLAGS: 00000006
[  726.243089] RAX: 0000000000000c00 RBX: 0000000000002710 RCX: 0000000000000006
[  726.243089] RDX: ffffffff81c2add8 RSI: 0000000000000002 RDI: 0000000000418958
[  726.243089] RBP: ffff88013fc03e18 R08: 000000000000000a R09: 0000000000000000
[  726.243089] R10: 0000000000000000 R11: 000000000000000d R12: ffffffff81c2f080
[  726.243089] R13: ffffffff81c2f080 R14: ffffffff81c2f140 R15: 0000000000000004
[  726.243089] FS:  00007f30e92a2700(0000) GS:ffff88013fc00000(0000) knlGS:0000000000000000
[  726.243089] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[  726.243089] CR2: 00007fcdc5b391a0 CR3: 0000000139f11000 CR4: 00000000000006f0
[  726.243089] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[  726.243089] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400[  726.243089] Process swapper (pid: 0, threadinfo ffffffff81c00000, task ffffffff81c25020)
[  726.243089] Stack:
[  726.243089]  ffffffff8101a342 000000000000000a ffff88013fc0cd40 ffff88013fc03e68
[  726.243089]  ffffffff810720eb 0000000000010740 ffffffff81c2f140 00000000ffffffff
[  726.243089]  0000000000000000 0000000000000000 0000000000000000 7fffffffffffffff
[  726.243089] Call Trace:
[  726.243089]  <IRQ> 
[  726.243089]  [<ffffffff8101a342>] ? arch_trigger_all_cpu_backtrace+0x62/0x80
[  726.243089]  [<ffffffff810720eb>] __rcu_pending+0x35b/0x380
[  726.243089]  [<ffffffff81072e53>] rcu_check_callbacks+0x103/0x120
[  726.243089]  [<ffffffff81041a83>] update_process_times+0x43/0x80
[  726.243089]  [<ffffffff8105ebbf>] tick_sched_timer+0x5f/0xb0
[  726.243089]  [<ffffffff8105326d>] __run_hrtimer.isra.34+0x4d/0x100
[  726.243089]  [<ffffffff81053a6f>] hrtimer_interrupt+0xdf/0x1f0
[  726.243089]  [<ffffffff810197c4>] smp_apic_timer_interrupt+0x64/0xa0
[  726.243089]  [<ffffffff8164e8cb>] apic_timer_interrupt+0x6b/0x70
[  726.243089]  <EOI> 
[  726.243089]  [<ffffffff810090b4>] ? default_idle+0x24/0x40
[  726.243089]  [<ffffffff810091c4>] amd_e400_idle+0x54/0x100
[  726.243089]  [<ffffffff81000818>] cpu_idle+0x78/0xc0
[  726.243089]  [<ffffffff8162cd99>] rest_init+0x6d/0x74
[  726.243089]  [<ffffffff81c91a76>] start_kernel+0x2ae/0x2b9
[  726.243089]  [<ffffffff81c912ee>] x86_64_start_reservations+0xfe/0x102
[  726.243089]  [<ffffffff81c913e2>] x86_64_start_kernel+0xf0/0xf7
[  726.243089] Code: 66 66 2e 0f 1f 84 00 00 00 00 00 48 ff c8 75 fb 48 ff c8 5d c3 66 0f 1f 44 00 00 55 48 89 e5 ff 15 0e 8d 96 00 5d c3 0f 1f 40 00 
[  726.243089]  48 8d 04 bd 00 00 00 00 65 48 8b 14 25 d8 06 01 00 48 69 d2 
[  726.243089] Call Trace:
[  726.243089]  <IRQ>  [<ffffffff8101a342>] ? arch_trigger_all_cpu_backtrace+0x62/0x80
[  726.243089]  [<ffffffff810720eb>] __rcu_pending+0x35b/0x380
[  726.243089]  [<ffffffff81072e53>] rcu_check_callbacks+0x103/0x120
[  726.243089]  [<ffffffff81041a83>] update_process_times+0x43/0x80
[  726.243089]  [<ffffffff8105ebbf>] tick_sched_timer+0x5f/0xb0
[  726.243089]  [<ffffffff8105326d>] __run_hrtimer.isra.34+0x4d/0x100
[  726.243089]  [<ffffffff81053a6f>] hrtimer_interrupt+0xdf/0x1f0
[  726.243089]  [<ffffffff810197c4>] smp_apic_timer_interrupt+0x64/0xa0
[  726.243089]  [<ffffffff8164e8cb>] apic_timer_interrupt+0x6b/0x70
[  726.243089]  <EOI>  [<ffffffff810090b4>] ? default_idle+0x24/0x40
[  726.243089]  [<ffffffff810091c4>] amd_e400_idle+0x54/0x100
[  726.243089]  [<ffffffff81000818>] cpu_idle+0x78/0xc0
[  726.243089]  [<ffffffff8162cd99>] rest_init+0x6d/0x74
[  726.243089]  [<ffffffff81c91a76>] start_kernel+0x2ae/0x2b9
[  726.243089]  [<ffffffff81c912ee>] x86_64_start_reservations+0xfe/0x102
[  726.243089]  [<ffffffff81c913e2>] x86_64_start_kernel+0xf0/0xf7


Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Valdis.Kletnieks-PjAqaU27lzQ]

[-- Attachment #1: Type: text/plain, Size: 601 bytes --]

On Wed, 17 Aug 2011 15:16:40 EDT, Justin Piszcz said:

> Tried with linux-3.1-rc2 and:
> 
> You don't even need to get a successful mount:
> 
> Does anyone know if any kernel supports CIFS w/out crashing? I'd like to 
> backup some CIFS shares, thanks.

3.1.0-rc1-next-20110808 is talking CIFS to our NetApp without any complaint.
Not heavily stress tested, but it mounted, I can see files, I can open files, I
specifically tried 'df' and didn't crash.

So I'm guessing there's some subtle difference between what my NetApp feeds my
laptop over the wire and what you're seeing from your Windows box?


[-- Attachment #2: Type: application/pgp-signature, Size: 227 bytes --]

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[J. R. Okajima]

Justin Piszcz:
> Does anyone know if any kernel supports CIFS w/out crashing? I'd like to 
> backup some CIFS shares, thanks.
>
>
> mount -t cifs //w2/x /mnt -o user=user,pass=pass
>
> [  881.388836] CIFS VFS: cifs_mount failed w/return code = -22
	:::

Since it failed mounting, this patch will help you. Although the patch
will fix one bug, there still may exist another problem.

http://marc.info/?l=linux-cifs&m=131345112022031&w=2


J. R. Okajima

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Thu, 18 Aug 2011, J. R. Okajima wrote:

>
> Justin Piszcz:
>> Does anyone know if any kernel supports CIFS w/out crashing? I'd like to
>> backup some CIFS shares, thanks.
>>
>>
>> mount -t cifs //w2/x /mnt -o user=user,pass=pass
>>
>> [  881.388836] CIFS VFS: cifs_mount failed w/return code = -22
> 	:::
>
> Since it failed mounting, this patch will help you. Although the patch
> will fix one bug, there still may exist another problem.
>
> http://marc.info/?l=linux-cifs&m=131345112022031&w=2

Hi,

Latest patch (this one) applied to linux-3.1-rc2 works, at least it mounted
this time and did not instantly crash the kernel!

I also tried the hostname again (and it did not crash the kernel, but it 
failed to mount).

Used the IP and it mounted successfully:
//10.0.0.11/x          28T  5.0T   23T  19% /mnt
//10.0.0.11/y          19T  1.2T   18T   7% /mnt2

It has not crashed yet (which is good), I'll apply this patch to my
production machine and test taking backups of this data and let you know
if it crashes again, thanks!

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Thu, 18 Aug 2011, Justin Piszcz wrote:

>
>
> On Thu, 18 Aug 2011, J. R. Okajima wrote:
>
>> 
>> Justin Piszcz:
>>> Does anyone know if any kernel supports CIFS w/out crashing? I'd like to
>>> backup some CIFS shares, thanks.
>>> 
>>> 
>>> mount -t cifs //w2/x /mnt -o user=user,pass=pass
>>> 
>>> [  881.388836] CIFS VFS: cifs_mount failed w/return code = -22
>> 	:::
>> 
>> Since it failed mounting, this patch will help you. Although the patch
>> will fix one bug, there still may exist another problem.
>> 
>> http://marc.info/?l=linux-cifs&m=131345112022031&w=2
>
> Hi,
>
> Latest patch (this one) applied to linux-3.1-rc2 works, at least it mounted
> this time and did not instantly crash the kernel!
>
> I also tried the hostname again (and it did not crash the kernel, but it 
> failed to mount).
>
> Used the IP and it mounted successfully:
> //10.0.0.11/x          28T  5.0T   23T  19% /mnt
> //10.0.0.11/y          19T  1.2T   18T   7% /mnt2
>
> It has not crashed yet (which is good), I'll apply this patch to my
> production machine and test taking backups of this data and let you know
> if it crashes again, thanks!
>
> Justin.


Hello,

It is working but very slowly:

Device eth6 [10.0.1.2] (1/1):
================================================================================
Incoming:                               Outgoing:
Curr: 37.60 MByte/s                     Curr: 0.44 MByte/s
Avg: 4.98 MByte/s                       Avg: 0.09 MByte/s
Min: 0.00 MByte/s                       Min: 0.00 MByte/s
Max: 40.79 MByte/s                      Max: 0.48 MByte/s
Ttl: 1.45 GByte                         Ttl: 26.77 MByte

Over 10GbE the other direction (Linux -> Windows (via Samba)) I get 
500MiB/s, is CIFS slow?

I'll look into options to tweak the speed but this is very poor speed when 
you have to transfer 5-10TB.  However, it is not crashing anymore, so any 
speed is better than that :)

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Thu, 18 Aug 2011, Justin Piszcz wrote:

>
>
> On Thu, 18 Aug 2011, Justin Piszcz wrote:
>
>> 
>> 
>> On Thu, 18 Aug 2011, J. R. Okajima wrote:
>> 
>>> 
>>> Justin Piszcz:
>>>> Does anyone know if any kernel supports CIFS w/out crashing? I'd like to
>>>> backup some CIFS shares, thanks.
>>>> 
>>>> 
>>>> mount -t cifs //w2/x /mnt -o user=user,pass=pass
>>>> 
>>>> [  881.388836] CIFS VFS: cifs_mount failed w/return code = -22
>>> 	:::
>>> 
>>> Since it failed mounting, this patch will help you. Although the patch
>>> will fix one bug, there still may exist another problem.
>>> 
>>> http://marc.info/?l=linux-cifs&m=131345112022031&w=2
>> 
>> Hi,
>> 
>> Latest patch (this one) applied to linux-3.1-rc2 works, at least it mounted
>> this time and did not instantly crash the kernel!
>> 
>> I also tried the hostname again (and it did not crash the kernel, but it 
>> failed to mount).
>> 
>> Used the IP and it mounted successfully:
>> //10.0.0.11/x          28T  5.0T   23T  19% /mnt
>> //10.0.0.11/y          19T  1.2T   18T   7% /mnt2
>> 
>> It has not crashed yet (which is good), I'll apply this patch to my
>> production machine and test taking backups of this data and let you know
>> if it crashes again, thanks!
>> 
>> Justin.
>
>
> Hello,
>
> It is working but very slowly:
>
> Device eth6 [10.0.1.2] (1/1):
> ================================================================================
> Incoming:                               Outgoing:
> Curr: 37.60 MByte/s                     Curr: 0.44 MByte/s
> Avg: 4.98 MByte/s                       Avg: 0.09 MByte/s
> Min: 0.00 MByte/s                       Min: 0.00 MByte/s
> Max: 40.79 MByte/s                      Max: 0.48 MByte/s
> Ttl: 1.45 GByte                         Ttl: 26.77 MByte
>
> Over 10GbE the other direction (Linux -> Windows (via Samba)) I get 500MiB/s, 
> is CIFS slow?
>
> I'll look into options to tweak the speed but this is very poor speed when 
> you have to transfer 5-10TB.  However, it is not crashing anymore, so any 
> speed is better than that :)
>
> Justin.

Hi,

Mounting with:
rw,uid=1000,gid=100,mode=0644,rsize=130048,wsize=1048576,credentials=/root/.cifs 
Same speed:
Device eth6 [10.0.1.2] (1/1):
================================================================================
Incoming:                               Outgoing:
Curr: 32.42 MByte/s                     Curr: 0.38 MByte/s
Avg: 30.72 MByte/s                      Avg: 0.39 MByte/s
Min: 0.00 MByte/s                       Min: 0.00 MByte/s
Max: 43.64 MByte/s                      Max: 0.59 MByte/s
Ttl: 20.15 GByte                        Ttl: 261.03 MByte

Thoughts?

Has anyone achieved > 30-40MB/s with CIFS?
This is a 10GbE link (and yes JUMBO frames are enabled on both sides, and 
again, samba from Linux->Windows = 500MB/s)

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Jeff Layton]

On Thu, 18 Aug 2011 08:22:44 -0400 (EDT)
Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:

> 
> 
> On Thu, 18 Aug 2011, Justin Piszcz wrote:
> 
> >
> >
> > On Thu, 18 Aug 2011, Justin Piszcz wrote:
> >
> >> 
> >> 
> >> On Thu, 18 Aug 2011, J. R. Okajima wrote:
> >> 
> >>> 
> >>> Justin Piszcz:
> >>>> Does anyone know if any kernel supports CIFS w/out crashing? I'd like to
> >>>> backup some CIFS shares, thanks.
> >>>> 
> >>>> 
> >>>> mount -t cifs //w2/x /mnt -o user=user,pass=pass
> >>>> 
> >>>> [  881.388836] CIFS VFS: cifs_mount failed w/return code = -22
> >>> 	:::
> >>> 
> >>> Since it failed mounting, this patch will help you. Although the patch
> >>> will fix one bug, there still may exist another problem.
> >>> 
> >>> http://marc.info/?l=linux-cifs&m=131345112022031&w=2
> >> 
> >> Hi,
> >> 
> >> Latest patch (this one) applied to linux-3.1-rc2 works, at least it mounted
> >> this time and did not instantly crash the kernel!
> >> 
> >> I also tried the hostname again (and it did not crash the kernel, but it 
> >> failed to mount).
> >> 
> >> Used the IP and it mounted successfully:
> >> //10.0.0.11/x          28T  5.0T   23T  19% /mnt
> >> //10.0.0.11/y          19T  1.2T   18T   7% /mnt2
> >> 
> >> It has not crashed yet (which is good), I'll apply this patch to my
> >> production machine and test taking backups of this data and let you know
> >> if it crashes again, thanks!
> >> 
> >> Justin.
> >
> >
> > Hello,
> >
> > It is working but very slowly:
> >
> > Device eth6 [10.0.1.2] (1/1):
> > ================================================================================
> > Incoming:                               Outgoing:
> > Curr: 37.60 MByte/s                     Curr: 0.44 MByte/s
> > Avg: 4.98 MByte/s                       Avg: 0.09 MByte/s
> > Min: 0.00 MByte/s                       Min: 0.00 MByte/s
> > Max: 40.79 MByte/s                      Max: 0.48 MByte/s
> > Ttl: 1.45 GByte                         Ttl: 26.77 MByte
> >
> > Over 10GbE the other direction (Linux -> Windows (via Samba)) I get 500MiB/s, 
> > is CIFS slow?
> >
> > I'll look into options to tweak the speed but this is very poor speed when 
> > you have to transfer 5-10TB.  However, it is not crashing anymore, so any 
> > speed is better than that :)
> >
> > Justin.
> 
> Hi,
> 
> Mounting with:
> rw,uid=1000,gid=100,mode=0644,rsize=130048,wsize=1048576,credentials=/root/.cifs 
> Same speed:
> Device eth6 [10.0.1.2] (1/1):
> ================================================================================
> Incoming:                               Outgoing:
> Curr: 32.42 MByte/s                     Curr: 0.38 MByte/s
> Avg: 30.72 MByte/s                      Avg: 0.39 MByte/s
> Min: 0.00 MByte/s                       Min: 0.00 MByte/s
> Max: 43.64 MByte/s                      Max: 0.59 MByte/s
> Ttl: 20.15 GByte                        Ttl: 261.03 MByte
> 
> Thoughts?
> 
> Has anyone achieved > 30-40MB/s with CIFS?
> This is a 10GbE link (and yes JUMBO frames are enabled on both sides, and 
> again, samba from Linux->Windows = 500MB/s)
> 
> Justin.
> 

To be clear -- incoming in this case is reads or writes?

Up until 3.0 cifs.ko didn't parallelize writes from a single thread. In
3.0 I added a patchset to increase the allowable wsize and to allow the
kernel to issue writes in parallel.

Reads still suffer from the same problem however. I'm working on a
patchset that should do the same thing for them, but it requires a
fairly substantial overhaul of the receive codepaths.

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Thu, 18 Aug 2011, Jeff Layton wrote:

> On Thu, 18 Aug 2011 08:22:44 -0400 (EDT)
> Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
>> Justin.
>>
>
> To be clear -- incoming in this case is reads or writes?
Reading from the CIFS share (Windows 7).

>
> Up until 3.0 cifs.ko didn't parallelize writes from a single thread. In
> 3.0 I added a patchset to increase the allowable wsize and to allow the
> kernel to issue writes in parallel.
Ahh, good to know, have not tried writes yet.

>
> Reads still suffer from the same problem however. I'm working on a
> patchset that should do the same thing for them, but it requires a
> fairly substantial overhaul of the receive codepaths.
Ok, that explains it then, thanks.

One other item, the rsync is currently running, what does mean/what option
would fix it/make the error go away?

[ 3730.897554] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3755.727255] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3760.509489] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3760.786650] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3762.854706] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3763.052163] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3764.962962] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3765.460372] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3765.855425] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3767.677672] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3781.958450] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3800.493896] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3805.911191] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3806.121384] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3817.356127] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3825.148050] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3833.687886] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3834.199746] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3834.448342] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3866.848508] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3867.126729] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3868.428427] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3872.206396] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3872.812745] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3877.655580] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3880.718684] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3880.769545] CIFS VFS: did not end path lookup where expected namelen is 0
[ 3882.026917] CIFS VFS: did not end path lookup where expected namelen is 0

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Jeff Layton]

On Thu, 18 Aug 2011 09:15:36 -0400 (EDT)
Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:

> 
> 
> On Thu, 18 Aug 2011, Jeff Layton wrote:
> 
> > On Thu, 18 Aug 2011 08:22:44 -0400 (EDT)
> > Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
> >
> >> Justin.
> >>
> >
> > To be clear -- incoming in this case is reads or writes?
> Reading from the CIFS share (Windows 7).
> 
> >
> > Up until 3.0 cifs.ko didn't parallelize writes from a single thread. In
> > 3.0 I added a patchset to increase the allowable wsize and to allow the
> > kernel to issue writes in parallel.
> Ahh, good to know, have not tried writes yet.
> 
> >
> > Reads still suffer from the same problem however. I'm working on a
> > patchset that should do the same thing for them, but it requires a
> > fairly substantial overhaul of the receive codepaths.
> Ok, that explains it then, thanks.
> 
> One other item, the rsync is currently running, what does mean/what option
> would fix it/make the error go away?
> 
> [ 3730.897554] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3755.727255] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3760.509489] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3760.786650] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3762.854706] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3763.052163] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3764.962962] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3765.460372] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3765.855425] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3767.677672] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3781.958450] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3800.493896] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3805.911191] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3806.121384] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3817.356127] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3825.148050] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3833.687886] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3834.199746] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3834.448342] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3866.848508] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3867.126729] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3868.428427] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3872.206396] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3872.812745] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3877.655580] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3880.718684] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3880.769545] CIFS VFS: did not end path lookup where expected namelen is 0
> [ 3882.026917] CIFS VFS: did not end path lookup where expected namelen is 0
> 
> Justin.
> 

I've sent a patch to Steve to turn those printk's into debug messages
for 3.1 and it should make its way there soon. It should be safe to
ignore them in the meantime.

In fact, that patch should probably go to 3.0-stable as well. Steve can
you add the 'Cc: stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org' line to that patch?

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Steve French]

On Thu, Aug 18, 2011 at 12:04 PM, Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
> On Thu, 18 Aug 2011 09:15:36 -0400 (EDT)
> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>> [ 3882.026917] CIFS VFS: did not end path lookup where expected namelen is 0
>>
>> Justin.
>>
>
> I've sent a patch to Steve to turn those printk's into debug messages
> for 3.1 and it should make its way there soon. It should be safe to
> ignore them in the meantime.
>
> In fact, that patch should probably go to 3.0-stable as well. Steve can
> you add the 'Cc: stable-DgEjT+Ai2ygdnm+yROfE0A@public.gmane.org' line to that patch?

Already done.

-- 
Thanks,

Steve

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Thu, 18 Aug 2011, Jeff Layton wrote:

> On Thu, 18 Aug 2011 09:15:36 -0400 (EDT)
> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>
>>
>>
>> On Thu, 18 Aug 2011, Jeff Layton wrote:
>>
>>> On Thu, 18 Aug 2011 08:22:44 -0400 (EDT)
>>> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>>>
>>>> Justin.
>>>>
>>>
>>> To be clear -- incoming in this case is reads or writes?
>> Reading from the CIFS share (Windows 7).
>>
>>>
>>> Up until 3.0 cifs.ko didn't parallelize writes from a single thread. In
>>> 3.0 I added a patchset to increase the allowable wsize and to allow the
>>> kernel to issue writes in parallel.
>> Ahh, good to know, have not tried writes yet.
>>
>>>
>>> Reads still suffer from the same problem however. I'm working on a
>>> patchset that should do the same thing for them, but it requires a
>>> fairly substantial overhaul of the receive codepaths.
>> Ok, that explains it then, thanks.


Hi,

Watching the rsync, it ran for a while, then:

rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot allocate memory (12)
rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_1.JPG": Cannot allocate memory (12)
rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_2.JPG": Cannot allocate memory (12)
rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot allocate memory (12)

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Steve French]

On Thu, Aug 18, 2011 at 12:16 PM, Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>
>
> On Thu, 18 Aug 2011, Jeff Layton wrote:
>
>> On Thu, 18 Aug 2011 09:15:36 -0400 (EDT)
>> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>>
>>>
>>>
>>> On Thu, 18 Aug 2011, Jeff Layton wrote:
>>>
>>>> On Thu, 18 Aug 2011 08:22:44 -0400 (EDT)
>>>> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>>>>
>>>>> Justin.
>>>>>
>>>>
>>>> To be clear -- incoming in this case is reads or writes?
>>>
>>> Reading from the CIFS share (Windows 7).
>>>
>>>>
>>>> Up until 3.0 cifs.ko didn't parallelize writes from a single thread. In
>>>> 3.0 I added a patchset to increase the allowable wsize and to allow the
>>>> kernel to issue writes in parallel.
>>>
>>> Ahh, good to know, have not tried writes yet.
>>>
>>>>
>>>> Reads still suffer from the same problem however. I'm working on a
>>>> patchset that should do the same thing for them, but it requires a
>>>> fairly substantial overhaul of the receive codepaths.
>>>
>>> Ok, that explains it then, thanks.
>
>
> Hi,
>
> Watching the rsync, it ran for a while, then:
>
> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot
> allocate memory (12)
> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_1.JPG": Cannot
> allocate memory (12)
> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_2.JPG": Cannot
> allocate memory (12)
> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot
> allocate memory (12)

When we were testing async write to Windows 7 Pavel mentioned to me
another WIndows 7 bug - which may be what you are hitting.

Under stress of simultaneous operations, Windows 7 server will sometimes start
responding with STATUS_INSUFF_SERVER_RESOURCES error code
(mapped to posix error ENOMEM by the Linux cifs kernel client)
He solved it by setting MaxWorkItems to 4096 in the Windows 7 registry.

If anyone knows whether Microsoft has fixed this or has a bug #, let us know
because it is easier to hit with Linux kernel 3.0 and later (to
Windows 7 server).



-- 
Thanks,

Steve

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Thu, 18 Aug 2011, Steve French wrote:

> On Thu, Aug 18, 2011 at 12:16 PM, Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>>
>>
>> On Thu, 18 Aug 2011, Jeff Layton wrote:
>>
>>> On Thu, 18 Aug 2011 09:15:36 -0400 (EDT)
>>> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>>>
>>>>
>>>>
>>>> On Thu, 18 Aug 2011, Jeff Layton wrote:
>>>>
>>>>> On Thu, 18 Aug 2011 08:22:44 -0400 (EDT)
>>>>> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>>>>>
>>>>>> Justin.
>>>>>>
>>>>>
>>>>> To be clear -- incoming in this case is reads or writes?
>>>>
>>>> Reading from the CIFS share (Windows 7).
>>>>
>>>>>
>>>>> Up until 3.0 cifs.ko didn't parallelize writes from a single thread. In
>>>>> 3.0 I added a patchset to increase the allowable wsize and to allow the
>>>>> kernel to issue writes in parallel.
>>>>
>>>> Ahh, good to know, have not tried writes yet.
>>>>
>>>>>
>>>>> Reads still suffer from the same problem however. I'm working on a
>>>>> patchset that should do the same thing for them, but it requires a
>>>>> fairly substantial overhaul of the receive codepaths.
>>>>
>>>> Ok, that explains it then, thanks.
>>
>>
>> Hi,
>>
>> Watching the rsync, it ran for a while, then:
>>
>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot
>> allocate memory (12)
>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_1.JPG": Cannot
>> allocate memory (12)
>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_2.JPG": Cannot
>> allocate memory (12)
>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot
>> allocate memory (12)
>
> When we were testing async write to Windows 7 Pavel mentioned to me
> another WIndows 7 bug - which may be what you are hitting.
>
> Under stress of simultaneous operations, Windows 7 server will sometimes start
> responding with STATUS_INSUFF_SERVER_RESOURCES error code
> (mapped to posix error ENOMEM by the Linux cifs kernel client)
> He solved it by setting MaxWorkItems to 4096 in the Windows 7 registry.
>
> If anyone knows whether Microsoft has fixed this or has a bug #, let us know
> because it is easier to hit with Linux kernel 3.0 and later (to
> Windows 7 server).

Ok, aside from sharing a few files, is there any recommended protocol to
transfer millions of files at 10GbE speeds (500MiB/s) in both directions
between Windows and Linux?

Specifically from Linux..

>From Windows, it achieves 500MiB/s no issues (From Linux/Samba).

The other direction:
- 30mb/s over 10GbE
- Cannot allocate memory

This should be good for moving a small subset of files, but for larger datasets
it will not work.  Any other filesystems/services/etc to try?

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Steve French]

If reading files - smbclient (ftp like syntax) should be able to reach
wire speeds (assuming the server disk can keep up) and for
writing it should be similar (perhaps a little slower but it won't
use i/o sizes as large as cifs kernel client).  I would expect
smbclient to/copy from Windows to be faster than Windows mount -> Samba.

I reached near wirespeeds for GigE with cifs client (writing) to
Winodws 2003/2008/r2
and Samba - but didn't have fast enough disks to test 10GigE although I expect
very good performance with that if you have fast enough server disks (and
am willing to put performance patches in, if you detect additional
changes that we should make for 10GigE - in particular allowing
more than 50 simultaneous requests).

If the registry fix for Windows 7 is in place (or if you copy to
WIndows 2008, Windows 2003, WIndows 2008r2) -
cifs kernel client is probably slightly faster for writes than alternatives,
smbclient much faster for reads.

If going the other direction (Windows client copying to/from Samba server) -
Samba 3.6 (server) with SMB2 support
turned on - is going to be faster than most alternatives.


On Thu, Aug 18, 2011 at 12:33 PM, Justin Piszcz <jpiszcz-BP4nVm5VUdOlno5B1MRpzg@public.gmane.orgm> wrote:
>
>
> On Thu, 18 Aug 2011, Steve French wrote:
>
>> On Thu, Aug 18, 2011 at 12:16 PM, Justin Piszcz <jpiszcz@lucidpixels.com>
>> wrote:
>>>
>>>
>>> On Thu, 18 Aug 2011, Jeff Layton wrote:
>>>
>>>> On Thu, 18 Aug 2011 09:15:36 -0400 (EDT)
>>>> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, 18 Aug 2011, Jeff Layton wrote:
>>>>>
>>>>>> On Thu, 18 Aug 2011 08:22:44 -0400 (EDT)
>>>>>> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>>>>>>
>>>>>>> Justin.
>>>>>>>
>>>>>>
>>>>>> To be clear -- incoming in this case is reads or writes?
>>>>>
>>>>> Reading from the CIFS share (Windows 7).
>>>>>
>>>>>>
>>>>>> Up until 3.0 cifs.ko didn't parallelize writes from a single thread.
>>>>>> In
>>>>>> 3.0 I added a patchset to increase the allowable wsize and to allow
>>>>>> the
>>>>>> kernel to issue writes in parallel.
>>>>>
>>>>> Ahh, good to know, have not tried writes yet.
>>>>>
>>>>>>
>>>>>> Reads still suffer from the same problem however. I'm working on a
>>>>>> patchset that should do the same thing for them, but it requires a
>>>>>> fairly substantial overhaul of the receive codepaths.
>>>>>
>>>>> Ok, that explains it then, thanks.
>>>
>>>
>>> Hi,
>>>
>>> Watching the rsync, it ran for a while, then:
>>>
>>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot
>>> allocate memory (12)
>>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_1.JPG": Cannot
>>> allocate memory (12)
>>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_2.JPG": Cannot
>>> allocate memory (12)
>>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot
>>> allocate memory (12)
>>
>> When we were testing async write to Windows 7 Pavel mentioned to me
>> another WIndows 7 bug - which may be what you are hitting.
>>
>> Under stress of simultaneous operations, Windows 7 server will sometimes
>> start
>> responding with STATUS_INSUFF_SERVER_RESOURCES error code
>> (mapped to posix error ENOMEM by the Linux cifs kernel client)
>> He solved it by setting MaxWorkItems to 4096 in the Windows 7 registry.
>>
>> If anyone knows whether Microsoft has fixed this or has a bug #, let us
>> know
>> because it is easier to hit with Linux kernel 3.0 and later (to
>> Windows 7 server).
>
> Ok, aside from sharing a few files, is there any recommended protocol to
> transfer millions of files at 10GbE speeds (500MiB/s) in both directions
> between Windows and Linux?
>
> Specifically from Linux..
>
> From Windows, it achieves 500MiB/s no issues (From Linux/Samba).
>
> The other direction:
> - 30mb/s over 10GbE
> - Cannot allocate memory
>
> This should be good for moving a small subset of files, but for larger
> datasets
> it will not work.  Any other filesystems/services/etc to try?
>
> Justin.
>
>
>



-- 
Thanks,

Steve

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Thu, 18 Aug 2011, Steve French wrote:

> If reading files - smbclient (ftp like syntax) should be able to reach
> wire speeds (assuming the server disk can keep up) and for
> writing it should be similar (perhaps a little slower but it won't
> use i/o sizes as large as cifs kernel client).  I would expect
> smbclient to/copy from Windows to be faster than Windows mount -> Samba.
>
> I reached near wirespeeds for GigE with cifs client (writing) to
> Winodws 2003/2008/r2
> and Samba - but didn't have fast enough disks to test 10GigE although I expect
> very good performance with that if you have fast enough server disks (and
> am willing to put performance patches in, if you detect additional
> changes that we should make for 10GigE - in particular allowing
> more than 50 simultaneous requests).
>
> If the registry fix for Windows 7 is in place (or if you copy to
> WIndows 2008, Windows 2003, WIndows 2008r2) -
> cifs kernel client is probably slightly faster for writes than alternatives,
> smbclient much faster for reads.
>
> If going the other direction (Windows client copying to/from Samba server) -
> Samba 3.6 (server) with SMB2 support
> turned on - is going to be faster than most alternatives.

Hi,

Downloading from an FTP Server on the windows machine:

<--- 150 Connection accepted
`file' at 418873344 (24%) 145.58M/s eta:9s [Receiving data]
`file' at 735510528 (42%) 162.93M/s eta:6s [Receiving data]
`file' at 1357185024 (78%) 172.89M/s eta:2s [Receiving data]
<--- 226 Transfer OK 
---- Got EOF on data connection
---- Closing data socket
1719784995 bytes transferred in 9 seconds (181.65M/s)

Same file (via CIFS): 50MB/s

Device eth6 [10.0.1.2] (1/1):
================================================================================
Incoming:                               Outgoing:
Curr: 49.17 MByte/s                     Curr: 0.58 MByte/s
Avg: 8.76 MByte/s                       Avg: 0.03 MByte/s
Min: 0.00 MByte/s                       Min: 0.00 MByte/s
Max: 197.68 MByte/s                     Max: 0.86 MByte/s
Ttl: 585.87 GByte                       Ttl: 6.97 GByte

0.02user 4.09system 0:33.84elapsed 12%CPU (0avgtext+0avgdata 3632maxresident)k

The RAID's on both systems can achieve > 750MiB/s sustained without any
problems (benchmarked in the past) and the system has 56 PCI-e 2.0 lanes
and a cable between two 10GbE nics.  Not sure where to start, Win7/box tweaks
or Linux/CIFS options?

Again, Samba -> Linux = 500MiB/s, so FTP > 3x faster than CIFS.
I know NFS gets a lot of testing (never had an issue with it/performance/etc)

However, regarding CIFS-- is there a test suite/or benchmark pack the CIFS 
developers use to make sure everything is working in order?

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Jeff Layton]

On Thu, 18 Aug 2011 14:18:15 -0400 (EDT)
Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:

> 
> 
> On Thu, 18 Aug 2011, Steve French wrote:
> 
> > If reading files - smbclient (ftp like syntax) should be able to reach
> > wire speeds (assuming the server disk can keep up) and for
> > writing it should be similar (perhaps a little slower but it won't
> > use i/o sizes as large as cifs kernel client).  I would expect
> > smbclient to/copy from Windows to be faster than Windows mount -> Samba.
> >
> > I reached near wirespeeds for GigE with cifs client (writing) to
> > Winodws 2003/2008/r2
> > and Samba - but didn't have fast enough disks to test 10GigE although I expect
> > very good performance with that if you have fast enough server disks (and
> > am willing to put performance patches in, if you detect additional
> > changes that we should make for 10GigE - in particular allowing
> > more than 50 simultaneous requests).
> >
> > If the registry fix for Windows 7 is in place (or if you copy to
> > WIndows 2008, Windows 2003, WIndows 2008r2) -
> > cifs kernel client is probably slightly faster for writes than alternatives,
> > smbclient much faster for reads.
> >
> > If going the other direction (Windows client copying to/from Samba server) -
> > Samba 3.6 (server) with SMB2 support
> > turned on - is going to be faster than most alternatives.
> 
> Hi,
> 
> Downloading from an FTP Server on the windows machine:
> 
> <--- 150 Connection accepted
> `file' at 418873344 (24%) 145.58M/s eta:9s [Receiving data]
> `file' at 735510528 (42%) 162.93M/s eta:6s [Receiving data]
> `file' at 1357185024 (78%) 172.89M/s eta:2s [Receiving data]
> <--- 226 Transfer OK 
> ---- Got EOF on data connection
> ---- Closing data socket
> 1719784995 bytes transferred in 9 seconds (181.65M/s)
> 
> Same file (via CIFS): 50MB/s
> 
> Device eth6 [10.0.1.2] (1/1):
> ================================================================================
> Incoming:                               Outgoing:
> Curr: 49.17 MByte/s                     Curr: 0.58 MByte/s
> Avg: 8.76 MByte/s                       Avg: 0.03 MByte/s
> Min: 0.00 MByte/s                       Min: 0.00 MByte/s
> Max: 197.68 MByte/s                     Max: 0.86 MByte/s
> Ttl: 585.87 GByte                       Ttl: 6.97 GByte
> 
> 0.02user 4.09system 0:33.84elapsed 12%CPU (0avgtext+0avgdata 3632maxresident)k
> 
> The RAID's on both systems can achieve > 750MiB/s sustained without any
> problems (benchmarked in the past) and the system has 56 PCI-e 2.0 lanes
> and a cable between two 10GbE nics.  Not sure where to start, Win7/box tweaks
> or Linux/CIFS options?
> 
> Again, Samba -> Linux = 500MiB/s, so FTP > 3x faster than CIFS.
> I know NFS gets a lot of testing (never had an issue with it/performance/etc)
> 
> However, regarding CIFS-- is there a test suite/or benchmark pack the CIFS 
> developers use to make sure everything is working in order?
> 
> Justin.
> 
> 

Like I said, read performance with cifs.ko just plain sucks currently.

Don't look for cifs.ko to achieve anywhere near NFS' performance unless
you jump through some very odd hoops (like multithreading your workload
in userspace, etc). cifs.ko just doesn't do a good job of keeping the
pipe "stuffed" as most calls are handled synchronously. A single task
can only handle one call on the wire in most cases. The exception here
is writes, but that just recently changed...

Reads are done using relatively small buffers and then copied to
pagecache. Part of what I'm working on will be to allow for much larger
reads directly into the pagecache. That should also help performance
significantly.

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Thu, 18 Aug 2011, Jeff Layton wrote:

> On Thu, 18 Aug 2011 14:18:15 -0400 (EDT)
> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>
>>
>
> Like I said, read performance with cifs.ko just plain sucks currently.
>
> Don't look for cifs.ko to achieve anywhere near NFS' performance unless
> you jump through some very odd hoops (like multithreading your workload
> in userspace, etc). cifs.ko just doesn't do a good job of keeping the
> pipe "stuffed" as most calls are handled synchronously. A single task
> can only handle one call on the wire in most cases. The exception here
> is writes, but that just recently changed...
>
> Reads are done using relatively small buffers and then copied to
> pagecache. Part of what I'm working on will be to allow for much larger
> reads directly into the pagecache. That should also help performance
> significantly.


Jeff,

Understood, thank you for the help thus far and the patch to stop the crashing
kernel problem, it is much appreciated, I'll look forward to the new patch sets
to improve the performance of CIFS mounts.

Justin.

>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Steve French]

On Thu, Aug 18, 2011 at 1:54 PM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>
>
> On Thu, 18 Aug 2011, Jeff Layton wrote:
>
>> On Thu, 18 Aug 2011 14:18:15 -0400 (EDT)
>> Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>>
>>>
>>
>> Like I said, read performance with cifs.ko just plain sucks currently.
>>
>> Don't look for cifs.ko to achieve anywhere near NFS' performance unless
>> you jump through some very odd hoops (like multithreading your workload
>> in userspace, etc). cifs.ko just doesn't do a good job of keeping the
>> pipe "stuffed" as most calls are handled synchronously. A single task
>> can only handle one call on the wire in most cases. The exception here
>> is writes, but that just recently changed...
>>
>> Reads are done using relatively small buffers and then copied to
>> pagecache. Part of what I'm working on will be to allow for much larger
>> reads directly into the pagecache. That should also help performance
>> significantly.

As Jeff notes, large file sequential read of one file at a time is going
to be cifs kernel client's worst case.   An earlier cifs async patchset
(prototype) showed dramatic improvements so it is not
inherent in the protocol and I am looking forward to seeing
Jeff's patches for this.

With multiple processes reading/writing cifs does pretty well
(since the worst case serialization is in cifs_readpages,
coupled with the relatively small buffer size on reads), as long as
you are accessing various files at one time.   I have seen
cifs beat nfs on broader tests such as dbench when lots of activity on the
wire from multiple processes - but meant to do a dbench run
on recent kernels to compare.   Would be interesting to compare
dbench now (with e.g. more than 30 processes) on nfs mount to
cifs mount.


-- 
Thanks,

Steve

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Jeff Layton]

On Thu, 18 Aug 2011 14:52:57 -0400
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:

> On Thu, 18 Aug 2011 14:18:15 -0400 (EDT)
> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
> 
> > 
> > 
> > On Thu, 18 Aug 2011, Steve French wrote:
> > 
> > > If reading files - smbclient (ftp like syntax) should be able to reach
> > > wire speeds (assuming the server disk can keep up) and for
> > > writing it should be similar (perhaps a little slower but it won't
> > > use i/o sizes as large as cifs kernel client).  I would expect
> > > smbclient to/copy from Windows to be faster than Windows mount -> Samba.
> > >
> > > I reached near wirespeeds for GigE with cifs client (writing) to
> > > Winodws 2003/2008/r2
> > > and Samba - but didn't have fast enough disks to test 10GigE although I expect
> > > very good performance with that if you have fast enough server disks (and
> > > am willing to put performance patches in, if you detect additional
> > > changes that we should make for 10GigE - in particular allowing
> > > more than 50 simultaneous requests).
> > >
> > > If the registry fix for Windows 7 is in place (or if you copy to
> > > WIndows 2008, Windows 2003, WIndows 2008r2) -
> > > cifs kernel client is probably slightly faster for writes than alternatives,
> > > smbclient much faster for reads.
> > >
> > > If going the other direction (Windows client copying to/from Samba server) -
> > > Samba 3.6 (server) with SMB2 support
> > > turned on - is going to be faster than most alternatives.
> > 
> > Hi,
> > 
> > Downloading from an FTP Server on the windows machine:
> > 
> > <--- 150 Connection accepted
> > `file' at 418873344 (24%) 145.58M/s eta:9s [Receiving data]
> > `file' at 735510528 (42%) 162.93M/s eta:6s [Receiving data]
> > `file' at 1357185024 (78%) 172.89M/s eta:2s [Receiving data]
> > <--- 226 Transfer OK 
> > ---- Got EOF on data connection
> > ---- Closing data socket
> > 1719784995 bytes transferred in 9 seconds (181.65M/s)
> > 
> > Same file (via CIFS): 50MB/s
> > 
> > Device eth6 [10.0.1.2] (1/1):
> > ================================================================================
> > Incoming:                               Outgoing:
> > Curr: 49.17 MByte/s                     Curr: 0.58 MByte/s
> > Avg: 8.76 MByte/s                       Avg: 0.03 MByte/s
> > Min: 0.00 MByte/s                       Min: 0.00 MByte/s
> > Max: 197.68 MByte/s                     Max: 0.86 MByte/s
> > Ttl: 585.87 GByte                       Ttl: 6.97 GByte
> > 
> > 0.02user 4.09system 0:33.84elapsed 12%CPU (0avgtext+0avgdata 3632maxresident)k
> > 
> > The RAID's on both systems can achieve > 750MiB/s sustained without any
> > problems (benchmarked in the past) and the system has 56 PCI-e 2.0 lanes
> > and a cable between two 10GbE nics.  Not sure where to start, Win7/box tweaks
> > or Linux/CIFS options?
> > 
> > Again, Samba -> Linux = 500MiB/s, so FTP > 3x faster than CIFS.
> > I know NFS gets a lot of testing (never had an issue with it/performance/etc)
> > 
> > However, regarding CIFS-- is there a test suite/or benchmark pack the CIFS 
> > developers use to make sure everything is working in order?
> > 
> > Justin.
> > 
> > 
> 
> Like I said, read performance with cifs.ko just plain sucks currently.
> 
> Don't look for cifs.ko to achieve anywhere near NFS' performance unless
> you jump through some very odd hoops (like multithreading your workload
> in userspace, etc). cifs.ko just doesn't do a good job of keeping the
> pipe "stuffed" as most calls are handled synchronously. A single task
> can only handle one call on the wire in most cases. The exception here
> is writes, but that just recently changed...
> 
> Reads are done using relatively small buffers and then copied to
> pagecache. Part of what I'm working on will be to allow for much larger
> reads directly into the pagecache. That should also help performance
> significantly.
> 

I just posted a patchset that should improve the performance of
buffered reads. It's rather large but should apply to current upstream
kernels. If you've had problems with cifs.ko read performance in the
past, then this would be a good time to step up and help test them.

If it makes things easier, then the patchset is also available via the
cifs-3.2 branch of my public git tree:

    http://git.kernel.org/?p=linux/kernel/git/jlayton/linux.git;a=shortlog;h=refs/heads/cifs-3.2

Thanks,
-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Fri, 26 Aug 2011, Jeff Layton wrote:

> On Thu, 18 Aug 2011 14:52:57 -0400
> Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
>
>> On Thu, 18 Aug 2011 14:18:15 -0400 (EDT)
>> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>>
>>>
>> Like I said, read performance with cifs.ko just plain sucks currently.
>>
>> Don't look for cifs.ko to achieve anywhere near NFS' performance unless
>> you jump through some very odd hoops (like multithreading your workload
>> in userspace, etc). cifs.ko just doesn't do a good job of keeping the
>> pipe "stuffed" as most calls are handled synchronously. A single task
>> can only handle one call on the wire in most cases. The exception here
>> is writes, but that just recently changed...
>>
>> Reads are done using relatively small buffers and then copied to
>> pagecache. Part of what I'm working on will be to allow for much larger
>> reads directly into the pagecache. That should also help performance
>> significantly.
>>
>
> I just posted a patchset that should improve the performance of
> buffered reads. It's rather large but should apply to current upstream
> kernels. If you've had problems with cifs.ko read performance in the
> past, then this would be a good time to step up and help test them.
>
> If it makes things easier, then the patchset is also available via the
> cifs-3.2 branch of my public git tree:
>
>    http://git.kernel.org/?p=linux/kernel/git/jlayton/linux.git;a=shortlog;h=refs/heads/cifs-3.2
>
> Thanks,
> -- 
> Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Hi Jeff,

This was working earlier..

# mount -t cifs //10.0.1.1/x /mnt -o user=XXXXXXX,pass=XXXXXXXX
mount error(12): Cannot allocate memory
Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

I'll need to contact the owner of the host to see if something has changed as 
this _was_ working previously.

Justin.

>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Steve French]

On Fri, Aug 26, 2011 at 5:57 PM, Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>
>
> On Fri, 26 Aug 2011, Jeff Layton wrote:
>
>> On Thu, 18 Aug 2011 14:52:57 -0400
>> Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
>>
>>> On Thu, 18 Aug 2011 14:18:15 -0400 (EDT)
>>> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>>>
>>>>
>>> Like I said, read performance with cifs.ko just plain sucks currently.
>>>
>>> Don't look for cifs.ko to achieve anywhere near NFS' performance unless
>>> you jump through some very odd hoops (like multithreading your workload
>>> in userspace, etc). cifs.ko just doesn't do a good job of keeping the
>>> pipe "stuffed" as most calls are handled synchronously. A single task
>>> can only handle one call on the wire in most cases. The exception here
>>> is writes, but that just recently changed...
>>>
>>> Reads are done using relatively small buffers and then copied to
>>> pagecache. Part of what I'm working on will be to allow for much larger
>>> reads directly into the pagecache. That should also help performance
>>> significantly.
>>>
>>
>> I just posted a patchset that should improve the performance of
>> buffered reads. It's rather large but should apply to current upstream
>> kernels. If you've had problems with cifs.ko read performance in the
>> past, then this would be a good time to step up and help test them.
>>
>> If it makes things easier, then the patchset is also available via the
>> cifs-3.2 branch of my public git tree:
>>
>>
>> http://git.kernel.org/?p=linux/kernel/git/jlayton/linux.git;a=shortlog;h=refs/heads/cifs-3.2
>>
>> Thanks,
>> --
>> Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>
>
> Hi Jeff,
>
> This was working earlier..
>
> # mount -t cifs //10.0.1.1/x /mnt -o user=XXXXXXX,pass=XXXXXXXX
> mount error(12): Cannot allocate memory
> Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)
>
> I'll need to contact the owner of the host to see if something has changed
> as this _was_ working previously.

Fails with Jeff's new 20+ patch series?


-- 
Thanks,

Steve

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Justin Piszcz]

On Fri, 26 Aug 2011, Steve French wrote:

> Fails with Jeff's new 20+ patch series?

Hi,

No, wanted to make sure it still mounted with 3.1-rc2+previous patches before 
doing anything, there have been quite a few patches applied recently on the
Windows host (updates) I'll need to check with the owner tomorrow; I want 
to ensure I can mount as before before applying Jeff's patches.

Justin.

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Jeff Layton]

On Fri, 26 Aug 2011 19:16:28 -0400 (EDT)
Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:

> 
> 
> On Fri, 26 Aug 2011, Steve French wrote:
> 
> > Fails with Jeff's new 20+ patch series?
> 
> Hi,
> 
> No, wanted to make sure it still mounted with 3.1-rc2+previous patches before 
> doing anything, there have been quite a few patches applied recently on the
> Windows host (updates) I'll need to check with the owner tomorrow; I want 
> to ensure I can mount as before before applying Jeff's patches.
> 
> Justin.
> 

Doubtful that it's related to your -ENOMEM problem, but I have found a
couple of problems with the async read patchset. You may want to wait on
testing it just yet. I've fixed a couple of the problems, but there's
one I'm still hunting down.

Steve, you may want to wait on merging any of it until I understand
what the problem actually is. I'll re-post the set once I get it
straightened out.

Thanks,
-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Jeff Layton]

On Thu, 18 Aug 2011 12:25:50 -0500
Steve French <smfrench-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> On Thu, Aug 18, 2011 at 12:16 PM, Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
> >
> >
> > On Thu, 18 Aug 2011, Jeff Layton wrote:
> >
> >> On Thu, 18 Aug 2011 09:15:36 -0400 (EDT)
> >> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
> >>
> >>>
> >>>
> >>> On Thu, 18 Aug 2011, Jeff Layton wrote:
> >>>
> >>>> On Thu, 18 Aug 2011 08:22:44 -0400 (EDT)
> >>>> Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
> >>>>
> >>>>> Justin.
> >>>>>
> >>>>
> >>>> To be clear -- incoming in this case is reads or writes?
> >>>
> >>> Reading from the CIFS share (Windows 7).
> >>>
> >>>>
> >>>> Up until 3.0 cifs.ko didn't parallelize writes from a single thread. In
> >>>> 3.0 I added a patchset to increase the allowable wsize and to allow the
> >>>> kernel to issue writes in parallel.
> >>>
> >>> Ahh, good to know, have not tried writes yet.
> >>>
> >>>>
> >>>> Reads still suffer from the same problem however. I'm working on a
> >>>> patchset that should do the same thing for them, but it requires a
> >>>> fairly substantial overhaul of the receive codepaths.
> >>>
> >>> Ok, that explains it then, thanks.
> >
> >
> > Hi,
> >
> > Watching the rsync, it ran for a while, then:
> >
> > rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot
> > allocate memory (12)
> > rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_1.JPG": Cannot
> > allocate memory (12)
> > rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_2.JPG": Cannot
> > allocate memory (12)
> > rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot
> > allocate memory (12)
> 
> When we were testing async write to Windows 7 Pavel mentioned to me
> another WIndows 7 bug - which may be what you are hitting.
> 
> Under stress of simultaneous operations, Windows 7 server will sometimes start
> responding with STATUS_INSUFF_SERVER_RESOURCES error code
> (mapped to posix error ENOMEM by the Linux cifs kernel client)
> He solved it by setting MaxWorkItems to 4096 in the Windows 7 registry.
> 
> If anyone knows whether Microsoft has fixed this or has a bug #, let us know
> because it is easier to hit with Linux kernel 3.0 and later (to
> Windows 7 server).
> 

You may be right, but I'd probably suggest doing a bit of debugging
before assuming that that is the problem. Sniffing with wireshark
should help determine if that is the cause.

Assuming that that is the case...

Since all of these seem to be barfing on open(), I wonder whether you're
hitting some limit on the number of open filehandles? Not sure how we
can reasonably deal with that if so...

On a semi-related note. Steve, how goes the patch to make cifs respect
the MaxMpx advertised by the server? That may be at least part of the
issue here.

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Kong Li]

http://support.microsoft.com/kb/307257

On Thu, Aug 18, 2011 at 10:25 AM, Steve French <smfrench@gmail.com> wrote:
> On Thu, Aug 18, 2011 at 12:16 PM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>>
>>
>> On Thu, 18 Aug 2011, Jeff Layton wrote:
>>
>>> On Thu, 18 Aug 2011 09:15:36 -0400 (EDT)
>>> Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>>>
>>>>
>>>>
>>>> On Thu, 18 Aug 2011, Jeff Layton wrote:
>>>>
>>>>> On Thu, 18 Aug 2011 08:22:44 -0400 (EDT)
>>>>> Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>>>>>
>>>>>> Justin.
>>>>>>
>>>>>
>>>>> To be clear -- incoming in this case is reads or writes?
>>>>
>>>> Reading from the CIFS share (Windows 7).
>>>>
>>>>>
>>>>> Up until 3.0 cifs.ko didn't parallelize writes from a single thread. In
>>>>> 3.0 I added a patchset to increase the allowable wsize and to allow the
>>>>> kernel to issue writes in parallel.
>>>>
>>>> Ahh, good to know, have not tried writes yet.
>>>>
>>>>>
>>>>> Reads still suffer from the same problem however. I'm working on a
>>>>> patchset that should do the same thing for them, but it requires a
>>>>> fairly substantial overhaul of the receive codepaths.
>>>>
>>>> Ok, that explains it then, thanks.
>>
>>
>> Hi,
>>
>> Watching the rsync, it ran for a while, then:
>>
>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot
>> allocate memory (12)
>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_1.JPG": Cannot
>> allocate memory (12)
>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_2.JPG": Cannot
>> allocate memory (12)
>> rsync: send_files failed to open "/cifs/w1/r1/data/hs12/f4_0.JPG": Cannot
>> allocate memory (12)
>
> When we were testing async write to Windows 7 Pavel mentioned to me
> another WIndows 7 bug - which may be what you are hitting.
>
> Under stress of simultaneous operations, Windows 7 server will sometimes start
> responding with STATUS_INSUFF_SERVER_RESOURCES error code
> (mapped to posix error ENOMEM by the Linux cifs kernel client)
> He solved it by setting MaxWorkItems to 4096 in the Windows 7 registry.
>
> If anyone knows whether Microsoft has fixed this or has a bug #, let us know
> because it is easier to hit with Linux kernel 3.0 and later (to
> Windows 7 server).
>
>
>
> --
> Thanks,
>
> Steve
> --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[J. R. Okajima]

Justin Piszcz:
> > It has not crashed yet (which is good), I'll apply this patch to my
> > production machine and test taking backups of this data and let you know
> > if it crashes again, thanks!
	:::
> It is working but very slowly:

As you might know, the patch is totally unrelated to the performance. It
just fixes a memory corruption at unmounting or when an error in mounting.


> you have to transfer 5-10TB.  However, it is not crashing anymore, so any 
> speed is better than that :)

But you are not satisifed obviously. :-)


J. R. Okajima

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Steve French]

Writing from cifs kernel client to WIndows or Samba server should be much faster
than the reverse (ie large file sequential file copy to server is
faster than copying
a file from the server)  cifs kernel client serializes reads from the same file
(unless mounting forcedirectio, in which case caching is disabled) and uses
a relatively smaller read size (16K) - while for writes they are sent
in parallel
even if to the same file (and the write size is much larger e.g. 126976 bytes,
and can be set even larger to Samba).  There may be a few cases (such
as copying to WIndowsXP or Windows7) where timeouts on the
server slow things down (writing from linux client to Windows XP
or Windows 7) but what is the server type?


On Thu, Aug 18, 2011 at 7:14 AM, Justin Piszcz <jpiszcz-BP4nVm5VUdNhbmWW9KSYcQ@public.gmane.org> wrote:
>
>
> On Thu, 18 Aug 2011, Justin Piszcz wrote:
>
>>
>>
>> On Thu, 18 Aug 2011, J. R. Okajima wrote:
>>
>>>
>>> Justin Piszcz:
>>>>
>>>> Does anyone know if any kernel supports CIFS w/out crashing? I'd like to
>>>> backup some CIFS shares, thanks.
>>>>
>>>>
>>>> mount -t cifs //w2/x /mnt -o user=user,pass=pass
>>>>
>>>> [  881.388836] CIFS VFS: cifs_mount failed w/return code = -22
>>>
>>>        :::
>>>
>>> Since it failed mounting, this patch will help you. Although the patch
>>> will fix one bug, there still may exist another problem.
>>>
>>> http://marc.info/?l=linux-cifs&m=131345112022031&w=2
>>
>> Hi,
>>
>> Latest patch (this one) applied to linux-3.1-rc2 works, at least it
>> mounted
>> this time and did not instantly crash the kernel!
>>
>> I also tried the hostname again (and it did not crash the kernel, but it
>> failed to mount).
>>
>> Used the IP and it mounted successfully:
>> //10.0.0.11/x          28T  5.0T   23T  19% /mnt
>> //10.0.0.11/y          19T  1.2T   18T   7% /mnt2
>>
>> It has not crashed yet (which is good), I'll apply this patch to my
>> production machine and test taking backups of this data and let you know
>> if it crashes again, thanks!
>>
>> Justin.
>
>
> Hello,
>
> It is working but very slowly:
>
> Device eth6 [10.0.1.2] (1/1):
> ================================================================================
> Incoming:                               Outgoing:
> Curr: 37.60 MByte/s                     Curr: 0.44 MByte/s
> Avg: 4.98 MByte/s                       Avg: 0.09 MByte/s
> Min: 0.00 MByte/s                       Min: 0.00 MByte/s
> Max: 40.79 MByte/s                      Max: 0.48 MByte/s
> Ttl: 1.45 GByte                         Ttl: 26.77 MByte
>
> Over 10GbE the other direction (Linux -> Windows (via Samba)) I get
> 500MiB/s, is CIFS slow?
>
> I'll look into options to tweak the speed but this is very poor speed when
> you have to transfer 5-10TB.  However, it is not crashing anymore, so any
> speed is better than that :)
>
> Justin.
>
>



-- 
Thanks,

Steve

Re: Kernel 3.0: Instant kernel crash when mounting CIFS (also crashes with linux-3.1-rc2

[Steve French]

I did some testing with 3.0 kernel (cifs client) and was getting 90%
wire speeds writing
cifs files to the server (Windows or Samba)

On Thu, Aug 18, 2011 at 11:01 AM, Steve French <smfrench-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> Writing from cifs kernel client to WIndows or Samba server should be much faster
> than the reverse (ie large file sequential file copy to server is
> faster than copying
> a file from the server)  cifs kernel client serializes reads from the same file
> (unless mounting forcedirectio, in which case caching is disabled) and uses
> a relatively smaller read size (16K) - while for writes they are sent
> in parallel
> even if to the same file (and the write size is much larger e.g. 126976 bytes,
> and can be set even larger to Samba).  There may be a few cases (such
> as copying to WIndowsXP or Windows7) where timeouts on the
> server slow things down (writing from linux client to Windows XP
> or Windows 7) but what is the server type?
>
>
> On Thu, Aug 18, 2011 at 7:14 AM, Justin Piszcz <jpiszcz@lucidpixels.com> wrote:
>>
>>
>> On Thu, 18 Aug 2011, Justin Piszcz wrote:
>>
>>>
>>>
>>> On Thu, 18 Aug 2011, J. R. Okajima wrote:
>>>
>>>>
>>>> Justin Piszcz:
>>>>>
>>>>> Does anyone know if any kernel supports CIFS w/out crashing? I'd like to
>>>>> backup some CIFS shares, thanks.
>>>>>
>>>>>
>>>>> mount -t cifs //w2/x /mnt -o user=user,pass=pass
>>>>>
>>>>> [  881.388836] CIFS VFS: cifs_mount failed w/return code = -22
>>>>
>>>>        :::
>>>>
>>>> Since it failed mounting, this patch will help you. Although the patch
>>>> will fix one bug, there still may exist another problem.
>>>>
>>>> http://marc.info/?l=linux-cifs&m=131345112022031&w=2
>>>
>>> Hi,
>>>
>>> Latest patch (this one) applied to linux-3.1-rc2 works, at least it
>>> mounted
>>> this time and did not instantly crash the kernel!
>>>
>>> I also tried the hostname again (and it did not crash the kernel, but it
>>> failed to mount).
>>>
>>> Used the IP and it mounted successfully:
>>> //10.0.0.11/x          28T  5.0T   23T  19% /mnt
>>> //10.0.0.11/y          19T  1.2T   18T   7% /mnt2
>>>
>>> It has not crashed yet (which is good), I'll apply this patch to my
>>> production machine and test taking backups of this data and let you know
>>> if it crashes again, thanks!
>>>
>>> Justin.
>>
>>
>> Hello,
>>
>> It is working but very slowly:
>>
>> Device eth6 [10.0.1.2] (1/1):
>> ================================================================================
>> Incoming:                               Outgoing:
>> Curr: 37.60 MByte/s                     Curr: 0.44 MByte/s
>> Avg: 4.98 MByte/s                       Avg: 0.09 MByte/s
>> Min: 0.00 MByte/s                       Min: 0.00 MByte/s
>> Max: 40.79 MByte/s                      Max: 0.48 MByte/s
>> Ttl: 1.45 GByte                         Ttl: 26.77 MByte
>>
>> Over 10GbE the other direction (Linux -> Windows (via Samba)) I get
>> 500MiB/s, is CIFS slow?
>>
>> I'll look into options to tweak the speed but this is very poor speed when
>> you have to transfer 5-10TB.  However, it is not crashing anymore, so any
>> speed is better than that :)
>>
>> Justin.
>>
>>
>
>
>
> --
> Thanks,
>
> Steve
>



-- 
Thanks,

Steve