* issues with CIFS on Ubuntu 20 mounting Windows 2019
@ 2020-05-01 4:46 Scott M. Lewandowski
[not found] ` <CAH2r5msOBGS7QYagUkby4upxFKbxCdQg=T65K4s9VhRKkWY+Xw@mail.gmail.com>
2020-05-04 10:11 ` Aurélien Aptel
0 siblings, 2 replies; 10+ messages in thread
From: Scott M. Lewandowski @ 2020-05-01 4:46 UTC (permalink / raw)
To: linux-cifs@vger.kernel.org
I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file):
//file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,nounix,iocharset=utf8 0 0
My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below.
Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg:
[ 179.625296] SMB2_read: 9 callbacks suppressed
[ 179.625298] CIFS VFS: Send error in read = -4
[ 179.704191] CIFS VFS: Send error in read = -4
[ 179.847514] CIFS VFS: Send error in read = -4
[ 180.031516] CIFS VFS: Send error in read = -4
[ 180.214848] CIFS VFS: Send error in read = -4
[ 180.282909] CIFS VFS: Send error in read = -4
[ 180.474829] CIFS VFS: Send error in read = -4
[ 180.621349] CIFS VFS: Send error in read = -4
[ 180.780465] CIFS VFS: Send error in read = -4
[ 180.823249] CIFS VFS: Send error in read = -4
[ 182.339351] CIFS VFS: Close unmatched open
[ 182.726829] CIFS VFS: Close unmatched open
[ 184.292663] CIFS VFS: Close unmatched open
[ 184.501185] CIFS VFS: Close unmatched open
[ 191.784922] SMB2_read: 4 callbacks suppressed
[ 191.784925] CIFS VFS: Send error in read = -4
[ 191.884186] CIFS VFS: Send error in read = -4
[ 192.270851] CIFS VFS: Send error in read = -4
[ 192.323467] CIFS VFS: Send error in read = -4
[ 192.381336] CIFS VFS: Send error in read = -4
[ 192.446711] CIFS VFS: Send error in read = -4
[ 192.507605] CIFS VFS: Send error in read = -4
[ 193.244397] CIFS VFS: Send error in read = -4
[ 193.307588] CIFS VFS: Send error in read = -4
[ 193.403410] CIFS VFS: Send error in read = -4
[ 193.797038] CIFS VFS: Close unmatched open
[ 196.784082] CIFS VFS: Close unmatched open
[ 198.247594] CIFS VFS: Close unmatched open
Here is the duplicate cookie info:
[ 21.713307] FS-Cache: Duplicate cookie detected
[ 21.713840] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af'
[ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1]
[ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5
[ 21.716141] FS-Cache: N-key=[8] '020001bdc0a801af'
[ 21.716591] FS-Cache: Duplicate cookie detected
[ 21.717217] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af'
[ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1]
[ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917
[ 21.719621] FS-Cache: N-key=[8] '020001bdc0a801af'
[ 21.720097] FS-Cache: Duplicate cookie detected
[ 21.720565] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1]
[ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c
[ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af'
[ 21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 fl=2 nc=0 na=1]
[ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa
[ 21.722941] FS-Cache: N-key=[8] '020001bdc0a801af'
Any ideas what could be going on?
Thanks for any help!
DebugData:
CIFS Version 2.23
Features: DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,CIFS_POSIX,UPCALL(SPNEGO),XATTR,ACL
CIFSMaxBufSize: 16384
Active VFS Requests: 0
Servers:
Number of credits: 8190 Dialect 0x300
1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x100080000021
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175
Number of credits: 8190 Dialect 0x300
2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000019
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175
3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: 0x10007c000061
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489
State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175
Number of credits: 8190 Dialect 0x300
4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1
Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000055
Shares:
0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0
PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0
Share Capabilities: None Share Flags: 0x30
tid: 0x1 Maximal Access: 0x11f01ff
1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df
PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0
Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0
tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff
MIDs:
Server interfaces: 2
0) Speed: 10000000000 bps
Capabilities:
IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806
1) Speed: 10000000000 bps
Capabilities:
IPv4: 192.168.1.175
^ permalink raw reply [flat|nested] 10+ messages in thread[parent not found: <CAH2r5msOBGS7QYagUkby4upxFKbxCdQg=T65K4s9VhRKkWY+Xw@mail.gmail.com>]
* RE: issues with CIFS on Ubuntu 20 mounting Windows 2019 [not found] ` <CAH2r5msOBGS7QYagUkby4upxFKbxCdQg=T65K4s9VhRKkWY+Xw@mail.gmail.com> @ 2020-05-01 5:18 ` Scott M. Lewandowski 2020-05-01 15:38 ` Scott M. Lewandowski 2020-05-02 6:10 ` Steve French 0 siblings, 2 replies; 10+ messages in thread From: Scott M. Lewandowski @ 2020-05-01 5:18 UTC (permalink / raw) To: Steve French; +Cc: CIFS > EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads Not using scp at the moment. > Could an app be sending signals trying to kill the process doing the reads? Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it. One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105 -scott From: Steve French <smfrench@gmail.com> Sent: Friday, May 1, 2020 12:07 AM To: Scott M. Lewandowski <vgerkernel@scottl.com> Cc: CIFS <linux-cifs@vger.kernel.org> Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:vgerkernel@scottl.com> wrote: I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file): //file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,nounix,iocharset=utf8 0 0 My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below. Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg: [ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS: Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4 [ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFS VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS: Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ 184.501185] CIFS VFS: Close unmatched open [ 191.784922] SMB2_read: 4 callbacks suppressed [ 191.784925] CIFS VFS: Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4 [ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFS VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS: Close unmatched open [ 198.247594] CIFS VFS: Close unmatched open Here is the duplicate cookie info: [ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8] '020001bdc0a801af' [ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8] '020001bdc0a801af' [ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' [ 21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] '020001bdc0a801af' Any ideas what could be going on? Thanks for any help! DebugData: CIFS Version 2.23 Features: DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,CIFS_POSIX,UPCALL(SPNEGO),XATTR,ACL CIFSMaxBufSize: 16384 Active VFS Requests: 0 Servers: Number of credits: 8190 Dialect 0x300 1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x100080000021 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175 Number of credits: 8190 Dialect 0x300 2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000019 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175 3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: 0x10007c000061 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553 Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175 Number of credits: 8190 Dialect 0x300 4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000055 Shares: 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 Share Capabilities: None Share Flags: 0x30 tid: 0x1 Maximal Access: 0x11f01ff 1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff MIDs: Server interfaces: 2 0) Speed: 10000000000 bps Capabilities: IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 1) Speed: 10000000000 bps Capabilities: IPv4: 192.168.1.175 ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: issues with CIFS on Ubuntu 20 mounting Windows 2019 2020-05-01 5:18 ` Scott M. Lewandowski @ 2020-05-01 15:38 ` Scott M. Lewandowski 2020-05-01 15:56 ` Steve French 2020-05-02 6:10 ` Steve French 1 sibling, 1 reply; 10+ messages in thread From: Scott M. Lewandowski @ 2020-05-01 15:38 UTC (permalink / raw) To: Scott M. Lewandowski, Steve French; +Cc: CIFS I’m also seeing entries like the following mixed in with the other messages: [27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd: 5 And [31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate inode 00000000b4733e96 > -----Original Message----- > From: Scott M. Lewandowski <vgerkernel@scottl.com> > Sent: Friday, May 1, 2020 12:20 AM > To: 'Steve French' <smfrench@gmail.com> > Cc: CIFS <linux-cifs@vger.kernel.org> > Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > EINTR on read ... Hundreds of times?! I have seen scp do that by > > sending signals but in writes not reads > > Not using scp at the moment. > > > Could an app be sending signals trying to kill the process doing the reads? > > Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look > into it. > > One other thing that came to mind that may be helpful is that one of the > processes doing the most IO is running under mono 6.8.0.105 > > -scott > > > > > From: Steve French <smfrench@gmail.com> > Sent: Friday, May 1, 2020 12:07 AM > To: Scott M. Lewandowski <vgerkernel@scottl.com> > Cc: CIFS <linux-cifs@vger.kernel.org> > Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > EINTR on read ... Hundreds of times?! I have seen scp do that by sending > signals but in writes not reads > > On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski > <mailto:vgerkernel@scottl.com> wrote: > I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. > Both boxes are fully patched. They are resident on the same hypervisor and > communicating directly via a vSwitch. I have 4 shares mounted; here is an > example from my fstab (yes, I know I should move the credential file): > > //file1/usenet /mnt/usenet cifs > vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,n > ounix,iocharset=utf8 0 0 > > My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of > DebugData are included below. > > Right after the mount, I see a duplicate cookie reported. It is unclear if this is > a problem or not. Following that, I periodically see messages similar to the > following repeated in dmesg: > > [ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS: > Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 > [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send > error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 > [ 180.282909] CIFS VFS: Send error in read = -4 [ 180.474829] CIFS VFS: Send > error in read = -4 [ 180.621349] CIFS VFS: Send error in read = -4 > [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send > error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open > [ 182.726829] CIFS VFS: Close unmatched open [ 184.292663] CIFS VFS: Close > unmatched open [ 184.501185] CIFS VFS: Close unmatched open > [ 191.784922] SMB2_read: 4 callbacks suppressed [ 191.784925] CIFS VFS: > Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 > [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send > error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 > [ 192.446711] CIFS VFS: Send error in read = -4 [ 192.507605] CIFS VFS: Send > error in read = -4 [ 193.244397] CIFS VFS: Send error in read = -4 > [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send > error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open > [ 196.784082] CIFS VFS: Close unmatched open [ 198.247594] CIFS VFS: Close > unmatched open > > Here is the duplicate cookie info: > > [ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS-Cache: O- > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c > [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' > [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 > fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee > n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8] '020001bdc0a801af' > [ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS-Cache: O- > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c > [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' > [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 > fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee > n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8] '020001bdc0a801af' > [ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS-Cache: O- > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c > [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' > [ 21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 > fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee > n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] '020001bdc0a801af' > > Any ideas what could be going on? > > Thanks for any help! > > > > > > > DebugData: > CIFS Version 2.23 > Features: > DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C > IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL > CIFSMaxBufSize: 16384 > Active VFS Requests: 0 > Servers: > Number of credits: 8190 Dialect 0x300 > 1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP > status: 1 Instance: 1 > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > 0x100080000021 > Shares: > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > Share Capabilities: None Share Flags: 0x30 > tid: 0x1 Maximal Access: 0x11f01ff > > 1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > MIDs: > > Server interfaces: 2 > 0) Speed: 10000000000 bps > Capabilities: > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > 1) Speed: 10000000000 bps > Capabilities: > IPv4: 192.168.1.175 > Number of credits: 8190 Dialect 0x300 > > 2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP > status: 1 Instance: 1 > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > 0x10006c000019 > Shares: > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > Share Capabilities: None Share Flags: 0x30 > tid: 0x1 Maximal Access: 0x11f01ff > > 1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > MIDs: > > Server interfaces: 2 > 0) Speed: 10000000000 bps > Capabilities: > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > 1) Speed: 10000000000 bps > Capabilities: > IPv4: 192.168.1.175 > > > 3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP > status: 1 Instance: 1 > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: > 0x10007c000061 > Shares: > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > Share Capabilities: None Share Flags: 0x30 > tid: 0x1 Maximal Access: 0x11f01ff > > 1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > MIDs: > State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 > State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553 > > Server interfaces: 2 > 0) Speed: 10000000000 bps > Capabilities: > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > 1) Speed: 10000000000 bps > Capabilities: > IPv4: 192.168.1.175 > > Number of credits: 8190 Dialect 0x300 > 4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP > status: 1 Instance: 1 > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > 0x10006c000055 > Shares: > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > Share Capabilities: None Share Flags: 0x30 > tid: 0x1 Maximal Access: 0x11f01ff > > 1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > MIDs: > > Server interfaces: 2 > 0) Speed: 10000000000 bps > Capabilities: > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > 1) Speed: 10000000000 bps > Capabilities: > IPv4: 192.168.1.175 ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 2020-05-01 15:38 ` Scott M. Lewandowski @ 2020-05-01 15:56 ` Steve French 2020-05-01 18:19 ` Scott M. Lewandowski 2020-05-01 19:37 ` Scott M. Lewandowski 0 siblings, 2 replies; 10+ messages in thread From: Steve French @ 2020-05-01 15:56 UTC (permalink / raw) To: Scott M. Lewandowski; +Cc: Scott M. Lewandowski, CIFS Are you comfortable with dynamic tracing ("trace-cmd record -e ..." ... the list of trace points can be seen in /sys/kernel/debug/tracing/events/cifs)? On Fri, May 1, 2020 at 10:40 AM Scott M. Lewandowski <vgerkernel@scottl.com> wrote: > > I’m also seeing entries like the following mixed in with the other messages: > > [27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd: 5 > > And > > [31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate inode 00000000b4733e96 > > > > > > -----Original Message----- > > From: Scott M. Lewandowski <vgerkernel@scottl.com> > > Sent: Friday, May 1, 2020 12:20 AM > > To: 'Steve French' <smfrench@gmail.com> > > Cc: CIFS <linux-cifs@vger.kernel.org> > > Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > > > EINTR on read ... Hundreds of times?! I have seen scp do that by > > > sending signals but in writes not reads > > > > Not using scp at the moment. > > > > > Could an app be sending signals trying to kill the process doing the reads? > > > > Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look > > into it. > > > > One other thing that came to mind that may be helpful is that one of the > > processes doing the most IO is running under mono 6.8.0.105 > > > > -scott > > > > > > > > > > From: Steve French <smfrench@gmail.com> > > Sent: Friday, May 1, 2020 12:07 AM > > To: Scott M. Lewandowski <vgerkernel@scottl.com> > > Cc: CIFS <linux-cifs@vger.kernel.org> > > Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > > EINTR on read ... Hundreds of times?! I have seen scp do that by sending > > signals but in writes not reads > > > > On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski > > <mailto:vgerkernel@scottl.com> wrote: > > I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. > > Both boxes are fully patched. They are resident on the same hypervisor and > > communicating directly via a vSwitch. I have 4 shares mounted; here is an > > example from my fstab (yes, I know I should move the credential file): > > > > //file1/usenet /mnt/usenet cifs > > vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,n > > ounix,iocharset=utf8 0 0 > > > > My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of > > DebugData are included below. > > > > Right after the mount, I see a duplicate cookie reported. It is unclear if this is > > a problem or not. Following that, I periodically see messages similar to the > > following repeated in dmesg: > > > > [ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS: > > Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read = -4 > > [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] CIFS VFS: Send > > error in read = -4 [ 180.214848] CIFS VFS: Send error in read = -4 > > [ 180.282909] CIFS VFS: Send error in read = -4 [ 180.474829] CIFS VFS: Send > > error in read = -4 [ 180.621349] CIFS VFS: Send error in read = -4 > > [ 180.780465] CIFS VFS: Send error in read = -4 [ 180.823249] CIFS VFS: Send > > error in read = -4 [ 182.339351] CIFS VFS: Close unmatched open > > [ 182.726829] CIFS VFS: Close unmatched open [ 184.292663] CIFS VFS: Close > > unmatched open [ 184.501185] CIFS VFS: Close unmatched open > > [ 191.784922] SMB2_read: 4 callbacks suppressed [ 191.784925] CIFS VFS: > > Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read = -4 > > [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] CIFS VFS: Send > > error in read = -4 [ 192.381336] CIFS VFS: Send error in read = -4 > > [ 192.446711] CIFS VFS: Send error in read = -4 [ 192.507605] CIFS VFS: Send > > error in read = -4 [ 193.244397] CIFS VFS: Send error in read = -4 > > [ 193.307588] CIFS VFS: Send error in read = -4 [ 193.403410] CIFS VFS: Send > > error in read = -4 [ 193.797038] CIFS VFS: Close unmatched open > > [ 196.784082] CIFS VFS: Close unmatched open [ 198.247594] CIFS VFS: Close > > unmatched open > > > > Here is the duplicate cookie info: > > > > [ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS-Cache: O- > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c > > [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' > > [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 > > fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee > > n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8] '020001bdc0a801af' > > [ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS-Cache: O- > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c > > [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' > > [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 > > fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee > > n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8] '020001bdc0a801af' > > [ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS-Cache: O- > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c > > [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' > > [ 21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 > > fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee > > n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] '020001bdc0a801af' > > > > Any ideas what could be going on? > > > > Thanks for any help! > > > > > > > > > > > > > > DebugData: > > CIFS Version 2.23 > > Features: > > DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C > > IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL > > CIFSMaxBufSize: 16384 > > Active VFS Requests: 0 > > Servers: > > Number of credits: 8190 Dialect 0x300 > > 1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP > > status: 1 Instance: 1 > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > 0x100080000021 > > Shares: > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > Share Capabilities: None Share Flags: 0x30 > > tid: 0x1 Maximal Access: 0x11f01ff > > > > 1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > MIDs: > > > > Server interfaces: 2 > > 0) Speed: 10000000000 bps > > Capabilities: > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > 1) Speed: 10000000000 bps > > Capabilities: > > IPv4: 192.168.1.175 > > Number of credits: 8190 Dialect 0x300 > > > > 2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP > > status: 1 Instance: 1 > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > 0x10006c000019 > > Shares: > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > Share Capabilities: None Share Flags: 0x30 > > tid: 0x1 Maximal Access: 0x11f01ff > > > > 1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > MIDs: > > > > Server interfaces: 2 > > 0) Speed: 10000000000 bps > > Capabilities: > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > 1) Speed: 10000000000 bps > > Capabilities: > > IPv4: 192.168.1.175 > > > > > > 3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP > > status: 1 Instance: 1 > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: > > 0x10007c000061 > > Shares: > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > Share Capabilities: None Share Flags: 0x30 > > tid: 0x1 Maximal Access: 0x11f01ff > > > > 1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > MIDs: > > State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 > > State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553 > > > > Server interfaces: 2 > > 0) Speed: 10000000000 bps > > Capabilities: > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > 1) Speed: 10000000000 bps > > Capabilities: > > IPv4: 192.168.1.175 > > > > Number of credits: 8190 Dialect 0x300 > > 4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP > > status: 1 Instance: 1 > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > 0x10006c000055 > > Shares: > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > Share Capabilities: None Share Flags: 0x30 > > tid: 0x1 Maximal Access: 0x11f01ff > > > > 1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > MIDs: > > > > Server interfaces: 2 > > 0) Speed: 10000000000 bps > > Capabilities: > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > 1) Speed: 10000000000 bps > > Capabilities: > > IPv4: 192.168.1.175 -- Thanks, Steve ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: issues with CIFS on Ubuntu 20 mounting Windows 2019 2020-05-01 15:56 ` Steve French @ 2020-05-01 18:19 ` Scott M. Lewandowski [not found] ` <CAH2r5mvscNft+arZuzzupNeB7df+c7fUmj=XRNZ5Y6PEts+Efw@mail.gmail.com> 2020-05-01 19:37 ` Scott M. Lewandowski 1 sibling, 1 reply; 10+ messages in thread From: Scott M. Lewandowski @ 2020-05-01 18:19 UTC (permalink / raw) To: Steve French; +Cc: CIFS I'm trying to become comfortable now 😊 I have it setup to capture all CIFS events, but I haven't seen the messages in dmesg since. I'm not sure if I've just gotten unlucky, or if the instrumentation is changing the behavior. > -----Original Message----- > From: Steve French <smfrench@gmail.com> > Sent: Friday, May 1, 2020 10:57 AM > To: Scott M. Lewandowski <vgerkernel@scottl.com> > Cc: Scott M. Lewandowski <scott@wynstonegroup.com>; CIFS <linux- > cifs@vger.kernel.org> > Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > Are you comfortable with dynamic tracing ("trace-cmd record -e ..." > ... the list of trace points can be seen in > /sys/kernel/debug/tracing/events/cifs)? > > On Fri, May 1, 2020 at 10:40 AM Scott M. Lewandowski > <vgerkernel@scottl.com> wrote: > > > > I’m also seeing entries like the following mixed in with the other messages: > > > > [27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd: > > 5 > > > > And > > > > [31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate > > inode 00000000b4733e96 > > > > > > > > > > > -----Original Message----- > > > From: Scott M. Lewandowski <vgerkernel@scottl.com> > > > Sent: Friday, May 1, 2020 12:20 AM > > > To: 'Steve French' <smfrench@gmail.com> > > > Cc: CIFS <linux-cifs@vger.kernel.org> > > > Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > > > > > EINTR on read ... Hundreds of times?! I have seen scp do that by > > > > sending signals but in writes not reads > > > > > > Not using scp at the moment. > > > > > > > Could an app be sending signals trying to kill the process doing the > reads? > > > > > > Not that I'm aware of. I'm not sure exactly how I'd check that, but > > > I will look into it. > > > > > > One other thing that came to mind that may be helpful is that one of > > > the processes doing the most IO is running under mono 6.8.0.105 > > > > > > -scott > > > > > > > > > > > > > > > From: Steve French <smfrench@gmail.com> > > > Sent: Friday, May 1, 2020 12:07 AM > > > To: Scott M. Lewandowski <vgerkernel@scottl.com> > > > Cc: CIFS <linux-cifs@vger.kernel.org> > > > Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > > > > EINTR on read ... Hundreds of times?! I have seen scp do that by > > > sending signals but in writes not reads > > > > > > On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski > > > <mailto:vgerkernel@scottl.com> wrote: > > > I have an Ubuntu 20.04 box mounting shares from a Windows 2019 > server. > > > Both boxes are fully patched. They are resident on the same > > > hypervisor and communicating directly via a vSwitch. I have 4 shares > > > mounted; here is an example from my fstab (yes, I know I should move > the credential file): > > > > > > //file1/usenet /mnt/usenet cifs > > > vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri > > > no,n > > > ounix,iocharset=utf8 0 0 > > > > > > My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The > > > contents of DebugData are included below. > > > > > > Right after the mount, I see a duplicate cookie reported. It is > > > unclear if this is a problem or not. Following that, I periodically > > > see messages similar to the following repeated in dmesg: > > > > > > [ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS: > > > Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read > > > = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] > > > CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send > > > error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4 > > > [ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFS > > > VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in > > > read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ > > > 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS: > > > Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ > > > 184.501185] CIFS VFS: Close unmatched open [ 191.784922] SMB2_read: > 4 callbacks suppressed [ 191.784925] CIFS VFS: > > > Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read > > > = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] > > > CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send > > > error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4 > > > [ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFS > > > VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in > > > read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ > > > 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS: > > > Close unmatched open [ 198.247594] CIFS VFS: Close unmatched open > > > > > > Here is the duplicate cookie info: > > > > > > [ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS- > Cache: O- > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee > n=00000000243f210c > > > [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' > > > [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c > [p=0000000079ef0365 > > > fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee > > > n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8] > '020001bdc0a801af' > > > [ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS- > Cache: O- > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee > n=00000000243f210c > > > [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' > > > [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 > [p=0000000079ef0365 > > > fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee > > > n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8] > '020001bdc0a801af' > > > [ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS- > Cache: O- > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee > n=00000000243f210c > > > [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' > > > [ 21.721994] FS-Cache: N-cookie c=00000000d479c294 > [p=0000000079ef0365 > > > fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee > > > n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] > '020001bdc0a801af' > > > > > > Any ideas what could be going on? > > > > > > Thanks for any help! > > > > > > > > > > > > > > > > > > > > > DebugData: > > > CIFS Version 2.23 > > > Features: > > > > DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C > > > IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL > > > CIFSMaxBufSize: 16384 > > > Active VFS Requests: 0 > > > Servers: > > > Number of credits: 8190 Dialect 0x300 > > > 1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > 0x100080000021 > > > Shares: > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > Number of credits: 8190 Dialect 0x300 > > > > > > 2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > 0x10006c000019 > > > Shares: > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > > > > > > > 3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: > > > 0x10007c000061 > > > Shares: > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 > > > State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid > > > 1211553 > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > > > > Number of credits: 8190 Dialect 0x300 > > > 4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > 0x10006c000055 > > > Shares: > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > > -- > Thanks, > > Steve ^ permalink raw reply [flat|nested] 10+ messages in thread
[parent not found: <CAH2r5mvscNft+arZuzzupNeB7df+c7fUmj=XRNZ5Y6PEts+Efw@mail.gmail.com>]
[parent not found: <40d5e83ebbd14de785be3d7ba82ec34a@wynstonegroup.com>]
* RE: issues with CIFS on Ubuntu 20 mounting Windows 2019 [not found] ` <40d5e83ebbd14de785be3d7ba82ec34a@wynstonegroup.com> @ 2020-05-02 20:28 ` Scott M. Lewandowski 0 siblings, 0 replies; 10+ messages in thread From: Scott M. Lewandowski @ 2020-05-02 20:28 UTC (permalink / raw) To: Steve French; +Cc: CIFS For anyone interested, there is an excerpt with the second trace set at https://www.dropbox.com/s/53fewmvsj8w2fau/trace-report-filtered-trimmed?dl=0 (it’s still not exactly succinct so I didn’t want to spam the list with it) I included the relevant dmesg output at the top of the file Steve, thx for your help and thoughts so far! From: Steve French <mailto:smfrench@gmail.com> Sent: Saturday, May 2, 2020 1:08 AM To: Scott M. Lewandowski <mailto:vgerkernel@scottl.com> Cc: CIFS <mailto:linux-cifs@vger.kernel.org> Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 if you have a short repro scenario then in window 1) trace-cmd record -e cifs window 2) repro the bug window 3) (immediately after the repro) trace-cmd show Or if you want to make it more narrow maybe something like "trace-cmd record -e smb3_read* -e smb3_reconnect* -e smb3_partial_send_reconnect" or perhaps a somewhat noisier one: "trace-cmd record -e smb3_read* -e smb3_reconnect* -e smb3_partial_send_reconnect -e smb3_cmd* -e smb3_slow_rsp -e smb3_enter -e smb3_exit*" On Fri, May 1, 2020 at 1:20 PM Scott M. Lewandowski <mailto:vgerkernel@scottl.com> wrote: I'm trying to become comfortable now 😊 I have it setup to capture all CIFS events, but I haven't seen the messages in dmesg since. I'm not sure if I've just gotten unlucky, or if the instrumentation is changing the behavior. > -----Original Message----- > From: Steve French <mailto:smfrench@gmail.com> > Sent: Friday, May 1, 2020 10:57 AM > To: Scott M. Lewandowski <mailto:vgerkernel@scottl.com> > Cc: Scott M. Lewandowski <mailto:scott@wynstonegroup.com>; CIFS <linux- > mailto:cifs@vger.kernel.org> > Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > Are you comfortable with dynamic tracing ("trace-cmd record -e ..." > ... the list of trace points can be seen in > /sys/kernel/debug/tracing/events/cifs)? > > On Fri, May 1, 2020 at 10:40 AM Scott M. Lewandowski > <mailto:vgerkernel@scottl.com> wrote: > > > > I’m also seeing entries like the following mixed in with the other messages: > > > > [27078.665192] CIFS VFS: file://file1 Cancelling wait for mid 13443257 cmd: > > 5 > > > > And > > > > [31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate > > inode 00000000b4733e96 > > > > > > > > > > > -----Original Message----- > > > From: Scott M. Lewandowski <mailto:vgerkernel@scottl.com> > > > Sent: Friday, May 1, 2020 12:20 AM > > > To: 'Steve French' <mailto:smfrench@gmail.com> > > > Cc: CIFS <mailto:linux-cifs@vger.kernel.org> > > > Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > > > > > EINTR on read ... Hundreds of times?! I have seen scp do that by > > > > sending signals but in writes not reads > > > > > > Not using scp at the moment. > > > > > > > Could an app be sending signals trying to kill the process doing the > reads? > > > > > > Not that I'm aware of. I'm not sure exactly how I'd check that, but > > > I will look into it. > > > > > > One other thing that came to mind that may be helpful is that one of > > > the processes doing the most IO is running under mono 6.8.0.105 > > > > > > -scott > > > > > > > > > > > > > > > From: Steve French <mailto:smfrench@gmail.com> > > > Sent: Friday, May 1, 2020 12:07 AM > > > To: Scott M. Lewandowski <mailto:vgerkernel@scottl.com> > > > Cc: CIFS <mailto:linux-cifs@vger.kernel.org> > > > Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > > > > EINTR on read ... Hundreds of times?! I have seen scp do that by > > > sending signals but in writes not reads > > > > > > On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski > > > <mailto:mailto:vgerkernel@scottl.com> wrote: > > > I have an Ubuntu 20.04 box mounting shares from a Windows 2019 > server. > > > Both boxes are fully patched. They are resident on the same > > > hypervisor and communicating directly via a vSwitch. I have 4 shares > > > mounted; here is an example from my fstab (yes, I know I should move > the credential file): > > > > > > //file1/usenet /mnt/usenet cifs > > > vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri > > > no,n > > > ounix,iocharset=utf8 0 0 > > > > > > My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The > > > contents of DebugData are included below. > > > > > > Right after the mount, I see a duplicate cookie reported. It is > > > unclear if this is a problem or not. Following that, I periodically > > > see messages similar to the following repeated in dmesg: > > > > > > [ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS: > > > Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read > > > = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] > > > CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send > > > error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4 > > > [ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFS > > > VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in > > > read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ > > > 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS: > > > Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ > > > 184.501185] CIFS VFS: Close unmatched open [ 191.784922] SMB2_read: > 4 callbacks suppressed [ 191.784925] CIFS VFS: > > > Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read > > > = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] > > > CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send > > > error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4 > > > [ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFS > > > VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in > > > read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ > > > 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS: > > > Close unmatched open [ 198.247594] CIFS VFS: Close unmatched open > > > > > > Here is the duplicate cookie info: > > > > > > [ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS- > Cache: O- > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee > n=00000000243f210c > > > [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' > > > [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c > [p=0000000079ef0365 > > > fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee > > > n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8] > '020001bdc0a801af' > > > [ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS- > Cache: O- > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee > n=00000000243f210c > > > [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' > > > [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 > [p=0000000079ef0365 > > > fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee > > > n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8] > '020001bdc0a801af' > > > [ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS- > Cache: O- > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee > n=00000000243f210c > > > [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' > > > [ 21.721994] FS-Cache: N-cookie c=00000000d479c294 > [p=0000000079ef0365 > > > fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee > > > n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] > '020001bdc0a801af' > > > > > > Any ideas what could be going on? > > > > > > Thanks for any help! > > > > > > > > > > > > > > > > > > > > > DebugData: > > > CIFS Version 2.23 > > > Features: > > > > DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C > > > IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL > > > CIFSMaxBufSize: 16384 > > > Active VFS Requests: 0 > > > Servers: > > > Number of credits: 8190 Dialect 0x300 > > > 1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > 0x100080000021 > > > Shares: > > > 0) IPC: file://file1/IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) file://file1/XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > Number of credits: 8190 Dialect 0x300 > > > > > > 2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > 0x10006c000019 > > > Shares: > > > 0) IPC: file://file1/IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) file://file1/usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > > > > > > > 3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: > > > 0x10007c000061 > > > Shares: > > > 0) IPC: file://file1/IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) file://file1/YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 > > > State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid > > > 1211553 > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > > > > Number of credits: 8190 Dialect 0x300 > > > 4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > 0x10006c000055 > > > Shares: > > > 0) IPC: file://file1/IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) file://file1/ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > > -- > Thanks, > > Steve -- Thanks, Steve ^ permalink raw reply [flat|nested] 10+ messages in thread
* RE: issues with CIFS on Ubuntu 20 mounting Windows 2019 2020-05-01 15:56 ` Steve French 2020-05-01 18:19 ` Scott M. Lewandowski @ 2020-05-01 19:37 ` Scott M. Lewandowski 2020-05-02 6:02 ` Steve French 1 sibling, 1 reply; 10+ messages in thread From: Scott M. Lewandowski @ 2020-05-01 19:37 UTC (permalink / raw) To: Steve French; +Cc: CIFS I've now got a trace that corresponds to the dmesg output below. I will take a look later, but upon quick glance, this doesn't look like anything that will mean much to me, unfortunately. If someone would like to look at the file I can find a way to get it to you. [53223.099221] CIFS VFS: Send error in read = -4 [53223.232373] CIFS VFS: Send error in read = -4 [53254.762914] CIFS VFS: Send error in read = -4 [53301.403925] CIFS VFS: Send error in read = -4 [53301.772568] CIFS VFS: Send error in read = -4 [53301.818566] CIFS VFS: Send error in read = -4 [53301.885510] CIFS VFS: Send error in read = -4 [53302.055959] CIFS VFS: Send error in read = -4 [53302.146337] CIFS VFS: Send error in read = -4 [53302.236322] CIFS VFS: Send error in read = -4 [53302.329525] CIFS VFS: Send error in read = -4 [53303.000869] CIFS VFS: Send error in read = -4 [53303.106961] CIFS VFS: Send error in read = -4 [53304.756998] CIFS VFS: Close unmatched open [53670.776832] CIFS VFS: Close unmatched open > -----Original Message----- > From: Steve French <smfrench@gmail.com> > Sent: Friday, May 1, 2020 10:57 AM > To: Scott M. Lewandowski <vgerkernel@scottl.com> > Cc: Scott M. Lewandowski <scott@wynstonegroup.com>; CIFS <linux- > cifs@vger.kernel.org> > Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > Are you comfortable with dynamic tracing ("trace-cmd record -e ..." > ... the list of trace points can be seen in > /sys/kernel/debug/tracing/events/cifs)? > > On Fri, May 1, 2020 at 10:40 AM Scott M. Lewandowski > <vgerkernel@scottl.com> wrote: > > > > I’m also seeing entries like the following mixed in with the other messages: > > > > [27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd: > > 5 > > > > And > > > > [31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate > > inode 00000000b4733e96 > > > > > > > > > > > -----Original Message----- > > > From: Scott M. Lewandowski <vgerkernel@scottl.com> > > > Sent: Friday, May 1, 2020 12:20 AM > > > To: 'Steve French' <smfrench@gmail.com> > > > Cc: CIFS <linux-cifs@vger.kernel.org> > > > Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > > > > > EINTR on read ... Hundreds of times?! I have seen scp do that by > > > > sending signals but in writes not reads > > > > > > Not using scp at the moment. > > > > > > > Could an app be sending signals trying to kill the process doing the > reads? > > > > > > Not that I'm aware of. I'm not sure exactly how I'd check that, but > > > I will look into it. > > > > > > One other thing that came to mind that may be helpful is that one of > > > the processes doing the most IO is running under mono 6.8.0.105 > > > > > > -scott > > > > > > > > > > > > > > > From: Steve French <smfrench@gmail.com> > > > Sent: Friday, May 1, 2020 12:07 AM > > > To: Scott M. Lewandowski <vgerkernel@scottl.com> > > > Cc: CIFS <linux-cifs@vger.kernel.org> > > > Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > > > > EINTR on read ... Hundreds of times?! I have seen scp do that by > > > sending signals but in writes not reads > > > > > > On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski > > > <mailto:vgerkernel@scottl.com> wrote: > > > I have an Ubuntu 20.04 box mounting shares from a Windows 2019 > server. > > > Both boxes are fully patched. They are resident on the same > > > hypervisor and communicating directly via a vSwitch. I have 4 shares > > > mounted; here is an example from my fstab (yes, I know I should move > the credential file): > > > > > > //file1/usenet /mnt/usenet cifs > > > vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri > > > no,n > > > ounix,iocharset=utf8 0 0 > > > > > > My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The > > > contents of DebugData are included below. > > > > > > Right after the mount, I see a duplicate cookie reported. It is > > > unclear if this is a problem or not. Following that, I periodically > > > see messages similar to the following repeated in dmesg: > > > > > > [ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS: > > > Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read > > > = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] > > > CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send > > > error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4 > > > [ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFS > > > VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in > > > read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ > > > 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS: > > > Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ > > > 184.501185] CIFS VFS: Close unmatched open [ 191.784922] SMB2_read: > 4 callbacks suppressed [ 191.784925] CIFS VFS: > > > Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read > > > = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] > > > CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send > > > error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4 > > > [ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFS > > > VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in > > > read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ > > > 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS: > > > Close unmatched open [ 198.247594] CIFS VFS: Close unmatched open > > > > > > Here is the duplicate cookie info: > > > > > > [ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS- > Cache: O- > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee > n=00000000243f210c > > > [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' > > > [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c > [p=0000000079ef0365 > > > fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee > > > n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8] > '020001bdc0a801af' > > > [ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS- > Cache: O- > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee > n=00000000243f210c > > > [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' > > > [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 > [p=0000000079ef0365 > > > fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee > > > n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8] > '020001bdc0a801af' > > > [ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS- > Cache: O- > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee > n=00000000243f210c > > > [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' > > > [ 21.721994] FS-Cache: N-cookie c=00000000d479c294 > [p=0000000079ef0365 > > > fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee > > > n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] > '020001bdc0a801af' > > > > > > Any ideas what could be going on? > > > > > > Thanks for any help! > > > > > > > > > > > > > > > > > > > > > DebugData: > > > CIFS Version 2.23 > > > Features: > > > > DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C > > > IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL > > > CIFSMaxBufSize: 16384 > > > Active VFS Requests: 0 > > > Servers: > > > Number of credits: 8190 Dialect 0x300 > > > 1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > 0x100080000021 > > > Shares: > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > Number of credits: 8190 Dialect 0x300 > > > > > > 2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > 0x10006c000019 > > > Shares: > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > > > > > > > 3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: > > > 0x10007c000061 > > > Shares: > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 > > > State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid > > > 1211553 > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > > > > Number of credits: 8190 Dialect 0x300 > > > 4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > TCP > > > status: 1 Instance: 1 > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > 0x10006c000055 > > > Shares: > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > Share Capabilities: None Share Flags: 0x30 > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > 1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > 0xa3c339a0 > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > MIDs: > > > > > > Server interfaces: 2 > > > 0) Speed: 10000000000 bps > > > Capabilities: > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > 1) Speed: 10000000000 bps > > > Capabilities: > > > IPv4: 192.168.1.175 > > > > -- > Thanks, > > Steve ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 2020-05-01 19:37 ` Scott M. Lewandowski @ 2020-05-02 6:02 ` Steve French 0 siblings, 0 replies; 10+ messages in thread From: Steve French @ 2020-05-02 6:02 UTC (permalink / raw) To: Scott M. Lewandowski; +Cc: CIFS I would expect the close unmatched open to be related to the session being closed (thus all files implicitly closed on the server) but the session drop overlapping the application closing a file. On Fri, May 1, 2020 at 2:39 PM Scott M. Lewandowski <vgerkernel@scottl.com> wrote: > > I've now got a trace that corresponds to the dmesg output below. I will take a look later, but upon quick glance, this doesn't look like anything that will mean much to me, unfortunately. If someone would like to look at the file I can find a way to get it to you. > > > [53223.099221] CIFS VFS: Send error in read = -4 > [53223.232373] CIFS VFS: Send error in read = -4 > [53254.762914] CIFS VFS: Send error in read = -4 > [53301.403925] CIFS VFS: Send error in read = -4 > [53301.772568] CIFS VFS: Send error in read = -4 > [53301.818566] CIFS VFS: Send error in read = -4 > [53301.885510] CIFS VFS: Send error in read = -4 > [53302.055959] CIFS VFS: Send error in read = -4 > [53302.146337] CIFS VFS: Send error in read = -4 > [53302.236322] CIFS VFS: Send error in read = -4 > [53302.329525] CIFS VFS: Send error in read = -4 > [53303.000869] CIFS VFS: Send error in read = -4 > [53303.106961] CIFS VFS: Send error in read = -4 > [53304.756998] CIFS VFS: Close unmatched open > [53670.776832] CIFS VFS: Close unmatched open > > > > -----Original Message----- > > From: Steve French <smfrench@gmail.com> > > Sent: Friday, May 1, 2020 10:57 AM > > To: Scott M. Lewandowski <vgerkernel@scottl.com> > > Cc: Scott M. Lewandowski <scott@wynstonegroup.com>; CIFS <linux- > > cifs@vger.kernel.org> > > Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > > Are you comfortable with dynamic tracing ("trace-cmd record -e ..." > > ... the list of trace points can be seen in > > /sys/kernel/debug/tracing/events/cifs)? > > > > On Fri, May 1, 2020 at 10:40 AM Scott M. Lewandowski > > <vgerkernel@scottl.com> wrote: > > > > > > I’m also seeing entries like the following mixed in with the other messages: > > > > > > [27078.665192] CIFS VFS: \\file1 Cancelling wait for mid 13443257 cmd: > > > 5 > > > > > > And > > > > > > [31833.351785] CIFS VFS: cifs_invalidate_mapping: could not invalidate > > > inode 00000000b4733e96 > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Scott M. Lewandowski <vgerkernel@scottl.com> > > > > Sent: Friday, May 1, 2020 12:20 AM > > > > To: 'Steve French' <smfrench@gmail.com> > > > > Cc: CIFS <linux-cifs@vger.kernel.org> > > > > Subject: RE: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > > > > > > > EINTR on read ... Hundreds of times?! I have seen scp do that by > > > > > sending signals but in writes not reads > > > > > > > > Not using scp at the moment. > > > > > > > > > Could an app be sending signals trying to kill the process doing the > > reads? > > > > > > > > Not that I'm aware of. I'm not sure exactly how I'd check that, but > > > > I will look into it. > > > > > > > > One other thing that came to mind that may be helpful is that one of > > > > the processes doing the most IO is running under mono 6.8.0.105 > > > > > > > > -scott > > > > > > > > > > > > > > > > > > > > From: Steve French <smfrench@gmail.com> > > > > Sent: Friday, May 1, 2020 12:07 AM > > > > To: Scott M. Lewandowski <vgerkernel@scottl.com> > > > > Cc: CIFS <linux-cifs@vger.kernel.org> > > > > Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > > > > > > > EINTR on read ... Hundreds of times?! I have seen scp do that by > > > > sending signals but in writes not reads > > > > > > > > On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski > > > > <mailto:vgerkernel@scottl.com> wrote: > > > > I have an Ubuntu 20.04 box mounting shares from a Windows 2019 > > server. > > > > Both boxes are fully patched. They are resident on the same > > > > hypervisor and communicating directly via a vSwitch. I have 4 shares > > > > mounted; here is an example from my fstab (yes, I know I should move > > the credential file): > > > > > > > > //file1/usenet /mnt/usenet cifs > > > > vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserveri > > > > no,n > > > > ounix,iocharset=utf8 0 0 > > > > > > > > My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The > > > > contents of DebugData are included below. > > > > > > > > Right after the mount, I see a duplicate cookie reported. It is > > > > unclear if this is a problem or not. Following that, I periodically > > > > see messages similar to the following repeated in dmesg: > > > > > > > > [ 179.625296] SMB2_read: 9 callbacks suppressed [ 179.625298] CIFS VFS: > > > > Send error in read = -4 [ 179.704191] CIFS VFS: Send error in read > > > > = -4 [ 179.847514] CIFS VFS: Send error in read = -4 [ 180.031516] > > > > CIFS VFS: Send error in read = -4 [ 180.214848] CIFS VFS: Send > > > > error in read = -4 [ 180.282909] CIFS VFS: Send error in read = -4 > > > > [ 180.474829] CIFS VFS: Send error in read = -4 [ 180.621349] CIFS > > > > VFS: Send error in read = -4 [ 180.780465] CIFS VFS: Send error in > > > > read = -4 [ 180.823249] CIFS VFS: Send error in read = -4 [ > > > > 182.339351] CIFS VFS: Close unmatched open [ 182.726829] CIFS VFS: > > > > Close unmatched open [ 184.292663] CIFS VFS: Close unmatched open [ > > > > 184.501185] CIFS VFS: Close unmatched open [ 191.784922] SMB2_read: > > 4 callbacks suppressed [ 191.784925] CIFS VFS: > > > > Send error in read = -4 [ 191.884186] CIFS VFS: Send error in read > > > > = -4 [ 192.270851] CIFS VFS: Send error in read = -4 [ 192.323467] > > > > CIFS VFS: Send error in read = -4 [ 192.381336] CIFS VFS: Send > > > > error in read = -4 [ 192.446711] CIFS VFS: Send error in read = -4 > > > > [ 192.507605] CIFS VFS: Send error in read = -4 [ 193.244397] CIFS > > > > VFS: Send error in read = -4 [ 193.307588] CIFS VFS: Send error in > > > > read = -4 [ 193.403410] CIFS VFS: Send error in read = -4 [ > > > > 193.797038] CIFS VFS: Close unmatched open [ 196.784082] CIFS VFS: > > > > Close unmatched open [ 198.247594] CIFS VFS: Close unmatched open > > > > > > > > Here is the duplicate cookie info: > > > > > > > > [ 21.713307] FS-Cache: Duplicate cookie detected [ 21.713840] FS- > > Cache: O- > > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > > [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee > > n=00000000243f210c > > > > [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' > > > > [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c > > [p=0000000079ef0365 > > > > fl=2 nc=0 na=1] [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee > > > > n=00000000dd1465c5 [ 21.716141] FS-Cache: N-key=[8] > > '020001bdc0a801af' > > > > [ 21.716591] FS-Cache: Duplicate cookie detected [ 21.717217] FS- > > Cache: O- > > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > > [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee > > n=00000000243f210c > > > > [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' > > > > [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 > > [p=0000000079ef0365 > > > > fl=2 nc=0 na=1] [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee > > > > n=00000000e757f917 [ 21.719621] FS-Cache: N-key=[8] > > '020001bdc0a801af' > > > > [ 21.720097] FS-Cache: Duplicate cookie detected [ 21.720565] FS- > > Cache: O- > > > > cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > > > > [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee > > n=00000000243f210c > > > > [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' > > > > [ 21.721994] FS-Cache: N-cookie c=00000000d479c294 > > [p=0000000079ef0365 > > > > fl=2 nc=0 na=1] [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee > > > > n=00000000ca1891fa [ 21.722941] FS-Cache: N-key=[8] > > '020001bdc0a801af' > > > > > > > > Any ideas what could be going on? > > > > > > > > Thanks for any help! > > > > > > > > > > > > > > > > > > > > > > > > > > > > DebugData: > > > > CIFS Version 2.23 > > > > Features: > > > > > > DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,C > > > > IFS_POSIX,UPCALL(SPNEGO),XATTR,ACL > > > > CIFSMaxBufSize: 16384 > > > > Active VFS Requests: 0 > > > > Servers: > > > > Number of credits: 8190 Dialect 0x300 > > > > 1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > > TCP > > > > status: 1 Instance: 1 > > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > > 0x100080000021 > > > > Shares: > > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > > Share Capabilities: None Share Flags: 0x30 > > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > > > 1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > > 0xa3c339a0 > > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > > > MIDs: > > > > > > > > Server interfaces: 2 > > > > 0) Speed: 10000000000 bps > > > > Capabilities: > > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > > 1) Speed: 10000000000 bps > > > > Capabilities: > > > > IPv4: 192.168.1.175 > > > > Number of credits: 8190 Dialect 0x300 > > > > > > > > 2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > > TCP > > > > status: 1 Instance: 1 > > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > > 0x10006c000019 > > > > Shares: > > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > > Share Capabilities: None Share Flags: 0x30 > > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > > > 1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > > 0xa3c339a0 > > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > > > MIDs: > > > > > > > > Server interfaces: 2 > > > > 0) Speed: 10000000000 bps > > > > Capabilities: > > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > > 1) Speed: 10000000000 bps > > > > Capabilities: > > > > IPv4: 192.168.1.175 > > > > > > > > > > > > 3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > > TCP > > > > status: 1 Instance: 1 > > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: > > > > 0x10007c000061 > > > > Shares: > > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > > Share Capabilities: None Share Flags: 0x30 > > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > > > 1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > > 0xa3c339a0 > > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > > > MIDs: > > > > State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 > > > > State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid > > > > 1211553 > > > > > > > > Server interfaces: 2 > > > > 0) Speed: 10000000000 bps > > > > Capabilities: > > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > > 1) Speed: 10000000000 bps > > > > Capabilities: > > > > IPv4: 192.168.1.175 > > > > > > > > Number of credits: 8190 Dialect 0x300 > > > > 4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 > > TCP > > > > status: 1 Instance: 1 > > > > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: > > > > 0x10006c000055 > > > > Shares: > > > > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > > > > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > > > > Share Capabilities: None Share Flags: 0x30 > > > > tid: 0x1 Maximal Access: 0x11f01ff > > > > > > > > 1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > > > > PathComponentMax: 255 Status: 1 type: DISK Serial Number: > > 0xa3c339a0 > > > > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > > > > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > > > > > > > MIDs: > > > > > > > > Server interfaces: 2 > > > > 0) Speed: 10000000000 bps > > > > Capabilities: > > > > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > > > > 1) Speed: 10000000000 bps > > > > Capabilities: > > > > IPv4: 192.168.1.175 > > > > > > > > -- > > Thanks, > > > > Steve -- Thanks, Steve ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 2020-05-01 5:18 ` Scott M. Lewandowski 2020-05-01 15:38 ` Scott M. Lewandowski @ 2020-05-02 6:10 ` Steve French 1 sibling, 0 replies; 10+ messages in thread From: Steve French @ 2020-05-02 6:10 UTC (permalink / raw) To: Scott M. Lewandowski; +Cc: CIFS I don't know much about mono but wonder if signal handling problems (in mono) could be related "EINTR" is suspicious on read ... Don't know if this is related but google search turned up this: "Mono uses signals on certain platforms to handle NullReferenceExceptions. It’s also becoming more and more common to add third-party libraries that detect and report crashes. The problem is when those third-party libraries install their own signal handlers, overwriting Mono’s signal handlers. When this happens, any NullReferenceException would be detected by such libraries as a fatal SIGSEGV, and the libraries would report it as a crash and terminate the process." On Fri, May 1, 2020 at 12:19 AM Scott M. Lewandowski <vgerkernel@scottl.com> wrote: > > > EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads > > Not using scp at the moment. > > > Could an app be sending signals trying to kill the process doing the reads? > > Not that I'm aware of. I'm not sure exactly how I'd check that, but I will look into it. > > One other thing that came to mind that may be helpful is that one of the processes doing the most IO is running under mono 6.8.0.105 > > -scott > > > > > From: Steve French <smfrench@gmail.com> > Sent: Friday, May 1, 2020 12:07 AM > To: Scott M. Lewandowski <vgerkernel@scottl.com> > Cc: CIFS <linux-cifs@vger.kernel.org> > Subject: Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 > > EINTR on read ... Hundreds of times?! I have seen scp do that by sending signals but in writes not reads > > On Thu, Apr 30, 2020, 23:48 Scott M. Lewandowski <mailto:vgerkernel@scottl.com> wrote: > I have an Ubuntu 20.04 box mounting shares from a Windows 2019 server. Both boxes are fully patched. They are resident on the same hypervisor and communicating directly via a vSwitch. I have 4 shares mounted; here is an example from my fstab (yes, I know I should move the credential file): > > //file1/usenet /mnt/usenet cifs vers=3.0,uid=scott,credentials=/home/scott/.smbcredentials,noserverino,nounix,iocharset=utf8 0 0 > > My kernel is 5.4.0-28-generic and mount.cifs version is 6.9. The contents of DebugData are included below. > > Right after the mount, I see a duplicate cookie reported. It is unclear if this is a problem or not. Following that, I periodically see messages similar to the following repeated in dmesg: > > [ 179.625296] SMB2_read: 9 callbacks suppressed > [ 179.625298] CIFS VFS: Send error in read = -4 > [ 179.704191] CIFS VFS: Send error in read = -4 > [ 179.847514] CIFS VFS: Send error in read = -4 > [ 180.031516] CIFS VFS: Send error in read = -4 > [ 180.214848] CIFS VFS: Send error in read = -4 > [ 180.282909] CIFS VFS: Send error in read = -4 > [ 180.474829] CIFS VFS: Send error in read = -4 > [ 180.621349] CIFS VFS: Send error in read = -4 > [ 180.780465] CIFS VFS: Send error in read = -4 > [ 180.823249] CIFS VFS: Send error in read = -4 > [ 182.339351] CIFS VFS: Close unmatched open > [ 182.726829] CIFS VFS: Close unmatched open > [ 184.292663] CIFS VFS: Close unmatched open > [ 184.501185] CIFS VFS: Close unmatched open > [ 191.784922] SMB2_read: 4 callbacks suppressed > [ 191.784925] CIFS VFS: Send error in read = -4 > [ 191.884186] CIFS VFS: Send error in read = -4 > [ 192.270851] CIFS VFS: Send error in read = -4 > [ 192.323467] CIFS VFS: Send error in read = -4 > [ 192.381336] CIFS VFS: Send error in read = -4 > [ 192.446711] CIFS VFS: Send error in read = -4 > [ 192.507605] CIFS VFS: Send error in read = -4 > [ 193.244397] CIFS VFS: Send error in read = -4 > [ 193.307588] CIFS VFS: Send error in read = -4 > [ 193.403410] CIFS VFS: Send error in read = -4 > [ 193.797038] CIFS VFS: Close unmatched open > [ 196.784082] CIFS VFS: Close unmatched open > [ 198.247594] CIFS VFS: Close unmatched open > > Here is the duplicate cookie info: > > [ 21.713307] FS-Cache: Duplicate cookie detected > [ 21.713840] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > [ 21.714330] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c > [ 21.714779] FS-Cache: O-key=[8] '020001bdc0a801af' > [ 21.715219] FS-Cache: N-cookie c=00000000afb49d9c [p=0000000079ef0365 fl=2 nc=0 na=1] > [ 21.715672] FS-Cache: N-cookie d=00000000173ea0ee n=00000000dd1465c5 > [ 21.716141] FS-Cache: N-key=[8] '020001bdc0a801af' > [ 21.716591] FS-Cache: Duplicate cookie detected > [ 21.717217] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > [ 21.717679] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c > [ 21.718220] FS-Cache: O-key=[8] '020001bdc0a801af' > [ 21.718681] FS-Cache: N-cookie c=0000000063665cf7 [p=0000000079ef0365 fl=2 nc=0 na=1] > [ 21.719166] FS-Cache: N-cookie d=00000000173ea0ee n=00000000e757f917 > [ 21.719621] FS-Cache: N-key=[8] '020001bdc0a801af' > [ 21.720097] FS-Cache: Duplicate cookie detected > [ 21.720565] FS-Cache: O-cookie c=00000000e8276279 [p=0000000079ef0365 fl=222 nc=0 na=1] > [ 21.721053] FS-Cache: O-cookie d=00000000173ea0ee n=00000000243f210c > [ 21.721518] FS-Cache: O-key=[8] '020001bdc0a801af' > [ 21.721994] FS-Cache: N-cookie c=00000000d479c294 [p=0000000079ef0365 fl=2 nc=0 na=1] > [ 21.722476] FS-Cache: N-cookie d=00000000173ea0ee n=00000000ca1891fa > [ 21.722941] FS-Cache: N-key=[8] '020001bdc0a801af' > > Any ideas what could be going on? > > Thanks for any help! > > > > > > > DebugData: > CIFS Version 2.23 > Features: DFS,FSCACHE,STATS,DEBUG,ALLOW_INSECURE_LEGACY,WEAK_PW_HASH,CIFS_POSIX,UPCALL(SPNEGO),XATTR,ACL > CIFSMaxBufSize: 16384 > Active VFS Requests: 0 > Servers: > Number of credits: 8190 Dialect 0x300 > 1) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x100080000021 > Shares: > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > Share Capabilities: None Share Flags: 0x30 > tid: 0x1 Maximal Access: 0x11f01ff > > 1) \\file1\XXX Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > MIDs: > > Server interfaces: 2 > 0) Speed: 10000000000 bps > Capabilities: > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > 1) Speed: 10000000000 bps > Capabilities: > IPv4: 192.168.1.175 > Number of credits: 8190 Dialect 0x300 > > 2) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000019 > Shares: > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > Share Capabilities: None Share Flags: 0x30 > tid: 0x1 Maximal Access: 0x11f01ff > > 1) \\file1\usenet Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > MIDs: > > Server interfaces: 2 > 0) Speed: 10000000000 bps > Capabilities: > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > 1) Speed: 10000000000 bps > Capabilities: > IPv4: 192.168.1.175 > > > 3) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 2 SessionId: 0x10007c000061 > Shares: > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > Share Capabilities: None Share Flags: 0x30 > tid: 0x1 Maximal Access: 0x11f01ff > > 1) \\file1\YYY Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > MIDs: > State: 2 com: 8 pid: 1218 cbdata: 00000000e45c59d2 mid 1211489 > State: 2 com: 8 pid: 1218 cbdata: 0000000018476195 mid 1211553 > > Server interfaces: 2 > 0) Speed: 10000000000 bps > Capabilities: > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > 1) Speed: 10000000000 bps > Capabilities: > IPv4: 192.168.1.175 > > Number of credits: 8190 Dialect 0x300 > 4) Name: 192.168.1.175 Uses: 1 Capability: 0x300067 Session Status: 1 TCP status: 1 Instance: 1 > Local Users To Server: 1 SecMode: 0x1 Req On Wire: 0 SessionId: 0x10006c000055 > Shares: > 0) IPC: \\file1\IPC$ Mounts: 1 DevInfo: 0x0 Attributes: 0x0 > PathComponentMax: 0 Status: 1 type: 0 Serial Number: 0x0 > Share Capabilities: None Share Flags: 0x30 > tid: 0x1 Maximal Access: 0x11f01ff > > 1) \\file1\ZZZ Mounts: 1 DevInfo: 0x20020 Attributes: 0x8500df > PathComponentMax: 255 Status: 1 type: DISK Serial Number: 0xa3c339a0 > Share Capabilities: None Aligned, Partition Aligned, Share Flags: 0x0 > tid: 0x5 Optimal sector size: 0x200 Maximal Access: 0x1f01ff > > MIDs: > > Server interfaces: 2 > 0) Speed: 10000000000 bps > Capabilities: > IPv6: fe80:0000:0000:0000:71cb:1839:1069:9806 > 1) Speed: 10000000000 bps > Capabilities: > IPv4: 192.168.1.175 -- Thanks, Steve ^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: issues with CIFS on Ubuntu 20 mounting Windows 2019 2020-05-01 4:46 issues with CIFS on Ubuntu 20 mounting Windows 2019 Scott M. Lewandowski [not found] ` <CAH2r5msOBGS7QYagUkby4upxFKbxCdQg=T65K4s9VhRKkWY+Xw@mail.gmail.com> @ 2020-05-04 10:11 ` Aurélien Aptel 1 sibling, 0 replies; 10+ messages in thread From: Aurélien Aptel @ 2020-05-04 10:11 UTC (permalink / raw) To: Scott M. Lewandowski, linux-cifs@vger.kernel.org Hi Scott, If you can reproduce this, could you try making a capture of mounting + waiting for the bug? https://wiki.samba.org/index.php/Bug_Reporting#cifs.ko Cheers, -- Aurélien Aptel / SUSE Labs Samba Team GPG: 1839 CB5F 9F5B FB9B AA97 8C99 03C8 A49B 521B D5D3 SUSE Software Solutions Germany GmbH, Maxfeldstr. 5, 90409 Nürnberg, DE GF: Felix Imendörffer, Mary Higgins, Sri Rasiah HRB 247165 (AG München) ^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2020-05-04 10:11 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2020-05-01 4:46 issues with CIFS on Ubuntu 20 mounting Windows 2019 Scott M. Lewandowski
[not found] ` <CAH2r5msOBGS7QYagUkby4upxFKbxCdQg=T65K4s9VhRKkWY+Xw@mail.gmail.com>
2020-05-01 5:18 ` Scott M. Lewandowski
2020-05-01 15:38 ` Scott M. Lewandowski
2020-05-01 15:56 ` Steve French
2020-05-01 18:19 ` Scott M. Lewandowski
[not found] ` <CAH2r5mvscNft+arZuzzupNeB7df+c7fUmj=XRNZ5Y6PEts+Efw@mail.gmail.com>
[not found] ` <40d5e83ebbd14de785be3d7ba82ec34a@wynstonegroup.com>
2020-05-02 20:28 ` Scott M. Lewandowski
2020-05-01 19:37 ` Scott M. Lewandowski
2020-05-02 6:02 ` Steve French
2020-05-02 6:10 ` Steve French
2020-05-04 10:11 ` Aurélien Aptel
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.