CIFS slowness & crashes

CIFS slowness & crashes

[Lasse Kärkkäinen / Tronic]

[-- Attachment #1: Type: text/plain, Size: 1993 bytes --]

I mailed sfrench@samba.org (the guy who wrote the driver) about this a
month ago, but didn't get any reply. Is anyone working on that driver
anymore?

The problems that I wrote him about were:

1. CIFS VFS hangs entirely if the server crashes or otherwise goes
offline. Every process touching the mount halts too and cannot be killed
(but they are not zombies). System loads start climbing and eventually
the entire system will die (after system loads reach about 500). It is
not possible to umount with either smbumount (hangs) nor umount -f
(prints errors but doesn't umount anything). It won't recover without
reboot, even if the server becomes back online.

This problem has been around as long as I have used SMBFS or CIFS. There
has only been slight variation from one version to another. Sometimes it
is possible to umount them (after some pretty long timeout), sometimes
it is not. It seems as if the problem was being fixed, but none of the
fixes really worked.

2. Occassionally the transmission speeds go extremely low for no
apparent reason. While writing this, I am getting 0.39 Mo/s over a
gigabit network. Using FTP to read the same file gives 40 Mo/s, which is
the speed that the file can be read locally on the server too.
Remounting the CIFS does not help, nor does restarting Samba. However,
using SMBFS I can get 20 Mo/s which is a bit better but still far from
what it should be. It is important to mention that sometimes CIFS does
work faster (about as quickly as SMBFS) and that this misbehavior occurs
randomly.

During CIFS transfer, both computers seem to be idling. The CPU usage
(including I/O wait) is almost none. During SMBFS transfer the server
smbd process uses about 15 % CPU and the client is almost idle. The
client is P4 3.4 GHz and the server is Athlon64 3000+.

I also tested with a Windows XP client machine and found out that this
slowness issue does not happen with it, using the very same Samba server
that the Linux CIFS mount is using.

- Tronic -

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]

Re: CIFS slowness & crashes

[Jesper Juhl]

On 7/21/05, Lasse Kärkkäinen / Tronic <tronic+lzID=lx43caky45@trn.iki.fi> wrote:
> I mailed sfrench@samba.org (the guy who wrote the driver) about this a
> month ago, but didn't get any reply. Is anyone working on that driver
> anymore?
> 
As far as I know Steve is still maintaining cifs. If you wrote him and
didn't get a response, then try again after a while (you might have
included him on CC for this mail) - maintainers don't always have time
to answer all mail in a timely fashion (or at all), and it's your
responsability to resend - that's not news.

You could also have written to the samba-technical@lists.samba.org
mailinglist (or copied it - it's listed in MAINTAINERS under "COMMON
INTERNET FILE SYSTEM (CIFS)").

[adding Stephen French to CC]

Personally I'd probably have send the mail
 To: Steve French <sfrench@samba.org>
 Cc: samba-technical@lists.samba.org
 Cc: linux-kernel@vger.kernel.org

> The problems that I wrote him about were:
> 
> 1. CIFS VFS hangs entirely if the server crashes or otherwise goes
> offline. Every process touching the mount halts too and cannot be killed
> (but they are not zombies). System loads start climbing and eventually
> the entire system will die (after system loads reach about 500). It is
> not possible to umount with either smbumount (hangs) nor umount -f
> (prints errors but doesn't umount anything). It won't recover without
> reboot, even if the server becomes back online.
> 
> This problem has been around as long as I have used SMBFS or CIFS. There
> has only been slight variation from one version to another. Sometimes it
> is possible to umount them (after some pretty long timeout), sometimes
> it is not. It seems as if the problem was being fixed, but none of the
> fixes really worked.
> 
> 2. Occassionally the transmission speeds go extremely low for no
> apparent reason. While writing this, I am getting 0.39 Mo/s over a
> gigabit network. Using FTP to read the same file gives 40 Mo/s, which is
> the speed that the file can be read locally on the server too.
> Remounting the CIFS does not help, nor does restarting Samba. However,
> using SMBFS I can get 20 Mo/s which is a bit better but still far from
> what it should be. It is important to mention that sometimes CIFS does
> work faster (about as quickly as SMBFS) and that this misbehavior occurs
> randomly.
> 
> During CIFS transfer, both computers seem to be idling. The CPU usage
> (including I/O wait) is almost none. During SMBFS transfer the server
> smbd process uses about 15 % CPU and the client is almost idle. The
> client is P4 3.4 GHz and the server is Athlon64 3000+.
> 
> I also tested with a Windows XP client machine and found out that this
> slowness issue does not happen with it, using the very same Samba server
> that the Linux CIFS mount is using.
> 
> - Tronic -
> 
>

Re: CIFS slowness & crashes

[Steve French]

On Thu, 2005-07-21 at 17:04, Jesper Juhl wrote:
> On 7/21/05, Lasse Kärkkäinen / Tronic <tronic+lzID=lx43caky45@trn.iki.fi> wrote:
> > I mailed sfrench@samba.org (the guy who wrote the driver) about this a
> > month ago, but didn't get any reply. Is anyone working on that driver
> > anymore?
> > 
> As far as I know Steve is still maintaining cifs. If you wrote him and
> didn't get a response, then try again after a while (you might have
> included him on CC for this mail) - maintainers don't always have time
> to answer all mail in a timely fashion (or at all), and it's your
> responsability to resend - that's not news.

CIFS is still being actively improved/maintained.  I was out on vacation
for over a week.  It might have gotten missed in the flood of email I
had to catch up on.  I am still maintaining cifs and evaluating patches
from others as well.  As you can see from the web git page, changesets
are still being added.  You can see the list of CIFS changesets (most
recent first) by the query:
	http://www.kernel.org/git/?p=linux%2Fkernel%2Fgit%2Fsfrench%2Fcifs-2.6.git&a=search&s=CIFS

> You could also have written to the samba-technical@lists.samba.org
> mailinglist (or copied it - it's listed in MAINTAINERS under "COMMON
> INTERNET FILE SYSTEM (CIFS)").
> 
> [adding Stephen French to CC]
> 
> Personally I'd probably have send the mail
>  To: Steve French <sfrench@samba.org>
>  Cc: samba-technical@lists.samba.org
>  Cc: linux-kernel@vger.kernel.org

Yes.  CCing those lists is recommended.  The best list to send to is and
which I and others in this area monitor regularly though is
linux-cifs-client@lists.samba.org

> > not possible to umount with either smbumount (hangs) 
smbumount can not be called on cifs vfs mounts (it is for the older
smbfs).  Although there is a somewhat similar umount.cifs utility
(umount.cifs.c in the samba 3 source) it will be obsolete when 
more general user umounting (of their own mounts) is permitted
in the kernel vfs itself.

> nor umount -f
> > (prints errors but doesn't umount anything). 

What are the errors?  What is the version of cifs.ko module?

> It won't recover without
> > reboot, even if the server becomes back online.
> > 
> > This problem has been around as long as I have used SMBFS or CIFS. There
> > has only been slight variation from one version to another. Sometimes it
> > is possible to umount them (after some pretty long timeout), sometimes
> > it is not. It seems as if the problem was being fixed, but none of the
> > fixes really worked.

My tests of reconnection after server crash or network reconnection with
smbfs works (for the past year or two) and also of course for cifs. 
cifs also reconnects state (open files) not just the connection to
\\server\share


> > 
> > 2. Occassionally the transmission speeds go extremely low for no
> > apparent reason. While writing this, I am getting 0.39 Mo/s over a
> > gigabit network.

My informal tests (cifs->samba) showed a maximum of about 20%
utilization of gigabit doing large file writes and about double that for
large file reads with a single cifs client to Samba over gigabit. Should
be somewhat simalar to Windows server.

The most common cause of widely varying speeds are the following:
1) memory fragmentation - some versions of the kernel badly fragment
slab allocations greater than 4 pages (cifs by default allocates 16.5K
buffers - which results in larger size allocations when more than 5
processes are accessing the mount and the cifs buffer pool is exceeded)
2) write behind due to oplock - smbfs does not do oplock, cifs does -
which means that cifs client caching can cause a large amount of
writebehind data to accumulate (with great performance for a while) -
then when memory gets tight due to the inode caching in linux's mm
layer, the cifs client is asked to write out a large amount of file data
at one time (which it does synchronously). 

Both of these are being improved.  You can bypass the inode caching on
the client by using the cifs mount option
	"forcedirectio"


>  Using FTP to read the same file gives 40 Mo/s,
Similarly NFS was somewhat better than that (as long as the file was in
memory already on the server).

Re: CIFS slowness & crashes

[Steve French]

> > 2. Occassionally the transmission speeds go extremely low for no
> > apparent reason. While writing this, I am getting 0.39 Mo/s over a
> > gigabit network.

It would help to know whether you are doing mostly writing to or reading
from the server.

Forgot to mention that another thing that may help (besides the large
number of improvements that already went into 2.6.12 already to
drastically reduce the expensive large buffer usage, and the even newer
experimental CIFS write code) are two configuration changes - either
dropping the buffer size to relieve pressure on the slab (e.g. to just
under 8K - perhaps 7800 bytes) or increasing the number of large buffers
in the pool (they are insmod parms of cifs.ko - you can do modinfo on
cifs.ko to see them) so fewer allocations are done.  This would only be
needed if multiple processes were accessing the mount at one time.

Re: CIFS slowness & crashes

[Jesper Juhl]

[-- Attachment #1: Type: text/plain, Size: 1222 bytes --]

On 21 Jul 2005 17:35:12 -0500, Steve French <smfltc@us.ibm.com> wrote:
> Yes.  CCing those lists is recommended.  The best list to send to is and
> which I and others in this area monitor regularly though is
> linux-cifs-client@lists.samba.org

If that's the best list to send to, then I suggest it be added to the
CIFS MAINTAINERS entry as pr the patch below (also attached since
gmail will probably mangle it).


Add linux-cifs-client@lists.samba.org to CIFS entry in MAINTAINERS

Signed-off-by: Jesper Juhl <jesper.juhl@gmail.com>
---

 MAINTAINERS |    1 +
 1 files changed, 1 insertion(+)

--- linux-2.6.13-rc3-orig/MAINTAINERS   2005-07-14 20:33:32.000000000 +0200
+++ linux-2.6.13-rc3/MAINTAINERS        2005-07-22 01:42:09.000000000 +0200
@@ -532,6 +532,7 @@
 COMMON INTERNET FILE SYSTEM (CIFS)
 P:     Steve French
 M:     sfrench@samba.org
+L:     linux-cifs-client@lists.samba.org
 L:     samba-technical@lists.samba.org
 W:     http://us1.samba.org/samba/Linux_CIFS_client.html
 S:     Supported


-- 
Jesper Juhl <jesper.juhl@gmail.com>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: CIFS_add_linux-cifs-client_list_to_MAINTAINERS.patch --]
[-- Type: text/x-patch; name="CIFS_add_linux-cifs-client_list_to_MAINTAINERS.patch", Size: 545 bytes --]

From: Jesper Juhl <jesper.juhl@gmail.com>

Add linux-cifs-client@lists.samba.org to CIFS entry in MAINTAINERS

Signed-off-by: Jesper Juhl <jesper.juhl@gmail.com>


--- linux-2.6.13-rc3-orig/MAINTAINERS	2005-07-14 20:33:32.000000000 +0200
+++ linux-2.6.13-rc3/MAINTAINERS	2005-07-22 01:42:09.000000000 +0200
@@ -532,6 +532,7 @@
 COMMON INTERNET FILE SYSTEM (CIFS)
 P:	Steve French
 M:	sfrench@samba.org
+L:	linux-cifs-client@lists.samba.org
 L:	samba-technical@lists.samba.org
 W:	http://us1.samba.org/samba/Linux_CIFS_client.html
 S:	Supported	

Re: CIFS slowness & crashes

[Lasse Kärkkäinen / Tronic]

[-- Attachment #1: Type: text/plain, Size: 3157 bytes --]

>>nor umount -f
> What are the errors?  What is the version of cifs.ko module?

umount2: Device or resource busy
umount: /tmpmnt: device is busy
umount2: Device or resource busy
umount: /tmpmnt: device is busy

Without -f it doesn't print those umount2 errors, just the other two.

The version is whatever comes with 2.6.12 or 2.6.13rc1.

> My tests of reconnection after server crash or network reconnection with
> smbfs works (for the past year or two) and also of course for cifs. 
> cifs also reconnects state (open files) not just the connection to
> \\server\share

Reconnection usually (or perhaps always, with new versions) works, but
nothing cannot be done if the server goes permanently offline (nor until
it comes back online). When the server is down, the programs that were
using the CIFS mount or that try to access it will halt.

> My informal tests (cifs->samba) showed a maximum of about 20%
> utilization of gigabit doing large file writes and about double that for
> large file reads with a single cifs client to Samba over gigabit. Should
> be somewhat simalar to Windows server.

This seems quite bad. Is it waiting for packet confirmations or what?

However, I have never got anything better than about 40 Mo/s with a
gigabit network so far. Jumbo frames and using direct cross-over cable
between the clients make no difference. Still, all protocols that I have
tried, except for SMB and CIFS, can reach that easily. I'll try to get
two Windows machines that I can test with. Perhaps the problem is with
Linux network drivers.

Anyway, 20 Mo/s or something like that would not be a big problem. The
real problem occurs when the speed drops under 3 Mo/s.

> The most common cause of widely varying speeds are the following:
> 1) memory fragmentation - some versions of the kernel badly fragment
> slab allocations greater than 4 pages (cifs by default allocates 16.5K
> buffers - which results in larger size allocations when more than 5
> processes are accessing the mount and the cifs buffer pool is exceeded)

Wouldn't this show up as increased system loads?

> 2) write behind due to oplock - smbfs does not do oplock, cifs does -
> which means that cifs client caching can cause a large amount of
> writebehind data to accumulate (with great performance for a while) -
> then when memory gets tight due to the inode caching in linux's mm
> layer, the cifs client is asked to write out a large amount of file data
> at one time (which it does synchronously). 
> 
> Both of these are being improved.  You can bypass the inode caching on
> the client by using the cifs mount option
> 	"forcedirectio"

This option has little or no effect. Still runs slowly (2.4 Mo/s right now).

My benchmark is reading data in 50 Mio chunks, as quickly as it can,
calculating the average read speed. The files being read are bigger than
4 Gio. I haven't tried writing anything (the shares that I play with are
read-only).

For the record, I am using O_RDONLY | O_LARGEFILE on open and then pread
 for all reads. However, I am getting similar results with all programs,
so I don't think that the reading method really matters that much.

- Tronic -


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 256 bytes --]