cifs client timeouts and hard/soft mounts

cifs client timeouts and hard/soft mounts

[Jeff Layton]

Introduction:
=============
Apologies for the wide distribution of this email, but I think this
topic is something that is something that is fundamental to anyone
attempting to implement a CIFS client. I also apologize in advance for
its long-windedness, but I've given this a lot of thought and want to
make sure that I communicate my rationale clearly.

We've begun a discussion concerning this on the cifs-protocol and pfif
lists, but I'm not sure this is of great interest to Microsoft but
probably is of wider interest to the readers of the mailing lists to
which I'm sending.

I'd like to use this email as a starting point for discussion to nail
down exactly how the transport layer in the Linux CIFS client should
behave. It may also be of interest for others implementing SMB clients.
I'll also point out that I recently sent a patchset to the linux-cifs
list that implements this design (for the most part) for the Linux CIFS
client, so I have a real interest in getting this behavior right.

The main questions boil down to:

1) When should a CIFS client give up on pending requests and reconnect
the socket?

2) What does "hard" and "soft" mean in the context of CIFS?

These are separate questions but the the answer to one affects the
other...

Timeouts:
=========
It's tempting to think of SMB as being very similar to NFS/RPC, but
when it comes to low-level transport, there are significant
differences. ONC-RPC was designed for connectionless transports and has
the concept of a retransmission. SMB however does not -- it was
originally layered on NetBIOS sessions and so has always been assumed
to run on a connection-based transport.

For that reason, we can never retransmit a SMB request on the same
connection. Our only recourse in the event of a communication breakdown
is to close the transport layer (aka the socket) and start over from
scratch.

OTOH, this design has some benefits. Because SMB is always on a
connection-oriented transport layer, we can generally assume that as
long as the server is responding to requests on the transport that it
received any previous request to which we haven't received a reply.

A significant design concern to consider is that reconnects for cifs
clients are horrifically expensive. Much of the state of the client is
intertwined with the socket. If we reconnect, we lose filesystem state
and have to reclaim it -- sessions, tree connects, open files, locks --
all of it.

Doing all of that is extremely costly, and in the case of locks we can
never be sure that another client hasn't raced in and stolen the lock
while we were reconnecting. That's a data-integrity issue -- there is
no lock reclaim grace period like with NLM. Thus, we should attempt to
avoid reconnects as much as we possibly can.

So, what does this mean for CIFS clients? I believe that the best
behavior for the client is to *never* time out an individual request,
aside from SMB echoes. When we haven't received a reply from the server
for some time (on the order of 30-60s), the client should issue an SMB
echo request. If the server doesn't reply within a reasonable amount of
time (maybe another 30-60s), we should close down the socket and
attempt to reconnect.

If the server is responding to the echo requests however, we should
assume that it's working on our earlier requests and continue to wait
for the reply indefinitely. That waiting should be interruptible by
fatal signals so that there is a "failsafe" for clients communicating
with misbehaving servers.

In short, timeouts should be a property of the socket as a whole and
not a property of individual requests on the wire. MS-CIFS and Windows'
behavior contradicts this to some degree, but MS isn't trying to
shoehorn a CIFS client into a unix-like OS either. They have their own
design concerns and they aren't necessarily the same as ours.

Hard and Soft mounts:
=====================

If we're not ever going to time out individual requests, what does this
mean for the "hard" and "soft" mount options?

I think that "hard" and "soft" should basically govern what happens to
outstanding requests once we've decided to try and reconnect the
socket. IOW, a socket disconnection should be treated more or less like
a major RPC timeout on NFS.

So in practical terms, let's assume for a moment that a server has
stopped responding at all while the client has outstanding requests. The
client then disconnects the socket and begins an attempt to reconnect.

If the mount is a hard mount, it should attempt to reissue the request
once the socket has been reconnected. Of course, open filehandles may
have changed, etc...so we may have to reencode requests but that's CIFS
for you. Callers should block until the socket has been reconnected and
the call reissued, but fatal signals should allow one to break out that
wait and return an error.

If the mount is a soft mount, we should return an error to the calling
application before or while attempting to reconnect the socket. That
allows the application to get the errors in timely fashion and deal
with them regardless of whether the reconnection is successful.

Soft mounts should also allow callers to tear down stateful objects
(files and locks, in particular) while the server is still down, so
that umounts can proceed in that case.

Open question here -- what should be done with new syscalls issued on
soft mounts while the socket is still unconnected? Should they block
until the socket is connected or should they return an immediate error?
I can see arguments for both. Maybe there should be a 3rd option?
(hard/soft/squishy)

Anyone have thoughts or comments?

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

Jeff Layton wrote:
:
:
> Timeouts:
> =========
> It's tempting to think of SMB as being very similar to NFS/RPC, but
> when it comes to low-level transport, there are significant
> differences. ONC-RPC was designed for connectionless transports and has
> the concept of a retransmission. SMB however does not -- it was
> originally layered on NetBIOS sessions and so has always been assumed
> to run on a connection-based transport.

...with the exception of the Direct Hosted IPX transport.  IPX is
connectionless.  In order to accommodate this connectionless transport, SMB
was actually modified (slightly).

That may seem to be in the "who cares" category, since those old transports
are essentially dead (much more dead than NBT, or even NBF).  Unfortunately,
the code to handle the old transports is still there in Windows, so there
are behaviors -- things like the timeouts you're talking about and the weird
VC=0 shutdown behvior -- that exist because of these old disused transports.

Ugh.  Eh?

Chris -)-----

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh-jFlgvBokg3lg9hUCZPvPmw@public.gmane.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh-zuGDro9SezXYtjvyW6yDsg@public.gmane.org

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Volker Lendecke]

On Fri, Dec 03, 2010 at 09:54:13PM -0600, Christopher R. Hertel wrote:
> That may seem to be in the "who cares" category, since those old transports
> are essentially dead (much more dead than NBT, or even NBF).  Unfortunately,
> the code to handle the old transports is still there in Windows, so there
> are behaviors -- things like the timeouts you're talking about and the weird
> VC=0 shutdown behvior -- that exist because of these old disused transports.

VC=0, how does Windows treat this facing NAT (masquerading)
networks? I've done tests in the past where Windows killed
valid connections from behind a NAT box when a new client
came in.

Volker

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Jeff Layton]

On Sat, 4 Dec 2010 09:13:21 +0100
Volker Lendecke <Volker.Lendecke-PS7XAnAlDA+VvDNblw4Uiw@public.gmane.org> wrote:

> On Fri, Dec 03, 2010 at 09:54:13PM -0600, Christopher R. Hertel wrote:
> > That may seem to be in the "who cares" category, since those old transports
> > are essentially dead (much more dead than NBT, or even NBF).  Unfortunately,
> > the code to handle the old transports is still there in Windows, so there
> > are behaviors -- things like the timeouts you're talking about and the weird
> > VC=0 shutdown behvior -- that exist because of these old disused transports.
> 
> VC=0, how does Windows treat this facing NAT (masquerading)
> networks? I've done tests in the past where Windows killed
> valid connections from behind a NAT box when a new client
> came in.
> 
> Volker

It seems like the best way to deal with this on the server side with
direct hosted TCP would be to treat VC=0 like any other VC number
(MS-CIFS says that this is allowed).

Ideally any new connection event from a host however should make the
server check the validity of any other connection from the same host.
That way you could release resources held by dead connections in case
the new one is a reconnect and needs to reclaim state.

The question is how to check that validity. Unfortunately, the best you
can probably do is rely on TCP keepalives.

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Shirish Pargaonkar]

On Sat, Dec 4, 2010 at 5:44 AM, Jeff Layton <jlayton@samba.org> wrote:

> On Sat, 4 Dec 2010 09:13:21 +0100
> Volker Lendecke <Volker.Lendecke@SerNet.DE> wrote:
>
> > On Fri, Dec 03, 2010 at 09:54:13PM -0600, Christopher R. Hertel wrote:
> > > That may seem to be in the "who cares" category, since those old
> transports
> > > are essentially dead (much more dead than NBT, or even NBF).
>  Unfortunately,
> > > the code to handle the old transports is still there in Windows, so
> there
> > > are behaviors -- things like the timeouts you're talking about and the
> weird
> > > VC=0 shutdown behvior -- that exist because of these old disused
> transports.
> >
> > VC=0, how does Windows treat this facing NAT (masquerading)
> > networks? I've done tests in the past where Windows killed
> > valid connections from behind a NAT box when a new client
> > came in.
> >
> > Volker
>
> It seems like the best way to deal with this on the server side with
> direct hosted TCP would be to treat VC=0 like any other VC number
> (MS-CIFS says that this is allowed).
>
> Ideally any new connection event from a host however should make the
> server check the validity of any other connection from the same host.
> That way you could release resources held by dead connections in case
> the new one is a reconnect and needs to reclaim state.
>
> The question is how to check that validity. Unfortunately, the best you
> can probably do is rely on TCP keepalives.
>
> --
> Jeff Layton <jlayton@samba.org>
>  --
> To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


Is SMB Echo command the only way to determine whether to reconnect or not?
The assumption here is SMB server is unresponsive.
There could be other circumstances on the server box (or even client box)
that are
slowing down the SMB server responses such as slow network, slow network
stack,
memory pressure etc.
So server could be fine all along and yet client would ask for reconnection!

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Jeff Layton]

On Sat, 4 Dec 2010 06:25:07 -0600
Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote:

> On Sat, Dec 4, 2010 at 5:44 AM, Jeff Layton <jlayton@samba.org> wrote:
> 
> > On Sat, 4 Dec 2010 09:13:21 +0100
> > Volker Lendecke <Volker.Lendecke@SerNet.DE> wrote:
> >
> > > On Fri, Dec 03, 2010 at 09:54:13PM -0600, Christopher R. Hertel wrote:
> > > > That may seem to be in the "who cares" category, since those old
> > transports
> > > > are essentially dead (much more dead than NBT, or even NBF).
> >  Unfortunately,
> > > > the code to handle the old transports is still there in Windows, so
> > there
> > > > are behaviors -- things like the timeouts you're talking about and the
> > weird
> > > > VC=0 shutdown behvior -- that exist because of these old disused
> > transports.
> > >
> > > VC=0, how does Windows treat this facing NAT (masquerading)
> > > networks? I've done tests in the past where Windows killed
> > > valid connections from behind a NAT box when a new client
> > > came in.
> > >
> > > Volker
> >
> > It seems like the best way to deal with this on the server side with
> > direct hosted TCP would be to treat VC=0 like any other VC number
> > (MS-CIFS says that this is allowed).
> >
> > Ideally any new connection event from a host however should make the
> > server check the validity of any other connection from the same host.
> > That way you could release resources held by dead connections in case
> > the new one is a reconnect and needs to reclaim state.
> >
> > The question is how to check that validity. Unfortunately, the best you
> > can probably do is rely on TCP keepalives.
> >
> > --
> > Jeff Layton <jlayton@samba.org>
> >  --
> > To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> > the body of a message to majordomo@vger.kernel.org
> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> 
> 
> Is SMB Echo command the only way to determine whether to reconnect or not?
> The assumption here is SMB server is unresponsive.
> There could be other circumstances on the server box (or even client box)
> that are
> slowing down the SMB server responses such as slow network, slow network
> stack,
> memory pressure etc.
> So server could be fine all along and yet client would ask for reconnection!

I think it's the best mechanism that the protocol has. If we aren't
going to use SMB echoes to detect an unresponsive server, then what
would you suggest? I don't think we can make calls wait indefinitely
for a response without a mechanism to determine when the server is gone
and attempt to reestablish the connection to it.

-- 
Jeff Layton <jlayton@samba.org>

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Shirish Pargaonkar]

On Sat, Dec 4, 2010 at 7:09 AM, Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
> On Sat, 4 Dec 2010 06:25:07 -0600
> Shirish Pargaonkar <shirishpargaonkar-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>> On Sat, Dec 4, 2010 at 5:44 AM, Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
>>
>> > On Sat, 4 Dec 2010 09:13:21 +0100
>> > Volker Lendecke <Volker.Lendecke-PS7XAnAlDA+VvDNblw4Uiw@public.gmane.org> wrote:
>> >
>> > > On Fri, Dec 03, 2010 at 09:54:13PM -0600, Christopher R. Hertel wrote:
>> > > > That may seem to be in the "who cares" category, since those old
>> > transports
>> > > > are essentially dead (much more dead than NBT, or even NBF).
>> >  Unfortunately,
>> > > > the code to handle the old transports is still there in Windows, so
>> > there
>> > > > are behaviors -- things like the timeouts you're talking about and the
>> > weird
>> > > > VC=0 shutdown behvior -- that exist because of these old disused
>> > transports.
>> > >
>> > > VC=0, how does Windows treat this facing NAT (masquerading)
>> > > networks? I've done tests in the past where Windows killed
>> > > valid connections from behind a NAT box when a new client
>> > > came in.
>> > >
>> > > Volker
>> >
>> > It seems like the best way to deal with this on the server side with
>> > direct hosted TCP would be to treat VC=0 like any other VC number
>> > (MS-CIFS says that this is allowed).
>> >
>> > Ideally any new connection event from a host however should make the
>> > server check the validity of any other connection from the same host.
>> > That way you could release resources held by dead connections in case
>> > the new one is a reconnect and needs to reclaim state.
>> >
>> > The question is how to check that validity. Unfortunately, the best you
>> > can probably do is rely on TCP keepalives.
>> >
>> > --
>> > Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>
>> >  --
>> > To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
>> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
>> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >
>>
>>
>> Is SMB Echo command the only way to determine whether to reconnect or not?
>> The assumption here is SMB server is unresponsive.
>> There could be other circumstances on the server box (or even client box)
>> that are
>> slowing down the SMB server responses such as slow network, slow network
>> stack,
>> memory pressure etc.
>> So server could be fine all along and yet client would ask for reconnection!
>
> I think it's the best mechanism that the protocol has. If we aren't
> going to use SMB echoes to detect an unresponsive server, then what
> would you suggest? I don't think we can make calls wait indefinitely
> for a response without a mechanism to determine when the server is gone
> and attempt to reestablish the connection to it.
>
> --
> Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>
>

I think we should use smb echo command as a means to let users know the
state of mount/server and let them decide.
If smb echo times out, cifs client should just log it and stop at that and a
very next request that receives a response (if any does) should log that server
is responding.

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Jeff Layton]

On Sat, 4 Dec 2010 08:06:46 -0600
Shirish Pargaonkar <shirishpargaonkar-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> On Sat, Dec 4, 2010 at 7:09 AM, Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
> > On Sat, 4 Dec 2010 06:25:07 -0600
> > Shirish Pargaonkar <shirishpargaonkar-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> >> On Sat, Dec 4, 2010 at 5:44 AM, Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
> >>
> >> > On Sat, 4 Dec 2010 09:13:21 +0100
> >> > Volker Lendecke <Volker.Lendecke-PS7XAnAlDA+VvDNblw4Uiw@public.gmane.org> wrote:
> >> >
> >> > > On Fri, Dec 03, 2010 at 09:54:13PM -0600, Christopher R. Hertel wrote:
> >> > > > That may seem to be in the "who cares" category, since those old
> >> > transports
> >> > > > are essentially dead (much more dead than NBT, or even NBF).
> >> >  Unfortunately,
> >> > > > the code to handle the old transports is still there in Windows, so
> >> > there
> >> > > > are behaviors -- things like the timeouts you're talking about and the
> >> > weird
> >> > > > VC=0 shutdown behvior -- that exist because of these old disused
> >> > transports.
> >> > >
> >> > > VC=0, how does Windows treat this facing NAT (masquerading)
> >> > > networks? I've done tests in the past where Windows killed
> >> > > valid connections from behind a NAT box when a new client
> >> > > came in.
> >> > >
> >> > > Volker
> >> >
> >> > It seems like the best way to deal with this on the server side with
> >> > direct hosted TCP would be to treat VC=0 like any other VC number
> >> > (MS-CIFS says that this is allowed).
> >> >
> >> > Ideally any new connection event from a host however should make the
> >> > server check the validity of any other connection from the same host.
> >> > That way you could release resources held by dead connections in case
> >> > the new one is a reconnect and needs to reclaim state.
> >> >
> >> > The question is how to check that validity. Unfortunately, the best you
> >> > can probably do is rely on TCP keepalives.
> >> >
> >> > --
> >> > Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>
> >> >  --
> >> > To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
> >> > the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> >
> >>
> >>
> >> Is SMB Echo command the only way to determine whether to reconnect or not?
> >> The assumption here is SMB server is unresponsive.
> >> There could be other circumstances on the server box (or even client box)
> >> that are
> >> slowing down the SMB server responses such as slow network, slow network
> >> stack,
> >> memory pressure etc.
> >> So server could be fine all along and yet client would ask for reconnection!
> >
> > I think it's the best mechanism that the protocol has. If we aren't
> > going to use SMB echoes to detect an unresponsive server, then what
> > would you suggest? I don't think we can make calls wait indefinitely
> > for a response without a mechanism to determine when the server is gone
> > and attempt to reestablish the connection to it.
> >
> > --
> > Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>
> >
> 
> I think we should use smb echo command as a means to let users know the
> state of mount/server and let them decide.
> If smb echo times out, cifs client should just log it and stop at that and a
> very next request that receives a response (if any does) should log that server
> is responding.

Could you elaborate a bit?

What do you mean by "let them decide" -- what steps would someone take
if the server stopped responding? Aside from killing the application,
what recourse would they have?

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Shirish Pargaonkar]

On Sat, Dec 4, 2010 at 8:22 AM, Jeff Layton <jlayton@samba.org> wrote:
> On Sat, 4 Dec 2010 08:06:46 -0600
> Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote:
>
>> On Sat, Dec 4, 2010 at 7:09 AM, Jeff Layton <jlayton@samba.org> wrote:
>> > On Sat, 4 Dec 2010 06:25:07 -0600
>> > Shirish Pargaonkar <shirishpargaonkar@gmail.com> wrote:
>> >
>> >> On Sat, Dec 4, 2010 at 5:44 AM, Jeff Layton <jlayton@samba.org> wrote:
>> >>
>> >> > On Sat, 4 Dec 2010 09:13:21 +0100
>> >> > Volker Lendecke <Volker.Lendecke@SerNet.DE> wrote:
>> >> >
>> >> > > On Fri, Dec 03, 2010 at 09:54:13PM -0600, Christopher R. Hertel wrote:
>> >> > > > That may seem to be in the "who cares" category, since those old
>> >> > transports
>> >> > > > are essentially dead (much more dead than NBT, or even NBF).
>> >> >  Unfortunately,
>> >> > > > the code to handle the old transports is still there in Windows, so
>> >> > there
>> >> > > > are behaviors -- things like the timeouts you're talking about and the
>> >> > weird
>> >> > > > VC=0 shutdown behvior -- that exist because of these old disused
>> >> > transports.
>> >> > >
>> >> > > VC=0, how does Windows treat this facing NAT (masquerading)
>> >> > > networks? I've done tests in the past where Windows killed
>> >> > > valid connections from behind a NAT box when a new client
>> >> > > came in.
>> >> > >
>> >> > > Volker
>> >> >
>> >> > It seems like the best way to deal with this on the server side with
>> >> > direct hosted TCP would be to treat VC=0 like any other VC number
>> >> > (MS-CIFS says that this is allowed).
>> >> >
>> >> > Ideally any new connection event from a host however should make the
>> >> > server check the validity of any other connection from the same host.
>> >> > That way you could release resources held by dead connections in case
>> >> > the new one is a reconnect and needs to reclaim state.
>> >> >
>> >> > The question is how to check that validity. Unfortunately, the best you
>> >> > can probably do is rely on TCP keepalives.
>> >> >
>> >> > --
>> >> > Jeff Layton <jlayton@samba.org>
>> >> >  --
>> >> > To unsubscribe from this list: send the line "unsubscribe linux-cifs" in
>> >> > the body of a message to majordomo@vger.kernel.org
>> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
>> >> >
>> >>
>> >>
>> >> Is SMB Echo command the only way to determine whether to reconnect or not?
>> >> The assumption here is SMB server is unresponsive.
>> >> There could be other circumstances on the server box (or even client box)
>> >> that are
>> >> slowing down the SMB server responses such as slow network, slow network
>> >> stack,
>> >> memory pressure etc.
>> >> So server could be fine all along and yet client would ask for reconnection!
>> >
>> > I think it's the best mechanism that the protocol has. If we aren't
>> > going to use SMB echoes to detect an unresponsive server, then what
>> > would you suggest? I don't think we can make calls wait indefinitely
>> > for a response without a mechanism to determine when the server is gone
>> > and attempt to reestablish the connection to it.
>> >
>> > --
>> > Jeff Layton <jlayton@samba.org>
>> >
>>
>> I think we should use smb echo command as a means to let users know the
>> state of mount/server and let them decide.
>> If smb echo times out, cifs client should just log it and stop at that and a
>> very next request that receives a response (if any does) should log that server
>> is responding.
>
> Could you elaborate a bit?
>
> What do you mean by "let them decide" -- what steps would someone take
> if the server stopped responding? Aside from killing the application,
> what recourse would they have?
>
> --
> Jeff Layton <jlayton@samba.org>
>

Jeff, I am not sure.  Basically I am coming from here:

I have a bug open, where an SMB server when slow to respond
(for a cifs client), if cifs client reconnects, causes data corruption
on the server. If left to its own, responses from server eventually
make through (without any intervention) and tests pass.

If an SMB server is unresponsive, how do we know it will respond to
a reconnect or a reconnect will help?  I do not know enough about
SMB servers to describe an unresponsive server i.e. how and when
it came to be unresponsive, how it handles transport layer then,
whether it corrects itself or how to correct it, how it handles
underlying physical file sytem etc..

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Jeff Layton]

On Sat, 4 Dec 2010 08:46:53 -0600
Shirish Pargaonkar <shirishpargaonkar-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> 
> Jeff, I am not sure.  Basically I am coming from here:
> 
> I have a bug open, where an SMB server when slow to respond
> (for a cifs client), if cifs client reconnects, causes data corruption
> on the server. If left to its own, responses from server eventually
> make through (without any intervention) and tests pass.
> 

I have a very similar bug open and that's what prompted me to go down
this road. You may want to test the patchset I proposed for cifs
against your reproducer.

> If an SMB server is unresponsive, how do we know it will respond to
> a reconnect or a reconnect will help? 
>
> I do not know enough about
> SMB servers to describe an unresponsive server i.e. how and when
> it came to be unresponsive, how it handles transport layer then,
> whether it corrects itself or how to correct it, how it handles
> underlying physical file sytem etc..

A reconnect may not help. The problem we have today however is that
Linux CIFS client is too cavalier with reconnects. It reconnects
the socket any time that a call has taken longer than an arbitrary
timeout. It tries to deal with that by varying timeouts with the
type of call, but I think that's a broken model that fails in many
situations.

It's impossible to predict how long it'll take the server to service a
particular call, as we can never be sure what the load on the
server and underlying storage is. A QPathInfo call may take just as
long as a write past EOF if the storage is being hammered.

The scheme I'm proposing makes the assumption that even when the server
is loaded, it'll still be able to respond to an echo. That may also
fail in certain situations, but empirical evidence has shown that
that it's generally true. This scheme won't fix every failure scenario,
but it should help the vast majority of situations where the server is
simply being slow to respond to a particular call.

I'm not opposed to what you're proposing, but it seems like a more
radical step than what I have proposed. We'd need to understand what
recourse the user would have in practice and what the behavior will be
in various failure scenarios. Leaving the processes hung and logging a
message when the server isn't responding isn't going to be very helpful
if there's nothing that can be done about it.

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

Shirish Pargaonkar wrote:
:
> Is SMB Echo command the only way to determine whether to reconnect or not?

The SMB Echo can only be sent from the client to the server.  The server has
no mechanism, within the SMB protocol, to "ping" the client.

The Echo is used by the client as a way to test whether it is the connection
that has failed or if the server is simply taking a very long time to
respond to a particular request.

> The assumption here is SMB server is unresponsive.
> There could be other circumstances on the server box (or even client
> box) that are slowing down the SMB server responses such as slow
> network, slow network stack, memory pressure etc.

Right.  So how do you know whether it is the transport connection or the SMB
server that is having trouble?  In theory, the SMB server should answer the
Echo right away, even if it is having other problems.

> So server could be fine all along and yet client would ask for reconnection!

If the network is slow, that's a transport problem.  It's also why the 45
second timeout value is adaptable.  See the earlier message in which I
pointed to the sections in [MS-CIFS] that explain the 45 second timeout and
point at the related KB articles.

Chris -)-----

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh-jFlgvBokg3lg9hUCZPvPmw@public.gmane.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh-zuGDro9SezXYtjvyW6yDsg@public.gmane.org

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Shirish Pargaonkar]

Is SMB Echo command the only way to determine whether to reconnect or not?
The assumption here is SMB server is unresponsive.
There could be other circumstances on the server box (or even client box)
that are slowing_down/grinding_down the SMB server responses such as
slow network, slow network stack, memory pressure etc.
SMB Server could be fine all along and yet client reconnects.

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Shirish Pargaonkar]

Is SMB Echo command the only way to determine whether to reconnect or not?
The assumption here is SMB server is unresponsive.
There could be other circumstances on the server box (or even client box)
that are slowing_down/grinding_down the SMB server responses such as
slow network, slow network stack, memory pressure etc.
SMB Server could be fine all along and yet client reconnects!

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

Jeff Layton wrote:
> On Sat, 4 Dec 2010 09:13:21 +0100
> Volker Lendecke <Volker.Lendecke-PS7XAnAlDA+VvDNblw4Uiw@public.gmane.org> wrote:
> 
>> On Fri, Dec 03, 2010 at 09:54:13PM -0600, Christopher R. Hertel wrote:
>>> That may seem to be in the "who cares" category, since those old transports
>>> are essentially dead (much more dead than NBT, or even NBF).  Unfortunately,
>>> the code to handle the old transports is still there in Windows, so there
>>> are behaviors -- things like the timeouts you're talking about and the weird
>>> VC=0 shutdown behvior -- that exist because of these old disused transports.
>> VC=0, how does Windows treat this facing NAT (masquerading)
>> networks? I've done tests in the past where Windows killed
>> valid connections from behind a NAT box when a new client
>> came in.
>>
>> Volker
> 
> It seems like the best way to deal with this on the server side with
> direct hosted TCP would be to treat VC=0 like any other VC number
> (MS-CIFS says that this is allowed).

The reasoning behind the VC=0 behavior does not apply to any contemporary
transport, so there is no reason to enforce that behavior.

> Ideally any new connection event from a host however should make the
> server check the validity of any other connection from the same host.
> That way you could release resources held by dead connections in case
> the new one is a reconnect and needs to reclaim state.

That's overkill, I think.  The transport, as you mention below, will take
care of dropped connections in due time.

> The question is how to check that validity. Unfortunately, the best you
> can probably do is rely on TCP keepalives.

...and that's probably the best you want to do.

Chris -)-----

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh-jFlgvBokg3lg9hUCZPvPmw@public.gmane.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh-zuGDro9SezXYtjvyW6yDsg@public.gmane.org

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

Volker Lendecke wrote:
> On Fri, Dec 03, 2010 at 09:54:13PM -0600, Christopher R. Hertel wrote:
>> That may seem to be in the "who cares" category, since those old transports
>> are essentially dead (much more dead than NBT, or even NBF).  Unfortunately,
>> the code to handle the old transports is still there in Windows, so there
>> are behaviors -- things like the timeouts you're talking about and the weird
>> VC=0 shutdown behvior -- that exist because of these old disused transports.
> 
> VC=0, how does Windows treat this facing NAT (masquerading)
> networks? I've done tests in the past where Windows killed
> valid connections from behind a NAT box when a new client
> came in.

I put a lot of info into [MS-CIFS] about how *and why* this works the way it
does.  See [MS-CIFS] 2.1.3.  Unfortunately, since [MS-CIFS] applies only to
W98, NT4, and earlier it doesn't tell you much about W2K and newer
implementations.  There's a KB article that talks about the problem a
little:  KB301673.

I thought that there was a registry setting that would allow you to tell the
Windows server NOT to close existing connections if it received a VC=0 from
the same IP.  Even when working with Microsoft I wasn't able to find
anything like that, beyond what's in that KB article.

Basically, they should give up on the VC=0 thing, since it's a throw-back to
the days when OS/2 could run SMB over connectionless IPX transport over
multiple modems in parallel.

Chris -)-----

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh@ubiqx.mn.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh@ubiqx.org

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Jeff Layton]

On Fri, 03 Dec 2010 21:54:13 -0600
"Christopher R. Hertel" <crh-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:

> Jeff Layton wrote:
> :
> :
> > Timeouts:
> > =========
> > It's tempting to think of SMB as being very similar to NFS/RPC, but
> > when it comes to low-level transport, there are significant
> > differences. ONC-RPC was designed for connectionless transports and has
> > the concept of a retransmission. SMB however does not -- it was
> > originally layered on NetBIOS sessions and so has always been assumed
> > to run on a connection-based transport.
> 
> ...with the exception of the Direct Hosted IPX transport.  IPX is
> connectionless.  In order to accommodate this connectionless transport, SMB
> was actually modified (slightly).
> 
> That may seem to be in the "who cares" category, since those old transports
> are essentially dead (much more dead than NBT, or even NBF).  Unfortunately,
> the code to handle the old transports is still there in Windows, so there
> are behaviors -- things like the timeouts you're talking about and the weird
> VC=0 shutdown behvior -- that exist because of these old disused transports.
> 
> Ugh.  Eh?
> 

Yeah, Direct Hosted IPX is a lot more like NFS/RPC. MS essentially
shoehorned in a way to do request retransmissions. It's a pretty
impressive kludge! I suppose we could theorize that that legacy is why
Windows times out individual requests so aggressively, but maybe there
are other design concerns behind that behavior.

In this case however it really is a "don't care" for us in Linux CIFS
land since we have never implemented it on top of IPX or any other
non-connected transport and have no plans to do so.

Thanks,
-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

Jeff Layton wrote:
> On Fri, 03 Dec 2010 21:54:13 -0600
> "Christopher R. Hertel" <crh@samba.org> wrote:
> 
>> Jeff Layton wrote:
>> :
>> :
>>> Timeouts:
>>> =========
>>> It's tempting to think of SMB as being very similar to NFS/RPC, but
>>> when it comes to low-level transport, there are significant
>>> differences. ONC-RPC was designed for connectionless transports and has
>>> the concept of a retransmission. SMB however does not -- it was
>>> originally layered on NetBIOS sessions and so has always been assumed
>>> to run on a connection-based transport.
>> ...with the exception of the Direct Hosted IPX transport.  IPX is
>> connectionless.  In order to accommodate this connectionless transport, SMB
>> was actually modified (slightly).
>>
>> That may seem to be in the "who cares" category, since those old transports
>> are essentially dead (much more dead than NBT, or even NBF).  Unfortunately,
>> the code to handle the old transports is still there in Windows, so there
>> are behaviors -- things like the timeouts you're talking about and the weird
>> VC=0 shutdown behvior -- that exist because of these old disused transports.
>>
>> Ugh.  Eh?
>>
> 
> Yeah, Direct Hosted IPX is a lot more like NFS/RPC. MS essentially
> shoehorned in a way to do request retransmissions. It's a pretty
> impressive kludge! I suppose we could theorize that that legacy is why
> Windows times out individual requests so aggressively, but maybe there
> are other design concerns behind that behavior.

I've seen the source code.  [Aiigh!  Bleach the eyes!]

I think that this is the primary reason for the timeout, but yeah... there
are other factors that are likely involved.

> In this case however it really is a "don't care" for us in Linux CIFS
> land since we have never implemented it on top of IPX or any other
> non-connected transport and have no plans to do so.

Yep!  I agree with Volker's vote that this is a case in which third-party
implementations should do "the right thing" rather than "what Windows does".

Chris -)-----

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh@ubiqx.mn.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh@ubiqx.org

Re: cifs client timeouts and hard/soft mounts

[Volker Lendecke]

On Fri, Dec 03, 2010 at 09:28:11PM -0500, Jeff Layton wrote:
> So, what does this mean for CIFS clients? I believe that the best
> behavior for the client is to *never* time out an individual request,
> aside from SMB echoes.

I like this concept.

Volker

Re: cifs client timeouts and hard/soft mounts

[Steve French]

On Sat, Dec 4, 2010 at 2:12 AM, Volker Lendecke
<Volker.Lendecke@sernet.de>wrote:

> On Fri, Dec 03, 2010 at 09:28:11PM -0500, Jeff Layton wrote:
> > So, what does this mean for CIFS clients? I believe that the best
> > behavior for the client is to *never* time out an individual request,
> > aside from SMB echoes.
>
> I like this concept.
>
>
That will break apps that can't take ctl-c though ...

-- 
Thanks,

Steve

Re: cifs client timeouts and hard/soft mounts

[Jeff Layton]

On Sun, 5 Dec 2010 19:42:30 -0600
Steve French <smfrench-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> On Sat, Dec 4, 2010 at 2:12 AM, Volker Lendecke
> <Volker.Lendecke-3ekOc4rQMZmzQB+pC5nmwQ@public.gmane.org>wrote:
> 
> > On Fri, Dec 03, 2010 at 09:28:11PM -0500, Jeff Layton wrote:
> > > So, what does this mean for CIFS clients? I believe that the best
> > > behavior for the client is to *never* time out an individual request,
> > > aside from SMB echoes.
> >
> > I like this concept.
> >
> >
> That will break apps that can't take ctl-c though ...
> 

How will waiting indefinitely for a response break applications?
Returning an error just because the server is slow seems far more
likely to break applications.

Now, in the (IMO unlikely) event that a server is responding to
echoes but not other calls, you'd have an that application will hang
until someone kills it. I think that's acceptable however:

It's an unlikely situation, and anyone who has a client faced with it
has a way to recover from the hang. They can kill the application. The
server in this case would be clearly broken however.

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: cifs client timeouts and hard/soft mounts

[Steve French]

On Sun, Dec 5, 2010 at 8:06 PM, Jeff Layton <jlayton@samba.org> wrote:

> On Sun, 5 Dec 2010 19:42:30 -0600
> Steve French <smfrench@gmail.com> wrote:
>
> > On Sat, Dec 4, 2010 at 2:12 AM, Volker Lendecke
> > <Volker.Lendecke@sernet.de>wrote:
> >
> > > On Fri, Dec 03, 2010 at 09:28:11PM -0500, Jeff Layton wrote:
> > > > So, what does this mean for CIFS clients? I believe that the best
> > > > behavior for the client is to *never* time out an individual request,
> > > > aside from SMB echoes.
> > >
> > > I like this concept.
> > >
> > >
> > That will break apps that can't take ctl-c though ...
> >
>
> How will waiting indefinitely for a response break applications?
> Returning an error just because the server is slow seems far more
> likely to break applications.
>
> Now, in the (IMO unlikely) event that a server is responding to
> echoes but not other calls, you'd have an that application will hang
> until someone kills it. I think that's acceptable however:
>
> It's an unlikely situation, and anyone who has a client faced with it
> has a way to recover from the hang. They can kill the application. The
> server in this case would be clearly broken however.
>
>
I am more worried about firewall rule changes and similar events
than about broken servers - but the idea of waiting forever on stat
to a server that is never going to respond seems odd.

-- 
Thanks,

Steve

Re: cifs client timeouts and hard/soft mounts

[Jeff Layton]

On Sun, 5 Dec 2010 20:16:46 -0600
Steve French <smfrench-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> On Sun, Dec 5, 2010 at 8:06 PM, Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
> 
> > On Sun, 5 Dec 2010 19:42:30 -0600
> > Steve French <smfrench-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
> >
> > > On Sat, Dec 4, 2010 at 2:12 AM, Volker Lendecke
> > > <Volker.Lendecke-3ekOc4rQMZmzQB+pC5nmwQ@public.gmane.org>wrote:
> > >
> > > > On Fri, Dec 03, 2010 at 09:28:11PM -0500, Jeff Layton wrote:
> > > > > So, what does this mean for CIFS clients? I believe that the best
> > > > > behavior for the client is to *never* time out an individual request,
> > > > > aside from SMB echoes.
> > > >
> > > > I like this concept.
> > > >
> > > >
> > > That will break apps that can't take ctl-c though ...
> > >
> >
> > How will waiting indefinitely for a response break applications?
> > Returning an error just because the server is slow seems far more
> > likely to break applications.
> >
> > Now, in the (IMO unlikely) event that a server is responding to
> > echoes but not other calls, you'd have an that application will hang
> > until someone kills it. I think that's acceptable however:
> >
> > It's an unlikely situation, and anyone who has a client faced with it
> > has a way to recover from the hang. They can kill the application. The
> > server in this case would be clearly broken however.
> >
> >
> I am more worried about firewall rule changes and similar events
> than about broken servers - but the idea of waiting forever on stat
> to a server that is never going to respond seems odd.
> 

But we won't wait forever...only if the server is still responding to
echoes. A server that responds to echoes but not other requests seems
very, very odd.

-- 
Jeff Layton <jlayton-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org>

Re: cifs client timeouts and hard/soft mounts

[Volker Lendecke]

Hi!

On Sun, Dec 05, 2010 at 08:16:46PM -0600, Steve French wrote:
> I am more worried about firewall rule changes and similar events
> than about broken servers - but the idea of waiting forever on stat
> to a server that is never going to respond seems odd.

That would be a strange fw rule that allows SMBEcho but not
other SMB requests. I think if someone puts up such a silly
rule, some pain is deserved :-)

Volker

Re: cifs client timeouts and hard/soft mounts

[Steve French]

On Sun, Dec 5, 2010 at 10:34 PM, Volker Lendecke
<Volker.Lendecke-3ekOc4rQMZmzQB+pC5nmwQ@public.gmane.org> wrote:
> Hi!
>
> On Sun, Dec 05, 2010 at 08:16:46PM -0600, Steve French wrote:
>> I am more worried about firewall rule changes and similar events
>> than about broken servers - but the idea of waiting forever on stat
>> to a server that is never going to respond seems odd.
>
> That would be a strange fw rule that allows SMBEcho but not
> other SMB requests. I think if someone puts up such a silly
> rule, some pain is deserved :-)

Aaah - remember the proxies that cut out "chatty" smb traffic by
responding on behalf of remote servers in the interest of optimizing
traffic over slow links :)


-- 
Thanks,

Steve

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[simo]

On Mon, 2010-12-06 at 10:28 -0600, Steve French wrote:
> On Sun, Dec 5, 2010 at 10:34 PM, Volker Lendecke
> <Volker.Lendecke@sernet.de> wrote:
> > Hi!
> >
> > On Sun, Dec 05, 2010 at 08:16:46PM -0600, Steve French wrote:
> >> I am more worried about firewall rule changes and similar events
> >> than about broken servers - but the idea of waiting forever on stat
> >> to a server that is never going to respond seems odd.
> >
> > That would be a strange fw rule that allows SMBEcho but not
> > other SMB requests. I think if someone puts up such a silly
> > rule, some pain is deserved :-)
> 
> Aaah - remember the proxies that cut out "chatty" smb traffic by
> responding on behalf of remote servers in the interest of optimizing
> traffic over slow links :)

They better send their own smb echos to remote servers then ...

Simo.

-- 
Simo Sorce
Samba Team GPL Compliance Officer <simo@samba.org>
Principal Software Engineer at Red Hat, Inc. <simo@redhat.com>

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

simo wrote:
> On Mon, 2010-12-06 at 10:28 -0600, Steve French wrote:
>> On Sun, Dec 5, 2010 at 10:34 PM, Volker Lendecke
>> <Volker.Lendecke-3ekOc4rQMZmzQB+pC5nmwQ@public.gmane.org> wrote:
>>> Hi!
>>>
>>> On Sun, Dec 05, 2010 at 08:16:46PM -0600, Steve French wrote:
>>>> I am more worried about firewall rule changes and similar events
>>>> than about broken servers - but the idea of waiting forever on stat
>>>> to a server that is never going to respond seems odd.
>>> That would be a strange fw rule that allows SMBEcho but not
>>> other SMB requests. I think if someone puts up such a silly
>>> rule, some pain is deserved :-)
>> Aaah - remember the proxies that cut out "chatty" smb traffic by
>> responding on behalf of remote servers in the interest of optimizing
>> traffic over slow links :)
> 
> They better send their own smb echos to remote servers then ...

The client-end proxy node is only interested in whether or not its peer node
is up and running.  The peer node, at the server end of the link, should be
responsible for knowing when the actual server is down.

...but this goes back to my question.  How much responsibility does the
Linux CIFS client have to ensure that the connection and server are both
working properly, and how much responsibility falls to the WAN accelerator?

I think it makes sense to do a little to mitigate WAN accelerator problems,
but the CIFS client needs to be as generic as possible so that it works well
in all environments.

Chris -)-----

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh-jFlgvBokg3lg9hUCZPvPmw@public.gmane.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh-zuGDro9SezXYtjvyW6yDsg@public.gmane.org

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Steve French]

On Mon, Dec 6, 2010 at 11:06 AM, Christopher R. Hertel <crh@samba.org> wrote:
> simo wrote:
>> On Mon, 2010-12-06 at 10:28 -0600, Steve French wrote:
>>> On Sun, Dec 5, 2010 at 10:34 PM, Volker Lendecke
>>> <Volker.Lendecke@sernet.de> wrote:
>>>> Hi!
>>>>
>>>> On Sun, Dec 05, 2010 at 08:16:46PM -0600, Steve French wrote:
>>>>> I am more worried about firewall rule changes and similar events
>>>>> than about broken servers - but the idea of waiting forever on stat
>>>>> to a server that is never going to respond seems odd.
>>>> That would be a strange fw rule that allows SMBEcho but not
>>>> other SMB requests. I think if someone puts up such a silly
>>>> rule, some pain is deserved :-)
>>> Aaah - remember the proxies that cut out "chatty" smb traffic by
>>> responding on behalf of remote servers in the interest of optimizing
>>> traffic over slow links :)
>>
>> They better send their own smb echos to remote servers then ...
>
> The client-end proxy node is only interested in whether or not its peer node
> is up and running.  The peer node, at the server end of the link, should be
> responsible for knowing when the actual server is down.
>
> ...but this goes back to my question.  How much responsibility does the
> Linux CIFS client have to ensure that the connection and server are both
> working properly, and how much responsibility falls to the WAN accelerator?
>
> I think it makes sense to do a little to mitigate WAN accelerator problems,
> but the CIFS client needs to be as generic as possible so that it works well
> in all environments.

That is a good broad "design goal"
Obviously using smb echo will help.  In the smb2 code, I issued an smb2
at mount time to do a rough estimate of link speed, but with both cifs and
smb2 code we need to be issuing an echo (at least) when the server is
unresponsive, but we may find that the action to take soft/hard, retry forver
vs. giveup after one or two cancel/retry takes some experimentation to
get right.



-- 
Thanks,

Steve

Re: cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

Steve,

I've done a lot of consulting work for companies that create this sort of
proxy.  Companies like Peribit (now part of Juniper), FineGround (now part
of Cisco), and Certeon (now part of history, I think).

You are correct.  A WAN accelerator that does proxy caching might very well
respond to an SMB Echo and various meta-data requests without actually
sending those requests to the "real" server.  If the link is down, or if the
real server is down, the proxy may still respond to some SMB messages.

To some extent, that is a problem inherent in the use of WAN accleration
systems.

Question (for which I do not have an answer):  How much work do you want to
do to make their intentionally broken model work?

WAN accelerators are actually cool technology.  I like them, but I also know
(and most of the companies that make them know) that there are inherent
problems when proxying SMB servers.

Chris -)-----

Steve French wrote:
> On Sun, Dec 5, 2010 at 10:34 PM, Volker Lendecke
> <Volker.Lendecke@sernet.de> wrote:
>> Hi!
>>
>> On Sun, Dec 05, 2010 at 08:16:46PM -0600, Steve French wrote:
>>> I am more worried about firewall rule changes and similar events
>>> than about broken servers - but the idea of waiting forever on stat
>>> to a server that is never going to respond seems odd.
>> That would be a strange fw rule that allows SMBEcho but not
>> other SMB requests. I think if someone puts up such a silly
>> rule, some pain is deserved :-)
> 
> Aaah - remember the proxies that cut out "chatty" smb traffic by
> responding on behalf of remote servers in the interest of optimizing
> traffic over slow links :)
> 
> 

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh@ubiqx.mn.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh@ubiqx.org

Re: cifs client timeouts and hard/soft mounts

[Jeff Layton]

On Mon, 06 Dec 2010 11:01:12 -0600
"Christopher R. Hertel" <crh-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:

> 
> Question (for which I do not have an answer):  How much work do you want to
> do to make their intentionally broken model work?
> 

Very little...live by the sword, die by the sword :)

I'm not opposed to trying to work around this sort of brokenness, but
not at the expense of more conventional configurations.

If the proxy keeps responding to echoes regardless of what happens on
the server side of its connection then that sounds really broken to me.
Does it do nothing to detect whether the server is still around? I don't
see a way for us to detect such a broken device if not.

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Re: cifs client timeouts and hard/soft mounts

[Jeremy Allison]

On Mon, Dec 06, 2010 at 12:42:55PM -0500, Jeff Layton wrote:
> On Mon, 06 Dec 2010 11:01:12 -0600
> "Christopher R. Hertel" <crh-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
> 
> > 
> > Question (for which I do not have an answer):  How much work do you want to
> > do to make their intentionally broken model work?
> > 
> 
> Very little...live by the sword, die by the sword :)
> 
> I'm not opposed to trying to work around this sort of brokenness, but
> not at the expense of more conventional configurations.
> 
> If the proxy keeps responding to echoes regardless of what happens on
> the server side of its connection then that sounds really broken to me.
> Does it do nothing to detect whether the server is still around? I don't
> see a way for us to detect such a broken device if not.

There's simply nothing you can do. If a WAN device fakes application
level probes (which is what SMBecho is) then you're toast. Just ignore
WAN devices. They're supposed to be transparent.

Jeremy.

Re: cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

Jeff Layton wrote:
> On Mon, 06 Dec 2010 11:01:12 -0600
> "Christopher R. Hertel" <crh-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
> 
>> Question (for which I do not have an answer):  How much work do you want to
>> do to make their intentionally broken model work?
>>
> 
> Very little...live by the sword, die by the sword :)
> 
> I'm not opposed to trying to work around this sort of brokenness, but
> not at the expense of more conventional configurations.
> 
> If the proxy keeps responding to echoes regardless of what happens on
> the server side of its connection then that sounds really broken to me.
> Does it do nothing to detect whether the server is still around? I don't
> see a way for us to detect such a broken device if not.

There are a lot of companies out there that do WAN acceleration and each of
them do things differently, so there is no good answer to your quite
rational question.  Some of the vendors are very focused on "just like the
real thing" behavior while others are willing to compromise behavior in
favor of acceleration.

The best-known product in this market would be Riverbed.  BlueCoat is
another, I think.

Chris -)-----

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh-jFlgvBokg3lg9hUCZPvPmw@public.gmane.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh-zuGDro9SezXYtjvyW6yDsg@public.gmane.org

Re: cifs client timeouts and hard/soft mounts

[Volker Lendecke]

On Mon, Dec 06, 2010 at 12:54:30PM -0600, Christopher R. Hertel wrote:
> There are a lot of companies out there that do WAN acceleration and each of
> them do things differently, so there is no good answer to your quite
> rational question.  Some of the vendors are very focused on "just like the
> real thing" behavior while others are willing to compromise behavior in
> favor of acceleration.
> 
> The best-known product in this market would be Riverbed.  BlueCoat is
> another, I think.

Sorry, but if a WAN accelerator does not have the smarts to
see that if a client sends smbechos, it is in trouble, then
that WAN accelerator is just broken. It does not necessarily
need to send these echos across the WAN link, but it must
trigger its own server liveliness check at this point.

Volker

Re: cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

Volker Lendecke wrote:
> On Mon, Dec 06, 2010 at 12:54:30PM -0600, Christopher R. Hertel wrote:
>> There are a lot of companies out there that do WAN acceleration and each of
>> them do things differently, so there is no good answer to your quite
>> rational question.  Some of the vendors are very focused on "just like the
>> real thing" behavior while others are willing to compromise behavior in
>> favor of acceleration.
>>
>> The best-known product in this market would be Riverbed.  BlueCoat is
>> another, I think.
> 
> Sorry, but if a WAN accelerator does not have the smarts to
> see that if a client sends smbechos, it is in trouble, then
> that WAN accelerator is just broken. It does not necessarily
> need to send these echos across the WAN link, but it must
> trigger its own server liveliness check at this point.

I agree.

That's why these companies hire me, though.  They have *no clue* when it
comes to CIFS and they run into brick walls at full speed.

When it comes down to it, though, anyone with smarts and half a clue would
avoid SMB/CIFS if at all possible.  Most of what we do with Samba and the
CIFS client is mitigate stupidity.  :)

...and we're good at it too!

Chris -)-----

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh@ubiqx.mn.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh@ubiqx.org

Re: cifs client timeouts and hard/soft mounts

[Volker Lendecke]

On Mon, Dec 06, 2010 at 02:38:37PM -0600, Christopher R. Hertel wrote:
> > Sorry, but if a WAN accelerator does not have the smarts to
> > see that if a client sends smbechos, it is in trouble, then
> > that WAN accelerator is just broken. It does not necessarily
> > need to send these echos across the WAN link, but it must
> > trigger its own server liveliness check at this point.
> 
> I agree.
> 
> That's why these companies hire me, though.  They have *no clue* when it
> comes to CIFS and they run into brick walls at full speed.
> 
> When it comes down to it, though, anyone with smarts and half a clue would
> avoid SMB/CIFS if at all possible.  Most of what we do with Samba and the
> CIFS client is mitigate stupidity.  :)

AFS to the rescue! (Sorry, wrong list I think... :-)

Volker

Re: cifs client timeouts and hard/soft mounts

[Steve French]

On Mon, Dec 6, 2010 at 2:38 PM, Christopher R. Hertel <crh-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org> wrote:
> Volker Lendecke wrote:
>> On Mon, Dec 06, 2010 at 12:54:30PM -0600, Christopher R. Hertel wrote:
>>> There are a lot of companies out there that do WAN acceleration and each of
>>> them do things differently, so there is no good answer to your quite
>>> rational question.  Some of the vendors are very focused on "just like the
>>> real thing" behavior while others are willing to compromise behavior in
>>> favor of acceleration.
>>>
>>> The best-known product in this market would be Riverbed.  BlueCoat is
>>> another, I think.
>>
>> Sorry, but if a WAN accelerator does not have the smarts to
>> see that if a client sends smbechos, it is in trouble, then
>> that WAN accelerator is just broken. It does not necessarily
>> need to send these echos across the WAN link, but it must
>> trigger its own server liveliness check at this point.
>
> I agree.
>
> That's why these companies hire me, though.  They have *no clue* when it
> comes to CIFS and they run into brick walls at full speed.
>
> When it comes down to it, though, anyone with smarts and half a clue would
> avoid SMB/CIFS if at all possible.  Most of what we do with Samba and the
> CIFS client is mitigate stupidity.  :)
>
> ...and we're good at it too!

Now that I have experimented with smb2,  it does seem better (and the
alternatives are worse)



-- 
Thanks,

Steve

Re: cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

Steve French wrote:
:
> Now that I have experimented with smb2,  it does seem better (and the
> alternatives are worse)

Yeah... but they're already headed down the same road that they took with
SMB.  Let's see where SMB2 is in 5, 10, 20 years.

I'll see what I can do to get José ready to take over for us when we're
shipped off to the home.

Chris -)-----

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh-jFlgvBokg3lg9hUCZPvPmw@public.gmane.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh-zuGDro9SezXYtjvyW6yDsg@public.gmane.org

Re: [cifs-protocol] cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

Christopher R. Hertel wrote:
> Steve French wrote:
> :
>> Now that I have experimented with smb2,  it does seem better (and the
>> alternatives are worse)
> 
> Yeah... but they're already headed down the same road that they took with
> SMB.  Let's see where SMB2 is in 5, 10, 20 years.

I want to apologize for the above comment.  It is much more negative
sounding than I had intended when I sent it off without double-checking it.

I agree with Steve that SMB2 is much better than SMB, and better than many
other alternatives.  The improved communication with Microsoft that
third-party developers have seen over the past three years or so will, I
hope, go a long way toward keeping it clean and avoiding the build-up of
obsolescent commands and subcommands that SMB experienced.

Now, if we could just get a little momentum behind the UNIXSMB2 effort...

> I'll see what I can do to get José ready to take over for us when we're
> shipped off to the home.

I'll still do this.  We have a fresh generation coming up, and I think
that's something to encourage.

Chrudz -)-----

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh-jFlgvBokg3lg9hUCZPvPmw@public.gmane.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh-zuGDro9SezXYtjvyW6yDsg@public.gmane.org

RE: cifs client timeouts and hard/soft mounts

[Matt Ficken (Insight Global)]

If the connection from the WAN Accelerator to the origin SMB server is down (ie the origin SMB server is down), shouldn't the WAN Accelerator realize that, and then not be responding to SMB echoes (at least responding as though it was the origin SMB server)?

-----Original Message-----
From: samba-technical-bounces@lists.samba.org [mailto:samba-technical-bounces@lists.samba.org] On Behalf Of Volker Lendecke
Sent: Monday, December 06, 2010 12:17 PM
To: Christopher R. Hertel
Cc: Steve French; cifs-protocol@samba.org; samba-technical@lists.samba.org; Jeff Layton; linux-cifs@vger.kernel.org
Subject: Re: cifs client timeouts and hard/soft mounts

On Mon, Dec 06, 2010 at 12:54:30PM -0600, Christopher R. Hertel wrote:
> There are a lot of companies out there that do WAN acceleration and each of
> them do things differently, so there is no good answer to your quite
> rational question.  Some of the vendors are very focused on "just like the
> real thing" behavior while others are willing to compromise behavior in
> favor of acceleration.
> 
> The best-known product in this market would be Riverbed.  BlueCoat is
> another, I think.

Sorry, but if a WAN accelerator does not have the smarts to
see that if a client sends smbechos, it is in trouble, then
that WAN accelerator is just broken. It does not necessarily
need to send these echos across the WAN link, but it must
trigger its own server liveliness check at this point.

Volker

Re: cifs client timeouts and hard/soft mounts

[Christopher R. Hertel]

Matt Ficken (Insight Global) wrote:
> If the connection from the WAN Accelerator to the origin SMB server is down (ie the origin SMB server is down), shouldn't the WAN Accelerator realize that, and then not be responding to SMB echoes (at least responding as though it was the origin SMB server)?

In theory, theory and practice are the same.  In practice they're not.

In other words, "should do" and "does do" are two very different things.  :)

...but I agree with Volker and Jeremy and others.  It's not our[1] job to
fix these things unless the fix is generally beneficial to the implementation.

Chris -)-----
[1]Where "our" refers to SMB/CIFS implementers in general.

> -----Original Message-----
> From: samba-technical-bounces-w/Ol4Ecudpl8XjKLYN78aQ@public.gmane.org [mailto:samba-technical-bounces-w/Ol4Ecudpl8XjKLYN78aQ@public.gmane.org] On Behalf Of Volker Lendecke
> Sent: Monday, December 06, 2010 12:17 PM
> To: Christopher R. Hertel
> Cc: Steve French; cifs-protocol-eUNUBHrolfbYtjvyW6yDsg@public.gmane.org; samba-technical-w/Ol4Ecudpl8XjKLYN78aQ@public.gmane.org; Jeff Layton; linux-cifs-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> Subject: Re: cifs client timeouts and hard/soft mounts
> 
> On Mon, Dec 06, 2010 at 12:54:30PM -0600, Christopher R. Hertel wrote:
>> There are a lot of companies out there that do WAN acceleration and each of
>> them do things differently, so there is no good answer to your quite
>> rational question.  Some of the vendors are very focused on "just like the
>> real thing" behavior while others are willing to compromise behavior in
>> favor of acceleration.
>>
>> The best-known product in this market would be Riverbed.  BlueCoat is
>> another, I think.
> 
> Sorry, but if a WAN accelerator does not have the smarts to
> see that if a client sends smbechos, it is in trouble, then
> that WAN accelerator is just broken. It does not necessarily
> need to send these echos across the WAN link, but it must
> trigger its own server liveliness check at this point.
> 
> Volker
> 

-- 
"Implementing CIFS - the Common Internet FileSystem" ISBN: 013047116X
Samba Team -- http://www.samba.org/     -)-----   Christopher R. Hertel
jCIFS Team -- http://jcifs.samba.org/   -)-----   ubiqx development, uninq.
ubiqx Team -- http://www.ubiqx.org/     -)-----   crh-jFlgvBokg3lg9hUCZPvPmw@public.gmane.org
OnLineBook -- http://ubiqx.org/cifs/    -)-----   crh-zuGDro9SezXYtjvyW6yDsg@public.gmane.org

Re: cifs client timeouts and hard/soft mounts

[Andrew Bartlett]

[-- Attachment #1: Type: text/plain, Size: 918 bytes --]

On Sat, 2010-12-04 at 09:12 +0100, Volker Lendecke wrote:
> On Fri, Dec 03, 2010 at 09:28:11PM -0500, Jeff Layton wrote:
> > So, what does this mean for CIFS clients? I believe that the best
> > behavior for the client is to *never* time out an individual request,
> > aside from SMB echoes.
> 
> I like this concept.

+1

If the server is up and running enough to respond with smb echo replies,
then it seems reasonable to assume it is still dealing with the other
requests, or if it wishes to abandon a pending request, it will respond
to it (eventually) with an error.  

Volker,

Is it correct to say that this is how Samba3 deals with slow calls and
Windows clients, using the echo handling child?

Andrew Bartlett

-- 
Andrew Bartlett                                http://samba.org/~abartlet/
Authentication Developer, Samba Team           http://samba.org
Samba Developer, Cisco Inc.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 190 bytes --]

Re: cifs client timeouts and hard/soft mounts

[Volker Lendecke]

On Mon, Dec 06, 2010 at 01:46:31PM +1100, Andrew Bartlett wrote:
> Is it correct to say that this is how Samba3 deals with slow calls and
> Windows clients, using the echo handling child?

Yes.

Volker