Writes greater than 64k fails with -ENOSPC

Writes greater than 64k fails with -ENOSPC

[Suresh Jayaraman]

Hi all,

I'm looking into a report on 3.0 based kernel (plus stable fixes)
where writes greater than 64k to a NAS (Hitachi NAS) is failing
(simple dd). The problem was not seen with a 2.6.32-ish kernel.
Also, note that the problem is not seen with other Servers such
as Windows 2003 or Windows 8 Servers.

The strace output shows the close() fall fails with -ENOSPC. 

The relevant cFYI snip

Jan 23 08:31:45 vsusix02 kernel: [1003552.227274] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
Jan 23 08:31:45 vsusix02 kernel: [1003552.227277] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 127044
Jan 23 08:31:45 vsusix02 kernel: [1003552.345848] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.393906] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssmb.c: async write at 1015808 8192 bytes
Jan 23 08:31:45 vsusix02 kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
Jan 23 08:31:45 vsusix02 kernel: [1003552.393914] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 8260
Jan 23 08:31:45 vsusix02 kernel: [1003552.479378] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.481215] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
Jan 23 08:31:45 vsusix02 kernel: [1003552.481260] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28


The problem seems to be that during close(), we try to flush the
buffers by calling cifs_flush() which in turn will call
filemap_write_and_wait() to wait on the pages under writeback to
complete. do_writepages() will invoke cifs_writepages() which is 
perhaps returning -ENOSPC and it is propagated back.

There are no quota restrictions or disk space problems. The
tcpdump output doesn't show any errors during the write or close.

I'm not sure what could be causing the problem and would
appreciate any clues or debugging suggestions.

Thanks

-- 
Suresh Jayaraman

Re: Writes greater than 64k fails with -ENOSPC

[Jeff Layton]

On Tue, 29 Jan 2013 17:54:21 +0530
Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:

> Hi all,
> 
> I'm looking into a report on 3.0 based kernel (plus stable fixes)
> where writes greater than 64k to a NAS (Hitachi NAS) is failing
> (simple dd). The problem was not seen with a 2.6.32-ish kernel.
> Also, note that the problem is not seen with other Servers such
> as Windows 2003 or Windows 8 Servers.
> 
> The strace output shows the close() fall fails with -ENOSPC. 
> 
> The relevant cFYI snip
> 
> Jan 23 08:31:45 vsusix02 kernel: [1003552.227274] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
> Jan 23 08:31:45 vsusix02 kernel: [1003552.227277] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 127044
> Jan 23 08:31:45 vsusix02 kernel: [1003552.345848] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
> Jan 23 08:31:45 vsusix02 kernel: [1003552.393906] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssmb.c: async write at 1015808 8192 bytes
> Jan 23 08:31:45 vsusix02 kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
> Jan 23 08:31:45 vsusix02 kernel: [1003552.393914] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 8260
> Jan 23 08:31:45 vsusix02 kernel: [1003552.479378] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
> Jan 23 08:31:45 vsusix02 kernel: [1003552.481215] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
> Jan 23 08:31:45 vsusix02 kernel: [1003552.481260] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28
> 
> 
> The problem seems to be that during close(), we try to flush the
> buffers by calling cifs_flush() which in turn will call
> filemap_write_and_wait() to wait on the pages under writeback to
> complete. do_writepages() will invoke cifs_writepages() which is 
> perhaps returning -ENOSPC and it is propagated back.
> 
> There are no quota restrictions or disk space problems. The
> tcpdump output doesn't show any errors during the write or close.
> 
> I'm not sure what could be causing the problem and would
> appreciate any clues or debugging suggestions.
> 
> Thanks
> 

I'd look for "short" write replies. We have this in
cifs_writev_callback:

                if (written < wdata->bytes)
                        wdata->result = -ENOSPC;

IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX
responses, but I remember it not being 100% clear...

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

RE: Writes greater than 64k fails with -ENOSPC

[Tom Talpey]

> -----Original Message-----
> From: linux-cifs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-cifs-
> owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Jeff Layton
> Sent: Tuesday, January 29, 2013 7:29 PM
> To: Suresh Jayaraman
> Cc: linux-cifs
> Subject: Re: Writes greater than 64k fails with -ENOSPC
> 
> On Tue, 29 Jan 2013 17:54:21 +0530
> Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> 
> > Hi all,
> >
> > I'm looking into a report on 3.0 based kernel (plus stable fixes)
> > where writes greater than 64k to a NAS (Hitachi NAS) is failing
> > (simple dd). The problem was not seen with a 2.6.32-ish kernel.
> > Also, note that the problem is not seen with other Servers such as
> > Windows 2003 or Windows 8 Servers.
> >
> > The strace output shows the close() fall fails with -ENOSPC.
> >
> > The relevant cFYI snip
> >
> > Jan 23 08:31:45 vsusix02 kernel: [1003552.227274]
> > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
> > ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
> > [1003552.227277]
> > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
> > ort.c: Sending smb:  total_len 127044 Jan 23 08:31:45 vsusix02 kernel:
> > [1003552.345848]
> > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
> > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > [1003552.393906]
> > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssm
> > b.c: async write at 1015808 8192 bytes Jan 23 08:31:45 vsusix02
> > kernel: [1003552.393911]
> > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
> > ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
> > [1003552.393914]
> > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
> > ort.c: Sending smb:  total_len 8260 Jan 23 08:31:45 vsusix02 kernel:
> > [1003552.479378]
> > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
> > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > [1003552.481215]
> > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
> > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > [1003552.481260]
> > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c
> > : Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28
> >
> >
> > The problem seems to be that during close(), we try to flush the
> > buffers by calling cifs_flush() which in turn will call
> > filemap_write_and_wait() to wait on the pages under writeback to
> > complete. do_writepages() will invoke cifs_writepages() which is
> > perhaps returning -ENOSPC and it is propagated back.
> >
> > There are no quota restrictions or disk space problems. The tcpdump
> > output doesn't show any errors during the write or close.
> >
> > I'm not sure what could be causing the problem and would appreciate
> > any clues or debugging suggestions.
> >
> > Thanks
> >
> 
> I'd look for "short" write replies. We have this in
> cifs_writev_callback:
> 
>                 if (written < wdata->bytes)
>                         wdata->result = -ENOSPC;

If the server does not return an error, then the client can conclude that at least something was written, based on the value of CountOfBytesWritten in the response. Unless this is zero, I think the client substituting ENOSPC may be incorrect.

> IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX responses,
> but I remember it not being 100% clear...

File a doc issue! Happy to clarify it if needed.

Tom.

Re: Writes greater than 64k fails with -ENOSPC

[Jeff Layton]

On Wed, 30 Jan 2013 14:06:19 +0000
Tom Talpey <ttalpey-0li6OtcxBFHby3iVrkZq2A@public.gmane.org> wrote:

> > -----Original Message-----
> > From: linux-cifs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-cifs-
> > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Jeff Layton
> > Sent: Tuesday, January 29, 2013 7:29 PM
> > To: Suresh Jayaraman
> > Cc: linux-cifs
> > Subject: Re: Writes greater than 64k fails with -ENOSPC
> > 
> > On Tue, 29 Jan 2013 17:54:21 +0530
> > Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> > 
> > > Hi all,
> > >
> > > I'm looking into a report on 3.0 based kernel (plus stable fixes)
> > > where writes greater than 64k to a NAS (Hitachi NAS) is failing
> > > (simple dd). The problem was not seen with a 2.6.32-ish kernel.
> > > Also, note that the problem is not seen with other Servers such as
> > > Windows 2003 or Windows 8 Servers.
> > >
> > > The strace output shows the close() fall fails with -ENOSPC.
> > >
> > > The relevant cFYI snip
> > >
> > > Jan 23 08:31:45 vsusix02 kernel: [1003552.227274]
> > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
> > > ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
> > > [1003552.227277]
> > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
> > > ort.c: Sending smb:  total_len 127044 Jan 23 08:31:45 vsusix02 kernel:
> > > [1003552.345848]
> > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
> > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > [1003552.393906]
> > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssm
> > > b.c: async write at 1015808 8192 bytes Jan 23 08:31:45 vsusix02
> > > kernel: [1003552.393911]
> > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
> > > ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
> > > [1003552.393914]
> > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transp
> > > ort.c: Sending smb:  total_len 8260 Jan 23 08:31:45 vsusix02 kernel:
> > > [1003552.479378]
> > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
> > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > [1003552.481215]
> > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connec
> > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > [1003552.481260]
> > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c
> > > : Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28
> > >
> > >
> > > The problem seems to be that during close(), we try to flush the
> > > buffers by calling cifs_flush() which in turn will call
> > > filemap_write_and_wait() to wait on the pages under writeback to
> > > complete. do_writepages() will invoke cifs_writepages() which is
> > > perhaps returning -ENOSPC and it is propagated back.
> > >
> > > There are no quota restrictions or disk space problems. The tcpdump
> > > output doesn't show any errors during the write or close.
> > >
> > > I'm not sure what could be causing the problem and would appreciate
> > > any clues or debugging suggestions.
> > >
> > > Thanks
> > >
> > 
> > I'd look for "short" write replies. We have this in
> > cifs_writev_callback:
> > 
> >                 if (written < wdata->bytes)
> >                         wdata->result = -ENOSPC;
> 
> If the server does not return an error, then the client can conclude that at least something was written, based on the value of CountOfBytesWritten in the response. Unless this is zero, I think the client substituting ENOSPC may be incorrect.
> 

That's probably the case, but short writes are really tricky to deal
with...

Say we're spraying out a bunch of 64k writes, but the server then
replies that only 63k of each was written. Now we have to go back and
redirty the pages that didn't get fully written. That's fine and not
too hard to do, but now you have a bunch of dirty pages sprinkled
around the file every 63k.

So, on the next pass through the dirty page radix tree, we go and try
to issue another WRITE_ANDX call. I'm not sure if we'd have to go back
and restart writepages ourselves at that point for the WB_SYNC_ALL case.

Note that this sort of server behavior was the crux of my argument with
Steve a few months ago about defaulting to 64k writes on servers w/o
POSIX extensions. We can be reasonably sure that most servers handle
64k writes OK since that's what Windows does.

The spec is not 100% clear on whether servers are *required* to support
arbitrarily large writes up to the 128k limit. Clearly there are some
that do not, and a larger default is problematic against those servers.

> > IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX responses,
> > but I remember it not being 100% clear...
> 
> File a doc issue! Happy to clarify it if needed.
> 

That might be nice to do once Suresh confirms whether that's the case.

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

RE: Writes greater than 64k fails with -ENOSPC

[Tom Talpey]

> -----Original Message-----
> From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> Sent: Wednesday, January 30, 2013 9:37 AM
> To: Tom Talpey
> Cc: Suresh Jayaraman; linux-cifs
> Subject: Re: Writes greater than 64k fails with -ENOSPC
> 
> On Wed, 30 Jan 2013 14:06:19 +0000
> Tom Talpey <ttalpey-0li6OtcxBFHby3iVrkZq2A@public.gmane.org> wrote:
> 
> > > -----Original Message-----
> > > From: linux-cifs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-cifs-
> > > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Jeff Layton
> > > Sent: Tuesday, January 29, 2013 7:29 PM
> > > To: Suresh Jayaraman
> > > Cc: linux-cifs
> > > Subject: Re: Writes greater than 64k fails with -ENOSPC
> > >
> > > On Tue, 29 Jan 2013 17:54:21 +0530
> > > Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm looking into a report on 3.0 based kernel (plus stable fixes)
> > > > where writes greater than 64k to a NAS (Hitachi NAS) is failing
> > > > (simple dd). The problem was not seen with a 2.6.32-ish kernel.
> > > > Also, note that the problem is not seen with other Servers such as
> > > > Windows 2003 or Windows 8 Servers.
> > > >
> > > > The strace output shows the close() fall fails with -ENOSPC.
> > > >
> > > > The relevant cFYI snip
> > > >
> > > > Jan 23 08:31:45 vsusix02 kernel: [1003552.227274]
> > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
> > > > ansp
> > > > ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
> > > > [1003552.227277]
> > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
> > > > ansp
> > > > ort.c: Sending smb:  total_len 127044 Jan 23 08:31:45 vsusix02 kernel:
> > > > [1003552.345848]
> > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
> > > > nnec
> > > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > > [1003552.393906]
> > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/ci
> > > > fssm
> > > > b.c: async write at 1015808 8192 bytes Jan 23 08:31:45 vsusix02
> > > > kernel: [1003552.393911]
> > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
> > > > ansp
> > > > ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
> > > > [1003552.393914]
> > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
> > > > ansp
> > > > ort.c: Sending smb:  total_len 8260 Jan 23 08:31:45 vsusix02 kernel:
> > > > [1003552.479378]
> > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
> > > > nnec
> > > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > > [1003552.481215]
> > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
> > > > nnec
> > > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > > [1003552.481260]
> > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/fi
> > > > le.c
> > > > : Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28
> > > >
> > > >
> > > > The problem seems to be that during close(), we try to flush the
> > > > buffers by calling cifs_flush() which in turn will call
> > > > filemap_write_and_wait() to wait on the pages under writeback to
> > > > complete. do_writepages() will invoke cifs_writepages() which is
> > > > perhaps returning -ENOSPC and it is propagated back.
> > > >
> > > > There are no quota restrictions or disk space problems. The
> > > > tcpdump output doesn't show any errors during the write or close.
> > > >
> > > > I'm not sure what could be causing the problem and would
> > > > appreciate any clues or debugging suggestions.
> > > >
> > > > Thanks
> > > >
> > >
> > > I'd look for "short" write replies. We have this in
> > > cifs_writev_callback:
> > >
> > >                 if (written < wdata->bytes)
> > >                         wdata->result = -ENOSPC;
> >
> > If the server does not return an error, then the client can conclude that at
> least something was written, based on the value of CountOfBytesWritten in
> the response. Unless this is zero, I think the client substituting ENOSPC may
> be incorrect.
> >
> 
> That's probably the case, but short writes are really tricky to deal with...
> 
> Say we're spraying out a bunch of 64k writes, but the server then replies that
> only 63k of each was written. Now we have to go back and redirty the pages
> that didn't get fully written. That's fine and not too hard to do, but now you
> have a bunch of dirty pages sprinkled around the file every 63k.
> 
> So, on the next pass through the dirty page radix tree, we go and try to issue
> another WRITE_ANDX call. I'm not sure if we'd have to go back and restart
> writepages ourselves at that point for the WB_SYNC_ALL case.
> 
> Note that this sort of server behavior was the crux of my argument with
> Steve a few months ago about defaulting to 64k writes on servers w/o POSIX
> extensions. We can be reasonably sure that most servers handle 64k writes
> OK since that's what Windows does.

Yes, I think you can count on the ability to handle up to 64KB from a server, you can certainly count on it from Windows.

But if the server does not indicate CAP_LARGE_WRITEX, then you can be certain it does not support >64KB. Signing has an effect on large writes too. See MS-SMB section 2.2.4.5.2.1 and 2.2.4.3.2, for example.

I still think it's questionable for the client to unconditionally signal a short write as an ENOSPC error. Yes, it's a very unhelpful server, but it did write some data. It seems wrong to ignore that, at this level.

> 
> The spec is not 100% clear on whether servers are *required* to support
> arbitrarily large writes up to the 128k limit. Clearly there are some that do
> not, and a larger default is problematic against those servers.

I'd be very interested to see traces of negotiate, large read and large write from such a server.

> 
> > > IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX
> > > responses, but I remember it not being 100% clear...
> >
> > File a doc issue! Happy to clarify it if needed.
> >
> 
> That might be nice to do once Suresh confirms whether that's the case.

Sounds good.

Tom.

Re: Writes greater than 64k fails with -ENOSPC

[Jeff Layton]

On Wed, 30 Jan 2013 16:06:26 +0000
Tom Talpey <ttalpey-0li6OtcxBFHby3iVrkZq2A@public.gmane.org> wrote:

> > -----Original Message-----
> > From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> > Sent: Wednesday, January 30, 2013 9:37 AM
> > To: Tom Talpey
> > Cc: Suresh Jayaraman; linux-cifs
> > Subject: Re: Writes greater than 64k fails with -ENOSPC
> > 
> > On Wed, 30 Jan 2013 14:06:19 +0000
> > Tom Talpey <ttalpey-0li6OtcxBFHby3iVrkZq2A@public.gmane.org> wrote:
> > 
> > > > -----Original Message-----
> > > > From: linux-cifs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-cifs-
> > > > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Jeff Layton
> > > > Sent: Tuesday, January 29, 2013 7:29 PM
> > > > To: Suresh Jayaraman
> > > > Cc: linux-cifs
> > > > Subject: Re: Writes greater than 64k fails with -ENOSPC
> > > >
> > > > On Tue, 29 Jan 2013 17:54:21 +0530
> > > > Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I'm looking into a report on 3.0 based kernel (plus stable fixes)
> > > > > where writes greater than 64k to a NAS (Hitachi NAS) is failing
> > > > > (simple dd). The problem was not seen with a 2.6.32-ish kernel.
> > > > > Also, note that the problem is not seen with other Servers such as
> > > > > Windows 2003 or Windows 8 Servers.
> > > > >
> > > > > The strace output shows the close() fall fails with -ENOSPC.
> > > > >
> > > > > The relevant cFYI snip
> > > > >
> > > > > Jan 23 08:31:45 vsusix02 kernel: [1003552.227274]
> > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
> > > > > ansp
> > > > > ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
> > > > > [1003552.227277]
> > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
> > > > > ansp
> > > > > ort.c: Sending smb:  total_len 127044 Jan 23 08:31:45 vsusix02 kernel:
> > > > > [1003552.345848]
> > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
> > > > > nnec
> > > > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > > > [1003552.393906]
> > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/ci
> > > > > fssm
> > > > > b.c: async write at 1015808 8192 bytes Jan 23 08:31:45 vsusix02
> > > > > kernel: [1003552.393911]
> > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
> > > > > ansp
> > > > > ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
> > > > > [1003552.393914]
> > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/tr
> > > > > ansp
> > > > > ort.c: Sending smb:  total_len 8260 Jan 23 08:31:45 vsusix02 kernel:
> > > > > [1003552.479378]
> > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
> > > > > nnec
> > > > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > > > [1003552.481215]
> > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/co
> > > > > nnec
> > > > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > > > [1003552.481260]
> > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/fi
> > > > > le.c
> > > > > : Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28
> > > > >
> > > > >
> > > > > The problem seems to be that during close(), we try to flush the
> > > > > buffers by calling cifs_flush() which in turn will call
> > > > > filemap_write_and_wait() to wait on the pages under writeback to
> > > > > complete. do_writepages() will invoke cifs_writepages() which is
> > > > > perhaps returning -ENOSPC and it is propagated back.
> > > > >
> > > > > There are no quota restrictions or disk space problems. The
> > > > > tcpdump output doesn't show any errors during the write or close.
> > > > >
> > > > > I'm not sure what could be causing the problem and would
> > > > > appreciate any clues or debugging suggestions.
> > > > >
> > > > > Thanks
> > > > >
> > > >
> > > > I'd look for "short" write replies. We have this in
> > > > cifs_writev_callback:
> > > >
> > > >                 if (written < wdata->bytes)
> > > >                         wdata->result = -ENOSPC;
> > >
> > > If the server does not return an error, then the client can conclude that at
> > least something was written, based on the value of CountOfBytesWritten in
> > the response. Unless this is zero, I think the client substituting ENOSPC may
> > be incorrect.
> > >
> > 
> > That's probably the case, but short writes are really tricky to deal with...
> > 
> > Say we're spraying out a bunch of 64k writes, but the server then replies that
> > only 63k of each was written. Now we have to go back and redirty the pages
> > that didn't get fully written. That's fine and not too hard to do, but now you
> > have a bunch of dirty pages sprinkled around the file every 63k.
> > 
> > So, on the next pass through the dirty page radix tree, we go and try to issue
> > another WRITE_ANDX call. I'm not sure if we'd have to go back and restart
> > writepages ourselves at that point for the WB_SYNC_ALL case.
> > 
> > Note that this sort of server behavior was the crux of my argument with
> > Steve a few months ago about defaulting to 64k writes on servers w/o POSIX
> > extensions. We can be reasonably sure that most servers handle 64k writes
> > OK since that's what Windows does.
> 
> Yes, I think you can count on the ability to handle up to 64KB from a server, you can certainly count on it from Windows.
> 
> But if the server does not indicate CAP_LARGE_WRITEX, then you can be certain it does not support >64KB. Signing has an effect on large writes too. See MS-SMB section 2.2.4.5.2.1 and 2.2.4.3.2, for example.
> 

Right, and we do handle that case correctly, AFAIK.

> I still think it's questionable for the client to unconditionally signal a short write as an ENOSPC error. Yes, it's a very unhelpful server, but it did write some data. It seems wrong to ignore that, at this level.
> 

Yes, almost certainly, but when I was reading the spec a couple of
years ago, that wasn't completely clear. IIRC, one way to interpret it
was that a short write meant the equivalent of an ENOSPC error.

Now that this code has been in the field for a bit, it seems like we
see similar responses when a server just can't handle writes larger than
some arbitrary size (usu. 64k).

> > 
> > The spec is not 100% clear on whether servers are *required* to support
> > arbitrarily large writes up to the 128k limit. Clearly there are some that do
> > not, and a larger default is problematic against those servers.
> 
> I'd be very interested to see traces of negotiate, large read and large write from such a server.
> 

It's almost assuredly sending CAP_LARGE_WRITEX or we'd cap this at the
MaxBufferSize. The spec says that that allows the client to exceed the
MaxBufferSize on a write, but it studiously does not say by how much. :)

It's clear in hindsight that a lot of server implementors just did the
bare minimum and that they only handle what Windows clients will send.

> > 
> > > > IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX
> > > > responses, but I remember it not being 100% clear...
> > >
> > > File a doc issue! Happy to clarify it if needed.
> > >
> > 
> > That might be nice to do once Suresh confirms whether that's the case.
> 
> Sounds good.
> 
> Tom.


-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

RE: Writes greater than 64k fails with -ENOSPC

[Tom Talpey]

> -----Original Message-----
> From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> Sent: Wednesday, January 30, 2013 12:50 PM
> To: Tom Talpey
> Cc: Suresh Jayaraman; linux-cifs
> Subject: Re: Writes greater than 64k fails with -ENOSPC
> 
> On Wed, 30 Jan 2013 16:06:26 +0000
> Tom Talpey <ttalpey-0li6OtcxBFHby3iVrkZq2A@public.gmane.org> wrote:
> 
> > > -----Original Message-----
> > > From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> > > Sent: Wednesday, January 30, 2013 9:37 AM
> > > To: Tom Talpey
> > > Cc: Suresh Jayaraman; linux-cifs
> > > Subject: Re: Writes greater than 64k fails with -ENOSPC
> > >
> > > On Wed, 30 Jan 2013 14:06:19 +0000
> > > Tom Talpey <ttalpey-0li6OtcxBFHby3iVrkZq2A@public.gmane.org> wrote:
> > >
> > > > > -----Original Message-----
> > > > > From: linux-cifs-owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org [mailto:linux-cifs-
> > > > > owner-u79uwXL29TY76Z2rM5mHXA@public.gmane.org] On Behalf Of Jeff Layton
> > > > > Sent: Tuesday, January 29, 2013 7:29 PM
> > > > > To: Suresh Jayaraman
> > > > > Cc: linux-cifs
> > > > > Subject: Re: Writes greater than 64k fails with -ENOSPC
> > > > >
> > > > > On Tue, 29 Jan 2013 17:54:21 +0530 Suresh Jayaraman
> > > > > <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I'm looking into a report on 3.0 based kernel (plus stable
> > > > > > fixes) where writes greater than 64k to a NAS (Hitachi NAS) is
> > > > > > failing (simple dd). The problem was not seen with a 2.6.32-ish
> kernel.
> > > > > > Also, note that the problem is not seen with other Servers
> > > > > > such as Windows 2003 or Windows 8 Servers.
> > > > > >
> > > > > > The strace output shows the close() fall fails with -ENOSPC.
> > > > > >
> > > > > > The relevant cFYI snip
> > > > > >
> > > > > > Jan 23 08:31:45 vsusix02 kernel: [1003552.227274]
> > > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif
> > > > > > s/tr
> > > > > > ansp
> > > > > > ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
> > > > > > [1003552.227277]
> > > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif
> > > > > > s/tr
> > > > > > ansp
> > > > > > ort.c: Sending smb:  total_len 127044 Jan 23 08:31:45 vsusix02
> kernel:
> > > > > > [1003552.345848]
> > > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif
> > > > > > s/co
> > > > > > nnec
> > > > > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > > > > [1003552.393906]
> > > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif
> > > > > > s/ci
> > > > > > fssm
> > > > > > b.c: async write at 1015808 8192 bytes Jan 23 08:31:45
> > > > > > vsusix02
> > > > > > kernel: [1003552.393911]
> > > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif
> > > > > > s/tr
> > > > > > ansp
> > > > > > ort.c: For smb_command 47 Jan 23 08:31:45 vsusix02 kernel:
> > > > > > [1003552.393914]
> > > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif
> > > > > > s/tr
> > > > > > ansp
> > > > > > ort.c: Sending smb:  total_len 8260 Jan 23 08:31:45 vsusix02 kernel:
> > > > > > [1003552.479378]
> > > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif
> > > > > > s/co
> > > > > > nnec
> > > > > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > > > > [1003552.481215]
> > > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif
> > > > > > s/co
> > > > > > nnec
> > > > > > t.c: rfc1002 length 0x33 Jan 23 08:31:45 vsusix02 kernel:
> > > > > > [1003552.481260]
> > > > > > /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cif
> > > > > > s/fi
> > > > > > le.c
> > > > > > : Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28
> > > > > >
> > > > > >
> > > > > > The problem seems to be that during close(), we try to flush
> > > > > > the buffers by calling cifs_flush() which in turn will call
> > > > > > filemap_write_and_wait() to wait on the pages under writeback
> > > > > > to complete. do_writepages() will invoke cifs_writepages()
> > > > > > which is perhaps returning -ENOSPC and it is propagated back.
> > > > > >
> > > > > > There are no quota restrictions or disk space problems. The
> > > > > > tcpdump output doesn't show any errors during the write or close.
> > > > > >
> > > > > > I'm not sure what could be causing the problem and would
> > > > > > appreciate any clues or debugging suggestions.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > >
> > > > > I'd look for "short" write replies. We have this in
> > > > > cifs_writev_callback:
> > > > >
> > > > >                 if (written < wdata->bytes)
> > > > >                         wdata->result = -ENOSPC;
> > > >
> > > > If the server does not return an error, then the client can
> > > > conclude that at
> > > least something was written, based on the value of
> > > CountOfBytesWritten in the response. Unless this is zero, I think
> > > the client substituting ENOSPC may be incorrect.
> > > >
> > >
> > > That's probably the case, but short writes are really tricky to deal with...
> > >
> > > Say we're spraying out a bunch of 64k writes, but the server then
> > > replies that only 63k of each was written. Now we have to go back
> > > and redirty the pages that didn't get fully written. That's fine and
> > > not too hard to do, but now you have a bunch of dirty pages sprinkled
> around the file every 63k.
> > >
> > > So, on the next pass through the dirty page radix tree, we go and
> > > try to issue another WRITE_ANDX call. I'm not sure if we'd have to
> > > go back and restart writepages ourselves at that point for the
> WB_SYNC_ALL case.
> > >
> > > Note that this sort of server behavior was the crux of my argument
> > > with Steve a few months ago about defaulting to 64k writes on
> > > servers w/o POSIX extensions. We can be reasonably sure that most
> > > servers handle 64k writes OK since that's what Windows does.
> >
> > Yes, I think you can count on the ability to handle up to 64KB from a server,
> you can certainly count on it from Windows.
> >
> > But if the server does not indicate CAP_LARGE_WRITEX, then you can be
> certain it does not support >64KB. Signing has an effect on large writes too.
> See MS-SMB section 2.2.4.5.2.1 and 2.2.4.3.2, for example.
> >
> 
> Right, and we do handle that case correctly, AFAIK.
> 
> > I still think it's questionable for the client to unconditionally signal a short
> write as an ENOSPC error. Yes, it's a very unhelpful server, but it did write
> some data. It seems wrong to ignore that, at this level.
> >
> 
> Yes, almost certainly, but when I was reading the spec a couple of years ago,
> that wasn't completely clear. IIRC, one way to interpret it was that a short
> write meant the equivalent of an ENOSPC error.
> 
> Now that this code has been in the field for a bit, it seems like we see similar
> responses when a server just can't handle writes larger than some arbitrary
> size (usu. 64k).
> 
> > >
> > > The spec is not 100% clear on whether servers are *required* to
> > > support arbitrarily large writes up to the 128k limit. Clearly there
> > > are some that do not, and a larger default is problematic against those
> servers.
> >
> > I'd be very interested to see traces of negotiate, large read and large write
> from such a server.
> >
> 
> It's almost assuredly sending CAP_LARGE_WRITEX or we'd cap this at the
> MaxBufferSize. The spec says that that allows the client to exceed the
> MaxBufferSize on a write, but it studiously does not say by how much. :)

Well, except that it's not known what this server is advertising for MaxBufferSize. Maybe it advertises a large value but can't actually handle it. The trace would tell.

Tom.


> 
> It's clear in hindsight that a lot of server implementors just did the bare
> minimum and that they only handle what Windows clients will send.
> 
> > >
> > > > > IIRC, this is spelled out in MS-CIFS in the section on
> > > > > WRITE_ANDX responses, but I remember it not being 100% clear...
> > > >
> > > > File a doc issue! Happy to clarify it if needed.
> > > >
> > >
> > > That might be nice to do once Suresh confirms whether that's the case.
> >
> > Sounds good.
> >
> > Tom.
> 
> 
> --
> Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Re: Writes greater than 64k fails with -ENOSPC

[Suresh Jayaraman]

On 01/30/2013 09:36 PM, Tom Talpey wrote:
>> -----Original Message-----
>> From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
>> Sent: Wednesday, January 30, 2013 9:37 AM
>> To: Tom Talpey
>> Cc: Suresh Jayaraman; linux-cifs
>> Subject: Re: Writes greater than 64k fails with -ENOSPC

>>
>> The spec is not 100% clear on whether servers are *required* to support
>> arbitrarily large writes up to the 128k limit. Clearly there are some that do
>> not, and a larger default is problematic against those servers.
> 
> I'd be very interested to see traces of negotiate, large read and large write from such a server.

I'm not sure whether I can share the complete trace (without customer's
permission) but I can get you the specific bits that might be
interesting. I have asked for a full trace (including negotiate protocol).


Thanks


-- 
Suresh Jayaraman

Re: Writes greater than 64k fails with -ENOSPC

[Jeff Layton]

On Thu, 31 Jan 2013 15:22:15 +0530
Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:

> On 01/30/2013 09:36 PM, Tom Talpey wrote:
> >> -----Original Message-----
> >> From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> >> Sent: Wednesday, January 30, 2013 9:37 AM
> >> To: Tom Talpey
> >> Cc: Suresh Jayaraman; linux-cifs
> >> Subject: Re: Writes greater than 64k fails with -ENOSPC
> 
> >>
> >> The spec is not 100% clear on whether servers are *required* to support
> >> arbitrarily large writes up to the 128k limit. Clearly there are some that do
> >> not, and a larger default is problematic against those servers.
> > 
> > I'd be very interested to see traces of negotiate, large read and large write from such a server.
> 
> I'm not sure whether I can share the complete trace (without customer's
> permission) but I can get you the specific bits that might be
> interesting. I have asked for a full trace (including negotiate protocol).
> 
> 
> Thanks
> 
> 

We probably don't need the whole capture. Just the "Capabilities" and
"MaxBufferSize" values from the NEGOTIATE would be enough...

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Re: Writes greater than 64k fails with -ENOSPC

[Suresh Jayaraman]

On 01/31/2013 04:54 PM, Jeff Layton wrote:
> On Thu, 31 Jan 2013 15:22:15 +0530
> Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> 
>> On 01/30/2013 09:36 PM, Tom Talpey wrote:
>>>> -----Original Message-----
>>>> From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
>>>> Sent: Wednesday, January 30, 2013 9:37 AM
>>>> To: Tom Talpey
>>>> Cc: Suresh Jayaraman; linux-cifs
>>>> Subject: Re: Writes greater than 64k fails with -ENOSPC
>>
>>>>
>>>> The spec is not 100% clear on whether servers are *required* to support
>>>> arbitrarily large writes up to the 128k limit. Clearly there are some that do
>>>> not, and a larger default is problematic against those servers.
>>>
>>> I'd be very interested to see traces of negotiate, large read and large write from such a server.
>>
>> I'm not sure whether I can share the complete trace (without customer's
>> permission) but I can get you the specific bits that might be
>> interesting. I have asked for a full trace (including negotiate protocol).
>>
>>
>> Thanks
>>
>>
> 
> We probably don't need the whole capture. Just the "Capabilities" and
> "MaxBufferSize" values from the NEGOTIATE would be enough...
> 

The commit ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 did fix the problem.

Finally, got the packet capture too. From the NEGOTIATE response,
the MaxBufferSize was 65280 and CAP_LARGE_WRITEX (and LARGE_READX) had been set.

The complete Capabilities were

.... .... .... .... .... .... .... ...0 = Raw Mode: Read Raw and Write Raw are not supported
.... .... .... .... .... .... .... ..0. = MPX Mode: Read Mpx and Write Mpx are not supported
.... .... .... .... .... .... .... .1.. = Unicode: Unicode strings are supported
.... .... .... .... .... .... .... 1... = Large Files: Large files are supported
.... .... .... .... .... .... ...1 .... = NT SMBs: NT SMBs are supported
.... .... .... .... .... .... ..1. .... = RPC Remote APIs: RPC remote APIs are supported
.... .... .... .... .... .... .1.. .... = NT Status Codes: NT status codes are supported
.... .... .... .... .... .... 1... .... = Level 2 Oplocks: Level 2 oplocks are supported
.... .... .... .... .... ...0 .... .... = Lock and Read: Lock and Read is not supported
.... .... .... .... .... ..1. .... .... = NT Find: NT Find is supported
.... .... .... .... ...1 .... .... .... = Dfs: Dfs is supported
.... .... .... .... ..1. .... .... .... = Infolevel Passthru: NT information level request passthrough is supported
.... .... .... .... .1.. .... .... .... = Large ReadX: Large Read andX is supported
.... .... .... .... 1... .... .... .... = Large WriteX: Large Write andX is supported
.... .... .... ...0 .... .... .... .... = LWIO: LWIO ioctl/fsctl is not supported
.... .... 0... .... .... .... .... .... = UNIX: UNIX extensions are not supported
.... ..0. .... .... .... .... .... .... = Compressed Data: Compressed data transfer is not supported
..0. .... .... .... .... .... .... .... = Dynamic Reauth: Dynamic Reauth is not supported
0... .... .... .... .... .... .... .... = Extended Security: Extended security exchanges are not supported


I'm not sure whether restricting it to MaxBufferSize even in case CAP_LARGE_WRITEX was set would make sense..



-- 
Suresh Jayaraman

Re: Writes greater than 64k fails with -ENOSPC

[Jeff Layton]

On Thu, 07 Feb 2013 14:22:04 +0530
Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:

> On 01/31/2013 04:54 PM, Jeff Layton wrote:
> > On Thu, 31 Jan 2013 15:22:15 +0530
> > Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> > 
> >> On 01/30/2013 09:36 PM, Tom Talpey wrote:
> >>>> -----Original Message-----
> >>>> From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> >>>> Sent: Wednesday, January 30, 2013 9:37 AM
> >>>> To: Tom Talpey
> >>>> Cc: Suresh Jayaraman; linux-cifs
> >>>> Subject: Re: Writes greater than 64k fails with -ENOSPC
> >>
> >>>>
> >>>> The spec is not 100% clear on whether servers are *required* to support
> >>>> arbitrarily large writes up to the 128k limit. Clearly there are some that do
> >>>> not, and a larger default is problematic against those servers.
> >>>
> >>> I'd be very interested to see traces of negotiate, large read and large write from such a server.
> >>
> >> I'm not sure whether I can share the complete trace (without customer's
> >> permission) but I can get you the specific bits that might be
> >> interesting. I have asked for a full trace (including negotiate protocol).
> >>
> >>
> >> Thanks
> >>
> >>
> > 
> > We probably don't need the whole capture. Just the "Capabilities" and
> > "MaxBufferSize" values from the NEGOTIATE would be enough...
> > 
> 
> The commit ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 did fix the problem.
> 
> Finally, got the packet capture too. From the NEGOTIATE response,
> the MaxBufferSize was 65280 and CAP_LARGE_WRITEX (and LARGE_READX) had been set.
> 
> The complete Capabilities were
> 
> .... .... .... .... .... .... .... ...0 = Raw Mode: Read Raw and Write Raw are not supported
> .... .... .... .... .... .... .... ..0. = MPX Mode: Read Mpx and Write Mpx are not supported
> .... .... .... .... .... .... .... .1.. = Unicode: Unicode strings are supported
> .... .... .... .... .... .... .... 1... = Large Files: Large files are supported
> .... .... .... .... .... .... ...1 .... = NT SMBs: NT SMBs are supported
> .... .... .... .... .... .... ..1. .... = RPC Remote APIs: RPC remote APIs are supported
> .... .... .... .... .... .... .1.. .... = NT Status Codes: NT status codes are supported
> .... .... .... .... .... .... 1... .... = Level 2 Oplocks: Level 2 oplocks are supported
> .... .... .... .... .... ...0 .... .... = Lock and Read: Lock and Read is not supported
> .... .... .... .... .... ..1. .... .... = NT Find: NT Find is supported
> .... .... .... .... ...1 .... .... .... = Dfs: Dfs is supported
> .... .... .... .... ..1. .... .... .... = Infolevel Passthru: NT information level request passthrough is supported
> .... .... .... .... .1.. .... .... .... = Large ReadX: Large Read andX is supported
> .... .... .... .... 1... .... .... .... = Large WriteX: Large Write andX is supported
> .... .... .... ...0 .... .... .... .... = LWIO: LWIO ioctl/fsctl is not supported
> .... .... 0... .... .... .... .... .... = UNIX: UNIX extensions are not supported
> .... ..0. .... .... .... .... .... .... = Compressed Data: Compressed data transfer is not supported
> ..0. .... .... .... .... .... .... .... = Dynamic Reauth: Dynamic Reauth is not supported
> 0... .... .... .... .... .... .... .... = Extended Security: Extended security exchanges are not supported
> 
> 
> I'm not sure whether restricting it to MaxBufferSize even in case CAP_LARGE_WRITEX was set would make sense..
> 

No, I think that server just happens to have an unusually large
MaxBufferSize. Most servers send values that are smaller than that. The
whole idea behind CAP_LARGE_WRITEX is that it allows you to send a write
that's larger than the MaxBufferSize.

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

RE: Writes greater than 64k fails with -ENOSPC

[Tom Talpey]

> -----Original Message-----
> From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> Sent: Thursday, February 7, 2013 6:50 AM
> To: Suresh Jayaraman
> Cc: Tom Talpey; linux-cifs
> Subject: Re: Writes greater than 64k fails with -ENOSPC
> 
> On Thu, 07 Feb 2013 14:22:04 +0530
> Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> 
> > On 01/31/2013 04:54 PM, Jeff Layton wrote:
> > > On Thu, 31 Jan 2013 15:22:15 +0530
> > > Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> > >
> > >> On 01/30/2013 09:36 PM, Tom Talpey wrote:
> > >>>> -----Original Message-----
> > >>>> From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> > >>>> Sent: Wednesday, January 30, 2013 9:37 AM
> > >>>> To: Tom Talpey
> > >>>> Cc: Suresh Jayaraman; linux-cifs
> > >>>> Subject: Re: Writes greater than 64k fails with -ENOSPC
> > >>
> > >>>>
> > >>>> The spec is not 100% clear on whether servers are *required* to
> > >>>> support arbitrarily large writes up to the 128k limit. Clearly
> > >>>> there are some that do not, and a larger default is problematic against
> those servers.
> > >>>
> > >>> I'd be very interested to see traces of negotiate, large read and large
> write from such a server.
> > >>
> > >> I'm not sure whether I can share the complete trace (without
> > >> customer's
> > >> permission) but I can get you the specific bits that might be
> > >> interesting. I have asked for a full trace (including negotiate protocol).
> > >>
> > >>
> > >> Thanks
> > >>
> > >>
> > >
> > > We probably don't need the whole capture. Just the "Capabilities"
> > > and "MaxBufferSize" values from the NEGOTIATE would be enough...
> > >
> >
> > The commit ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 did fix the
> problem.
> >
> > Finally, got the packet capture too. From the NEGOTIATE response, the
> > MaxBufferSize was 65280 and CAP_LARGE_WRITEX (and LARGE_READX)
> had been set.
> >
> > The complete Capabilities were
> >
> > .... .... .... .... .... .... .... ...0 = Raw Mode: Read Raw and Write
> > Raw are not supported .... .... .... .... .... .... .... ..0. = MPX
> > Mode: Read Mpx and Write Mpx are not supported .... .... .... ....
> > .... .... .... .1.. = Unicode: Unicode strings are supported .... ....
> > .... .... .... .... .... 1... = Large Files: Large files are supported
> > .... .... .... .... .... .... ...1 .... = NT SMBs: NT SMBs are
> > supported .... .... .... .... .... .... ..1. .... = RPC Remote APIs:
> > RPC remote APIs are supported .... .... .... .... .... .... .1.. ....
> > = NT Status Codes: NT status codes are supported .... .... .... ....
> > .... .... 1... .... = Level 2 Oplocks: Level 2 oplocks are supported
> > .... .... .... .... .... ...0 .... .... = Lock and Read: Lock and Read
> > is not supported .... .... .... .... .... ..1. .... .... = NT Find: NT
> > Find is supported .... .... .... .... ...1 .... .... .... = Dfs: Dfs
> > is supported .... .... .... .... ..1. .... .... .... = Infolevel
> > Passthru: NT information level request passthrough is supported ....
> > .... .... .... .1.. .... .... .... = Large ReadX: Large Read andX is
> > supported .... .... .... .... 1... .... .... .... = Large WriteX:
> > Large Write andX is supported .... .... .... ...0 .... .... .... ....
> > = LWIO: LWIO ioctl/fsctl is not supported .... .... 0... .... ....
> > .... .... .... = UNIX: UNIX extensions are not supported .... ..0.
> > .... .... .... .... .... .... = Compressed Data: Compressed data
> > transfer is not supported ..0. .... .... .... .... .... .... .... =
> > Dynamic Reauth: Dynamic Reauth is not supported 0... .... .... ....
> > .... .... .... .... = Extended Security: Extended security exchanges
> > are not supported
> >
> >
> > I'm not sure whether restricting it to MaxBufferSize even in case
> CAP_LARGE_WRITEX was set would make sense..
> >
> 
> No, I think that server just happens to have an unusually large
> MaxBufferSize. Most servers send values that are smaller than that. The
> whole idea behind CAP_LARGE_WRITEX is that it allows you to send a write
> that's larger than the MaxBufferSize.

I agree, but it's still unhelpful that this server accepts only 64KB in response to the client's 124KB large write. In my opinion the server should not be offering CAP_LARGE_WRITEX, although it's certainly legal for it to do so. Perhaps your client should be ignoring it, when MaxBufferSize permits >64KB.

BTW, we're investigating the large write size behaviors for a related MS-SMB doc bug. I don't have the full picture yet, but it appears that Windows SMB servers will process somewhere between 124KB and 128KB-1 bytes when issued a large WRITE_ANDX in this situation.

Tom.

Re: Writes greater than 64k fails with -ENOSPC

[Jeff Layton]

On Thu, 7 Feb 2013 13:37:00 +0000
Tom Talpey <ttalpey-0li6OtcxBFHby3iVrkZq2A@public.gmane.org> wrote:

> > -----Original Message-----
> > From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> > Sent: Thursday, February 7, 2013 6:50 AM
> > To: Suresh Jayaraman
> > Cc: Tom Talpey; linux-cifs
> > Subject: Re: Writes greater than 64k fails with -ENOSPC
> > 
> > On Thu, 07 Feb 2013 14:22:04 +0530
> > Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> > 
> > > On 01/31/2013 04:54 PM, Jeff Layton wrote:
> > > > On Thu, 31 Jan 2013 15:22:15 +0530
> > > > Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> > > >
> > > >> On 01/30/2013 09:36 PM, Tom Talpey wrote:
> > > >>>> -----Original Message-----
> > > >>>> From: Jeff Layton [mailto:jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org]
> > > >>>> Sent: Wednesday, January 30, 2013 9:37 AM
> > > >>>> To: Tom Talpey
> > > >>>> Cc: Suresh Jayaraman; linux-cifs
> > > >>>> Subject: Re: Writes greater than 64k fails with -ENOSPC
> > > >>
> > > >>>>
> > > >>>> The spec is not 100% clear on whether servers are *required* to
> > > >>>> support arbitrarily large writes up to the 128k limit. Clearly
> > > >>>> there are some that do not, and a larger default is problematic against
> > those servers.
> > > >>>
> > > >>> I'd be very interested to see traces of negotiate, large read and large
> > write from such a server.
> > > >>
> > > >> I'm not sure whether I can share the complete trace (without
> > > >> customer's
> > > >> permission) but I can get you the specific bits that might be
> > > >> interesting. I have asked for a full trace (including negotiate protocol).
> > > >>
> > > >>
> > > >> Thanks
> > > >>
> > > >>
> > > >
> > > > We probably don't need the whole capture. Just the "Capabilities"
> > > > and "MaxBufferSize" values from the NEGOTIATE would be enough...
> > > >
> > >
> > > The commit ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 did fix the
> > problem.
> > >
> > > Finally, got the packet capture too. From the NEGOTIATE response, the
> > > MaxBufferSize was 65280 and CAP_LARGE_WRITEX (and LARGE_READX)
> > had been set.
> > >
> > > The complete Capabilities were
> > >
> > > .... .... .... .... .... .... .... ...0 = Raw Mode: Read Raw and Write
> > > Raw are not supported .... .... .... .... .... .... .... ..0. = MPX
> > > Mode: Read Mpx and Write Mpx are not supported .... .... .... ....
> > > .... .... .... .1.. = Unicode: Unicode strings are supported .... ....
> > > .... .... .... .... .... 1... = Large Files: Large files are supported
> > > .... .... .... .... .... .... ...1 .... = NT SMBs: NT SMBs are
> > > supported .... .... .... .... .... .... ..1. .... = RPC Remote APIs:
> > > RPC remote APIs are supported .... .... .... .... .... .... .1.. ....
> > > = NT Status Codes: NT status codes are supported .... .... .... ....
> > > .... .... 1... .... = Level 2 Oplocks: Level 2 oplocks are supported
> > > .... .... .... .... .... ...0 .... .... = Lock and Read: Lock and Read
> > > is not supported .... .... .... .... .... ..1. .... .... = NT Find: NT
> > > Find is supported .... .... .... .... ...1 .... .... .... = Dfs: Dfs
> > > is supported .... .... .... .... ..1. .... .... .... = Infolevel
> > > Passthru: NT information level request passthrough is supported ....
> > > .... .... .... .1.. .... .... .... = Large ReadX: Large Read andX is
> > > supported .... .... .... .... 1... .... .... .... = Large WriteX:
> > > Large Write andX is supported .... .... .... ...0 .... .... .... ....
> > > = LWIO: LWIO ioctl/fsctl is not supported .... .... 0... .... ....
> > > .... .... .... = UNIX: UNIX extensions are not supported .... ..0.
> > > .... .... .... .... .... .... = Compressed Data: Compressed data
> > > transfer is not supported ..0. .... .... .... .... .... .... .... =
> > > Dynamic Reauth: Dynamic Reauth is not supported 0... .... .... ....
> > > .... .... .... .... = Extended Security: Extended security exchanges
> > > are not supported
> > >
> > >
> > > I'm not sure whether restricting it to MaxBufferSize even in case
> > CAP_LARGE_WRITEX was set would make sense..
> > >
> > 
> > No, I think that server just happens to have an unusually large
> > MaxBufferSize. Most servers send values that are smaller than that. The
> > whole idea behind CAP_LARGE_WRITEX is that it allows you to send a write
> > that's larger than the MaxBufferSize.
> 
> I agree, but it's still unhelpful that this server accepts only 64KB in response to the client's 124KB large write. In my opinion the server should not be offering CAP_LARGE_WRITEX, although it's certainly legal for it to do so. Perhaps your client should be ignoring it, when MaxBufferSize permits >64KB.
> 

Right. That was the crux of my question a few months ago -- the spec
doesn't say how much larger than the MaxBufferSize a server is required
to support. In practice, it seems a lot of server vendors didn't do any
more than they perceived they had to.

> BTW, we're investigating the large write size behaviors for a related MS-SMB doc bug. I don't have the full picture yet, but it appears that Windows SMB servers will process somewhere between 124KB and 128KB-1 bytes when issued a large WRITE_ANDX in this situation.
> 
> Tom.
> 

It's been a while since I wrote this code, but I originally did quite a
bit of testing vs. windows versions that were current at the time
(win2k8?). At that time, I never saw windows return a short write, but
we would generally issue writes that were no larger than 124k or so.

We generally use a max of 2^17-1, minus the length of the WRITE_ANDX
header (which is small), and then squeeze as many pages as we can fit
within it.

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>

Re: Writes greater than 64k fails with -ENOSPC

[Suresh Jayaraman]

On 01/30/2013 05:59 AM, Jeff Layton wrote:
> On Tue, 29 Jan 2013 17:54:21 +0530
> Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> 
>> Hi all,
>>
>> I'm looking into a report on 3.0 based kernel (plus stable fixes)
>> where writes greater than 64k to a NAS (Hitachi NAS) is failing
>> (simple dd). The problem was not seen with a 2.6.32-ish kernel.
>> Also, note that the problem is not seen with other Servers such
>> as Windows 2003 or Windows 8 Servers.
>>
>> The strace output shows the close() fall fails with -ENOSPC. 
>>
>> The relevant cFYI snip
>>
>> Jan 23 08:31:45 vsusix02 kernel: [1003552.227274] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
>> Jan 23 08:31:45 vsusix02 kernel: [1003552.227277] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 127044
>> Jan 23 08:31:45 vsusix02 kernel: [1003552.345848] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
>> Jan 23 08:31:45 vsusix02 kernel: [1003552.393906] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssmb.c: async write at 1015808 8192 bytes
>> Jan 23 08:31:45 vsusix02 kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
>> Jan 23 08:31:45 vsusix02 kernel: [1003552.393914] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 8260
>> Jan 23 08:31:45 vsusix02 kernel: [1003552.479378] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
>> Jan 23 08:31:45 vsusix02 kernel: [1003552.481215] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
>> Jan 23 08:31:45 vsusix02 kernel: [1003552.481260] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28
>>
>>
>> The problem seems to be that during close(), we try to flush the
>> buffers by calling cifs_flush() which in turn will call
>> filemap_write_and_wait() to wait on the pages under writeback to
>> complete. do_writepages() will invoke cifs_writepages() which is 
>> perhaps returning -ENOSPC and it is propagated back.
>>
>> There are no quota restrictions or disk space problems. The
>> tcpdump output doesn't show any errors during the write or close.
>>
>> I'm not sure what could be causing the problem and would
>> appreciate any clues or debugging suggestions.
>>
>> Thanks
>>
> 
> I'd look for "short" write replies. We have this in
> cifs_writev_callback:

I think you are spot-on. The packet capture looks little odd:

Client is requesting a WRITE of 126976 bytes
[DataLengthHigh -> 1 (multiple by 64k), DataLengthLow -> 61440,
WordCount -> 14]

Server seems to have written only 65536 bytes
[CountHigh -> 1, CountLow -> 0, Remaining -> 65535]
However, [File RW Length] is set to 126976

There are many such writes. I don't have the full capture to verify

Sounds similar to the problem fixed by commit
ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 (Some server not handling large
wsize correctly?

From: Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
Date: Tue, 17 Jan 2012 16:08:51 -0500
Subject: cifs: lower default wsize when unix extensions are not used


>                 if (written < wdata->bytes)
>                         wdata->result = -ENOSPC;
> 
> IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX
> responses, but I remember it not being 100% clear...
> 

Did you mean that the spec suggests that in case of -ENOSPC, the SMB/NT
status code should still be STATUS_SUCCESS and the POSIX error should be
set to -ENOSPC?


Thanks

-- 
Suresh Jayaraman

Re: Writes greater than 64k fails with -ENOSPC

[Jeff Layton]

On Thu, 31 Jan 2013 15:17:28 +0530
Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:

> On 01/30/2013 05:59 AM, Jeff Layton wrote:
> > On Tue, 29 Jan 2013 17:54:21 +0530
> > Suresh Jayaraman <sjayaraman-IBi9RG/b67k@public.gmane.org> wrote:
> > 
> >> Hi all,
> >>
> >> I'm looking into a report on 3.0 based kernel (plus stable fixes)
> >> where writes greater than 64k to a NAS (Hitachi NAS) is failing
> >> (simple dd). The problem was not seen with a 2.6.32-ish kernel.
> >> Also, note that the problem is not seen with other Servers such
> >> as Windows 2003 or Windows 8 Servers.
> >>
> >> The strace output shows the close() fall fails with -ENOSPC. 
> >>
> >> The relevant cFYI snip
> >>
> >> Jan 23 08:31:45 vsusix02 kernel: [1003552.227274] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
> >> Jan 23 08:31:45 vsusix02 kernel: [1003552.227277] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 127044
> >> Jan 23 08:31:45 vsusix02 kernel: [1003552.345848] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
> >> Jan 23 08:31:45 vsusix02 kernel: [1003552.393906] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/cifssmb.c: async write at 1015808 8192 bytes
> >> Jan 23 08:31:45 vsusix02 kernel: [1003552.393911] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: For smb_command 47
> >> Jan 23 08:31:45 vsusix02 kernel: [1003552.393914] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/transport.c: Sending smb:  total_len 8260
> >> Jan 23 08:31:45 vsusix02 kernel: [1003552.479378] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
> >> Jan 23 08:31:45 vsusix02 kernel: [1003552.481215] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/connect.c: rfc1002 length 0x33
> >> Jan 23 08:31:45 vsusix02 kernel: [1003552.481260] /usr/src/packages/BUILD/kernel-default-3.0.42/linux-3.0/fs/cifs/file.c: Flush inode ffff88004e1655c0 file ffff8800379d48c0 rc -28
> >>
> >>
> >> The problem seems to be that during close(), we try to flush the
> >> buffers by calling cifs_flush() which in turn will call
> >> filemap_write_and_wait() to wait on the pages under writeback to
> >> complete. do_writepages() will invoke cifs_writepages() which is 
> >> perhaps returning -ENOSPC and it is propagated back.
> >>
> >> There are no quota restrictions or disk space problems. The
> >> tcpdump output doesn't show any errors during the write or close.
> >>
> >> I'm not sure what could be causing the problem and would
> >> appreciate any clues or debugging suggestions.
> >>
> >> Thanks
> >>
> > 
> > I'd look for "short" write replies. We have this in
> > cifs_writev_callback:
> 
> I think you are spot-on. The packet capture looks little odd:
> 
> Client is requesting a WRITE of 126976 bytes
> [DataLengthHigh -> 1 (multiple by 64k), DataLengthLow -> 61440,
> WordCount -> 14]
> 
> Server seems to have written only 65536 bytes
> [CountHigh -> 1, CountLow -> 0, Remaining -> 65535]
> However, [File RW Length] is set to 126976
> 
> There are many such writes. I don't have the full capture to verify
> 
> Sounds similar to the problem fixed by commit
> ce91acb3acae26f4163c5a6f1f695d1a1e8d9009 (Some server not handling large
> wsize correctly?
> 
> From: Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>
> Date: Tue, 17 Jan 2012 16:08:51 -0500
> Subject: cifs: lower default wsize when unix extensions are not used
> 

Yep, though it's only "fixed" in that that patch makes the defaults
match what Windows does. You can still make the wsize larger (up to the
128k limit), but at that point we assume that you know what you're
doing. :)

> 
> >                 if (written < wdata->bytes)
> >                         wdata->result = -ENOSPC;
> > 
> > IIRC, this is spelled out in MS-CIFS in the section on WRITE_ANDX
> > responses, but I remember it not being 100% clear...
> > 
> 
> Did you mean that the spec suggests that in case of -ENOSPC, the SMB/NT
> status code should still be STATUS_SUCCESS and the POSIX error should be
> set to -ENOSPC?
> 
> 

See section 2.2.4.43.2. The "Error Codes" chart shows 2 entries at the
top that seem to describe short writes. I'll note that they *do*
indicate that the server ought to return a count of 0 in that
situation. It does not indicate how the client should interpret a
non-zero, but short write count.

As Tom suggests, the current behavior in that situation is clearly
wrong. We ought to just go back and redirty_page_for_writepage those
pages that didn't get written out.

If the wsize is larger than the server can handle though, it's likely
to make performance suck. Doing writeback may take several passes
through writepages since the client will have to go back and fill in
the gaps. It would be nice to indicate that to the admin somehow, but
I'm not sure how best to do so.

-- 
Jeff Layton <jlayton-H+wXaHxf7aLQT0dZR+AlfA@public.gmane.org>