Re: [RFC] extending splice for copy offloading

Re: [RFC] extending splice for copy offloading

[Steve French]

>>> I suppose, but can't the app achieve a nice middle ground by copying the
>>> file in smaller syscalls?  Avoid bulk data motion back to the client,
>>> but still get notification every, I dunno, few hundred meg?
>> Yes.  And if "cp"  could just be switched from a read+write syscall
>> pair to a single splice syscall using the same buffer size.
> Will the various magic fs-specific copy operations become inefficient
> when the range copied is too small?

Yes - it is much less efficient for the network file system cases when
copy size is small.   Reasonable minimum is probably at least 1MB.
Windows will use up to 16MB, but a saner approach to this would base
the copy chunk size on either response time or on network bandwidth
for the connection.

Copy offload has been done for a long time with CIFS/SMB2/SMB3
protocol (and obviously helps a lot more over the network for file
copies than locally), but only recently have we added support for this
in Samba through David Disseldorp's work.   i have kernel patches
almost ready to post for cifs.ko for the client side to do copy
offload (cp --reflink) via CopyChunk fsctl over SMB3 which is
supported by most all servers now.

Windows clients seem to max out at 16MB chunk size when doing copy
offload.   I would like to increase chunk size larger than that if
network bandwidth (returned at mount time in SMB3 on the query network
interfaces FSCTL) is large enough, and response time is not more than
100 (?) milliseconds.



-- 
Thanks,

Steve

Re: [RFC] extending splice for copy offloading

[David Disseldorp]

Hi Steve,

On Thu, 26 Sep 2013 12:22:49 -0500
Steve French <smfrench-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

> Windows clients seem to max out at 16MB chunk size when doing copy
> offload.   I would like to increase chunk size larger than that if
> network bandwidth (returned at mount time in SMB3 on the query network
> interfaces FSCTL) is large enough, and response time is not more than
> 100 (?) milliseconds.

Sounds good, though with Samba and Btrfs the request latency will be
heavily effected by whether the FSCTL_SRV_COPYCHUNK wire request byte
offsets and lengths align to the Btrfs sector-size (4K by default
IIRC).
If they do align, then each chunk in the request can be mapped to a
BTRFS_IOC_CLONE_RANGE ioctl. Otherwise, we need to fall back to a
server-side local read/write.

We should be able to provide underlying filesystem alignment
requirements to SMB clients via the new FileFSSectorSizeInformation
ioctl, but it's not currently supported by Samba.

Cheers, David

Re: [RFC] extending splice for copy offloading

[J. Bruce Fields]

On Thu, Sep 26, 2013 at 12:22:49PM -0500, Steve French wrote:
> >>> I suppose, but can't the app achieve a nice middle ground by copying the
> >>> file in smaller syscalls?  Avoid bulk data motion back to the client,
> >>> but still get notification every, I dunno, few hundred meg?
> >> Yes.  And if "cp"  could just be switched from a read+write syscall
> >> pair to a single splice syscall using the same buffer size.
> > Will the various magic fs-specific copy operations become inefficient
> > when the range copied is too small?
> 
> Yes - it is much less efficient for the network file system cases when
> copy size is small.   Reasonable minimum is probably at least 1MB.
> Windows will use up to 16MB, but a saner approach to this would base
> the copy chunk size on either response time or on network bandwidth
> for the connection.
> 
> Copy offload has been done for a long time with CIFS/SMB2/SMB3
> protocol (and obviously helps a lot more over the network for file
> copies than locally), but only recently have we added support for this
> in Samba through David Disseldorp's work.   i have kernel patches
> almost ready to post for cifs.ko for the client side to do copy
> offload (cp --reflink) via CopyChunk fsctl over SMB3 which is
> supported by most all servers now.
> 
> Windows clients seem to max out at 16MB chunk size when doing copy
> offload.   I would like to increase chunk size larger than that if
> network bandwidth (returned at mount time in SMB3 on the query network
> interfaces FSCTL) is large enough, and response time is not more than
> 100 (?) milliseconds.

I'm confused--copy offload means no data's going over the network, so
why would network bandwidth be a factor at all?

(Or are you talking about some kind of server-to-server bandwidth?)

--b.

Re: [RFC] extending splice for copy offloading

[Steve French]

You are right, network bandwidth is not the issue - but we can get
info on the underlying filesystem, and perhaps use the FS info (on
sector size etc.) that David noted to control the size we request -
and the number of credits and variations in response time as hints to
control how many copychunk requests we send at one time.

On Tue, Oct 1, 2013 at 4:05 PM, J. Bruce Fields <bfields-uC3wQj2KruNg9hUCZPvPmw@public.gmane.org> wrote:
> On Thu, Sep 26, 2013 at 12:22:49PM -0500, Steve French wrote:
>> >>> I suppose, but can't the app achieve a nice middle ground by copying the
>> >>> file in smaller syscalls?  Avoid bulk data motion back to the client,
>> >>> but still get notification every, I dunno, few hundred meg?
>> >> Yes.  And if "cp"  could just be switched from a read+write syscall
>> >> pair to a single splice syscall using the same buffer size.
>> > Will the various magic fs-specific copy operations become inefficient
>> > when the range copied is too small?
>>
>> Yes - it is much less efficient for the network file system cases when
>> copy size is small.   Reasonable minimum is probably at least 1MB.
>> Windows will use up to 16MB, but a saner approach to this would base
>> the copy chunk size on either response time or on network bandwidth
>> for the connection.
>>
>> Copy offload has been done for a long time with CIFS/SMB2/SMB3
>> protocol (and obviously helps a lot more over the network for file
>> copies than locally), but only recently have we added support for this
>> in Samba through David Disseldorp's work.   i have kernel patches
>> almost ready to post for cifs.ko for the client side to do copy
>> offload (cp --reflink) via CopyChunk fsctl over SMB3 which is
>> supported by most all servers now.
>>
>> Windows clients seem to max out at 16MB chunk size when doing copy
>> offload.   I would like to increase chunk size larger than that if
>> network bandwidth (returned at mount time in SMB3 on the query network
>> interfaces FSCTL) is large enough, and response time is not more than
>> 100 (?) milliseconds.
>
> I'm confused--copy offload means no data's going over the network, so
> why would network bandwidth be a factor at all?
>
> (Or are you talking about some kind of server-to-server bandwidth?)
>
> --b.



-- 
Thanks,

Steve

Re: [RFC] extending splice for copy offloading

[Steve French]

On Thu, Sep 26, 2013 at 2:34 PM, David Disseldorp <ddiss@suse.de> wrote:
> Hi Steve,
>
> On Thu, 26 Sep 2013 12:22:49 -0500
> Steve French <smfrench@gmail.com> wrote:
>
>> Windows clients seem to max out at 16MB chunk size when doing copy
>> offload.   I would like to increase chunk size larger than that if
>> network bandwidth (returned at mount time in SMB3 on the query network
>> interfaces FSCTL) is large enough, and response time is not more than
>> 100 (?) milliseconds.
>
> Sounds good, though with Samba and Btrfs the request latency will be
> heavily effected by whether the FSCTL_SRV_COPYCHUNK wire request byte
> offsets and lengths align to the Btrfs sector-size (4K by default
> IIRC).
> If they do align, then each chunk in the request can be mapped to a
> BTRFS_IOC_CLONE_RANGE ioctl. Otherwise, we need to fall back to a
> server-side local read/write.
>
> We should be able to provide underlying filesystem alignment
> requirements to SMB clients via the new FileFSSectorSizeInformation
> ioctl, but it's not currently supported by Samba.

I just posted a patch for the kernel client to request this information (when
SMB3 is requested) but Windows 8 for both SSD and spinning disk
was returning 512 byte for logical and physical and best
performance sector sizes.  So alignment for the CopyChunk chunks
that I will request on refcopy request will be straightforward.

It was great that the system reports the SSD vs. non-SSD correctly
with this new FSCTL.  Hopefully we can add a Samba VFS routine
to report this info - although not sure how we can request info
from the kernel to fill in these flags and fields.

#define SSINFO_FLAGS_ALIGNED_DEVICE        0x00000001
#define SSINFO_FLAGS_PARTITION_ALIGNED_ON_DEVICE 0x00000002
#define SSINFO_FLAGS_NO_SEEK_PENALTY        0x00000004 /* SSD */
#define SSINFO_FLAGS_TRIM_ENABLED        0x00000008

/* sector size info struct for new level 11 Query FS Info */
struct smb3_fs_ss_info {
    __le32 LogicalBytesPerSector;
    __le32 PhysicalBytesPerSectorForAtomicity;
    __le32 PhysicalBytesPerSectorForPerf;
    __le32 FileSystemEffectivePhysicalBytesPerSectorForAtomicity;
    __le32 Flags;
    __le32 ByteOffsetForSectorAlignment;
    __le32 ByteOffsetForPartitionAlignment;
}

-- 
Thanks,

Steve