punching holes in files

punching holes in files

[Steve French]

madvise_remove (in Linux) is used to free the backing store associated
with pages (punching a hole in a file).   This is one of the vfs
operations that we do not send over the wire to Samba (so this call
would return -ENOSYS locally).   Any thoughts on whether this could be
done with an obscure SetFileInfo level or FCNTL or whether it is worth
adding to the CIFS POSIX Extensions?

A second interesting question is whether to implement fallocate over
the wire.  sys_fallocate takes a range (and also a flag which
indicates whether to extend the size beyond end of file).
SET_FILE_ALLOCATION_INFO takes a size field rather than a range.  For
the case of the range starting at zero, we could simply call Trans2
set pathinfo (or file info) with SET_FILE_ALLOCATION_INFO.   Should we
add a CIFS POSIX Extension operation for this too?

By the way it looks like OCFS2 (not just ext4) implement these.

-- 
Thanks,

Steve

Re: punching holes in files

[Anton Altaparmakov]

On 1 Nov 2007, at 19:16, Steve French wrote:
> madvise_remove (in Linux) is used to free the backing store associated
> with pages (punching a hole in a file).   This is one of the vfs
> operations that we do not send over the wire to Samba (so this call
> would return -ENOSYS locally).   Any thoughts on whether this could be
> done with an obscure SetFileInfo level or FCNTL or whether it is worth
> adding to the CIFS POSIX Extensions?

Don't know about SMB/CIFS but on Windows locally you would need to  
first set the file sparse, and then to punch the hole.  These two  
things are accomplished like so on Windows (STARTING_OFFSET is the  
first byte of the hole to be punched and END_OFFSET is the first byte  
after the hole to be punched):

	HANDLE f; // This is obtained from a CreateFile() call...
	DWORD bw;

	/* Set file sparse. */
	if (!DeviceIoControl(f, FSCTL_SET_SPARSE, NULL, 0, NULL, 0, &bw,  
NULL)) {
		// Failed.
		return 1;
	}

	FILE_ZERO_DATA_INFORMATION z;

	/* Create a hole. */
	z.FileOffset.QuadPart = STARTING_OFFSET;
	z.BeyondFinalZero.QuadPart = END_OFFSET;
	if (!DeviceIoControl(f, FSCTL_SET_ZERO_DATA, &z, sizeof(z), NULL, 0,  
&bw, NULL)) {
		// Failed.
		return 1;
	}

> A second interesting question is whether to implement fallocate over
> the wire.  sys_fallocate takes a range (and also a flag which
> indicates whether to extend the size beyond end of file).
> SET_FILE_ALLOCATION_INFO takes a size field rather than a range.  For
> the case of the range starting at zero, we could simply call Trans2
> set pathinfo (or file info) with SET_FILE_ALLOCATION_INFO.   Should we
> add a CIFS POSIX Extension operation for this too?
>
> By the way it looks like OCFS2 (not just ext4) implement these.


Again, I don't know about SMB/CIFS but Windows locally supports this,  
too.  To pre-allocate some space without changing the file size (i.e.  
this only allocates space on disk, assigns it to the file, and changes  
the allocated_size on NTFS appropriately but does not change the  
data_size), one would call (note BYTE_OFFSET_IN_FILE_TO_PREALLOCATE_TO  
is the logical file offset to which to pre-allocate to):

	HANDLE f; // This is obtained from a CreateFile() call...
	FILE_ALLOCATION_INFO ai;

	ai.AllocationSize.QuadPart = BYTE_OFFSET_IN_FILE_TO_PREALLOCATE_TO;
	if (!SetFileInformationByHandle(f, FileAllocationInfo, &ai,  
sizeof(ai))) {
		// Failed.
		return 1;
	}

Hope this is of some use to use...

Best regards,

	Anton
-- 
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @)
Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK
Linux NTFS maintainer, http://www.linux-ntfs.org/

Re: punching holes in files

[Volker Lendecke]

[-- Attachment #1.1: Type: text/plain, Size: 969 bytes --]

On Thu, Nov 01, 2007 at 11:17:32PM +0000, Anton Altaparmakov wrote:
> On 1 Nov 2007, at 19:16, Steve French wrote:
> >madvise_remove (in Linux) is used to free the backing store associated
> >with pages (punching a hole in a file).   This is one of the vfs
> >operations that we do not send over the wire to Samba (so this call
> >would return -ENOSYS locally).   Any thoughts on whether this could be
> >done with an obscure SetFileInfo level or FCNTL or whether it is worth
> >adding to the CIFS POSIX Extensions?
> 
> Don't know about SMB/CIFS but on Windows locally you would need to  
> first set the file sparse, and then to punch the hole.  These two  
> things are accomplished like so on Windows (STARTING_OFFSET is the  
> first byte of the hole to be punched and END_OFFSET is the first byte  
> after the hole to be punched):

It would be really great if you could try this with a file
on a Windows server and send us the sniffs!

Volker

[-- Attachment #1.2: Type: application/pgp-signature, Size: 189 bytes --]

[-- Attachment #2: Type: text/plain, Size: 172 bytes --]

_______________________________________________
linux-cifs-client mailing list
linux-cifs-client@lists.samba.org
https://lists.samba.org/mailman/listinfo/linux-cifs-client

Large SMBwriteX testing.

[Jeremy Allison]

Hi Steve,

	I've finished adding the ability for smbd to support up to
16MB writeX calls in the latest git 3.2 tree.

To enable, set the parameter :

min receivefile size = XXX

where XXX is the smallest writeX you want to handle with recvfile.

The linux kernel doesn't yet support zerocopy from network to
file (ie. splice only works one way currently) so it's emulated
in userspace (with a 128k staging buffer) for now.

Also it must be an unsigned connection (for obvious reasons).

Once you've set this param smbd will start reporting
CIFS_UNIX_LARGE_WRITE_CAP on a SMB_QUERY_CIFS_UNIX_INFO:
call and you should be good to go. You'll need to use
a writeX call identical to Windows (14 wct with a 1 byte
pad field) in order to trigger the new code.

Let me know if you get the chance to test it and if
it makes a speed difference for CIFSFS.

Cheers,

	Jeremy.

Re: Large SMBwriteX testing.

[Steve French]

I have verified that it works for the case in which "min receivefile
size" is under 128K.  When I set it to 250000 and tried to read 148000
there were two or three problems (reply_write_and_X in Samba is
calling smb_len instead of smb_len_large and it is looking for
"req->unread_bytes" incorrectly in a few places in reply.c and
fileio.c

On Nov 2, 2007 6:43 PM, Jeremy Allison <jra@samba.org> wrote:
> Hi Steve,
>
>         I've finished adding the ability for smbd to support up to
> 16MB writeX calls in the latest git 3.2 tree.
>
> To enable, set the parameter :
>
> min receivefile size = XXX
>
> where XXX is the smallest writeX you want to handle with recvfile.
>
> The linux kernel doesn't yet support zerocopy from network to
> file (ie. splice only works one way currently) so it's emulated
> in userspace (with a 128k staging buffer) for now.
>
> Also it must be an unsigned connection (for obvious reasons).
>
> Once you've set this param smbd will start reporting
> CIFS_UNIX_LARGE_WRITE_CAP on a SMB_QUERY_CIFS_UNIX_INFO:
> call and you should be good to go. You'll need to use
> a writeX call identical to Windows (14 wct with a 1 byte
> pad field) in order to trigger the new code.
>
> Let me know if you get the chance to test it and if
> it makes a speed difference for CIFSFS.
>
> Cheers,
>
>         Jeremy.
>



-- 
Thanks,

Steve