algif_hash: splice of data > 2**16

algif_hash: splice of data > 2**16

[Stephan Mueller]

Hi Herbert,

While testing the vmsplice/splice interface of algif_hash I was made 
aware of the problem that data blobs larger than 16 pages do not seem to 
be hashed properly.

For testing, a file is mmap()ed and handed to vmsplice / splice. If the 
file is smaller than 2**16, the interface returns the proper hash. 
However, when the file is larger, only the first 2**16 bytes seem to be 
used.

When adding printk's to hash_sendpage, I see that this function is 
invoked exactly 16 times where the first 15 invocations have the 
MSG_MORE flag set and the last invocation does not have MSG_MORE.

To invoke the vmsplice/splice interface, I use my current implementation 
of libkcapi available at [1]. That library uses the following calls:

vmsplice(handle->pipes[1], iov, iovlen, SPLICE_F_GIFT);
splice(handle->pipes[0], NULL, handle->opfd, NULL, inlen, 0);
read(handle->opfd, out, outlen);

The return code of splice() is always 65536 irrespectively of the 
following variations:

- have one iovec pointing to the entire buffer of the mmap()ed file

- have multiple iovecs with all pointing to at most 2**16 bytes

- invoke splice/read multiple times with at most 2**16 bytes

My guess of the problem is that the maximum buffer size for the pipe is 
16 pages. These 16 pages are filled with the repeated invocation of 
hash_sendpage.

Is there a specific way of how the splice interface is to be called so 
that the pages are released? Or is there a need to change hash_sendpage 
such that after the af_alg_wait_for_completion call, some form of 
"releasing/putting" of the used page needs to be added?

Note, hashing the large mmap()ed file with one iovec using the sendmsg() 
interface works flawlessly.

[1] http://www.chronox.de/libkcapi.html

Thanks
Stephan

Re: algif_hash: splice of data > 2**16

[leroy christophe]

Le 18/12/2014 13:15, Stephan Mueller a écrit :
> Hi Herbert,
>
> While testing the vmsplice/splice interface of algif_hash I was made
> aware of the problem that data blobs larger than 16 pages do not seem to
> be hashed properly.
>
> For testing, a file is mmap()ed and handed to vmsplice / splice. If the
> file is smaller than 2**16, the interface returns the proper hash.
> However, when the file is larger, only the first 2**16 bytes seem to be
> used.
>
> When adding printk's to hash_sendpage, I see that this function is
> invoked exactly 16 times where the first 15 invocations have the
> MSG_MORE flag set and the last invocation does not have MSG_MORE.
>
>
Hi Stephan,

I have already noticed the same issue and proposed a patch, but I never 
got any feedback and it has never been merged, allthought I pinged it a 
few times.

See https://lkml.org/lkml/2014/4/18/276

Christophe

Re: algif_hash: splice of data > 2**16

[Stephan Mueller]

Am Donnerstag, 18. Dezember 2014, 13:22:20 schrieb leroy christophe:

Hi Leroy, Herbert,

>Le 18/12/2014 13:15, Stephan Mueller a écrit :
>> Hi Herbert,
>> 
>> While testing the vmsplice/splice interface of algif_hash I was made
>> aware of the problem that data blobs larger than 16 pages do not seem
>> to be hashed properly.
>> 
>> For testing, a file is mmap()ed and handed to vmsplice / splice. If
>> the file is smaller than 2**16, the interface returns the proper
>> hash. However, when the file is larger, only the first 2**16 bytes
>> seem to be used.
>> 
>> When adding printk's to hash_sendpage, I see that this function is
>> invoked exactly 16 times where the first 15 invocations have the
>> MSG_MORE flag set and the last invocation does not have MSG_MORE.
>
>Hi Stephan,
>
>I have already noticed the same issue and proposed a patch, but I never
>got any feedback and it has never been merged, allthought I pinged it
>a few times.
>
>See https://lkml.org/lkml/2014/4/18/276

Thanks a lot for pointing to this patch.

Herbert, do you see a problem in this patch? If no, would it be possible 
that you ping the VFS maintainer as I guess your word carries more 
weight?

In any case, I will test that patch tonight.

Thanks
Stephan

Re: algif_hash: splice of data > 2**16

[Stephan Mueller]

Am Donnerstag, 18. Dezember 2014, 13:22:20 schrieb leroy christophe:

Hi Christophe,

> Le 18/12/2014 13:15, Stephan Mueller a écrit :
> > Hi Herbert,
> > 
> > While testing the vmsplice/splice interface of algif_hash I was made
> > aware of the problem that data blobs larger than 16 pages do not seem to
> > be hashed properly.
> > 
> > For testing, a file is mmap()ed and handed to vmsplice / splice. If the
> > file is smaller than 2**16, the interface returns the proper hash.
> > However, when the file is larger, only the first 2**16 bytes seem to be
> > used.
> > 
> > When adding printk's to hash_sendpage, I see that this function is
> > invoked exactly 16 times where the first 15 invocations have the
> > MSG_MORE flag set and the last invocation does not have MSG_MORE.
> 
> Hi Stephan,
> 
> I have already noticed the same issue and proposed a patch, but I never
> got any feedback and it has never been merged, allthought I pinged it a
> few times.
> 
> See https://lkml.org/lkml/2014/4/18/276

After testing, this patch does not work for me. The operation still stops 
after 16 pages.
> 
> Christophe


-- 
Ciao
Stephan

Re: algif_hash: splice of data > 2**16

[leroy christophe]

Le 20/12/2014 07:37, Stephan Mueller a écrit :
> Am Donnerstag, 18. Dezember 2014, 13:22:20 schrieb leroy christophe:
>
> Hi Christophe,
>
>> Le 18/12/2014 13:15, Stephan Mueller a écrit :
>>> Hi Herbert,
>>>
>>> While testing the vmsplice/splice interface of algif_hash I was made
>>> aware of the problem that data blobs larger than 16 pages do not seem to
>>> be hashed properly.
>>>
>>> For testing, a file is mmap()ed and handed to vmsplice / splice. If the
>>> file is smaller than 2**16, the interface returns the proper hash.
>>> However, when the file is larger, only the first 2**16 bytes seem to be
>>> used.
>>>
>>> When adding printk's to hash_sendpage, I see that this function is
>>> invoked exactly 16 times where the first 15 invocations have the
>>> MSG_MORE flag set and the last invocation does not have MSG_MORE.
>> Hi Stephan,
>>
>> I have already noticed the same issue and proposed a patch, but I never
>> got any feedback and it has never been merged, allthought I pinged it a
>> few times.
>>
>> See https://lkml.org/lkml/2014/4/18/276
> After testing, this patch does not work for me. The operation still stops
> after 16 pages.
Yes, it looks like the function I fixed is exclusively used by 
sendfile() system call.
So there is probably the same kind of fix to be done in another function.

Christophe

Re: algif_hash: splice of data > 2**16

[Stephan Mueller]

Am Dienstag, 23. Dezember 2014, 18:16:01 schrieb leroy christophe:

Hi leroy,

> Le 20/12/2014 07:37, Stephan Mueller a écrit :
> > Am Donnerstag, 18. Dezember 2014, 13:22:20 schrieb leroy christophe:
> > 
> > Hi Christophe,
> > 
> >> Le 18/12/2014 13:15, Stephan Mueller a écrit :
> >>> Hi Herbert,
> >>> 
> >>> While testing the vmsplice/splice interface of algif_hash I was made
> >>> aware of the problem that data blobs larger than 16 pages do not seem to
> >>> be hashed properly.
> >>> 
> >>> For testing, a file is mmap()ed and handed to vmsplice / splice. If the
> >>> file is smaller than 2**16, the interface returns the proper hash.
> >>> However, when the file is larger, only the first 2**16 bytes seem to be
> >>> used.
> >>> 
> >>> When adding printk's to hash_sendpage, I see that this function is
> >>> invoked exactly 16 times where the first 15 invocations have the
> >>> MSG_MORE flag set and the last invocation does not have MSG_MORE.
> >> 
> >> Hi Stephan,
> >> 
> >> I have already noticed the same issue and proposed a patch, but I never
> >> got any feedback and it has never been merged, allthought I pinged it a
> >> few times.
> >> 
> >> See https://lkml.org/lkml/2014/4/18/276
> > 
> > After testing, this patch does not work for me. The operation still stops
> > after 16 pages.
> 
> Yes, it looks like the function I fixed is exclusively used by
> sendfile() system call.
> So there is probably the same kind of fix to be done in another function.

I do not believe that is the case. IMHO the blocking issue is found in the 
following code:

splice_from_pipe_feed walks the pipe->nrbufs. And vmsplice_to_pipe defines the 
maximum number of nrbufs as PIPE_DEF_BUFFERS -- which is 16. As subsequent 
functions allocate memory based on PIPE_DEF_BUFFERS, there is no trivial way 
to increase the number of pages to be processed.

Thus I see that the vmsplice/splice combo can at most operate on a chunk of 16 
pages. Thus, you have to segment your input buffer into chunks of that size 
and invoke the vmsplice/splice syscalls for each segment.

-- 
Ciao
Stephan

Re: algif_hash: splice of data > 2**16

[leroy christophe]

Le 24/12/2014 10:03, Stephan Mueller a écrit :
> Am Dienstag, 23. Dezember 2014, 18:16:01 schrieb leroy christophe:
>
> Hi leroy,
>
>> Le 20/12/2014 07:37, Stephan Mueller a écrit :
>>> Am Donnerstag, 18. Dezember 2014, 13:22:20 schrieb leroy christophe:
>>>
>>> Hi Christophe,
>>>
>>>> Le 18/12/2014 13:15, Stephan Mueller a écrit :
>>>>> Hi Herbert,
>>>>>
>>>>> While testing the vmsplice/splice interface of algif_hash I was made
>>>>> aware of the problem that data blobs larger than 16 pages do not seem to
>>>>> be hashed properly.
>>>>>
>>>>> For testing, a file is mmap()ed and handed to vmsplice / splice. If the
>>>>> file is smaller than 2**16, the interface returns the proper hash.
>>>>> However, when the file is larger, only the first 2**16 bytes seem to be
>>>>> used.
>>>>>
>>>>> When adding printk's to hash_sendpage, I see that this function is
>>>>> invoked exactly 16 times where the first 15 invocations have the
>>>>> MSG_MORE flag set and the last invocation does not have MSG_MORE.
>>>> Hi Stephan,
>>>>
>>>> I have already noticed the same issue and proposed a patch, but I never
>>>> got any feedback and it has never been merged, allthought I pinged it a
>>>> few times.
>>>>
>>>> See https://lkml.org/lkml/2014/4/18/276
>>> After testing, this patch does not work for me. The operation still stops
>>> after 16 pages.
>> Yes, it looks like the function I fixed is exclusively used by
>> sendfile() system call.
>> So there is probably the same kind of fix to be done in another function.
> I do not believe that is the case. IMHO the blocking issue is found in the
> following code:
>
> splice_from_pipe_feed walks the pipe->nrbufs. And vmsplice_to_pipe defines the
> maximum number of nrbufs as PIPE_DEF_BUFFERS -- which is 16. As subsequent
> functions allocate memory based on PIPE_DEF_BUFFERS, there is no trivial way
> to increase the number of pages to be processed.
>
> Thus I see that the vmsplice/splice combo can at most operate on a chunk of 16
> pages. Thus, you have to segment your input buffer into chunks of that size
> and invoke the vmsplice/splice syscalls for each segment.
>
Yes your are probably right. There splice needs to be called with 
SPLICE_F_MORE flag, hope that works.

Re: algif_hash: splice of data > 2**16

[Stephan Mueller]

Am Mittwoch, 24. Dezember 2014, 15:10:14 schrieb leroy christophe:

Hi leroy,

> Le 24/12/2014 10:03, Stephan Mueller a écrit :
> > Am Dienstag, 23. Dezember 2014, 18:16:01 schrieb leroy christophe:
> > 
> > Hi leroy,
> > 
> >> Le 20/12/2014 07:37, Stephan Mueller a écrit :
> >>> Am Donnerstag, 18. Dezember 2014, 13:22:20 schrieb leroy christophe:
> >>> 
> >>> Hi Christophe,
> >>> 
> >>>> Le 18/12/2014 13:15, Stephan Mueller a écrit :
> >>>>> Hi Herbert,
> >>>>> 
> >>>>> While testing the vmsplice/splice interface of algif_hash I was made
> >>>>> aware of the problem that data blobs larger than 16 pages do not seem
> >>>>> to
> >>>>> be hashed properly.
> >>>>> 
> >>>>> For testing, a file is mmap()ed and handed to vmsplice / splice. If
> >>>>> the
> >>>>> file is smaller than 2**16, the interface returns the proper hash.
> >>>>> However, when the file is larger, only the first 2**16 bytes seem to
> >>>>> be
> >>>>> used.
> >>>>> 
> >>>>> When adding printk's to hash_sendpage, I see that this function is
> >>>>> invoked exactly 16 times where the first 15 invocations have the
> >>>>> MSG_MORE flag set and the last invocation does not have MSG_MORE.
> >>>> 
> >>>> Hi Stephan,
> >>>> 
> >>>> I have already noticed the same issue and proposed a patch, but I never
> >>>> got any feedback and it has never been merged, allthought I pinged it a
> >>>> few times.
> >>>> 
> >>>> See https://lkml.org/lkml/2014/4/18/276
> >>> 
> >>> After testing, this patch does not work for me. The operation still
> >>> stops
> >>> after 16 pages.
> >> 
> >> Yes, it looks like the function I fixed is exclusively used by
> >> sendfile() system call.
> >> So there is probably the same kind of fix to be done in another function.
> > 
> > I do not believe that is the case. IMHO the blocking issue is found in the
> > following code:
> > 
> > splice_from_pipe_feed walks the pipe->nrbufs. And vmsplice_to_pipe defines
> > the maximum number of nrbufs as PIPE_DEF_BUFFERS -- which is 16. As
> > subsequent functions allocate memory based on PIPE_DEF_BUFFERS, there is
> > no trivial way to increase the number of pages to be processed.
> > 
> > Thus I see that the vmsplice/splice combo can at most operate on a chunk
> > of 16 pages. Thus, you have to segment your input buffer into chunks of
> > that size and invoke the vmsplice/splice syscalls for each segment.
> 
> Yes your are probably right. There splice needs to be called with
> SPLICE_F_MORE flag, hope that works.

That is not the only one. Vmsplice works on a pipe. A pipe is backed by 16 
pages max. Thus, you have to call vmsplice once per 16 pages. The current code 
I am testing which seems to work is the following:

	while (inlen) {
		size_t datalen = (inlen > MAXLEN) ? MAXLEN : inlen;

		iov.iov_len = datalen;
		ret = _kcapi_common_vmsplice_data(handle, &iov, 1, datalen, 
1);
		if (0 > ret)
			return ret;
		processed += ret;
		iov.iov_base = (void*)(uintptr_t)(in + processed);
		inlen -= ret;
	}
	return _kcapi_common_read_data(handle, out, outlen);


-- 
Ciao
Stephan

Re: algif_hash: splice of data > 2**16

[Stephan Mueller]

Am Mittwoch, 24. Dezember 2014, 16:12:53 schrieb Stephan Mueller:

Hi Christophe,

> Am Mittwoch, 24. Dezember 2014, 15:10:14 schrieb leroy christophe:
> 
> Hi leroy,
> 
> > Le 24/12/2014 10:03, Stephan Mueller a écrit :
> > > Am Dienstag, 23. Dezember 2014, 18:16:01 schrieb leroy christophe:
> > > 
> > > Hi leroy,
> > > 
> > >> Le 20/12/2014 07:37, Stephan Mueller a écrit :
> > >>> Am Donnerstag, 18. Dezember 2014, 13:22:20 schrieb leroy christophe:
> > >>> 
> > >>> Hi Christophe,
> > >>> 
> > >>>> Le 18/12/2014 13:15, Stephan Mueller a écrit :
> > >>>>> Hi Herbert,
> > >>>>> 
> > >>>>> While testing the vmsplice/splice interface of algif_hash I was made
> > >>>>> aware of the problem that data blobs larger than 16 pages do not
> > >>>>> seem
> > >>>>> to
> > >>>>> be hashed properly.
> > >>>>> 
> > >>>>> For testing, a file is mmap()ed and handed to vmsplice / splice. If
> > >>>>> the
> > >>>>> file is smaller than 2**16, the interface returns the proper hash.
> > >>>>> However, when the file is larger, only the first 2**16 bytes seem to
> > >>>>> be
> > >>>>> used.
> > >>>>> 
> > >>>>> When adding printk's to hash_sendpage, I see that this function is
> > >>>>> invoked exactly 16 times where the first 15 invocations have the
> > >>>>> MSG_MORE flag set and the last invocation does not have MSG_MORE.
> > >>>> 
> > >>>> Hi Stephan,
> > >>>> 
> > >>>> I have already noticed the same issue and proposed a patch, but I
> > >>>> never
> > >>>> got any feedback and it has never been merged, allthought I pinged it
> > >>>> a
> > >>>> few times.
> > >>>> 
> > >>>> See https://lkml.org/lkml/2014/4/18/276
> > >>> 
> > >>> After testing, this patch does not work for me. The operation still
> > >>> stops
> > >>> after 16 pages.
> > >> 
> > >> Yes, it looks like the function I fixed is exclusively used by
> > >> sendfile() system call.
> > >> So there is probably the same kind of fix to be done in another
> > >> function.
> > > 
> > > I do not believe that is the case. IMHO the blocking issue is found in
> > > the
> > > following code:
> > > 
> > > splice_from_pipe_feed walks the pipe->nrbufs. And vmsplice_to_pipe
> > > defines
> > > the maximum number of nrbufs as PIPE_DEF_BUFFERS -- which is 16. As
> > > subsequent functions allocate memory based on PIPE_DEF_BUFFERS, there is
> > > no trivial way to increase the number of pages to be processed.
> > > 
> > > Thus I see that the vmsplice/splice combo can at most operate on a chunk
> > > of 16 pages. Thus, you have to segment your input buffer into chunks of
> > > that size and invoke the vmsplice/splice syscalls for each segment.
> > 
> > Yes your are probably right. There splice needs to be called with
> > SPLICE_F_MORE flag, hope that works.
> 
> That is not the only one. Vmsplice works on a pipe. A pipe is backed by 16
> pages max. Thus, you have to call vmsplice once per 16 pages. The current
> code I am testing which seems to work is the following:
> 
> 	while (inlen) {
> 		size_t datalen = (inlen > MAXLEN) ? MAXLEN : inlen;
> 
> 		iov.iov_len = datalen;
> 		ret = _kcapi_common_vmsplice_data(handle, &iov, 1, datalen,
> 1);
> 		if (0 > ret)
> 			return ret;
> 		processed += ret;
> 		iov.iov_base = (void*)(uintptr_t)(in + processed);
> 		inlen -= ret;
> 	}
> 	return _kcapi_common_read_data(handle, out, outlen);

As I do not see an easy fix where a single vmsplice/splice syscall combo can 
be applied to an input data stream of arbitrary length, I added appropriate 
handling code in libkcapi version 0.6.2 [1].

This code follows the aforementioned approach to segment the input into 16 
PAGE_SIZE segments and send them to the kernel individually.

[1] http://www.chronox.de/libkcapi.html
-- 
Ciao
Stephan