transferring bvecs over the network in drbd

transferring bvecs over the network in drbd

[Christoph Hellwig]

Hi all,

I recently went over code that directly access the bio_vec bv_page/
bv_offset members and the code in _drbd_send_bio/_drbd_send_zc_bio
came to my attention.

It iterates the bio to kmap all segments, and then either does a
sock_sendmsg on a newly created kvec iter, or one one a new bvec iter
for each segment.  The former can't work on highmem systems and both
versions are rather inefficient.

What is preventing drbd from doing a single sock_sendmsg with the
bvec payload?  nvme-tcp (nvme_tcp_init_iter0 is a good example for
doing that, or the sunrpc svcsock code using it's local bvec list
(svc_tcp_sendmsg).

transferring bvecs over the network in drbd

[Christoph Hellwig]

Hi all,

I recently went over code that directly access the bio_vec bv_page/
bv_offset members and the code in _drbd_send_bio/_drbd_send_zc_bio
came to my attention.

It iterates the bio to kmap all segments, and then either does a
sock_sendmsg on a newly created kvec iter, or one one a new bvec iter
for each segment.  The former can't work on highmem systems and both
versions are rather inefficient.

What is preventing drbd from doing a single sock_sendmsg with the
bvec payload?  nvme-tcp (nvme_tcp_init_iter0 is a good example for
doing that, or the sunrpc svcsock code using it's local bvec list
(svc_tcp_sendmsg).

Re: transferring bvecs over the network in drbd

[Lars Ellenberg]

On Wed, May 07, 2025 at 11:45:50PM -0700, Christoph Hellwig wrote:
> Hi all,
> 
> I recently went over code that directly access the bio_vec bv_page/
> bv_offset members and the code in _drbd_send_bio/_drbd_send_zc_bio
> came to my attention.
> 
> It iterates the bio to kmap all segments, and then either does a
> sock_sendmsg on a newly created kvec iter, or one one a new bvec iter
> for each segment.  The former can't work on highmem systems and both
> versions are rather inefficient.
> 
> What is preventing drbd from doing a single sock_sendmsg with the
> bvec payload?  nvme-tcp (nvme_tcp_init_iter0 is a good example for
> doing that, or the sunrpc svcsock code using it's local bvec list
> (svc_tcp_sendmsg).

For async replication, we want to actually copy data into send buffer,
we cannot have the network stack hold a reference to a page for which
we signalled io completion already.

For sync replication we want to avoid additional data copy if possible,
so try to use "zero copy sendpage".

That's why we have two variants of what looks to be the same thing.

Why we do it that way: probably when we wrote that part,
a better infrastructure was not available, or we were not aware of it.
Thanks for the pointers, we'll look into it.
Using more efficient ways to do stuff sounds good.

    Lars

Re: transferring bvecs over the network in drbd

[Lars Ellenberg]

On Wed, May 07, 2025 at 11:45:50PM -0700, Christoph Hellwig wrote:
> Hi all,
> 
> I recently went over code that directly access the bio_vec bv_page/
> bv_offset members and the code in _drbd_send_bio/_drbd_send_zc_bio
> came to my attention.
> 
> It iterates the bio to kmap all segments, and then either does a
> sock_sendmsg on a newly created kvec iter, or one one a new bvec iter
> for each segment.  The former can't work on highmem systems and both
> versions are rather inefficient.
> 
> What is preventing drbd from doing a single sock_sendmsg with the
> bvec payload?  nvme-tcp (nvme_tcp_init_iter0 is a good example for
> doing that, or the sunrpc svcsock code using it's local bvec list
> (svc_tcp_sendmsg).

For async replication, we want to actually copy data into send buffer,
we cannot have the network stack hold a reference to a page for which
we signalled io completion already.

For sync replication we want to avoid additional data copy if possible,
so try to use "zero copy sendpage".

That's why we have two variants of what looks to be the same thing.

Why we do it that way: probably when we wrote that part,
a better infrastructure was not available, or we were not aware of it.
Thanks for the pointers, we'll look into it.
Using more efficient ways to do stuff sounds good.

    Lars

Re: transferring bvecs over the network in drbd

[Christoph Hellwig]

On Thu, May 08, 2025 at 10:39:56AM +0200, Lars Ellenberg wrote:
> For async replication, we want to actually copy data into send buffer,
> we cannot have the network stack hold a reference to a page for which
> we signalled io completion already.
> 
> For sync replication we want to avoid additional data copy if possible,
> so try to use "zero copy sendpage".

I didn't even complain about having both variants :)

> 
> That's why we have two variants of what looks to be the same thing.
> 
> Why we do it that way: probably when we wrote that part,
> a better infrastructure was not available, or we were not aware of it.

Yes.  While the iov_iter and the bvec version of have been around
for a long time, drbd probably still predates them.

> Thanks for the pointers, we'll look into it.
> Using more efficient ways to do stuff sounds good.

thanks.  Note that now that ->sendpage has been replaced with the
MSG_SPLICE_PAGES flag you can actually share most code for both
variants as well.