splice/vmsplice performance test results

splice/vmsplice performance test results

[Jim Schutt]

Hi,

I've done some testing to see how splice/vmsplice perform
vs. other alternatives on transferring a large file across
a fast network.  One option I tested was to use vmsplice 
to get a 1-copy receive, but it didn't perform as well 
as I had hoped.  I was wondering if my results were at odds
with what other people have observed.

I've two systems, each with:
  Tyan S2895 motherboard
  2 ea. 2.6 GHz Opteron
  1 GiB memory
  Myricom Myri-10G 10 Gb/s NIC (PCIe x8)
  2.6.19-rc5-g134a11f0 on FC4

In addition, one system has a 3ware 9590-8ML (PCIe) and a 3ware
9550SX-8LP (PCI-X), with 16 Seagate Barracuda 7200.10 SATA drives 
(250 GB ea., NCQ enabled).  Write caching is enabled on the 3ware
cards.

The Myricom cards are connected back-to-back using 9000 byte MTU. 
I baseline the network performance with 'iperf -w 1M -l 64K'
and get 6.9 Gb/s.

After a fair amount of testing, I settled on a 4-way software
RAID0 on top of 4-way hardware RAID0 units as giving the best
streaming performance.  The file system is XFS, with the stripe
unit set to the hardware RAID chunk size, and the stripe width 
16 times that.  

Disk tuning parameters in /sys/block/sd*/queue are default
values, except queue/nr_requests = 5 gives me best performance.
(It seems like the 3ware cards slow down a little if I feed them 
too much data on the streaming write test I'm using.)

I baseline file write performance with 
  sync; time { dd if=/dev/zero of=./zero bs=32k count=512k; sync; }
and get 465-520 MB/s (highly variable).

I test baseline file read performance with
  time dd if=./zero of=/dev/null bs=32k count=512k
and get 950 MB/s (fairly repeatable).

My test program can do one of the following:

send data:
 A) read() from file into buffer, write() buffer into socket
 B) mmap() section of file, write() that into socket, munmap()
 C) splice() from file to pipe, splice() from pipe to socket

receive data:
 1) read() from socket into buffer, write() buffer into file
 2) ftruncate() to extend file, mmap() new extent, read() 
      from socket into new extent, munmap()
 3) read() from socket into buffer, vmsplice() buffer to 
     pipe, splice() pipe to file (using the double-buffer trick)

Here's the results, using:
 - 64 KiB buffer, mmap extent, or splice
 - 1 MiB TCP window
 - 16 GiB data sent across network

A) from /dev/zero -> 1) to /dev/null : 857 MB/s (6.86 Gb/s)

A) from file      -> 1) to /dev/null : 472 MB/s (3.77 Gb/s)
B) from file      -> 1) to /dev/null : 366 MB/s (2.93 Gb/s)
C) from file      -> 1) to /dev/null : 854 MB/s (6.83 Gb/s)

A) from /dev/zero -> 1) to file      : 375 MB/s (3.00 Gb/s)
A) from /dev/zero -> 2) to file      : 150 MB/s (1.20 Gb/s)
A) from /dev/zero -> 3) to file      : 286 MB/s (2.29 Gb/s)

I had (naively) hoped the read/vmsplice/splice combination would 
run at the same speed I can write a file, i.e. at about 450 MB/s
on my setup.  Do any of my numbers seem bogus, so I should look 
harder at my test program?

Or is read+write really the fastest way to get data off a
socket and into a file?

-- Jim Schutt

(Please Cc: me, as I'm not subscribed to lkml.)

Re: splice/vmsplice performance test results

[Jens Axboe]

On Thu, Nov 16 2006, Jim Schutt wrote:
> Hi,
> 
> I've done some testing to see how splice/vmsplice perform
> vs. other alternatives on transferring a large file across
> a fast network.  One option I tested was to use vmsplice 
> to get a 1-copy receive, but it didn't perform as well 
> as I had hoped.  I was wondering if my results were at odds
> with what other people have observed.
> 
> I've two systems, each with:
>   Tyan S2895 motherboard
>   2 ea. 2.6 GHz Opteron
>   1 GiB memory
>   Myricom Myri-10G 10 Gb/s NIC (PCIe x8)
>   2.6.19-rc5-g134a11f0 on FC4
> 
> In addition, one system has a 3ware 9590-8ML (PCIe) and a 3ware
> 9550SX-8LP (PCI-X), with 16 Seagate Barracuda 7200.10 SATA drives 
> (250 GB ea., NCQ enabled).  Write caching is enabled on the 3ware
> cards.
> 
> The Myricom cards are connected back-to-back using 9000 byte MTU. 
> I baseline the network performance with 'iperf -w 1M -l 64K'
> and get 6.9 Gb/s.
> 
> After a fair amount of testing, I settled on a 4-way software
> RAID0 on top of 4-way hardware RAID0 units as giving the best
> streaming performance.  The file system is XFS, with the stripe
> unit set to the hardware RAID chunk size, and the stripe width 
> 16 times that.  
> 
> Disk tuning parameters in /sys/block/sd*/queue are default
> values, except queue/nr_requests = 5 gives me best performance.
> (It seems like the 3ware cards slow down a little if I feed them 
> too much data on the streaming write test I'm using.)
> 
> I baseline file write performance with 
>   sync; time { dd if=/dev/zero of=./zero bs=32k count=512k; sync; }
> and get 465-520 MB/s (highly variable).
> 
> I test baseline file read performance with
>   time dd if=./zero of=/dev/null bs=32k count=512k
> and get 950 MB/s (fairly repeatable).
> 
> My test program can do one of the following:
> 
> send data:
>  A) read() from file into buffer, write() buffer into socket
>  B) mmap() section of file, write() that into socket, munmap()
>  C) splice() from file to pipe, splice() from pipe to socket
> 
> receive data:
>  1) read() from socket into buffer, write() buffer into file
>  2) ftruncate() to extend file, mmap() new extent, read() 
>       from socket into new extent, munmap()
>  3) read() from socket into buffer, vmsplice() buffer to 
>      pipe, splice() pipe to file (using the double-buffer trick)
> 
> Here's the results, using:
>  - 64 KiB buffer, mmap extent, or splice
>  - 1 MiB TCP window
>  - 16 GiB data sent across network
> 
> A) from /dev/zero -> 1) to /dev/null : 857 MB/s (6.86 Gb/s)
> 
> A) from file      -> 1) to /dev/null : 472 MB/s (3.77 Gb/s)
> B) from file      -> 1) to /dev/null : 366 MB/s (2.93 Gb/s)
> C) from file      -> 1) to /dev/null : 854 MB/s (6.83 Gb/s)
> 
> A) from /dev/zero -> 1) to file      : 375 MB/s (3.00 Gb/s)
> A) from /dev/zero -> 2) to file      : 150 MB/s (1.20 Gb/s)
> A) from /dev/zero -> 3) to file      : 286 MB/s (2.29 Gb/s)
> 
> I had (naively) hoped the read/vmsplice/splice combination would 
> run at the same speed I can write a file, i.e. at about 450 MB/s
> on my setup.  Do any of my numbers seem bogus, so I should look 
> harder at my test program?

Could be read-ahead playing in here, I'd have to take a closer look at
the generated io patterns to say more about that. Any chance you can
capture iostat or blktrace info for such a run to compare that goes to
the disk? Can you pass along the test program?

> Or is read+write really the fastest way to get data off a
> socket and into a file?

splice() should be just as fast of course, and more efficient. Not a lot
of real-life performance tuning has gone into it yet, so I would not be
surprised if we need to smoothen a few edges.

-- 
Jens Axboe

Re: splice/vmsplice performance test results

[Jim Schutt]

On Thu, 2006-11-16 at 21:25 +0100, Jens Axboe wrote:
> On Thu, Nov 16 2006, Jim Schutt wrote:
> > Hi,
> > 

> > 
> > My test program can do one of the following:
> > 
> > send data:
> >  A) read() from file into buffer, write() buffer into socket
> >  B) mmap() section of file, write() that into socket, munmap()
> >  C) splice() from file to pipe, splice() from pipe to socket
> > 
> > receive data:
> >  1) read() from socket into buffer, write() buffer into file
> >  2) ftruncate() to extend file, mmap() new extent, read() 
> >       from socket into new extent, munmap()
> >  3) read() from socket into buffer, vmsplice() buffer to 
> >      pipe, splice() pipe to file (using the double-buffer trick)
> > 
> > Here's the results, using:
> >  - 64 KiB buffer, mmap extent, or splice
> >  - 1 MiB TCP window
> >  - 16 GiB data sent across network
> > 
> > A) from /dev/zero -> 1) to /dev/null : 857 MB/s (6.86 Gb/s)
> > 
> > A) from file      -> 1) to /dev/null : 472 MB/s (3.77 Gb/s)
> > B) from file      -> 1) to /dev/null : 366 MB/s (2.93 Gb/s)
> > C) from file      -> 1) to /dev/null : 854 MB/s (6.83 Gb/s)
> > 
> > A) from /dev/zero -> 1) to file      : 375 MB/s (3.00 Gb/s)
> > A) from /dev/zero -> 2) to file      : 150 MB/s (1.20 Gb/s)
> > A) from /dev/zero -> 3) to file      : 286 MB/s (2.29 Gb/s)
> > 
> > I had (naively) hoped the read/vmsplice/splice combination would 
> > run at the same speed I can write a file, i.e. at about 450 MB/s
> > on my setup.  Do any of my numbers seem bogus, so I should look 
> > harder at my test program?
> 
> Could be read-ahead playing in here, I'd have to take a closer look at
> the generated io patterns to say more about that. Any chance you can
> capture iostat or blktrace info for such a run to compare that goes to
> the disk? 

I can try.  Do you prefer iostat or blktrace, or would you like both?
Can you point me at some instructions?

> Can you pass along the test program?

Inserted inline below.

> 
> > Or is read+write really the fastest way to get data off a
> > socket and into a file?
> 
> splice() should be just as fast of course, and more efficient. Not a lot
> of real-life performance tuning has gone into it yet, so I would not be
> surprised if we need to smoothen a few edges.
> 

I'm glad I can help a little here.

-- Jim

Here's a splice.h I'm using, based on your example:
-----------------------

/* Implement splice syscall support for glibc versions that don't
 * have it.
 */

#ifndef __do_splice_syscall_h__
#define __do_splice_syscall_h__

#include <sys/syscall.h>
#include <unistd.h>

#if defined(__i386__)

/* From kernel tree include/asm-i386/unistd.h
 */
#ifndef __NR_splice
#define __NR_splice   313
#endif
#ifndef __NR_vmsplice
#define __NR_vmsplice 316
#endif

#elif defined(__x86_64__)

/* From kernel tree include/asm-x86_64/unistd.h
 */
#ifndef __NR_splice
#define __NR_splice   275
#endif
#ifndef __NR_vmsplice
#define __NR_vmsplice 278
#endif

#else
#error unsupported architecture
#endif

/* From kernel tree include/linux/pipe_fs_i.h
 */
#define SPLICE_F_MOVE	(0x01)	 /* move pages instead of copying */
#define SPLICE_F_NONBLOCK (0x02) /* don't block on the pipe splicing
(but */
				 /* we may still block on the fd we splice */
				 /* from/to, of course */
#define SPLICE_F_MORE   (0x04)	 /* expect more data */
#define SPLICE_F_GIFT   (0x08)	 /* pages passed in are a gift */

#ifndef SYS_splice
#define SYS_splice __NR_splice
#endif
#ifndef SYS_vmsplice
#define SYS_vmsplice __NR_vmsplice
#endif


#ifndef _LARGEFILE64_SOURCE
#error need -D_LARGEFILE64_SOURCE
#endif

static inline
int splice(int fd_in, off64_t *off_in, int fd_out, off64_t *off_out,
	   size_t len, unsigned int flags)
{
	return syscall(SYS_splice, fd_in, off_in,
		       fd_out, off_out, len, flags);
}

struct iovec;

static inline
int vmsplice(int fd, const struct iovec *iov,
	     unsigned long nr_segs, unsigned int flags)
{
	return syscall(SYS_vmsplice, fd, iov, nr_segs, flags);
}


#endif /* __do_splice_syscall_h__ */

------------------

And here's the test program itself:


/* Copyright 2006 Sandia Corporation.
 *
 * This file is free software; you can redistribute it and/or modify
 * it under the terms of version 2 of the GNU General Public License,
 * as published by the Free Software Foundation.
 */

/* Compile with -DHAVE_SPLICE_SYSCALL if you _know_ your kernel
 * has splice/vmsplice support.
 */

#ifndef _LARGEFILE64_SOURCE
#error need -D_LARGEFILE64_SOURCE
#endif


#include <stdint.h>
#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>
#include <errno.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <sys/time.h>
#include <sys/mman.h>
#include <sys/sendfile.h>

#include <netdb.h>
#include <sys/socket.h>
#include <netinet/tcp.h>
#include <arpa/inet.h>
#include <resolv.h>

#define ALIGN(value,size) (((value) + (size) - 1) & ~((size) - 1))

/* Once glibc implements splice(), SYS_splice will be defined in 
 * system headers.  Until then we need to use our own stuff to access
 * these syscalls in new kernels.
 */
#ifdef HAVE_SPLICE_SYSCALL
#ifndef SYS_splice
#include <splice.h>
#endif
#endif

#define OPTION_FLAGS "b:c:hl:m:op:tw:"

const char usage[] = "\n\
dnd [OPTION] ... <file> \n\
\n\
	Performs a timed disk-network-disk data transfer, using TCP/IP \n\
	as the network protocol. \n\
\n\
	When dnd is invoked with \"-c <remote-host>\", it connects to \n\
	<remote-host> and sends the contents of <file>.  Otherwise, it \n\
	accepts a connection and writes the data received over the \n\
	connection into <file>.  On the sender side, timing starts just \n\
	before the first byte is read from the file, and stops just after \n\
	the last byte of data is sent.  On the receiver side, timing starts \n\
	just before the first byte is received, and stops just after the \n\
	last byte is synced to disk. \n\
\n\
	-b <bsz> \n\
		Use a buffer of size <bsz> bytes to move data.  The default \n\
		value is 65536 bytes. The value for <bsz> may be suffixed \n\
		with one of the following multipliers: \n\
		  k	*1000 \n\
		  M	*1000*1000 \n\
		  G	*1000*1000*1000 \n\
		  Ki	*1024 \n\
		  Mi	*1024*1024 \n\
		  Gi	*1024*1024*1024 \n\
	-c <remote-host> \n\
		Connect to <rhost> to send the data in <file>. \n\
		If not specified, listen for connections and receive data \n\
		into <file>. \n\
	-h \n\
		Print this message. \n\
	-l <sz> \n\
		Limit the transfer to at most <sz> bytes.  The value for \n\
		  <sz> may be suffixed as for the '-b' option.  Valid only \n\
		  if the '-c' option is also present.\n\
	-m <method> \n\
		Select one of the following methods: \n\
		mmap	Use mmap system call on the file descriptor and \n\
			  read/write system calls on the socket descriptor. \n\
		rw	Use read/write system calls on both the file \n\
			  descriptor and the socket descriptor. (Default) \n\
		sendfile \n\
			Use the sendfile system call to send data. \n\
		splice	Use the splice system call. Currently only supports \n\
			  a splice from the file to the socket. \n\
		vmsplice \n\
			Use the read system call to receive data from the \n\
			  socket into memory, and the vmsplice system call \n\
			  to move the data into the file. \n\
	-o \n\
		If writing to <file> and it already exists, overwrite its \n\
		data and truncate it to the total number of bytes received.\n\
	-p <port> \n\
		Either listen on <port>, or attempt to connect to \n\
		<remote_host>:<port>.  The default port is 13931. \n\
	-t \n\
		If writing to <file> and it already exists, truncate it \n\
		to zero length before writing to it.\n\
	-w <wsz> \n\
		Use a TCP window size of <wsz> bytes, which may be suffixed \n\
		as for <bsz> above.  The default value is 131072 bytes.\n\
\n\
";


enum method {MMAP, RW, SENDFILE, SPLICE, VMSPLICE};

struct options {
	enum method use;

	unsigned int truncate:1;
	unsigned int overwrite:1;
	unsigned int readfile:1;

	uint64_t limit;
	int win_size;
	size_t buf_size;
	unsigned short def_port;

	char *host_str;
	char *file;
	char *buf;
	void *private;
};

typedef uint64_t (*move_function)(const struct options *opts,
				  int from_fd, int to_fd);

union pipe_fd {
	int fda[2];
	struct {
		int r;
		int w;
	} fd;
};

static struct options dnd_opts = {
	.use = RW,
	.buf_size = 64 * 1024,
	.win_size = 128 * 1024,
	.def_port = 13931
};

uint64_t dt_usec(struct timeval *start, struct timeval *stop)
{
	uint64_t dt;

	dt = stop->tv_sec - start->tv_sec;
	dt *= 1000000;
	dt += stop->tv_usec - start->tv_usec;
	return dt;
}

static
uint64_t suffix(const char *str)
{
	uint64_t s = 1;

	switch (*str) {
	case 'k':
		s *= 1000;
		break;
	case 'K':
		if (*(str+1) == 'i')
			s *= 1024;
		break;
	case 'M':
		if (*(str+1) == 'i')
			s *= 1024*1024;
		else
			s *= 1000*1000;
		break;
	case 'G':
		if (*(str+1) == 'i')
			s *= 1024*1024*1024;
		else
			s *= 1000*1000*1000;
		break;
	}

	return s;
}

int open_file(const struct options *opts)
{
	int fd, err;
	mode_t o_mode = 0644;
	int o_flags = O_LARGEFILE;

	if (opts->use == MMAP)
		o_flags |= O_RDWR;
	else {
		if (opts->readfile)
			o_flags |= O_RDONLY;
		else
			o_flags |= O_WRONLY;
	}

	if (!opts->readfile) {
		o_flags |= O_CREAT;
		if (opts->truncate)
			o_flags |= O_TRUNC;
		else if (!opts->overwrite)
			o_flags |= O_EXCL;
	}
	fd = open64(opts->file, o_flags, o_mode);
	if (fd < 0) {
		perror("Open data file");
		exit(EXIT_FAILURE);
	}
	return fd;
}

int connect_to(const struct options *opts, struct sockaddr_in6 *saddr)
{
	int fd;
	int optval;
	socklen_t optlen;
	int err;

	struct hostent *tgt;

	/* Turn on IPv6 resolver action - gethostbyname() will always
	 * return IPv6 addresses.
	 */
	res_init();
	_res.options |= RES_USE_INET6;

	tgt = gethostbyname(opts->host_str);
	if (!tgt) {
		herror("gethostbyname/IPv6");
		exit(EXIT_FAILURE);
	}
	if (tgt->h_addrtype != AF_INET6) {
		fprintf(stderr, 
			"Error: got non-IPv6 address from gethostbyname!\n");
		exit(EXIT_FAILURE);
	}
#if 1
	{
		char buf[INET6_ADDRSTRLEN+1] = {0,};
		char **ptr;

		printf("connecting to host: %s\n", opts->host_str);
		printf("  canonical name: %s\n", tgt->h_name);
		ptr = tgt->h_aliases;
		while (*ptr) {
			printf("  alias: %s\n", *ptr);
			++ptr;
		}
		ptr = tgt->h_addr_list;
		while (*ptr) {
			if (!inet_ntop(tgt->h_addrtype, *ptr,
				       buf, INET6_ADDRSTRLEN+1)) {
				perror("inet_ntop/IPv6");
				exit(EXIT_FAILURE);
			}
			printf("  address: %s\n", buf);
			++ptr;
		}
	}
#endif
	fd = socket(PF_INET6, SOCK_STREAM, 0);
	if (fd == -1) {
		perror("Open IPv6 socket");
		exit(EXIT_FAILURE);
	}

	optval = opts->win_size;
	optlen = sizeof(optval);
	err = setsockopt(fd, SOL_SOCKET, SO_SNDBUF, &optval, optlen);
	if (err) {
		perror("Set IPv6 socket SO_SNDBUF");
		exit(EXIT_FAILURE);
	}
	optlen = sizeof(optval);
	err = getsockopt(fd, SOL_SOCKET, SO_SNDBUF, &optval, &optlen);
	if (err) {
		perror("Get IPv6 socket SO_SNDBUF");
		exit(EXIT_FAILURE);
	}
	if (optval != opts->win_size)
		printf("TCP send window size: requested %d actual %d\n",
		       opts->win_size, optval);

	saddr->sin6_family = AF_INET6;
	saddr->sin6_port = htons(opts->def_port);
	saddr->sin6_addr = *(struct in6_addr *)tgt->h_addr_list[0];

	err = connect(fd, (struct sockaddr *)saddr, sizeof(*saddr));
	if (err) {
		perror("Connect to remote host");
		exit(EXIT_FAILURE);
	}

	optval = 1;
	optlen = sizeof(optval);
	err = setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &optval, optlen);
	if (err) {
		perror("Set IPv6 socket TCP_NODELAY");
		exit(EXIT_FAILURE);
	}
	return fd;
}

int listen_for(const struct options *opts, struct sockaddr_in6 *saddr)
{
	int fd, lfd;
	int optval;
	socklen_t optlen;
	int err;

	lfd = socket(PF_INET6, SOCK_STREAM, 0);
	if (lfd == -1) {
		perror("Open IPv6 socket");
		exit(EXIT_FAILURE);
	}
	optval = 1;
	err = setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR,
			 &optval, sizeof(optval));
	if (err) {
		perror("Set IPv6 socket SO_REUSEADDR");
		exit(EXIT_FAILURE);
	}

	optval = opts->win_size;
	optlen = sizeof(optval);
	err = setsockopt(lfd, SOL_SOCKET, SO_RCVBUF, &optval, optlen);
	if (err) {
		perror("Set IPv6 socket SO_RCVBUF");
		exit(EXIT_FAILURE);
	}
	optlen = sizeof(optval);
	err = getsockopt(lfd, SOL_SOCKET, SO_RCVBUF, &optval, &optlen);
	if (err) {
		perror("Get IPv6 socket SO_RCVBUF");
		exit(EXIT_FAILURE);
	}
	if (optval != opts->win_size)
		printf("TCP receive window size: requested %d actual %d\n",
		       opts->win_size, optval);

	saddr->sin6_family = AF_INET6;
	saddr->sin6_port = htons(opts->def_port);
	saddr->sin6_addr = in6addr_any;

	err = bind(lfd, (struct sockaddr *)saddr, sizeof(*saddr));
	if (err) {
		perror("Bind IPv6 address");
		exit(EXIT_FAILURE);
	}
	err = listen(lfd, 1);
	if (err) {
		perror("Listen on IPv6 address");
		exit(EXIT_FAILURE);
	}
	fd = accept(lfd, NULL, 0);
	if (fd < 0) {
		perror("Accept new connection");
		exit(EXIT_FAILURE);
	}
	close(lfd);

	optval = 1;
	optlen = sizeof(optval);
	err = setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &optval, optlen);
	if (err) {
		perror("Set IPv6 socket TCP_NODELAY");
		exit(EXIT_FAILURE);
	}
	return fd;
}

int wait_on_connection(const struct options *opts)
{
	int sock_fd, err;
	struct sockaddr_in6 saddr;
	socklen_t saddr_len = sizeof(saddr);
	char buf[INET6_ADDRSTRLEN+1];

	if (opts->host_str)
		sock_fd = connect_to(opts, &saddr);
	else
		sock_fd = listen_for(opts, &saddr);

	err = getpeername(sock_fd, (struct sockaddr *)&saddr, &saddr_len);
	if (err) {
		perror("getpeername");
		exit(EXIT_FAILURE);
	}
	if (saddr.sin6_family != AF_INET6) {
		fprintf(stderr, 
			"Error: got non-IPv6 address from getpeername!\n");
		exit(EXIT_FAILURE);
	}
	if (!inet_ntop(AF_INET6, &saddr.sin6_addr,
		       buf, INET6_ADDRSTRLEN+1)) {
		perror("inet_ntop/IPv6");
		exit(EXIT_FAILURE);
	}
	printf("Connected to %s port %d\n",
	       buf, (int)ntohs(saddr.sin6_port));

	return sock_fd;
}

void setup_mmap(struct options *opts, int fd)
{
	struct stat64 sb;
	size_t pg_sz = sysconf(_SC_PAGESIZE);

	opts->buf_size = ALIGN(opts->buf_size, pg_sz);

	/* We'll just get the file size here so we don't time it later...
	 */
	if (fstat64(fd, &sb) < 0) {
		perror("Stat data file");
		exit(EXIT_FAILURE);
	}
	opts->private = malloc(sizeof(off64_t));
	if (!opts->private) {
		perror("Allocating private data");
		exit(EXIT_FAILURE);
	}
	*((off64_t *)opts->private) = sb.st_size;
}

uint64_t mmap_send(const struct options *opts, int fd, int sd)
{
	size_t bufl = opts->buf_size;
	off64_t fsz = *((off64_t *)opts->private);

	uint64_t bytes = 0;
	ssize_t n;
	size_t m, l;
	char *mem;
	int err;

	if (opts->limit && opts->limit < (uint64_t)fsz)
		fsz = opts->limit;

again:
	mem = mmap64(NULL, bufl, PROT_READ, MAP_SHARED, fd, bytes);
	if (mem == MAP_FAILED) {
		fprintf(stderr, "mmap %llu @ offset %llu: %s\n",
			(unsigned long long)bufl,
			(unsigned long long)bytes, strerror(errno));
		exit(EXIT_FAILURE);
	}
#ifdef USE_MADVISE
	err = madvise(mem, bufl, MADV_WILLNEED);
	if (err && errno != EAGAIN) {
		perror("madvise");
		exit(EXIT_FAILURE);
	}
#endif
	if (bytes + bufl < (uint64_t)fsz)
		l = bufl;
	else
		l = fsz - bytes;

	m = 0;
	while (l) {

	again2:
		n = write(sd, mem + m, l);
		if (n < 0) {
			if (errno == EINTR)
				goto again2;
			perror("write");
			exit(EXIT_FAILURE);
		}
		bytes += n;
		m += n;
		l -= n;
	}
	err = munmap(mem, bufl);
	if (err < 0) {
		fprintf(stderr, "munmap %llu: %s\n",
			(unsigned long long)bufl, strerror(errno));
		exit(EXIT_FAILURE);
	}
	if (bytes == (uint64_t)fsz)
		return bytes;

	goto again;
}

uint64_t mmap_recv(const struct options *opts, int sd, int fd)
{
	size_t bufl = opts->buf_size;

	uint64_t bytes = 0;
	ssize_t n;
	size_t m, l;
	char *mem;
	int err;

again:
	l = bufl;

	err = ftruncate64(fd, bytes + bufl);
	if (err < 0) {
		fprintf(stderr, "ftruncate to %llu: %s\n",
			(unsigned long long)bytes + bufl, strerror(errno));
		exit(EXIT_FAILURE);
	}
	mem = mmap64(NULL, bufl, PROT_WRITE, MAP_SHARED, fd, bytes);
	if (mem == MAP_FAILED) {
		fprintf(stderr, "mmap %llu @ offset %llu: %s\n",
			(unsigned long long)bufl,
			(unsigned long long)bytes, strerror(errno));
		exit(EXIT_FAILURE);
	}

	m = 0;
	while (l) {

	again2:
		n = read(sd, mem + m, l);
		if (n < 0) {
			if (errno == EINTR)
				goto again2;
			perror("Read");
			exit(EXIT_FAILURE);
		}
		if (n == 0) {
			err = munmap(mem, bufl);
			if (err < 0) {
				fprintf(stderr, "munmap %llu: %s\n",
					(unsigned long long)bufl,
					strerror(errno));
				exit(EXIT_FAILURE);
			}
			err = ftruncate64(fd, bytes);
			if (err < 0) {
				fprintf(stderr, "ftruncate to %llu: %s\n",
					(unsigned long long)bytes,
					strerror(errno));
				exit(EXIT_FAILURE);
			}
			fdatasync(fd);
			return bytes;
		}
		bytes += n;
		m += n;
		l -= n;
	}
	err = munmap(mem, bufl);
	if (err < 0) {
		fprintf(stderr, "munmap %llu: %s\n",
			(unsigned long long)bufl, strerror(errno));
		exit(EXIT_FAILURE);
	}
	goto again;
}

void setup_rw(struct options *opts)
{
	opts->buf = malloc(opts->buf_size);
	if (!opts->buf) {
		perror("Allocating data buffer");
		exit(EXIT_FAILURE);
	}
}

uint64_t read_write(const struct options *opts, int rfd, int wfd)
{
	char *buf = opts->buf;
	size_t bufl = opts->buf_size;

	uint64_t bytes = 0;
	ssize_t n, m, l;

again:
	if (opts->limit && bufl > opts->limit - bytes)
		bufl = opts->limit - bytes;

	l = read(rfd, buf, bufl);
	if (l < 0) {
		if (errno == EINTR)
			goto again;
		perror("Read");
		exit(EXIT_FAILURE);
	}
	if (l == 0 || (opts->limit && opts->limit == bytes)) {
		if (opts->readfile)
			close(wfd);
		else {
			ftruncate64(wfd, bytes);
			fdatasync(wfd);
		}
		return bytes;
	}

	m = 0;
	while (l) {

	again2:
		n = write(wfd, buf + m, l);
		if (n < 0) {
			if (errno == EINTR)
				goto again2;
			perror("Write");
			exit(EXIT_FAILURE);
		}
		m += n;
		l -= n;
	}
	bytes += m;
	goto again;
}

uint64_t sendfile_send(const struct options *opts, int fd, int sd)
{
	size_t bufl = opts->buf_size;

	uint64_t bytes = 0;
	off64_t os = 0;
	ssize_t l;

again:
	if (opts->limit && bufl > opts->limit - bytes)
		bufl = opts->limit - bytes;

	l = sendfile64(sd, fd, &os, bufl);
	if (l < 0) {
		perror("sendfile from file");
		exit(EXIT_FAILURE);
	}
	if (l == 0) {
		close(sd);
		return bytes;
	}
	bytes += l;

	goto again;
}

/* At least for now (as of 2.6.18-rc5), splice seems to hang if you
 * try to splice more data than the pipe can handle, rather than doing
 * what it can and returning what that was.  So, coerce user buffer down
 * to maximum pipe buffer size.
 */
#ifndef MAX_SPLICE_SIZE
#define MAX_SPLICE_SIZE (64 * 1024)
#endif

void setup_splice(struct options *opts)
{
	union pipe_fd *p;
	size_t pg_sz = sysconf(_SC_PAGESIZE);

	p = malloc(sizeof(union pipe_fd));
	if (!p) {
		perror("allocate pipe fds");
		exit(EXIT_FAILURE);
	}
	if (pipe(p->fda) < 0) {
		perror("opening pipe");
		exit(EXIT_FAILURE);
	}
	opts->private = p;

	if (opts->buf_size > MAX_SPLICE_SIZE)
		opts->buf_size = MAX_SPLICE_SIZE;

	opts->buf_size = ALIGN(opts->buf_size, pg_sz);
}

#ifdef HAVE_SPLICE_SYSCALL
uint64_t splice_send(const struct options *opts, int fd, int sd)
{
	union pipe_fd *p = opts->private;
	size_t bufl = opts->buf_size;

	uint64_t bytes = 0;
	ssize_t n, l;

again:
	if (opts->limit && bufl > opts->limit - bytes)
		bufl = opts->limit - bytes;

	l = splice(fd, NULL, p->fd.w, NULL, bufl, 
		   SPLICE_F_MORE | SPLICE_F_MOVE);
	if (l < 0) {
		perror("splice from file");
		exit(EXIT_FAILURE);
	}
	if (l == 0) {
		close(sd);
		return bytes;
	}

	while (l) {

		n = splice(p->fd.r, NULL, sd, NULL, l,
			   SPLICE_F_MORE | SPLICE_F_MOVE);
		if (n < 0) {
			perror("splice to socket");
			exit(EXIT_FAILURE);
		}
		l -= n;
		bytes += n;
	}
	goto again;
}
#else
move_function splice_send = NULL;
#endif

/* vmsplice moves pages backing a user address range to a pipe.
However,
 * you don't want the application changing data in that address range
 * after the pages have been moved to the pipe, but before they have
been
 * consumed at their destination.
 *
 * The solution is to double buffer: 
 *   load buffer A, vmsplice to pipe
 *   load buffer B, vmsplice to pipe
 * When the B->splice->pipe call completes, there can no longer be any
 * references in the pipe to the pages backing buffer A, since it is now
 * filled with references to the pages backing buffer B.  So, it is safe
 * to load new data into buffer A.
 */
void setup_vmsplice(struct options *opts)
{
	union pipe_fd *p;
	size_t pg_sz = sysconf(_SC_PAGESIZE);

	p = malloc(sizeof(union pipe_fd));
	if (!p) {
		perror("allocate pipe fds");
		exit(EXIT_FAILURE);
	}
	if (pipe(p->fda) < 0) {
		perror("opening pipe");
		exit(EXIT_FAILURE);
	}
	opts->private = p;

	if (opts->buf_size > MAX_SPLICE_SIZE)
		opts->buf_size = MAX_SPLICE_SIZE;

	opts->buf_size = ALIGN(opts->buf_size, pg_sz);

	opts->buf = malloc(2*opts->buf_size + pg_sz);
	if (!opts->buf) {
		perror("Allocating data buffer");
		exit(EXIT_FAILURE);
	}
	opts->buf = (char *)ALIGN((unsigned long)opts->buf,
				  (unsigned long)pg_sz);
}

#ifdef HAVE_SPLICE_SYSCALL
uint64_t vmsplice_recv(const struct options *opts, int sd, int fd)
{
	union pipe_fd *p = opts->private;
	struct iovec iov;

	uint64_t bytes = 0;
	ssize_t n, m, l;
	unsigned i = 1;

again:
	i = (i + 1) & 1;
	iov.iov_base = opts->buf + i * opts->buf_size;

again2:
	l = read(sd, iov.iov_base, opts->buf_size);
	if (l < 0) {
		if (errno == EINTR)
			goto again2;
		perror("Read");
		exit(EXIT_FAILURE);
	}
	if (l == 0) {
		fdatasync(fd);
		return bytes;
	}

	while (l) {
		iov.iov_len = l;

		n = vmsplice(p->fd.w, &iov, 1, 0);
		if (n < 0) {
			perror("vmsplice to pipe");
			exit(EXIT_FAILURE);
		}

		while (n) {
			m = splice(p->fd.r, NULL, fd, NULL, n, SPLICE_F_MORE);
			if (m < 0) {
				perror("splice to file");
				exit(EXIT_FAILURE);
			}
			n -= m;

			l -= m;
			bytes += m;
			iov.iov_base += m;
		}
	}
	goto again;
}
#else
move_function vmsplice_recv = NULL;
#endif

uint64_t move(const struct options *opts, int fd, int sd,
	      move_function do_send, move_function do_recv)
{
	uint64_t bytes;

	if (opts->readfile) {
		if (do_send)
			bytes = do_send(opts, fd, sd);
		else
			goto no_support;
	}
	else {
		if (do_recv)
			bytes = do_recv(opts, sd, fd);
		else
			goto no_support;
	}
	return bytes;

no_support:
	fprintf(stderr, "Sorry, method not implemented.\n");
	exit(EXIT_FAILURE);
}

int main(int argc, char *argv[])
{
	int sd, fd;

	uint64_t byte_cnt;

	struct timeval start;
	struct timeval stop;
	uint64_t et_usec;

	move_function do_send;
	move_function do_recv;

	while (1) {
		char *next_char;
		int c = getopt(argc, argv, OPTION_FLAGS);
		if (c == -1)
			break;

		switch (c) {
		case 'b':
		{
			uint64_t sz = strtoull(optarg, &next_char, 0);
			sz *= suffix(next_char);
			dnd_opts.buf_size = sz;
			if ((uint64_t)dnd_opts.buf_size != sz) {
				fprintf(stderr,
					"Error: invalid buffer size\n");
				exit(EXIT_FAILURE);
			}
		}
			break;
		case 'c':
			dnd_opts.host_str = strdup(optarg);
			dnd_opts.readfile = 1;
			break;
		case 'h':
			printf("%s", usage);
			exit(EXIT_SUCCESS);
		case 'l':
		{
			uint64_t sz = strtoull(optarg, &next_char, 0);
			sz *= suffix(next_char);
			dnd_opts.limit = sz;
		}
			break;
		case 'm':
			if (strncmp(optarg, "mmap", 32) == 0)
				dnd_opts.use = MMAP;
			else if (strncmp(optarg, "rw", 32) == 0)
				dnd_opts.use = RW;
			else if (strncmp(optarg, "sendfile", 32) == 0)
				dnd_opts.use = SENDFILE;
			else if (strncmp(optarg, "splice", 32) == 0)
				dnd_opts.use = SPLICE;
			else if (strncmp(optarg, "vmsplice", 32) == 0)
				dnd_opts.use = VMSPLICE;
			else {
				fprintf(stderr, 
					"Error: unknown method '%s'\n",
					optarg);
				exit(EXIT_FAILURE);
			}
			break;
		case 'o':
			dnd_opts.overwrite = 1;
                        break;
		case 'p':
		{
			unsigned long port = strtoul(optarg, NULL, 0);
			dnd_opts.def_port = port;
			if (dnd_opts.def_port == 0 ||
			    (unsigned long)dnd_opts.def_port != port) {
				fprintf(stderr,
					"Error: invalid port\n");
				exit(EXIT_FAILURE);
			}
		}
			break;
		case 't':
			dnd_opts.truncate = 1;
			break;
		case 'w':
 		{
			uint64_t wsz = strtoull(optarg, &next_char, 0);
			wsz *= suffix(next_char);
			dnd_opts.win_size = wsz;
			if ((uint64_t)dnd_opts.win_size != wsz) {
				fprintf(stderr,
					"Error: invalid window size\n");
				exit(EXIT_FAILURE);
			}
		}
			break;
		}
	}
	if (dnd_opts.limit && !dnd_opts.readfile) {
		fprintf(stderr, "Error: can only limit transfer as sender!\n"
			"  (I.e., when also using '-c' option.)\n");
		exit(EXIT_FAILURE);
	}
	if (dnd_opts.truncate && dnd_opts.overwrite) {
		fprintf(stderr, "Error: cannot both overwrite "
			"and truncate data file!\n");
		exit(EXIT_FAILURE);
	}
	if (argc == 1) {
		printf("%s", usage);
		exit(EXIT_SUCCESS);
	}
	if (optind+1 != argc) {
		fprintf(stderr, "Error: need a filename\n");
		exit(EXIT_FAILURE);
	}
	dnd_opts.file = strdup(argv[optind]);

	fd = open_file(&dnd_opts);
	sd = wait_on_connection(&dnd_opts);

	switch (dnd_opts.use) {
	case MMAP:
		printf("Using mmap with read/write calls\n");
		setup_mmap(&dnd_opts, fd);
		do_send = mmap_send;
		do_recv = mmap_recv;
		break;
	case RW:
		printf("Using read/write calls\n");
		setup_rw(&dnd_opts);
		do_send = read_write;
		do_recv = read_write;
		break;
	case SENDFILE:
		printf("Using sendfile calls\n");
		do_send = sendfile_send;
		do_recv = NULL;
		break;
	case SPLICE:
		printf("Using splice calls\n");
		setup_splice(&dnd_opts);
		do_send = splice_send;
		do_recv = NULL;
		break;
	case VMSPLICE:
		printf("Using vmsplice with read/write calls\n");
		setup_vmsplice(&dnd_opts);
		do_send = NULL;
		do_recv = vmsplice_recv;
		break;
	}
	printf("Buffer size %u bytes\n", (unsigned)dnd_opts.buf_size);

	gettimeofday(&start, NULL);

	byte_cnt = move(&dnd_opts, fd, sd, do_send, do_recv);

	gettimeofday(&stop, NULL);
	et_usec = dt_usec(&start, &stop);

	printf("\n%s %llu KiB in %.3f sec: %.3f MB/s (%.3f Gb/s)\n\n",
	       (dnd_opts.readfile ? "Sent" : "Received"),
	       (unsigned long long)byte_cnt/1024, (1.e-6 * et_usec),
	       ((float)byte_cnt/et_usec),
	       8*((float)byte_cnt/et_usec)/1000);

	exit(EXIT_SUCCESS);
}

Re: splice/vmsplice performance test results

[Jim Schutt]

[-- Attachment #1: Type: text/plain, Size: 2124 bytes --]

On Thu, 2006-11-16 at 21:25 +0100, Jens Axboe wrote:
> On Thu, Nov 16 2006, Jim Schutt wrote:
> > Hi,
> > 

> > My test program can do one of the following:
> > 
> > send data:
> >  A) read() from file into buffer, write() buffer into socket
> >  B) mmap() section of file, write() that into socket, munmap()
> >  C) splice() from file to pipe, splice() from pipe to socket
> > 
> > receive data:
> >  1) read() from socket into buffer, write() buffer into file
> >  2) ftruncate() to extend file, mmap() new extent, read() 
> >       from socket into new extent, munmap()
> >  3) read() from socket into buffer, vmsplice() buffer to 
> >      pipe, splice() pipe to file (using the double-buffer trick)
> > 
> > Here's the results, using:
> >  - 64 KiB buffer, mmap extent, or splice
> >  - 1 MiB TCP window
> >  - 16 GiB data sent across network
> > 
> > A) from /dev/zero -> 1) to /dev/null : 857 MB/s (6.86 Gb/s)
> > 
> > A) from file      -> 1) to /dev/null : 472 MB/s (3.77 Gb/s)
> > B) from file      -> 1) to /dev/null : 366 MB/s (2.93 Gb/s)
> > C) from file      -> 1) to /dev/null : 854 MB/s (6.83 Gb/s)
> > 
> > A) from /dev/zero -> 1) to file      : 375 MB/s (3.00 Gb/s)
> > A) from /dev/zero -> 2) to file      : 150 MB/s (1.20 Gb/s)
> > A) from /dev/zero -> 3) to file      : 286 MB/s (2.29 Gb/s)
> > 
> > I had (naively) hoped the read/vmsplice/splice combination would 
> > run at the same speed I can write a file, i.e. at about 450 MB/s
> > on my setup.  Do any of my numbers seem bogus, so I should look 
> > harder at my test program?
> 
> Could be read-ahead playing in here, I'd have to take a closer look at
> the generated io patterns to say more about that. Any chance you can
> capture iostat or blktrace info for such a run to compare that goes to
> the disk?

I've attached a file with iostat and vmstat results for the case
where I read from a socket and write a file, vs. the case where I
read from a socket and use vmsplice/splice to write the file.
(Sorry it's not inline - my mailer locks up when I try to
include the file.)

Would you still like blktrace info for these two cases?

-- Jim


[-- Attachment #2: iostat.txt --]
[-- Type: text/plain, Size: 19790 bytes --]


Here's 'iostat -dx 5' on the receiver, when it reads from a socket
and writes to a file:

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdd          0.00   0.00  0.00  0.20    0.00    0.40     0.00     0.20     2.00     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0          0.00   0.00  0.00  0.20    0.00    0.40     0.00     0.20     2.00     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.20  0.00 571.26    0.00 142574.45     0.00 71287.23   249.58     0.89    1.56   0.43  24.51
sdc          0.00   0.00  0.00 576.85    0.00 142558.08     0.00 71279.04   247.13     1.02    1.77   0.42  23.95
sdd          0.00   1.20  0.60 577.45    4.79 142545.91     2.40 71272.95   246.61     1.16    2.00   0.54  31.46
sde          0.00   0.60  0.00 573.85    0.00 142607.58     0.00 71303.79   248.51     1.34    2.33   0.54  30.74
md0          0.00   0.00  0.60 2301.40    4.79 570286.03     2.40 285143.01   247.74     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.20  0.00 750.40    0.00 187265.80     0.00 93632.90   249.55     1.10    1.46   0.42  31.36
sdc          0.00   1.40  0.00 755.80    0.00 187236.80     0.00 93618.40   247.73     1.30    1.72   0.43  32.64
sdd          0.00   2.80  0.00 751.20    0.00 187253.40     0.00 93626.70   249.27     1.53    2.04   0.55  41.36
sde          0.00   0.20  0.00 767.40    0.00 187235.20     0.00 93617.60   243.99     1.65    2.15   0.54  41.52
md0          0.00   0.00  0.00 3029.40    0.00 748991.20     0.00 374495.60   247.24     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   1.00  0.00 771.26    0.00 187299.20     0.00 93649.60   242.85     1.21    1.57   0.44  33.61
sdc          0.00   1.60  0.00 765.87    0.00 187262.28     0.00 93631.14   244.51     1.33    1.73   0.45  34.49
sdd          0.00   1.00  0.00 739.72    0.00 187258.48     0.00 93629.24   253.15     1.43    1.93   0.55  40.56
sde          0.00   0.80  0.00 744.31    0.00 187227.15     0.00 93613.57   251.54     1.85    2.48   0.56  41.68
md0          0.00   0.00  0.00 3025.55    0.00 749047.11     0.00 374523.55   247.57     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.40  0.20 751.30    3.19 185520.36     1.60 92760.18   246.87     1.15    1.52   0.45  33.85
sdc          0.00   1.20  0.00 741.12    0.00 185344.51     0.00 92672.26   250.09     1.38    1.87   0.45  33.21
sdd          0.00   2.59  0.00 756.09    0.00 185470.06     0.00 92735.03   245.30     1.53    2.03   0.54  40.64
sde          0.00   0.40  0.00 744.71    0.00 185379.64     0.00 92689.82   248.93     1.72    2.31   0.54  40.16
md0          0.00   0.00  0.20 3002.79    3.19 742992.02     1.60 371496.01   247.42     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00 753.80    0.00 186913.80     0.00 93456.90   247.96     1.20    1.59   0.45  34.00
sdc          0.00   1.80  0.00 756.40    0.00 186971.20     0.00 93485.60   247.19     1.35    1.79   0.44  33.60
sdd          0.00   1.00  0.00 751.00    0.00 186895.40     0.00 93447.70   248.86     1.54    2.05   0.55  41.20
sde          0.00   1.00  0.00 755.60    0.00 186832.00     0.00 93416.00   247.26     1.76    2.33   0.56  42.00
md0          0.00   0.00  0.00 3019.20    0.00 747254.00     0.00 373627.00   247.50     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00 744.89    0.00 184891.58     0.00 92445.79   248.21     1.17    1.58   0.45  33.59
sdc          0.00   4.21  0.00 742.08    0.00 184734.27     0.00 92367.13   248.94     1.41    1.90   0.45  33.75
sdd          0.00   0.60  0.00 759.72    0.00 184695.19     0.00 92347.60   243.11     1.52    2.00   0.55  41.68
sde          0.00   0.80  0.00 737.88    0.00 184989.18     0.00 92494.59   250.71     1.79    2.43   0.56  41.68
md0          0.00   0.00  0.00 2991.18    0.00 739566.73     0.00 369783.37   247.25     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.80  0.00 736.53    0.00 184289.42     0.00 92144.71   250.21     1.19    1.61   0.46  34.09
sdc          0.00   1.00  0.00 743.51    0.00 184225.15     0.00 92112.57   247.78     1.39    1.86   0.46  34.57
sdd          0.00   0.60  0.00 747.70    0.00 184270.86     0.00 92135.43   246.45     1.60    2.13   0.56  41.68
sde          0.00   0.80  0.00 754.69    0.00 184199.60     0.00 92099.80   244.07     1.69    2.24   0.56  42.16
md0          0.00   0.00  0.00 2985.83    0.00 737036.13     0.00 368518.06   246.84     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   1.00  0.00 753.09    0.00 183984.23     0.00 91992.12   244.30     1.20    1.59   0.45  33.61
sdc          0.00   0.80  0.00 741.52    0.00 184097.41     0.00 92048.70   248.27     1.35    1.83   0.45  33.05
sdd          0.00   0.60  0.00 735.13    0.00 184121.56     0.00 92060.78   250.46     1.48    2.01   0.55  40.72
sde          0.00   0.40  0.00 742.12    0.00 183975.65     0.00 91987.82   247.91     1.76    2.38   0.56  41.20
md0          0.00   0.00  0.00 2969.86    0.00 734952.50     0.00 367476.25   247.47     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   1.60  0.00 741.40    0.00 183925.00     0.00 91962.50   248.08     1.11    1.50   0.45  33.44
sdc          0.00   0.80  0.00 738.80    0.00 183760.00     0.00 91880.00   248.73     1.36    1.85   0.45  33.36
sdd          0.00   0.40  0.00 749.40    0.00 183790.60     0.00 91895.30   245.25     1.52    2.02   0.55  41.04
sde          0.00   0.40  0.00 742.20    0.00 183752.80     0.00 91876.40   247.58     1.71    2.30   0.55  40.80
md0          0.00   0.00  0.00 2980.20    0.00 736559.60     0.00 368279.80   247.15     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00 201.40    0.00 50070.60     0.00 25035.30   248.61     0.30    1.48   0.46   9.20
sdc          0.00   0.40  0.00 198.40    0.00 50134.40     0.00 25067.20   252.69     0.38    1.90   0.46   9.20
sdd          0.00   0.20  0.00 203.00    0.00 50102.60     0.00 25051.30   246.81     0.49    2.40   0.56  11.36
sde          0.00   0.00  0.00 209.80    0.00 50146.00     0.00 25073.00   239.02     0.48    2.28   0.53  11.04
md0          0.00   0.00  0.00 808.00    0.00 199122.40     0.00 99561.20   246.44     0.00    0.00   0.00   0.00


and here's 'vmstat 5' during the same time, at 5 sec intervals:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 0  0    596 931948    244  18152    0    0     0     0  359    31  0  0 100  0
 1  0    596 279084    240 648088    0    0     2 318336 25708  3892  0 54 40  7
 1  0    596   7108    108 910932    0    0     0 374689 28552  5537  0 64 29  7
 1  0    596   8216    108 908124    0    0     0 375773 28805  3841  0 67 26  7
 2  1    596   7812     88 908512    0    0     2 372315 28616  3348  0 66 26  7
 1  0    596   7904     88 908368    0    0     0 372839 28687  3455  0 67 26  7
 2  0    596   7632     88 908624    0    0     0 369063 28381  3380  0 68 26  7
 1  0    596   8540    100 907712    0    0     2 369179 28405  3394  0 68 25  7
 1  0    596   7088    120 909272    0    0     6 368593 28351  3430  0 67 25  8
 1  0    596   7876    108 908492    0    0     0 367826 28282  3342  0 68 24  7


And here's 'iostat -dx 5' on the receiver, when it reads from a socket
and uses vmsplice/splice to write the file:

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdc          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
sdd          0.00   0.20  0.00  0.40    0.00    3.40     0.00     1.70     8.50     0.00    0.00   0.00   0.00
sde          0.00   0.00  0.00  0.00    0.00    0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00
md0          0.00   0.00  0.00  0.60    0.00    3.40     0.00     1.70     5.67     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00 250.30    0.00 62179.24     0.00 31089.62   248.42     0.34    1.34   0.40   9.90
sdc          0.00   0.00  0.60 249.10    4.79 62166.87     2.40 31083.43   248.98     0.42    1.67   0.41  10.14
sdd          0.00   1.00  0.00 253.89    0.00 62200.40     0.00 31100.20   244.99     0.49    1.94   0.53  13.49
sde          0.00   0.00  0.00 249.10    0.00 62166.87     0.00 31083.43   249.56     0.56    2.26   0.53  13.25
md0          0.00   0.00  0.60 1003.39    4.79 248713.37     2.40 124356.69   247.73     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00 551.60    0.00 138981.40     0.00 69490.70   251.96     0.75    1.37   0.43  23.52
sdc          0.00   0.40  0.00 545.40    0.00 138878.80     0.00 69439.40   254.64     1.06    1.93   0.44  23.76
sdd          0.00   0.00  0.00 582.80    0.00 138822.40     0.00 69411.20   238.20     1.28    2.20   0.55  31.92
sde          0.00   0.40  0.00 579.80    0.00 138780.80     0.00 69390.40   239.36     1.19    2.05   0.54  31.20
md0          0.00   0.00  0.00 2260.60    0.00 555465.00     0.00 277732.50   245.72     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00 539.68    0.00 137531.66     0.00 68765.83   254.84     0.75    1.39   0.43  23.09
sdc          0.00   0.20  0.00 537.47    0.00 137494.39     0.00 68747.19   255.82     1.05    1.96   0.43  23.01
sdd          0.00   0.00  0.00 578.36    0.00 137462.12     0.00 68731.06   237.68     1.32    2.28   0.58  33.43
sde          0.00   0.60  0.00 578.76    0.00 137487.78     0.00 68743.89   237.56     1.22    2.12   0.57  33.27
md0          0.00   0.00  0.00 2234.87    0.00 549974.35     0.00 274987.17   246.09     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00 537.45    0.00 137488.65     0.00 68744.32   255.82     0.71    1.33   0.42  22.63
sdc          0.00   0.20  0.00 537.85    0.00 137495.42     0.00 68747.71   255.64     1.03    1.92   0.43  23.11
sdd          0.00   0.00  0.00 579.08    0.00 137461.35     0.00 68730.68   237.38     1.28    2.22   0.56  32.35
sde          0.00   0.20  0.00 573.31    0.00 137447.01     0.00 68723.51   239.74     1.17    2.04   0.55  31.39
md0          0.00   0.00  0.00 2228.09    0.00 549892.43     0.00 274946.22   246.80     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00 540.40    0.00 137667.40     0.00 68833.70   254.75     0.74    1.38   0.42  22.96
sdc          0.00   0.20  0.00 538.20    0.00 137634.60     0.00 68817.30   255.73     1.05    1.96   0.44  23.52
sdd          0.00   0.00  0.00 581.60    0.00 137643.20     0.00 68821.60   236.66     1.30    2.23   0.57  33.36
sde          0.00   0.00  0.00 580.60    0.00 137641.60     0.00 68820.80   237.07     1.22    2.10   0.56  32.72
md0          0.00   0.00  0.00 2241.00    0.00 550586.80     0.00 275293.40   245.69     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00 536.33    0.00 137057.29     0.00 68528.64   255.55     0.73    1.35   0.43  22.91
sdc          0.00   0.20  0.00 535.53    0.00 136950.70     0.00 68475.35   255.73     1.04    1.93   0.43  22.91
sdd          0.00   0.20  0.00 578.64    0.00 136950.10     0.00 68475.05   236.67     1.28    2.20   0.56  32.65
sde          0.00   0.00  0.00 577.84    0.00 137124.15     0.00 68562.08   237.30     1.20    2.07   0.56  32.26
md0          0.00   0.00  0.00 2233.73    0.00 549359.68     0.00 274679.84   245.94     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.20  0.00 547.60    0.00 136721.80     0.00 68360.90   249.67     0.78    1.43   0.44  24.24
sdc          0.00   0.20  0.00 535.00    0.00 136815.40     0.00 68407.70   255.73     1.06    1.99   0.46  24.40
sdd          0.00   0.60  0.00 565.00    0.00 136840.00     0.00 68420.00   242.19     1.25    2.22   0.59  33.28
sde          0.00   0.00  0.00 571.20    0.00 136664.00     0.00 68332.00   239.26     1.22    2.14   0.56  32.16
md0          0.00   0.00  0.00 2214.80    0.00 545761.20     0.00 272880.60   246.42     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.20  0.00 541.12    0.00 136218.96     0.00 68109.48   251.74     0.77    1.42   0.45  24.35
sdc          0.00   0.20  0.00 532.53    0.00 136133.53     0.00 68066.77   255.63     1.07    2.01   0.45  23.79
sdd          0.00   0.20  0.00 562.87    0.00 136132.53     0.00 68066.27   241.85     1.25    2.22   0.55  30.98
sde          0.00   0.40  0.00 571.66    0.00 136114.97     0.00 68057.49   238.11     1.15    2.01   0.52  29.94
md0          0.00   0.00  0.00 2209.18    0.00 544600.00     0.00 272300.00   246.52     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.20  0.00 547.60    0.00 135878.80     0.00 67939.40   248.14     0.80    1.46   0.45  24.72
sdc          0.00   0.20  0.00 538.20    0.00 135837.20     0.00 67918.60   252.39     1.03    1.91   0.46  24.72
sdd          0.00   0.00  0.00 554.80    0.00 135854.40     0.00 67927.20   244.87     1.15    2.07   0.56  30.80
sde          0.00   0.00  0.00 560.60    0.00 135904.00     0.00 67952.00   242.43     1.26    2.24   0.55  30.80
md0          0.00   0.00  0.00 2201.60    0.00 543474.40     0.00 271737.20   246.85     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.40  0.00 553.00    0.00 136131.40     0.00 68065.70   246.17     0.88    1.60   0.45  25.04
sdc          0.00   0.40  0.00 552.80    0.00 136140.80     0.00 68070.40   246.27     0.99    1.79   0.44  24.08
sdd          0.00   0.00  0.00 547.80    0.00 136147.20     0.00 68073.60   248.53     1.12    2.05   0.56  30.72
sde          0.00   0.80  0.00 549.80    0.00 136028.80     0.00 68014.40   247.42     1.31    2.37   0.56  30.96
md0          0.00   0.00  0.00 2206.40    0.00 544779.40     0.00 272389.70   246.91     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00 549.60    0.00 135710.60     0.00 67855.30   246.93     0.85    1.54   0.47  26.08
sdc          0.00   0.20  0.00 544.80    0.00 135724.20     0.00 67862.10   249.13     1.02    1.88   0.47  25.36
sdd          0.00   0.20  0.00 548.20    0.00 135739.20     0.00 67869.60   247.61     1.15    2.09   0.57  31.52
sde          0.00   2.00  0.00 553.80    0.00 135891.20     0.00 67945.60   245.38     1.24    2.25   0.56  31.28
md0          0.00   0.00  0.00 2197.40    0.00 542734.00     0.00 271367.00   246.99     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.00  0.00 553.60    0.00 135733.00     0.00 67866.50   245.18     0.84    1.53   0.47  25.76
sdc          0.00   0.20  0.00 545.00    0.00 135703.40     0.00 67851.70   249.00     1.02    1.87   0.46  24.96
sdd          0.00   0.40  0.00 548.60    0.00 135649.60     0.00 67824.80   247.27     1.13    2.06   0.56  30.72
sde          0.00   0.00  0.00 548.60    0.00 135673.60     0.00 67836.80   247.31     1.27    2.32   0.56  30.64
md0          0.00   0.00  0.00 2196.40    0.00 542759.60     0.00 271379.80   247.11     0.00    0.00   0.00   0.00

Device:    rrqm/s wrqm/s   r/s   w/s  rsec/s  wsec/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb          0.00   0.40  0.00 445.00    0.00 109865.80     0.00 54932.90   246.89     0.70    1.58   0.46  20.64
sdc          0.00   1.00  0.00 447.00    0.00 109896.20     0.00 54948.10   245.85     0.79    1.77   0.45  20.24
sdd          0.00   0.60  0.00 448.00    0.00 109880.60     0.00 54940.30   245.27     0.94    2.10   0.55  24.64
sde          0.00   0.40  0.00 441.80    0.00 109851.20     0.00 54925.60   248.64     1.07    2.43   0.56  24.64
md0          0.00   0.00  0.00 1784.20    0.00 439493.80     0.00 219746.90   246.33     0.00    0.00   0.00   0.00


and here's 'vmstat 5' during the same time, at 5 sec intervals:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
 1  0    596 867408    192  77884    0    0     2     1 1163    30  0  2 98  0
 1  0    596 299956    196 628000    0    0     2 275379 21385  3112  0 58 36  5
 1  0    596  92256    196 829024    0    0     0 276103 20903  3997  0 59 35  5
 2  0    596   8320    100 909068    0    0     0 275221 20929  3106  0 61 35  5
 1  0    596   8276     88 908564    0    0     0 275253 21041  2410  0 62 33  5
 1  0    596   7100     88 909980    0    0     0 275288 21001  2243  0 63 33  4
 1  0    596   7524     88 909356    0    0     0 274507 20956  2207  0 62 33  5
 1  1    596   7520     88 909396    0    0     0 272799 20833  2236  0 63 33  4
 1  0    596   7040     88 909936    0    0     0 271638 20771  2060  0 63 32  5
 1  0    596   7996     88 909008    0    0     0 271984 20773  2191  0 63 32  5
 1  0    596   7940     88 909084    0    0     0 271769 20755  2155  0 63 31  5
 2  0    596   7824     88 908980    0    0     0 272193 20743  2158  0 63 32  5
 1  0    596   7144     88 909812    0    0     0 271100 20736  2121  0 64 31  5

Re: splice/vmsplice performance test results

[Jens Axboe]

On Fri, Nov 17 2006, Jim Schutt wrote:
> On Thu, 2006-11-16 at 21:25 +0100, Jens Axboe wrote:
> > On Thu, Nov 16 2006, Jim Schutt wrote:
> > > Hi,
> > > 
> 
> > > My test program can do one of the following:
> > > 
> > > send data:
> > >  A) read() from file into buffer, write() buffer into socket
> > >  B) mmap() section of file, write() that into socket, munmap()
> > >  C) splice() from file to pipe, splice() from pipe to socket
> > > 
> > > receive data:
> > >  1) read() from socket into buffer, write() buffer into file
> > >  2) ftruncate() to extend file, mmap() new extent, read() 
> > >       from socket into new extent, munmap()
> > >  3) read() from socket into buffer, vmsplice() buffer to 
> > >      pipe, splice() pipe to file (using the double-buffer trick)
> > > 
> > > Here's the results, using:
> > >  - 64 KiB buffer, mmap extent, or splice
> > >  - 1 MiB TCP window
> > >  - 16 GiB data sent across network
> > > 
> > > A) from /dev/zero -> 1) to /dev/null : 857 MB/s (6.86 Gb/s)
> > > 
> > > A) from file      -> 1) to /dev/null : 472 MB/s (3.77 Gb/s)
> > > B) from file      -> 1) to /dev/null : 366 MB/s (2.93 Gb/s)
> > > C) from file      -> 1) to /dev/null : 854 MB/s (6.83 Gb/s)
> > > 
> > > A) from /dev/zero -> 1) to file      : 375 MB/s (3.00 Gb/s)
> > > A) from /dev/zero -> 2) to file      : 150 MB/s (1.20 Gb/s)
> > > A) from /dev/zero -> 3) to file      : 286 MB/s (2.29 Gb/s)
> > > 
> > > I had (naively) hoped the read/vmsplice/splice combination would 
> > > run at the same speed I can write a file, i.e. at about 450 MB/s
> > > on my setup.  Do any of my numbers seem bogus, so I should look 
> > > harder at my test program?
> > 
> > Could be read-ahead playing in here, I'd have to take a closer look at
> > the generated io patterns to say more about that. Any chance you can
> > capture iostat or blktrace info for such a run to compare that goes to
> > the disk?
> 
> I've attached a file with iostat and vmstat results for the case
> where I read from a socket and write a file, vs. the case where I
> read from a socket and use vmsplice/splice to write the file.
> (Sorry it's not inline - my mailer locks up when I try to
> include the file.)
> 
> Would you still like blktrace info for these two cases?

No, I think the iostat data is fine, I don't think the blktrace info
would give me any more insight on this problem. I'll set up a test to
reproduce it here, looks like the write out path could be optimized some
more.

-- 
Jens Axboe

Re: splice/vmsplice performance test results

[Jens Axboe]

On Mon, Nov 20 2006, Jens Axboe wrote:
> On Fri, Nov 17 2006, Jim Schutt wrote:
> > On Thu, 2006-11-16 at 21:25 +0100, Jens Axboe wrote:
> > > On Thu, Nov 16 2006, Jim Schutt wrote:
> > > > Hi,
> > > > 
> > 
> > > > My test program can do one of the following:
> > > > 
> > > > send data:
> > > >  A) read() from file into buffer, write() buffer into socket
> > > >  B) mmap() section of file, write() that into socket, munmap()
> > > >  C) splice() from file to pipe, splice() from pipe to socket
> > > > 
> > > > receive data:
> > > >  1) read() from socket into buffer, write() buffer into file
> > > >  2) ftruncate() to extend file, mmap() new extent, read() 
> > > >       from socket into new extent, munmap()
> > > >  3) read() from socket into buffer, vmsplice() buffer to 
> > > >      pipe, splice() pipe to file (using the double-buffer trick)
> > > > 
> > > > Here's the results, using:
> > > >  - 64 KiB buffer, mmap extent, or splice
> > > >  - 1 MiB TCP window
> > > >  - 16 GiB data sent across network
> > > > 
> > > > A) from /dev/zero -> 1) to /dev/null : 857 MB/s (6.86 Gb/s)
> > > > 
> > > > A) from file      -> 1) to /dev/null : 472 MB/s (3.77 Gb/s)
> > > > B) from file      -> 1) to /dev/null : 366 MB/s (2.93 Gb/s)
> > > > C) from file      -> 1) to /dev/null : 854 MB/s (6.83 Gb/s)
> > > > 
> > > > A) from /dev/zero -> 1) to file      : 375 MB/s (3.00 Gb/s)
> > > > A) from /dev/zero -> 2) to file      : 150 MB/s (1.20 Gb/s)
> > > > A) from /dev/zero -> 3) to file      : 286 MB/s (2.29 Gb/s)
> > > > 
> > > > I had (naively) hoped the read/vmsplice/splice combination would 
> > > > run at the same speed I can write a file, i.e. at about 450 MB/s
> > > > on my setup.  Do any of my numbers seem bogus, so I should look 
> > > > harder at my test program?
> > > 
> > > Could be read-ahead playing in here, I'd have to take a closer look at
> > > the generated io patterns to say more about that. Any chance you can
> > > capture iostat or blktrace info for such a run to compare that goes to
> > > the disk?
> > 
> > I've attached a file with iostat and vmstat results for the case
> > where I read from a socket and write a file, vs. the case where I
> > read from a socket and use vmsplice/splice to write the file.
> > (Sorry it's not inline - my mailer locks up when I try to
> > include the file.)
> > 
> > Would you still like blktrace info for these two cases?
> 
> No, I think the iostat data is fine, I don't think the blktrace info
> would give me any more insight on this problem. I'll set up a test to
> reproduce it here, looks like the write out path could be optimized some
> more.

While I get that setup, can you repeat your testing without using
SPLICE_F_MORE (you don't really use that flag correctly, but it does not
matter for your case afaict) and SPLICE_F_MOVE? The latter will cost
some CPU, but vmsplice/splice for network receive to a file is not
really optimal in the first place. When we get splice() from socket fd
support that'll improve, right now you are doing the best you can.

-- 
Jens Axboe

Re: splice/vmsplice performance test results

[Jim Schutt]

On Mon, 2006-11-20 at 09:24 +0100, Jens Axboe wrote:
> On Mon, Nov 20 2006, Jens Axboe wrote:
> > On Fri, Nov 17 2006, Jim Schutt wrote:
> > > On Thu, 2006-11-16 at 21:25 +0100, Jens Axboe wrote:
> > > > On Thu, Nov 16 2006, Jim Schutt wrote:
> > > > > Hi,
> > > > > 
> > > 
> > > > > My test program can do one of the following:
> > > > > 
> > > > > send data:
> > > > >  A) read() from file into buffer, write() buffer into socket
> > > > >  B) mmap() section of file, write() that into socket, munmap()
> > > > >  C) splice() from file to pipe, splice() from pipe to socket
> > > > > 
> > > > > receive data:
> > > > >  1) read() from socket into buffer, write() buffer into file
> > > > >  2) ftruncate() to extend file, mmap() new extent, read() 
> > > > >       from socket into new extent, munmap()
> > > > >  3) read() from socket into buffer, vmsplice() buffer to 
> > > > >      pipe, splice() pipe to file (using the double-buffer trick)
> > > > > 
> > > > > Here's the results, using:
> > > > >  - 64 KiB buffer, mmap extent, or splice
> > > > >  - 1 MiB TCP window
> > > > >  - 16 GiB data sent across network
> > > > > 
> > > > > A) from /dev/zero -> 1) to /dev/null : 857 MB/s (6.86 Gb/s)
> > > > > 
> > > > > A) from file      -> 1) to /dev/null : 472 MB/s (3.77 Gb/s)
> > > > > B) from file      -> 1) to /dev/null : 366 MB/s (2.93 Gb/s)
> > > > > C) from file      -> 1) to /dev/null : 854 MB/s (6.83 Gb/s)
> > > > > 
> > > > > A) from /dev/zero -> 1) to file      : 375 MB/s (3.00 Gb/s)
> > > > > A) from /dev/zero -> 2) to file      : 150 MB/s (1.20 Gb/s)
> > > > > A) from /dev/zero -> 3) to file      : 286 MB/s (2.29 Gb/s)
> > > > > 
> > > > > I had (naively) hoped the read/vmsplice/splice combination would 
> > > > > run at the same speed I can write a file, i.e. at about 450 MB/s
> > > > > on my setup.  Do any of my numbers seem bogus, so I should look 
> > > > > harder at my test program?
> > > > 
> > > > Could be read-ahead playing in here, I'd have to take a closer look at
> > > > the generated io patterns to say more about that. Any chance you can
> > > > capture iostat or blktrace info for such a run to compare that goes to
> > > > the disk?
> > > 
> > > I've attached a file with iostat and vmstat results for the case
> > > where I read from a socket and write a file, vs. the case where I
> > > read from a socket and use vmsplice/splice to write the file.
> > > (Sorry it's not inline - my mailer locks up when I try to
> > > include the file.)
> > > 
> > > Would you still like blktrace info for these two cases?
> > 
> > No, I think the iostat data is fine, I don't think the blktrace info
> > would give me any more insight on this problem. I'll set up a test to
> > reproduce it here, looks like the write out path could be optimized some
> > more.

Great, let me know if you need testing from me.

> 
> While I get that setup, can you repeat your testing without using
> SPLICE_F_MORE (you don't really use that flag correctly, but it does not
> matter for your case afaict) and SPLICE_F_MOVE?

Done.  Removing these flags from any call to splice or vmsplice made 
no difference to throughput for me, either when sending or receiving 
a file.

>  The latter will cost
> some CPU, but vmsplice/splice for network receive to a file is not
> really optimal in the first place. When we get splice() from socket fd
> support that'll improve, right now you are doing the best you can.
> 
Thanks for reviewing my test program.

Re: splice/vmsplice performance test results

[Jens Axboe]

On Mon, Nov 20 2006, Jim Schutt wrote:
> On Mon, 2006-11-20 at 09:24 +0100, Jens Axboe wrote:
> > On Mon, Nov 20 2006, Jens Axboe wrote:
> > > On Fri, Nov 17 2006, Jim Schutt wrote:
> > > > On Thu, 2006-11-16 at 21:25 +0100, Jens Axboe wrote:
> > > > > On Thu, Nov 16 2006, Jim Schutt wrote:
> > > > > > Hi,
> > > > > > 
> > > > 
> > > > > > My test program can do one of the following:
> > > > > > 
> > > > > > send data:
> > > > > >  A) read() from file into buffer, write() buffer into socket
> > > > > >  B) mmap() section of file, write() that into socket, munmap()
> > > > > >  C) splice() from file to pipe, splice() from pipe to socket
> > > > > > 
> > > > > > receive data:
> > > > > >  1) read() from socket into buffer, write() buffer into file
> > > > > >  2) ftruncate() to extend file, mmap() new extent, read() 
> > > > > >       from socket into new extent, munmap()
> > > > > >  3) read() from socket into buffer, vmsplice() buffer to 
> > > > > >      pipe, splice() pipe to file (using the double-buffer trick)
> > > > > > 
> > > > > > Here's the results, using:
> > > > > >  - 64 KiB buffer, mmap extent, or splice
> > > > > >  - 1 MiB TCP window
> > > > > >  - 16 GiB data sent across network
> > > > > > 
> > > > > > A) from /dev/zero -> 1) to /dev/null : 857 MB/s (6.86 Gb/s)
> > > > > > 
> > > > > > A) from file      -> 1) to /dev/null : 472 MB/s (3.77 Gb/s)
> > > > > > B) from file      -> 1) to /dev/null : 366 MB/s (2.93 Gb/s)
> > > > > > C) from file      -> 1) to /dev/null : 854 MB/s (6.83 Gb/s)
> > > > > > 
> > > > > > A) from /dev/zero -> 1) to file      : 375 MB/s (3.00 Gb/s)
> > > > > > A) from /dev/zero -> 2) to file      : 150 MB/s (1.20 Gb/s)
> > > > > > A) from /dev/zero -> 3) to file      : 286 MB/s (2.29 Gb/s)
> > > > > > 
> > > > > > I had (naively) hoped the read/vmsplice/splice combination would 
> > > > > > run at the same speed I can write a file, i.e. at about 450 MB/s
> > > > > > on my setup.  Do any of my numbers seem bogus, so I should look 
> > > > > > harder at my test program?
> > > > > 
> > > > > Could be read-ahead playing in here, I'd have to take a closer look at
> > > > > the generated io patterns to say more about that. Any chance you can
> > > > > capture iostat or blktrace info for such a run to compare that goes to
> > > > > the disk?
> > > > 
> > > > I've attached a file with iostat and vmstat results for the case
> > > > where I read from a socket and write a file, vs. the case where I
> > > > read from a socket and use vmsplice/splice to write the file.
> > > > (Sorry it's not inline - my mailer locks up when I try to
> > > > include the file.)
> > > > 
> > > > Would you still like blktrace info for these two cases?
> > > 
> > > No, I think the iostat data is fine, I don't think the blktrace info
> > > would give me any more insight on this problem. I'll set up a test to
> > > reproduce it here, looks like the write out path could be optimized some
> > > more.
> 
> Great, let me know if you need testing from me.

I found some suboptimal behaviour in your test app - you don't check for
short reads and splice would really like things to be aligned for the
best performance. I did some testing with the original app here, and I
get 114.769MB/s for read-from-socket -> write-to-file and 109.878MB/s
for read-from-socket -> vmsplice-splice-to-file. If I fix up the read to
always get the full buffer size before doing the vmsplice+splice, the
performance is up to the same as the read/write.

Since it's doing buffered writes, the results do vary a lot though (as
you also indicated). A raw /dev/zero -> /dev/null is 3 times faster with
vmsplice/splice.

-- 
Jens Axboe

Re: splice/vmsplice performance test results

[Jim Schutt]

On Tue, 2006-11-21 at 14:54 +0100, Jens Axboe wrote: 
> On Mon, Nov 20 2006, Jim Schutt wrote:
> > On Mon, 2006-11-20 at 09:24 +0100, Jens Axboe wrote:
> > > On Mon, Nov 20 2006, Jens Axboe wrote:
> > > > On Fri, Nov 17 2006, Jim Schutt wrote:
> > > > > On Thu, 2006-11-16 at 21:25 +0100, Jens Axboe wrote:
> > > > > > On Thu, Nov 16 2006, Jim Schutt wrote:
> > > > > > > Hi,
> > > > > > > 
> > > > > 
> > > > > > > My test program can do one of the following:
> > > > > > > 
> > > > > > > send data:
> > > > > > >  A) read() from file into buffer, write() buffer into socket
> > > > > > >  B) mmap() section of file, write() that into socket, munmap()
> > > > > > >  C) splice() from file to pipe, splice() from pipe to socket
> > > > > > > 
> > > > > > > receive data:
> > > > > > >  1) read() from socket into buffer, write() buffer into file
> > > > > > >  2) ftruncate() to extend file, mmap() new extent, read() 
> > > > > > >       from socket into new extent, munmap()
> > > > > > >  3) read() from socket into buffer, vmsplice() buffer to 
> > > > > > >      pipe, splice() pipe to file (using the double-buffer trick)
> > > > > > > 
> > > > > > > Here's the results, using:
> > > > > > >  - 64 KiB buffer, mmap extent, or splice
> > > > > > >  - 1 MiB TCP window
> > > > > > >  - 16 GiB data sent across network
> > > > > > > 
> > > > > > > A) from /dev/zero -> 1) to /dev/null : 857 MB/s (6.86 Gb/s)
> > > > > > > 
> > > > > > > A) from file      -> 1) to /dev/null : 472 MB/s (3.77 Gb/s)
> > > > > > > B) from file      -> 1) to /dev/null : 366 MB/s (2.93 Gb/s)
> > > > > > > C) from file      -> 1) to /dev/null : 854 MB/s (6.83 Gb/s)
> > > > > > > 
> > > > > > > A) from /dev/zero -> 1) to file      : 375 MB/s (3.00 Gb/s)
> > > > > > > A) from /dev/zero -> 2) to file      : 150 MB/s (1.20 Gb/s)
> > > > > > > A) from /dev/zero -> 3) to file      : 286 MB/s (2.29 Gb/s)
> > > > > > > 
> > > > > > > I had (naively) hoped the read/vmsplice/splice combination would 
> > > > > > > run at the same speed I can write a file, i.e. at about 450 MB/s
> > > > > > > on my setup.  Do any of my numbers seem bogus, so I should look 
> > > > > > > harder at my test program?
> > > > > > 
> > > > > > Could be read-ahead playing in here, I'd have to take a closer look at
> > > > > > the generated io patterns to say more about that. Any chance you can
> > > > > > capture iostat or blktrace info for such a run to compare that goes to
> > > > > > the disk?
> > > > > 
> > > > > I've attached a file with iostat and vmstat results for the case
> > > > > where I read from a socket and write a file, vs. the case where I
> > > > > read from a socket and use vmsplice/splice to write the file.
> > > > > (Sorry it's not inline - my mailer locks up when I try to
> > > > > include the file.)
> > > > > 
> > > > > Would you still like blktrace info for these two cases?
> > > > 
> > > > No, I think the iostat data is fine, I don't think the blktrace info
> > > > would give me any more insight on this problem. I'll set up a test to
> > > > reproduce it here, looks like the write out path could be optimized some
> > > > more.
> > 
> > Great, let me know if you need testing from me.
> 
> I found some suboptimal behaviour in your test app - you don't check for
> short reads and splice would really like things to be aligned for the
> best performance. I did some testing with the original app here, and I
> get 114.769MB/s for read-from-socket -> write-to-file and 109.878MB/s
> for read-from-socket -> vmsplice-splice-to-file. If I fix up the read to
> always get the full buffer size before doing the vmsplice+splice, the
> performance is up to the same as the read/write.

Sorry - I had assumed my network was so much faster than my
disk subsystem I'd never get a short read from a socket except at
the end of the transfer.   Pretty silly of me, in hindsight.

I can see now how even one short read early would screw up 
the alignment for splicing into a file for the rest of the 
transfer, right?

Here's some new results:

Run w/check for short read on socket in vmsplice case:
- /dev/zero -> /dev/null w/ socket read + file write: 1130 MB/s
   (Man, my network is running fast today.  I don't know why.)
- /dev/zero -> /dev/null w/ socket read + vmsplice/splice: 1028 MB/s
- /dev/zero -> file w/ socket read + vmsplice/splice: 336 MB/s

Rerun w/original:
- /dev/zero -> /dev/null w/ socket read + vmsplice/splice: 1026 MB/s
- /dev/zero -> file w/ socket read + vmsplice/splice: 285 MB/s
- /dev/zero -> file w/ socket read + file write: 382 MB/s

So I was losing 50 MB/s due to short reads on the socket 
screwing up the alignment for splice.  Sorry to waste your
time on that.

But, it looks like socket-read + file-write is still ~50 MB/s 
faster than socket-read + vmsplice/splice (assuming I didn't 
screw up my short read fix - see patch below). I assume that's 
still unexpected?

> 
> Since it's doing buffered writes, the results do vary a lot though (as
> you also indicated). A raw /dev/zero -> /dev/null is 3 times faster with
> vmsplice/splice.
> 

Hmmm.  Is it worth me trying to do some sort of kernel 
profiling to see if there is anything unexpected with 
my setup?  If so, do you have a preference as to what 
I would use?  

Here's how I fixed my app to fix up (I think) short reads.
Maybe I missed your point?

diff --git a/src/dnd.c b/src/dnd.c
index 01bd7b8..aa70102 100644
--- a/src/dnd.c
+++ b/src/dnd.c
@@ -773,18 +773,26 @@ uint64_t vmsplice_recv(const struct opti
 again:
 	i = (i + 1) & 1;
 	iov.iov_base = opts->buf + i * opts->buf_size;
+	l = 0;
 
 again2:
-	l = read(sd, iov.iov_base, opts->buf_size);
-	if (l < 0) {
+	m = read(sd, iov.iov_base + l, opts->buf_size - l);
+	if (m < 0) {
 		if (errno == EINTR)
 			goto again2;
 		perror("Read");
 		exit(EXIT_FAILURE);
 	}
-	if (l == 0) {
-		fdatasync(fd);
-		return bytes;
+	if (m == 0) {
+		if (l == 0) {
+			fdatasync(fd);
+			return bytes;
+		}
+	}
+	else {
+		l += m;
+		if (l != opts->buf_size)
+			goto again2;
 	}
 
 	while (l) {

Re: splice/vmsplice performance test results

[Jens Axboe]

On Tue, Nov 21 2006, Jim Schutt wrote:
> On Tue, 2006-11-21 at 14:54 +0100, Jens Axboe wrote: 
> > On Mon, Nov 20 2006, Jim Schutt wrote:
> > > On Mon, 2006-11-20 at 09:24 +0100, Jens Axboe wrote:
> > > > On Mon, Nov 20 2006, Jens Axboe wrote:
> > > > > On Fri, Nov 17 2006, Jim Schutt wrote:
> > > > > > On Thu, 2006-11-16 at 21:25 +0100, Jens Axboe wrote:
> > > > > > > On Thu, Nov 16 2006, Jim Schutt wrote:
> > > > > > > > Hi,
> > > > > > > > 
> > > > > > 
> > > > > > > > My test program can do one of the following:
> > > > > > > > 
> > > > > > > > send data:
> > > > > > > >  A) read() from file into buffer, write() buffer into socket
> > > > > > > >  B) mmap() section of file, write() that into socket, munmap()
> > > > > > > >  C) splice() from file to pipe, splice() from pipe to socket
> > > > > > > > 
> > > > > > > > receive data:
> > > > > > > >  1) read() from socket into buffer, write() buffer into file
> > > > > > > >  2) ftruncate() to extend file, mmap() new extent, read() 
> > > > > > > >       from socket into new extent, munmap()
> > > > > > > >  3) read() from socket into buffer, vmsplice() buffer to 
> > > > > > > >      pipe, splice() pipe to file (using the double-buffer trick)
> > > > > > > > 
> > > > > > > > Here's the results, using:
> > > > > > > >  - 64 KiB buffer, mmap extent, or splice
> > > > > > > >  - 1 MiB TCP window
> > > > > > > >  - 16 GiB data sent across network
> > > > > > > > 
> > > > > > > > A) from /dev/zero -> 1) to /dev/null : 857 MB/s (6.86 Gb/s)
> > > > > > > > 
> > > > > > > > A) from file      -> 1) to /dev/null : 472 MB/s (3.77 Gb/s)
> > > > > > > > B) from file      -> 1) to /dev/null : 366 MB/s (2.93 Gb/s)
> > > > > > > > C) from file      -> 1) to /dev/null : 854 MB/s (6.83 Gb/s)
> > > > > > > > 
> > > > > > > > A) from /dev/zero -> 1) to file      : 375 MB/s (3.00 Gb/s)
> > > > > > > > A) from /dev/zero -> 2) to file      : 150 MB/s (1.20 Gb/s)
> > > > > > > > A) from /dev/zero -> 3) to file      : 286 MB/s (2.29 Gb/s)
> > > > > > > > 
> > > > > > > > I had (naively) hoped the read/vmsplice/splice combination would 
> > > > > > > > run at the same speed I can write a file, i.e. at about 450 MB/s
> > > > > > > > on my setup.  Do any of my numbers seem bogus, so I should look 
> > > > > > > > harder at my test program?
> > > > > > > 
> > > > > > > Could be read-ahead playing in here, I'd have to take a closer look at
> > > > > > > the generated io patterns to say more about that. Any chance you can
> > > > > > > capture iostat or blktrace info for such a run to compare that goes to
> > > > > > > the disk?
> > > > > > 
> > > > > > I've attached a file with iostat and vmstat results for the case
> > > > > > where I read from a socket and write a file, vs. the case where I
> > > > > > read from a socket and use vmsplice/splice to write the file.
> > > > > > (Sorry it's not inline - my mailer locks up when I try to
> > > > > > include the file.)
> > > > > > 
> > > > > > Would you still like blktrace info for these two cases?
> > > > > 
> > > > > No, I think the iostat data is fine, I don't think the blktrace info
> > > > > would give me any more insight on this problem. I'll set up a test to
> > > > > reproduce it here, looks like the write out path could be optimized some
> > > > > more.
> > > 
> > > Great, let me know if you need testing from me.
> > 
> > I found some suboptimal behaviour in your test app - you don't check for
> > short reads and splice would really like things to be aligned for the
> > best performance. I did some testing with the original app here, and I
> > get 114.769MB/s for read-from-socket -> write-to-file and 109.878MB/s
> > for read-from-socket -> vmsplice-splice-to-file. If I fix up the read to
> > always get the full buffer size before doing the vmsplice+splice, the
> > performance is up to the same as the read/write.
> 
> Sorry - I had assumed my network was so much faster than my
> disk subsystem I'd never get a short read from a socket except at
> the end of the transfer.   Pretty silly of me, in hindsight.

That may be true if you are doing sync io, but the tests here do not.
Hence queueing can be much quicker than you can pull the data off the
net.

> I can see now how even one short read early would screw up 
> the alignment for splicing into a file for the rest of the 
> transfer, right?

Yep, it would hurt a lot. As your numbers show :-)

> Here's some new results:
> 
> Run w/check for short read on socket in vmsplice case:
> - /dev/zero -> /dev/null w/ socket read + file write: 1130 MB/s
>    (Man, my network is running fast today.  I don't know why.)
> - /dev/zero -> /dev/null w/ socket read + vmsplice/splice: 1028 MB/s
> - /dev/zero -> file w/ socket read + vmsplice/splice: 336 MB/s
> 
> Rerun w/original:
> - /dev/zero -> /dev/null w/ socket read + vmsplice/splice: 1026 MB/s
> - /dev/zero -> file w/ socket read + vmsplice/splice: 285 MB/s
> - /dev/zero -> file w/ socket read + file write: 382 MB/s
> 
> So I was losing 50 MB/s due to short reads on the socket 
> screwing up the alignment for splice.  Sorry to waste your
> time on that.
> 
> But, it looks like socket-read + file-write is still ~50 MB/s 
> faster than socket-read + vmsplice/splice (assuming I didn't 
> screw up my short read fix - see patch below). I assume that's 
> still unexpected?

Maybe unexpected is not the right word, but it certainly would be nice
if it was faster. As I originally mentioned, your test case of socket
read + vmsplice + splice is not the best way to pull data off the
network, but it's the best until splice-from-socket works. Your zero ->
null transfer doesn't seem to take a large hit becaue of that, so it's
still very surprising that we lose 50MB/s on real io performance.

> > Since it's doing buffered writes, the results do vary a lot though (as
> > you also indicated). A raw /dev/zero -> /dev/null is 3 times faster with
> > vmsplice/splice.
> > 
> 
> Hmmm.  Is it worth me trying to do some sort of kernel 
> profiling to see if there is anything unexpected with 
> my setup?  If so, do you have a preference as to what 
> I would use?  

Not sure that profiling would be that interesting, as the problem
probably lies in where we are _not_ spending the time. But it certainly
can't hurt. Try to oprofile the kernel for a 10-20 sec interval while
the test is running. Do 3 such runs for the two test cases
(write-to-file, vmsplice/splice-to-file).

See the bottom of Documentation/basic_profiling.txt for how to profile
the kernel easily.

> Here's how I fixed my app to fix up (I think) short reads.
> Maybe I missed your point?

That looks about right.

-- 
Jens Axboe

Re: splice/vmsplice performance test results

[Jim Schutt]

[-- Attachment #1: Type: text/plain, Size: 1019 bytes --]


On Wed, 2006-11-22 at 09:57 +0100, Jens Axboe wrote:
> On Tue, Nov 21 2006, Jim Schutt wrote:
[snip]
> > 
> > Hmmm.  Is it worth me trying to do some sort of kernel 
> > profiling to see if there is anything unexpected with 
> > my setup?  If so, do you have a preference as to what 
> > I would use?  
> 
> Not sure that profiling would be that interesting, as the problem
> probably lies in where we are _not_ spending the time. But it certainly
> can't hurt. Try to oprofile the kernel for a 10-20 sec interval while
> the test is running. Do 3 such runs for the two test cases
> (write-to-file, vmsplice/splice-to-file).
> 

OK, I've attached results for 20 second profiles of three
runs of each test: read-from-socket + write-to-file, and
read-from-socket + vmsplice/splice-to-file.

The test case and throughput is in the name: e.g. rvs-1-306MBps
is trial 1 of read/vmsplice/splice case, which ran at 306 MB/s.

Let me know if I can help with more testing, and thanks
again for looking into this.

-- Jim Schutt

[-- Attachment #2: oprofile-rvs-1-306MBps.txt.bz2 --]
[-- Type: application/x-bzip, Size: 4508 bytes --]

[-- Attachment #3: oprofile-rvs-2-324MBps.txt.bz2 --]
[-- Type: application/x-bzip, Size: 4516 bytes --]

[-- Attachment #4: oprofile-rvs-3-314MBps.txt.bz2 --]
[-- Type: application/x-bzip, Size: 4593 bytes --]

[-- Attachment #5: oprofile-rw-1-334MBps.txt.bz2 --]
[-- Type: application/x-bzip, Size: 4544 bytes --]

[-- Attachment #6: oprofile-rw-2-348MBps.txt.bz2 --]
[-- Type: application/x-bzip, Size: 4682 bytes --]

[-- Attachment #7: oprofile-rw-3-361MBps.txt.bz2 --]
[-- Type: application/x-bzip, Size: 4654 bytes --]

Re: splice/vmsplice performance test results

[Jens Axboe]

On Wed, Nov 22 2006, Jim Schutt wrote:
> 
> On Wed, 2006-11-22 at 09:57 +0100, Jens Axboe wrote:
> > On Tue, Nov 21 2006, Jim Schutt wrote:
> [snip]
> > > 
> > > Hmmm.  Is it worth me trying to do some sort of kernel 
> > > profiling to see if there is anything unexpected with 
> > > my setup?  If so, do you have a preference as to what 
> > > I would use?  
> > 
> > Not sure that profiling would be that interesting, as the problem
> > probably lies in where we are _not_ spending the time. But it certainly
> > can't hurt. Try to oprofile the kernel for a 10-20 sec interval while
> > the test is running. Do 3 such runs for the two test cases
> > (write-to-file, vmsplice/splice-to-file).
> > 
> 
> OK, I've attached results for 20 second profiles of three
> runs of each test: read-from-socket + write-to-file, and
> read-from-socket + vmsplice/splice-to-file.
> 
> The test case and throughput is in the name: e.g. rvs-1-306MBps
> is trial 1 of read/vmsplice/splice case, which ran at 306 MB/s.
> 
> Let me know if I can help with more testing, and thanks
> again for looking into this.

As I suspected, nothing sticks out in these logs as the problem here is
not due to a maxed out system. The traces look fairly identical, less
time spent in copy_user with the splice approach.

Comparing the generic_file_buffered_write() and splice-to-file path,
there really isn't a whole lot of difference. It would be interesting to
try and eliminate some of the differences between the two approaches -
could you try and change the vmsplice to a write-to-pipe instead? And
add SPLICE_F_NONBLOCK to the splice-to-file as well. Basically I'm
interested in a something that only really tests splice-to-file vs
write-to-file. Perhaps easier if you can just run fio to test that, I'm
inlining a job file to test that specifically.

; -- start job file

[global]
bs=64k
rw=write
overwrite=0
size=16g
end_fsync=1
direct=0
unlink

[write]
ioengine=sync

[splice]
stonewall
ioengine=splice

; -- end job file

You can grab a fio snapshot here:

http://brick.kernel.dk/snaps/fio-git-20061123122325.tar.gz

You probably want to run that a few times to see how stable the results
are, buffered io is always a little problematic from a consistency point
of view in benchmark results.

-- 
Jens Axboe

Re: splice/vmsplice performance test results

[Jim Schutt]

On Thu, 2006-11-23 at 12:24 +0100, Jens Axboe wrote: 
> On Wed, Nov 22 2006, Jim Schutt wrote:
> > 
> > On Wed, 2006-11-22 at 09:57 +0100, Jens Axboe wrote:
> > > On Tue, Nov 21 2006, Jim Schutt wrote:
> > [snip]
> > > > 
> > > > Hmmm.  Is it worth me trying to do some sort of kernel 
> > > > profiling to see if there is anything unexpected with 
> > > > my setup?  If so, do you have a preference as to what 
> > > > I would use?  
> > > 
> > > Not sure that profiling would be that interesting, as the problem
> > > probably lies in where we are _not_ spending the time. But it certainly
> > > can't hurt. Try to oprofile the kernel for a 10-20 sec interval while
> > > the test is running. Do 3 such runs for the two test cases
> > > (write-to-file, vmsplice/splice-to-file).
> > > 
> > 
> > OK, I've attached results for 20 second profiles of three
> > runs of each test: read-from-socket + write-to-file, and
> > read-from-socket + vmsplice/splice-to-file.
> > 
> > The test case and throughput is in the name: e.g. rvs-1-306MBps
> > is trial 1 of read/vmsplice/splice case, which ran at 306 MB/s.
> > 
> > Let me know if I can help with more testing, and thanks
> > again for looking into this.
> 
> As I suspected, nothing sticks out in these logs as the problem here is
> not due to a maxed out system. The traces look fairly identical, less
> time spent in copy_user with the splice approach.
> 
> Comparing the generic_file_buffered_write() and splice-to-file path,
> there really isn't a whole lot of difference. It would be interesting to
> try and eliminate some of the differences between the two approaches -
> could you try and change the vmsplice to a write-to-pipe instead? And
> add SPLICE_F_NONBLOCK to the splice-to-file as well. Basically I'm
> interested in a something that only really tests splice-to-file vs
> write-to-file. Perhaps easier if you can just run fio to test that, I'm
> inlining a job file to test that specifically.
> 

Sorry for the delayed reply.

Here's results from three fio runs, using the job file
you gave me, and fio-git-20061124142507.tar.gz:

---
write: (g=0): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=sync, iodepth=1
splice: (g=1): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=splice, iodepth=1
write: (g=0): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=sync, iodepth=1
splice: (g=1): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=splice, iodepth=1
write: (g=0): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=sync, iodepth=1
splice: (g=1): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=splice, iodepth=1
write: (groupid=0): err= 0:
  write: io= 16384MiB, bw=558404KiB/s, runt= 30766msec
    slat (msec): min=    0, max=    0, avg= 0.00, dev= 0.00
    clat (msec): min=    0, max=   35, avg= 0.00, dev= 0.08
    bw (KiB/s) : min=    0, max=644481, per=98.90%, avg=552243.95,
dev=561417.03
  cpu          : usr=0.91%, sys=85.03%, ctx=14121
splice: (groupid=1): err= 0:
  write: io= 16384MiB, bw=486144KiB/s, runt= 35339msec
    slat (msec): min=    0, max=    0, avg= 0.00, dev= 0.00
    clat (msec): min=    0, max=   30, avg= 0.00, dev= 0.10
    bw (KiB/s) : min=    0, max=555745, per=99.06%, avg=481567.28,
dev=488565.60
  cpu          : usr=0.85%, sys=88.87%, ctx=12956

Run status group 0 (all jobs):
  WRITE: io=16384MiB, aggrb=558404, minb=558404, maxb=558404,
mint=30766msec, maxt=30766msec

Run status group 1 (all jobs):
  WRITE: io=16384MiB, aggrb=486144, minb=486144, maxb=486144,
mint=35339msec, maxt=35339msec

Disk stats (read/write):
  md0: ios=20/270938, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
---
write: (g=0): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=sync, iodepth=1
splice: (g=1): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=splice, iodepth=1
write: (g=0): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=sync, iodepth=1
splice: (g=1): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=splice, iodepth=1
write: (g=0): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=sync, iodepth=1
splice: (g=1): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=splice, iodepth=1
write: (groupid=0): err= 0:
  write: io= 16384MiB, bw=547234KiB/s, runt= 31394msec
    slat (msec): min=    0, max=    1, avg= 0.00, dev= 0.00
    clat (msec): min=    0, max=   57, avg= 0.00, dev= 0.14
    bw (KiB/s) : min=    0, max=662568, per=98.94%, avg=541406.71,
dev=550958.67
  cpu          : usr=0.79%, sys=82.80%, ctx=16560
splice: (groupid=1): err= 0:
  write: io= 16384MiB, bw=473313KiB/s, runt= 36297msec
    slat (msec): min=    0, max=    1, avg= 0.00, dev= 0.00
    clat (msec): min=    0, max=   27, avg= 0.00, dev= 0.11
    bw (KiB/s) : min=    0, max=562298, per=99.12%, avg=469142.21,
dev=476426.36
  cpu          : usr=1.05%, sys=84.78%, ctx=16043

Run status group 0 (all jobs):
  WRITE: io=16384MiB, aggrb=547234, minb=547234, maxb=547234,
mint=31394msec, maxt=31394msec

Run status group 1 (all jobs):
  WRITE: io=16384MiB, aggrb=473313, minb=473313, maxb=473313,
mint=36297msec, maxt=36297msec

Disk stats (read/write):
  md0: ios=17/270784, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
---
write: (g=0): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=sync, iodepth=1
splice: (g=1): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=splice, iodepth=1
write: (g=0): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=sync, iodepth=1
splice: (g=1): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=splice, iodepth=1
write: (g=0): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=sync, iodepth=1
splice: (g=1): rw=write, odir=0, bs=64K-64K/64K-64K, rate=0,
ioengine=splice, iodepth=1
write: (groupid=0): err= 0:
  write: io= 16384MiB, bw=561140KiB/s, runt= 30616msec
    slat (msec): min=    0, max=    1, avg= 0.00, dev= 0.00
    clat (msec): min=    0, max=   26, avg= 0.00, dev= 0.08
    bw (KiB/s) : min=    0, max=665976, per=98.94%, avg=555198.98,
dev=564632.52
  cpu          : usr=0.82%, sys=85.12%, ctx=14287
splice: (groupid=1): err= 0:
  write: io= 16384MiB, bw=487192KiB/s, runt= 35263msec
    slat (msec): min=    0, max=    0, avg= 0.00, dev= 0.00
    clat (msec): min=    0, max=   29, avg= 0.00, dev= 0.09
    bw (KiB/s) : min=    0, max=566099, per=99.08%, avg=482706.32,
dev=489726.36
  cpu          : usr=0.85%, sys=86.96%, ctx=13072

Run status group 0 (all jobs):
  WRITE: io=16384MiB, aggrb=561140, minb=561140, maxb=561140,
mint=30616msec, maxt=30616msec

Run status group 1 (all jobs):
  WRITE: io=16384MiB, aggrb=487192, minb=487192, maxb=487192,
mint=35263msec, maxt=35263msec

Disk stats (read/write):
  md0: ios=18/270851, merge=0/0, ticks=0/0, in_queue=0, util=0.00%
---

Do these results give you something to work with, or would 
you like me to run some other tests?

Let me know.

Thanks -- Jim Schutt

Re: splice/vmsplice performance test results

[David Miller]

From: "Jim Schutt" <jaschut@sandia.gov>
Date: Thu, 16 Nov 2006 11:08:59 -0700

> Or is read+write really the fastest way to get data off a
> socket and into a file?

There is still no explicit TCP support for splice/vmsplice so things
get copied around and most of the other advantaves of splice/vmplice
aren't obtained either.  So perhaps that explains your numbers.

Jens Axboe tries to get things working, and others have looked into it
too, but adding TCP support is hard and for several reasons folks like
Alexey Kuznetsov and Evgeniy Polyakov believe that sys_receivefile()
is an interface much better suited for TCP receive.

splice/vmsplice has a lot of state connected to a transaction, and
perhaps that is part of why Evgeniy and Alexey have trouble wrapping
their brains around an efficient implementation.

Re: splice/vmsplice performance test results

[Jens Axboe]

On Thu, Nov 16 2006, David Miller wrote:
> From: "Jim Schutt" <jaschut@sandia.gov>
> Date: Thu, 16 Nov 2006 11:08:59 -0700
> 
> > Or is read+write really the fastest way to get data off a
> > socket and into a file?
> 
> There is still no explicit TCP support for splice/vmsplice so things
> get copied around and most of the other advantaves of splice/vmplice
> aren't obtained either.  So perhaps that explains your numbers.

There should not be any copying for tcp send, at least no more than what
sendfile() did/does. Hmm?

> Jens Axboe tries to get things working, and others have looked into it
> too, but adding TCP support is hard and for several reasons folks like
> Alexey Kuznetsov and Evgeniy Polyakov believe that sys_receivefile()
> is an interface much better suited for TCP receive.
> 
> splice/vmsplice has a lot of state connected to a transaction, and
> perhaps that is part of why Evgeniy and Alexey have trouble wrapping
> their brains around an efficient implementation.

I hope to try and see if I can help get some of that done, however I
need all the help I can get on the networking side. Not sure I
understand why it has to be so difficult, if we need to define a wrapper
container instead of passing down a pipe that is completely fine with
me. The networking code basically just needs to hang on to the
pipe_buffer and release it on ack for send, receive is somewhat more
involved (and I don't know enough about networking to voice any
half-intelligent opinion on that!).

I would just consider it a damn shame if we cannot complete the splice
family and need to punt to something else for net receive.

-- 
Jens Axboe

Re: splice/vmsplice performance test results

[David Miller]

From: Jens Axboe <jens.axboe@oracle.com>
Date: Thu, 16 Nov 2006 22:21:07 +0100

> On Thu, Nov 16 2006, David Miller wrote:
> > From: "Jim Schutt" <jaschut@sandia.gov>
> > Date: Thu, 16 Nov 2006 11:08:59 -0700
> > 
> > > Or is read+write really the fastest way to get data off a
> > > socket and into a file?
> > 
> > There is still no explicit TCP support for splice/vmsplice so things
> > get copied around and most of the other advantaves of splice/vmplice
> > aren't obtained either.  So perhaps that explains your numbers.
> 
> There should not be any copying for tcp send, at least no more than what
> sendfile() did/does. Hmm?

That's true on send, correct.

> > Jens Axboe tries to get things working, and others have looked into it
> > too, but adding TCP support is hard and for several reasons folks like
> > Alexey Kuznetsov and Evgeniy Polyakov believe that sys_receivefile()
> > is an interface much better suited for TCP receive.
> > 
> > splice/vmsplice has a lot of state connected to a transaction, and
> > perhaps that is part of why Evgeniy and Alexey have trouble wrapping
> > their brains around an efficient implementation.
> 
> I hope to try and see if I can help get some of that done, however I
> need all the help I can get on the networking side. Not sure I
> understand why it has to be so difficult, if we need to define a wrapper
> container instead of passing down a pipe that is completely fine with
> me. The networking code basically just needs to hang on to the
> pipe_buffer and release it on ack for send, receive is somewhat more
> involved (and I don't know enough about networking to voice any
> half-intelligent opinion on that!).
> 
> I would just consider it a damn shame if we cannot complete the splice
> family and need to punt to something else for net receive.

I'm sure that the folks on netdev@vger.kernel.org are more than
willing to give you a hand in this area :-)