vmsplice bad address

vmsplice bad address

[Guillaume Dargaud]

Hello all,
I'm trying to use the vmsplice/splice combination to so socket writes in zero-copy mode since it seems to be the big 
bottle neck in my application.

Here's a simplified code:

// This buffer is actually outside kernel space,
// in reserved contiguous physical memory (custom kernel compilation)
#define BUFFER_PHY_BASE 0x7000000
#define BUFFER_SIZE 0x1000000
memfd=open("/dev/mem", O_RDWR | O_SYNC)
Base=(unsigned char*)mmap(0, BUFFER_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, memfd, BUFFER_PHY_BASE);

sockfd = socket(AF_INET, SOCK_STREAM, 0)

int pfd[2];	pipe(pfd);

struct iovec iov;
iov.iov_base=Base;
iov.iov_len=BUFFER_SIZE;
printf("Pipe %d/%d, base:0x%x, size:0x%x\n", pfd[0], pfd[1], iov.iov_base, iov.iov_len);
vmsplice(pfd[1], &iov, 1, SPLICE_F_GIFT)
splice(pfd[0], NULL, sockfd, NULL, BUFFER_SIZE, SPLICE_F_MOVE)

I get the following:
Pipe 6/7, base:0x480cf000, size:0x1000000
vmsplice: Bad address


Any idea what is wrong in my use of vmsplice ?
Thanks
-- 
Guillaume Dargaud
http://www.gdargaud.net/

Re: vmsplice bad address

[Benjamin Herrenschmidt]

On Thu, 2011-03-03 at 14:01 +0100, Guillaume Dargaud wrote:
> Hello all,
> I'm trying to use the vmsplice/splice combination to so socket writes in zero-copy mode since it seems to be the big 
> bottle neck in my application.
> 
> Here's a simplified code:
> 
> // This buffer is actually outside kernel space,
> // in reserved contiguous physical memory (custom kernel compilation)
> #define BUFFER_PHY_BASE 0x7000000
> #define BUFFER_SIZE 0x1000000
> memfd=open("/dev/mem", O_RDWR | O_SYNC)
> Base=(unsigned char*)mmap(0, BUFFER_SIZE, PROT_READ | PROT_WRITE, MAP_SHARED, memfd, BUFFER_PHY_BASE);
> 
> sockfd = socket(AF_INET, SOCK_STREAM, 0)
> 
> int pfd[2];	pipe(pfd);
> 
> struct iovec iov;
> iov.iov_base=Base;
> iov.iov_len=BUFFER_SIZE;
> printf("Pipe %d/%d, base:0x%x, size:0x%x\n", pfd[0], pfd[1], iov.iov_base, iov.iov_len);
> vmsplice(pfd[1], &iov, 1, SPLICE_F_GIFT)
> splice(pfd[0], NULL, sockfd, NULL, BUFFER_SIZE, SPLICE_F_MOVE)
> 
> I get the following:
> Pipe 6/7, base:0x480cf000, size:0x1000000
> vmsplice: Bad address
> 
> 
> Any idea what is wrong in my use of vmsplice ?

I'm not completely surprised ... I wouldn't expect vmsplice to work
on /dev/mem ... No struct page backing the memory for example.

Ben.

Re: vmsplice bad address

[Guillaume Dargaud]

On Monday 07 March 2011 10:35:44 Benjamin Herrenschmidt wrote:

> I'm not completely surprised ... I wouldn't expect vmsplice to work
> on /dev/mem ... No struct page backing the memory for example.

So how would that work to do zero-copy ? 
Somebody gave me a hint of marking the memory regions as iomem, do a ioremap then mmap and vmsplice but that's a 
mouthful...
-- 
Guillaume Dargaud
http://www.gdargaud.net/