Testing NVME Split BIO

Testing NVME Split BIO

[Jeffrey Lien]

We are trying to create a testcase to verify the nvme split bio function.   Does anyone have any ideas or suggestions for a simple way to force a split bio request?   

Thanks

Jeff Lien

Jeff.Lien at hgst.com
507 322-2416/23-2416
3605 Hwy 52 N.  
Rochester, MN 55901

Testing NVME Split BIO

[Sam Bradshaw (sbradshaw)]

> -----Original Message-----
> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On
> Behalf Of Jeffrey Lien
> Sent: Wednesday, December 03, 2014 10:19 AM
> To: linux-nvme at lists.infradead.org
> Subject: Testing NVME Split BIO
> 
> We are trying to create a testcase to verify the nvme split bio
> function.   Does anyone have any ideas or suggestions for a simple way
> to force a split bio request?

We built a device mapper shim layer that does this.  Using a 4k transfer, single bvec, the mapper would: 
1) bio_alloc a bio with 2 bvecs
2) point the first bvec at the original bvec but truncate the size to 2k
3) alloc a 4k page, kmap it, memcpy the upper 2k from the original bvec page into a non-zero offset into the allocated page
4) point the second bvec in the allocated bio at the bvec with non-zero offset
5) hand it off to the driver

Make sure your allocated bio callback does any cleanup then calls the original callback with a pointer to the original bio.

A device mapper shim approach may not be the "easiest" but has other benefits for traffic shaping beyond this use case.

-Sam

Testing NVME Split BIO

[Andrey Kuzmin]

Assuming an NVME device/namespace formatted with 4K block size, I'd
give a shot to allocating, say, 8K in user-space with memalign (so
that my virtual 8K occupy exactly 2 physical pages), and then issuing
a 4K write to the respective nvme block device using any offset into
the allocated memory you'd like, such as 2K suggested by Sam in that
other email.
Regards,
Andrey


On Wed, Dec 3, 2014 at 9:42 PM, Sam Bradshaw (sbradshaw)
<sbradshaw@micron.com> wrote:
>
>
>> -----Original Message-----
>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On
>> Behalf Of Jeffrey Lien
>> Sent: Wednesday, December 03, 2014 10:19 AM
>> To: linux-nvme at lists.infradead.org
>> Subject: Testing NVME Split BIO
>>
>> We are trying to create a testcase to verify the nvme split bio
>> function.   Does anyone have any ideas or suggestions for a simple way
>> to force a split bio request?
>
> We built a device mapper shim layer that does this.  Using a 4k transfer, single bvec, the mapper would:
> 1) bio_alloc a bio with 2 bvecs
> 2) point the first bvec at the original bvec but truncate the size to 2k
> 3) alloc a 4k page, kmap it, memcpy the upper 2k from the original bvec page into a non-zero offset into the allocated page
> 4) point the second bvec in the allocated bio at the bvec with non-zero offset
> 5) hand it off to the driver
>
> Make sure your allocated bio callback does any cleanup then calls the original callback with a pointer to the original bio.
>
> A device mapper shim approach may not be the "easiest" but has other benefits for traffic shaping beyond this use case.
>
> -Sam
>
> _______________________________________________
> Linux-nvme mailing list
> Linux-nvme at lists.infradead.org
> http://lists.infradead.org/mailman/listinfo/linux-nvme

Testing NVME Split BIO

[Sam Bradshaw (sbradshaw)]

> -----Original Message-----
> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On
> Behalf Of Andrey Kuzmin
> Sent: Wednesday, December 03, 2014 11:27 AM
> To: Jeffrey Lien
> Cc: Sam Bradshaw (sbradshaw); linux-nvme at lists.infradead.org
> Subject: Re: Testing NVME Split BIO
> 
> Assuming an NVME device/namespace formatted with 4K block size, I'd
> give a shot to allocating, say, 8K in user-space with memalign (so that
> my virtual 8K occupy exactly 2 physical pages), and then issuing a 4K
> write to the respective nvme block device using any offset into the
> allocated memory you'd like, such as 2K suggested by Sam in that other
> email.
> Regards,
> Andrey

Using this approach, you'll still end up with a virtually contiguous block, which won't be split.

Testing NVME Split BIO

[Andrey Kuzmin]

On Wed, Dec 3, 2014 at 11:02 PM, Sam Bradshaw (sbradshaw)
<sbradshaw@micron.com> wrote:
>
>
>> -----Original Message-----
>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On
>> Behalf Of Andrey Kuzmin
>> Sent: Wednesday, December 03, 2014 11:27 AM
>> To: Jeffrey Lien
>> Cc: Sam Bradshaw (sbradshaw); linux-nvme at lists.infradead.org
>> Subject: Re: Testing NVME Split BIO
>>
>> Assuming an NVME device/namespace formatted with 4K block size, I'd
>> give a shot to allocating, say, 8K in user-space with memalign (so that
>> my virtual 8K occupy exactly 2 physical pages), and then issuing a 4K
>> write to the respective nvme block device using any offset into the
>> allocated memory you'd like, such as 2K suggested by Sam in that other
>> email.
>> Regards,
>> Andrey
>
> Using this approach, you'll still end up with a virtually contiguous block, which won't be split.

You're right, it will be split at the nvme (PRP construction) level,
and it's not the same as split bio.

Regards,
Andrey

Testing NVME Split BIO

[Keith Busch]

I don't think it's possible to synthesize split conditions from user
space if you're formatted 4k physical block size.

This will force splits for any <4k block format.
---
#define _GNU_SOURCE

#include <fcntl.h>
#include <string.h>
#include <stdio.h>

int main(int argc, char **argv)
{
 	struct iovec iov[2];
 	unsigned char *buffer;
 	int fd, err;

 	if (posix_memalign((void **)&buffer, 0x4000, 0x1000))
 		return 1;
 	if (argc < 2) {
 		fprintf(stderr, "usage: %s <device>\n", argv[0]);
 		return 1;
 	}

 	fd = open(argv[1], O_RDONLY | O_DIRECT);
 	if (fd <= 0) {
 		perror(argv[1]);
 		return 1;
 	}

 	memset(buffer, 0, 0x4000);

 	iov[0].iov_base = &buffer[0x800];
 	iov[1].iov_base = &buffer[0x2800];
 	iov[0].iov_len = 0x1000;
 	iov[1].iov_len = 0x1000;

 	err = readv(fd, iov, 2);
 	if (err)
 		perror("readv");
 	return err;
}
--

On Wed, 3 Dec 2014, Andrey Kuzmin wrote:
> Assuming an NVME device/namespace formatted with 4K block size, I'd
> give a shot to allocating, say, 8K in user-space with memalign (so
> that my virtual 8K occupy exactly 2 physical pages), and then issuing
> a 4K write to the respective nvme block device using any offset into
> the allocated memory you'd like, such as 2K suggested by Sam in that
> other email.
> Regards,
> Andrey
>
>
> On Wed, Dec 3, 2014 at 9:42 PM, Sam Bradshaw (sbradshaw)
> <sbradshaw@micron.com> wrote:
>>
>>
>>> -----Original Message-----
>>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On
>>> Behalf Of Jeffrey Lien
>>> Sent: Wednesday, December 03, 2014 10:19 AM
>>> To: linux-nvme at lists.infradead.org
>>> Subject: Testing NVME Split BIO
>>>
>>> We are trying to create a testcase to verify the nvme split bio
>>> function.   Does anyone have any ideas or suggestions for a simple way
>>> to force a split bio request?
>>
>> We built a device mapper shim layer that does this.  Using a 4k transfer, single bvec, the mapper would:
>> 1) bio_alloc a bio with 2 bvecs
>> 2) point the first bvec at the original bvec but truncate the size to 2k
>> 3) alloc a 4k page, kmap it, memcpy the upper 2k from the original bvec page into a non-zero offset into the allocated page
>> 4) point the second bvec in the allocated bio at the bvec with non-zero offset
>> 5) hand it off to the driver
>>
>> Make sure your allocated bio callback does any cleanup then calls the original callback with a pointer to the original bio.
>>
>> A device mapper shim approach may not be the "easiest" but has other benefits for traffic shaping beyond this use case.
>>
>> -Sam

Testing NVME Split BIO

[Jeffrey Lien]

Thanks for the suggestions and ideas.   

-----Original Message-----
From: Keith Busch [mailto:keith.busch@intel.com] 
Sent: Wednesday, December 3, 2014 2:57 PM
To: Andrey Kuzmin
Cc: Jeffrey Lien; Sam Bradshaw (sbradshaw); linux-nvme at lists.infradead.org
Subject: Re: Testing NVME Split BIO

I don't think it's possible to synthesize split conditions from user space if you're formatted 4k physical block size.

This will force splits for any <4k block format.
---
#define _GNU_SOURCE

#include <fcntl.h>
#include <string.h>
#include <stdio.h>

int main(int argc, char **argv)
{
 	struct iovec iov[2];
 	unsigned char *buffer;
 	int fd, err;

 	if (posix_memalign((void **)&buffer, 0x4000, 0x1000))
 		return 1;
 	if (argc < 2) {
 		fprintf(stderr, "usage: %s <device>\n", argv[0]);
 		return 1;
 	}

 	fd = open(argv[1], O_RDONLY | O_DIRECT);
 	if (fd <= 0) {
 		perror(argv[1]);
 		return 1;
 	}

 	memset(buffer, 0, 0x4000);

 	iov[0].iov_base = &buffer[0x800];
 	iov[1].iov_base = &buffer[0x2800];
 	iov[0].iov_len = 0x1000;
 	iov[1].iov_len = 0x1000;

 	err = readv(fd, iov, 2);
 	if (err)
 		perror("readv");
 	return err;
}
--

On Wed, 3 Dec 2014, Andrey Kuzmin wrote:
> Assuming an NVME device/namespace formatted with 4K block size, I'd 
> give a shot to allocating, say, 8K in user-space with memalign (so 
> that my virtual 8K occupy exactly 2 physical pages), and then issuing 
> a 4K write to the respective nvme block device using any offset into 
> the allocated memory you'd like, such as 2K suggested by Sam in that 
> other email.
> Regards,
> Andrey
>
>
> On Wed, Dec 3, 2014 at 9:42 PM, Sam Bradshaw (sbradshaw) 
> <sbradshaw@micron.com> wrote:
>>
>>
>>> -----Original Message-----
>>> From: Linux-nvme [mailto:linux-nvme-bounces at lists.infradead.org] On 
>>> Behalf Of Jeffrey Lien
>>> Sent: Wednesday, December 03, 2014 10:19 AM
>>> To: linux-nvme at lists.infradead.org
>>> Subject: Testing NVME Split BIO
>>>
>>> We are trying to create a testcase to verify the nvme split bio
>>> function.   Does anyone have any ideas or suggestions for a simple way
>>> to force a split bio request?
>>
>> We built a device mapper shim layer that does this.  Using a 4k transfer, single bvec, the mapper would:
>> 1) bio_alloc a bio with 2 bvecs
>> 2) point the first bvec at the original bvec but truncate the size to 
>> 2k
>> 3) alloc a 4k page, kmap it, memcpy the upper 2k from the original 
>> bvec page into a non-zero offset into the allocated page
>> 4) point the second bvec in the allocated bio at the bvec with 
>> non-zero offset
>> 5) hand it off to the driver
>>
>> Make sure your allocated bio callback does any cleanup then calls the original callback with a pointer to the original bio.
>>
>> A device mapper shim approach may not be the "easiest" but has other benefits for traffic shaping beyond this use case.
>>
>> -Sam