HugeTLB: Driver example

HugeTLB: Driver example

[Alexey Korolev]

There is an example of simple driver which provides huge pages mapping
for user level applications. The  procedure for mapping of huge pages
to userspace by the driver is:

1. Create a hugetlb file on vfs mount of hugetlbfs (h_file)

2. File operations of /dev/hpage_map do the following:
In file open we  associate mappings of /dev/xxx with the file on
hugetlbfs (like it is done in ipc/shm.c)
       file->f_mapping = h_file->f_mapping;
In get_unmapped_area we should tell about addressing constraints in
case of huge pages by calling hugetlbfs procedures. (as in ipc/shm.c)
       return get_unmapped_area(h_file, addr, len, pgoff, flags);

3 In mmap get huge page in order to DMA or for something else
(hugetlb_get_user_page call).
..................
4 Remove hugetlbfs file

---
#include <linux/module.h>
#include <linux/mm.h>
#include <linux/file.h>
#include <linux/pagemap.h>
#include <linux/hugetlb.h>
#include <linux/pagevec.h>
#include <linux/miscdevice.h>
#include <asm/io.h>
#include <asm/ioctl.h>

#define HFILE_SIZE 16UL*1024*1024
static struct file	*h_file;

static int hpage_map_mmap(struct file *file, struct vm_area_struct *vma)
{
	int ret;
	struct page *page;
	struct hstate *h;
	unsigned long addr = vma->vm_start;

	if ((ret = h_file->f_op->mmap(h_file, vma)))
	    return ret;

	h = hstate_file(h_file);
	
	while (addr < vma->vm_end) {
		page = hugetlb_get_user_page(vma, addr);
		if (IS_ERR(page))
		        return -EFAULT;
		addr += huge_page_size(h);
		/* Add code to configure DMA here */
	}
	return 0;
}

static unsigned long hpage_map_get_unmapped_area(struct file *file,
	unsigned long addr, unsigned long len, unsigned long pgoff,
	unsigned long flags)
{
	/* Tell about addressing constrains in case of huge pages,
	 * hugetlbfs knows how to do this */
	return get_unmapped_area(h_file, addr, len, pgoff, flags);
}
static int hpage_map_open(struct inode * inode, struct file * file)
{
	/* Associate mappings of /dev/xxx with the file on hugetlbfs
	 * like it is done in ipc/shm.c */
	file->f_mapping = h_file->f_mapping;
	return 0;
}

/*
 * The file operations for /dev/hpage_map
 */
static const struct file_operations hpage_map_fops = {
	.owner		= THIS_MODULE,
	.mmap		= hpage_map_mmap,
	.open 		= hpage_map_open,
	.get_unmapped_area	= hpage_map_get_unmapped_area,
};

static struct miscdevice hpage_map_dev = {
	MISC_DYNAMIC_MINOR,
	"hpage_map",
	&hpage_map_fops
};

static int __init
hpage_map_init(void)
{
	int ret;
	struct user_struct *user = NULL;

	/* Create the device in the /sys/class/misc directory. */
	if ((ret = misc_register(&hpage_map_dev)))
		return ret;

	/* Create file on hugetlbfs */
	h_file = hugetlb_file_setup("hpage_map_dev", HFILE_SIZE, 0, &user,
				HUGETLB_DEVBACK_INODE);
	if (IS_ERR(h_file)) {
		misc_deregister(&hpage_map_dev);
		ret = -ENOENT;
	}
	return ret;
}

module_init(hpage_map_init);

static void __exit
hpage_map_exit(void)
{
	fput(h_file);
	misc_deregister(&hpage_map_dev);
}

module_exit(hpage_map_exit);

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Alexey Korolev");
MODULE_DESCRIPTION("Example of driver with hugetlb mapping");
MODULE_VERSION("1.0");

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: HugeTLB: Driver example

[Christoph Hellwig]

On Mon, Sep 14, 2009 at 05:30:12PM +1200, Alexey Korolev wrote:
> There is an example of simple driver which provides huge pages mapping
> for user level applications. The  procedure for mapping of huge pages
> to userspace by the driver is:
> 
> 1. Create a hugetlb file on vfs mount of hugetlbfs (h_file)

Note that to get your support code included at all you'll need a real
intree driver, not just an example.  That is if VM people are happy with
the general concept.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: HugeTLB: Driver example

[Alexey Korolev]

>> There is an example of simple driver which provides huge pages mapping
>> for user level applications. The  procedure for mapping of huge pages
>> to userspace by the driver is:
>>
>> 1. Create a hugetlb file on vfs mount of hugetlbfs (h_file)
>
> Note that to get your support code included at all you'll need a real
> intree driver, not just an example.  That is if VM people are happy with
> the general concept.
Hi,
The driver example listed here takes the same approach as already done
inside ipc/shm.c. So people can refer this file for development. The
patches just make existing functions more usable by drivers and this
example is an extract of ipc/shm.c in order to give pretty simple
how-to.
Seems I gave a not so good description for this patch set so it caused
a lot of misunderstanding, sorry about that.

Thanks,
Alexey

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: HugeTLB: Driver example

[Mel Gorman]

On Thu, Sep 17, 2009 at 06:42:20PM +1200, Alexey Korolev wrote:
> >> There is an example of simple driver which provides huge pages mapping
> >> for user level applications. The  procedure for mapping of huge pages
> >> to userspace by the driver is:
> >>
> >> 1. Create a hugetlb file on vfs mount of hugetlbfs (h_file)
> >
> > Note that to get your support code included at all you'll need a real
> > intree driver, not just an example.  That is if VM people are happy with
> > the general concept.
>
> Hi,
> The driver example listed here takes the same approach as already done
> inside ipc/shm.c. So people can refer this file for development. The
> patches just make existing functions more usable by drivers and this
> example is an extract of ipc/shm.c in order to give pretty simple
> how-to.

I think Christoph's point is that there should be an in-kernel user of the
altered interface to hugetlbfs before the patches are merged. This example
driver could move to samples/ and then add another patch converting some
existing driver to use the new interface. Looking at the example driver,
I'm hoping that converting an existing driver of interest would not be a
massive undertaking.

I tend to agree with him.

As I'm having trouble envisioning what a real driver would look like,
converting an in-kernel driver ensures that the interface was sane instead
of exporting symbols that turn out to be unusable later. It'll also force
any objectors out of the closet sooner rather than later.

As it stands, I have no problems with the patches as such other than they
need a bit more spit and polish. The basic principal seems ok.

> Seems I gave a not so good description for this patch set so it caused
> a lot of misunderstanding, sorry about that.
> 

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: HugeTLB: Driver example

[Alexey Korolev]

Mel,

> I think Christoph's point is that there should be an in-kernel user of the
> altered interface to hugetlbfs before the patches are merged. This example
> driver could move to samples/ and then add another patch converting some
> existing driver to use the new interface. Looking at the example driver,
> I'm hoping that converting an existing driver of interest would not be a
> massive undertaking.

Converting an existing driver may be a very difficult task as this
assumes involving in development process of the particular driver i.e.
having enough details about h/w and drivers and  having ability to
test the results. Also it is necessary to motivate maintainers to
accept this conversion. So I likely would not to be able change the
third party drivers for these reasons, but I'm open to help other
people to migrate if they want.
I heard that other people were asking you about driver interfaces for
huge pages, if this was about in-tree drivers then we could help each
other. Could you put me in touch with other developers you know of who
are interested in using htlb in drivers? It makes sense to get this
feature merged as it provides a quite efficient way for performance
increase. According to our test data, applying these little changes
gives about 7-10% gain.


> I tend to agree with him.
>
> As I'm having trouble envisioning what a real driver would look like,
> converting an in-kernel driver ensures that the interface was sane instead
> of exporting symbols that turn out to be unusable later. It'll also force
> any objectors out of the closet sooner rather than later.
>
> As it stands, I have no problems with the patches as such other than they
> need a bit more spit and polish. The basic principal seems ok.

Thanks,
Alexey

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: HugeTLB: Driver example

[Mel Gorman]

On Mon, Sep 21, 2009 at 05:16:26PM +1200, Alexey Korolev wrote:
> Mel,
> 
> > I think Christoph's point is that there should be an in-kernel user of the
> > altered interface to hugetlbfs before the patches are merged. This example
> > driver could move to samples/ and then add another patch converting some
> > existing driver to use the new interface. Looking at the example driver,
> > I'm hoping that converting an existing driver of interest would not be a
> > massive undertaking.
> 
> Converting an existing driver may be a very difficult task as this
> assumes involving in development process of the particular driver i.e.
> having enough details about h/w and drivers and  having ability to
> test the results.

I had assumed that you had a driver in mind as you discussed test data.
I assumed you had a prototype conversion of some driver and with
performance gains like that, they would be willing to get it in a
mergeable state.

> Also it is necessary to motivate maintainers to
> accept this conversion. So I likely would not to be able change the
> third party drivers for these reasons, but I'm open to help other
> people to migrate if they want.
> I heard that other people were asking you about driver interfaces for
> huge pages, if this was about in-tree drivers then we could help each
> other. Could you put me in touch with other developers you know of who
> are interested in using htlb in drivers?

I can't. The request was from 9-10 months ago for drivers about about 18
months ago for Xen when they were looking at this area. The authors are no
longer working in that area AFAIK as once it was discussed what would need
to be done to use huge pages, they balked. It's far easier now as most of
the difficulties have been addressed but the interested parties are not
likely to be looking at this area for some time.

The most recent expression of interest was from KVM developers within the
company. On discussion, using huge pages was something they were going to
push out as there are other concerns that should be addressed first. I'd
say it'd be at least a year before they looked at huge pages for KVM.

> It makes sense to get this
> feature merged as it provides a quite efficient way for performance
> increase. According to our test data, applying these little changes
> gives about 7-10% gain.
> 

What is this test data based on?

> > I tend to agree with him.
> >
> > As I'm having trouble envisioning what a real driver would look like,
> > converting an in-kernel driver ensures that the interface was sane instead
> > of exporting symbols that turn out to be unusable later. It'll also force
> > any objectors out of the closet sooner rather than later.
> >
> > As it stands, I have no problems with the patches as such other than they
> > need a bit more spit and polish. The basic principal seems ok.
> 

-- 
Mel Gorman
Part-time Phd Student                          Linux Technology Center
University of Limerick                         IBM Dublin Software Lab

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: HugeTLB: Driver example

[Alexey Korolev]

>
> I can't. The request was from 9-10 months ago for drivers about about 18
> months ago for Xen when they were looking at this area. The authors are no
> longer working in that area AFAIK as once it was discussed what would need
> to be done to use huge pages, they balked. It's far easier now as most of
> the difficulties have been addressed but the interested parties are not
> likely to be looking at this area for some time.
>
> The most recent expression of interest was from KVM developers within the
> company. On discussion, using huge pages was something they were going to
> push out as there are other concerns that should be addressed first. I'd
> say it'd be at least a year before they looked at huge pages for KVM.
>
Ok. I see. Thanks.

>> It makes sense to get this
>> feature merged as it provides a quite efficient way for performance
>> increase. According to our test data, applying these little changes
>> gives about 7-10% gain.
>>
>
> What is this test data based on?

The test is based on the throughput of network packets processing. We
read the data from DMA buffers whose are mmaped to user space and then
parse packets by applications. If mapping is based on huge pages we
have gain ~7-10% (more mbps).  Actually I was surprised a bit when
find out that there is no possibility to have huge page mappings for
device drivers. Probably people just don't know about this.

Thanks,
Alexey

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>