Re: [PATCH] libibverbs: Add huge page support to ibv_madvise_range()

[Alex Vainman <alexonlists-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org>]

Roland Dreier Wrote:
>  > ibv_reg_mr() fails to register a memory region allocated on huge page and not
>  > the default page size. This happens because ibv_madvise_range() aligns memory
>  > region to the default system page size before calling to madvise() which fails
>  > with EINVAL error. madvise() fails because it expects that the start and end
>  > pointer of the memory range be huge page aligned.
> 
> Seems unfortunate.  I wonder if there's a way the kernel madvise could
> help us here?
> 
>  > +/*
>  > + * Get the kernel default huge page size.
>  > + */
>  > +static int get_huge_page_size()
>  > +{
>  > +	int fd;
>  > +	char buf[MEMINFO_SIZE];
>  > +	int mem_file_len;
>  > +	char *p_hpage_val = NULL;
>  > +	char *end_pointer = NULL;
>  > +	char file_name[] = "/proc/meminfo";
>  > +	const char label[] = "Hugepagesize:";
>  > +	int ret_val = 0;
>  > +
>  > +	fd = open(file_name, O_RDONLY);
>  > +	if (fd < 0)
>  > +		return fd;
>  > +
>  > +	mem_file_len = read(fd, buf, sizeof(buf) - 1);
>  > +
>  > +	close(fd);
>  > +	if (mem_file_len < 0)
>  > +		return mem_file_len;
>  > +
>  > +	buf[mem_file_len] = '\0';
>  > +
>  > +	p_hpage_val = strstr(buf, label);
>  > +	if (!p_hpage_val) {
>  > +		errno = EINVAL;
>  > +		return -1;
>  > +	}
>  > +	p_hpage_val += strlen(label);
>  > +
>  > +	errno = 0;
>  > +	ret_val = strtol(p_hpage_val, &end_pointer, 0);
>  > +
>  > +	if (errno != 0)
>  > +		return -1;
>  > +
>  > +	return ret_val * 1024;
>  > +}
> 
> This seems to duplicate but only partially a similar function from
> libhugetlbfs.  Is there any way we can just use that directly?  eg
> libhugetlbfs handles the case where there are multiple huge page sizes
> (and that exists even on mainstream x86 with 2MB and 1GB pages possible
> on the same system).
> 
>  - R.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 

Hi Roland,

After the patches, which handle madvise failure, are applied(these pathces were submited under the topic:"libibverbs: Undo changes in memory range tree when madvise() fails"), I would like to renew the discussion about this patch, which actually depends on the above patches, since it may cause madvise failure.

>This seems to duplicate but only partially a similar function from
>libhugetlbfs.  Is there any way we can just use that directly?  eg
>libhugetlbfs handles the case where there are multiple huge page sizes
>(and that exists even on mainstream x86 with 2MB and 1GB pages possible
>on the same system).

In order to avoid adding additional dependency to libibverbs, maybe we should just to enhance the get_huge_page_size() so it will support multiple huge page sizes?

-Alex


--
To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
the body of a message to majordomo-u79uwXL29TY76Z2rM5mHXA@public.gmane.org
More majordomo info at  http://vger.kernel.org/majordomo-info.html