hugetlbfs not working

hugetlbfs not working

[Alexander Graf]

Hi,

I'm currently trying to get hugetlbfs working on the current git  
version and am quite puzzled to see it not working. It appears as if  
the ftruncate call fails:

open("/dev/hugetlbfs//kvm.vI3G8z", O_RDWR|O_CREAT|O_EXCL, 0600) = 7
unlink("/dev/hugetlbfs//kvm.vI3G8z")    = 0
ftruncate(7, 157286400)                 = -1 EINVAL (Invalid argument)
dup(2)                                  = 8
fcntl(8, F_GETFL)                       = 0x8001 (flags O_WRONLY| 
O_LARGEFILE)
close(8)                                = 0
write(2, "ftruncate: Invalid argument\n", 28ftruncate: Invalid argument
) = 28
close(7)                                = 0

My host kernel is a 2.6.22.

Is this supposed to work? The first version did not have the ftruncate  
call, so maybe it doesn't work at all with hugetlbfs?

Alex

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: hugetlbfs not working

[Avi Kivity]

Alexander Graf wrote:
> Hi,
>
> I'm currently trying to get hugetlbfs working on the current git  
> version and am quite puzzled to see it not working. It appears as if  
> the ftruncate call fails:
>
> open("/dev/hugetlbfs//kvm.vI3G8z", O_RDWR|O_CREAT|O_EXCL, 0600) = 7
> unlink("/dev/hugetlbfs//kvm.vI3G8z")    = 0
> ftruncate(7, 157286400)                 = -1 EINVAL (Invalid argument)
> dup(2)                                  = 8
> fcntl(8, F_GETFL)                       = 0x8001 (flags O_WRONLY| 
> O_LARGEFILE)
> close(8)                                = 0
> write(2, "ftruncate: Invalid argument\n", 28ftruncate: Invalid argument
> ) = 28
> close(7)                                = 0
>
> My host kernel is a 2.6.22.
>
> Is this supposed to work? The first version did not have the ftruncate  
> call, so maybe it doesn't work at all with hugetlbfs?
>   

Are you on i386 non-pae?  that has 4MB pages, while you're asking for 
for 37.5 4MB pages.

Try adding 2MB to the requested memory size.


-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: hugetlbfs not working

[Alexander Graf]

On Mar 27, 2008, at 4:19 PM, Avi Kivity wrote:

> Alexander Graf wrote:
>> Hi,
>>
>> I'm currently trying to get hugetlbfs working on the current git   
>> version and am quite puzzled to see it not working. It appears as  
>> if  the ftruncate call fails:
>>
>> open("/dev/hugetlbfs//kvm.vI3G8z", O_RDWR|O_CREAT|O_EXCL, 0600) = 7
>> unlink("/dev/hugetlbfs//kvm.vI3G8z")    = 0
>> ftruncate(7, 157286400)                 = -1 EINVAL (Invalid  
>> argument)
>> dup(2)                                  = 8
>> fcntl(8, F_GETFL)                       = 0x8001 (flags O_WRONLY|  
>> O_LARGEFILE)
>> close(8)                                = 0
>> write(2, "ftruncate: Invalid argument\n", 28ftruncate: Invalid  
>> argument
>> ) = 28
>> close(7)                                = 0
>>
>> My host kernel is a 2.6.22.
>>
>> Is this supposed to work? The first version did not have the  
>> ftruncate  call, so maybe it doesn't work at all with hugetlbfs?
>>
>
> Are you on i386 non-pae?  that has 4MB pages, while you're asking  
> for for 37.5 4MB pages.
>
> Try adding 2MB to the requested memory size.

I'm on x86_64 and /proc/meminfo tells me:

HugePages_Total:  1435
HugePages_Free:   1435
HugePages_Rsvd:      0
Hugepagesize:     2048 kB

I also reserved all available huge pages:

% cat /proc/sys/vm/nr_hugepages
1435

Changing the guest memory size using -m does not help.

Alex

-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: hugetlbfs not working

[Anthony Liguori]

Alexander Graf wrote:
> On Mar 27, 2008, at 4:19 PM, Avi Kivity wrote:
>
>   
>> Alexander Graf wrote:
>>     
>>> Hi,
>>>
>>> I'm currently trying to get hugetlbfs working on the current git   
>>> version and am quite puzzled to see it not working. It appears as  
>>> if  the ftruncate call fails:
>>>
>>> open("/dev/hugetlbfs//kvm.vI3G8z", O_RDWR|O_CREAT|O_EXCL, 0600) = 7
>>> unlink("/dev/hugetlbfs//kvm.vI3G8z")    = 0
>>> ftruncate(7, 157286400)                 = -1 EINVAL (Invalid  
>>> argument)
>>> dup(2)                                  = 8
>>> fcntl(8, F_GETFL)                       = 0x8001 (flags O_WRONLY|  
>>> O_LARGEFILE)
>>> close(8)                                = 0
>>> write(2, "ftruncate: Invalid argument\n", 28ftruncate: Invalid  
>>> argument
>>> ) = 28
>>> close(7)                                = 0
>>>
>>> My host kernel is a 2.6.22.
>>>
>>> Is this supposed to work? The first version did not have the  
>>> ftruncate  call, so maybe it doesn't work at all with hugetlbfs?
>>>
>>>       
>> Are you on i386 non-pae?  that has 4MB pages, while you're asking  
>> for for 37.5 4MB pages.
>>
>> Try adding 2MB to the requested memory size.
>>     
>
> I'm on x86_64 and /proc/meminfo tells me:
>
> HugePages_Total:  1435
> HugePages_Free:   1435
> HugePages_Rsvd:      0
> Hugepagesize:     2048 kB
>
> I also reserved all available huge pages:
>
> % cat /proc/sys/vm/nr_hugepages
> 1435
>
> Changing the guest memory size using -m does not help.
>   

If you don't bail when ftruncate fails, does it work as expected for 
you?  Perhaps older versions of hugetlbfs didn't support truncate.

Regards,

Anthony Liguori

> Alex
>
> -------------------------------------------------------------------------
> Check out the new SourceForge.net Marketplace.
> It's the best place to buy or sell services for
> just about anything Open Source.
> http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
> _______________________________________________
> kvm-devel mailing list
> kvm-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/kvm-devel
>   


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: hugetlbfs not working

[Avi Kivity]

Anthony Liguori wrote: 
>
> If you don't bail when ftruncate fails, does it work as expected for 
> you?  Perhaps older versions of hugetlbfs didn't support truncate.

If you don't truncate, how can you change the file size?  hugetlbfs 
doesn't support write().

I vaugely recall using ftruncate() on 2.4.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: hugetlbfs not working

[Marcelo Tosatti]

On Thu, Mar 27, 2008 at 06:09:56PM +0200, Avi Kivity wrote:
> Anthony Liguori wrote: 
> >
> > If you don't bail when ftruncate fails, does it work as expected for 
> > you?  Perhaps older versions of hugetlbfs didn't support truncate.
> 
> If you don't truncate, how can you change the file size?  hugetlbfs 
> doesn't support write().
> 
> I vaugely recall using ftruncate() on 2.4.

hugetlbfs will automatically adjust the file size on mmap().


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: hugetlbfs not working

[Marcelo Tosatti]

On Thu, Mar 27, 2008 at 06:09:56PM +0200, Avi Kivity wrote:
> Anthony Liguori wrote: 
> >
> > If you don't bail when ftruncate fails, does it work as expected for 
> > you?  Perhaps older versions of hugetlbfs didn't support truncate.
> 
> If you don't truncate, how can you change the file size?  hugetlbfs 
> doesn't support write().
> 
> I vaugely recall using ftruncate() on 2.4.

commit 7aa91e104028b87ff13f5eeb7a0d7ffe7b5a2348
Author: Ken Chen <kenchen@google.com>
Date:   Tue Oct 16 01:26:21 2007 -0700

    hugetlb: allow extending ftruncate on hugetlbfs

    For historical reason, expanding ftruncate that increases file size on
    hugetlbfs is not allowed due to pages were pre-faulted and lack of fault
    handler.  Now that we have demand faulting on hugetlb since 2.6.15, there
    is no reason to hold back that limitation.

    This will make hugetlbfs behave more like a normal fs.  I'm writing a user
    level code that uses hugetlbfs but will fall back to tmpfs if there are no
    hugetlb page available in the system.  Having hugetlbfs specific ftruncate
    behavior is a bit quirky and I would like to remove that artificial
    limitation.

    Signed-off-by: <kenchen@google.com>
    Acked-by: Wiliam Irwin <wli@holomorphy.com>
    Cc: Adam Litke <agl@us.ibm.com>
    Cc: David Gibson <david@gibson.dropbear.id.au>
    Cc: Nishanth Aravamudan <nacc@us.ibm.com>
    Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
    Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

mmap() should fail if anything goes wrong with ftruncate and the file
length is not extented on tmpfs.

--- vl.c.orig   2008-03-27 18:51:31.000000000 -0300
+++ vl.c        2008-03-27 18:52:40.000000000 -0300
@@ -8749,11 +8749,7 @@

     memory = (memory+hpagesize-1) & ~(hpagesize-1);

-    if (ftruncate(fd, memory) == -1) {
-       perror("ftruncate");
-       close(fd);
-       return NULL;
-    }
+    ftruncate(fd, memory);

     area = mmap(0, memory, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
     if (area == MAP_FAILED) {




-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: hugetlbfs not working

[Marcelo Tosatti]

On Fri, Mar 28, 2008 at 09:41:06AM +0300, Avi Kivity wrote:
> >mmap() should fail if anything goes wrong with ftruncate and the file
> >length is not extented on tmpfs.
> >
> >--- vl.c.orig   2008-03-27 18:51:31.000000000 -0300
> >+++ vl.c        2008-03-27 18:52:40.000000000 -0300
> >@@ -8749,11 +8749,7 @@
> >
> >     memory = (memory+hpagesize-1) & ~(hpagesize-1);
> >
> >-    if (ftruncate(fd, memory) == -1) {
> >-       perror("ftruncate");
> >-       close(fd);
> >-       return NULL;
> >-    }
> >+    ftruncate(fd, memory);
> >  
> 
> I'm sure a patch will follow to add a missing error check, so how about 
> a comment here?
> 
> btw, we use signoffs on kvm-userspace, too.

QEMU/KVM: ftruncate() is not supported by hugetlbfs on older hosts

Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>

diff --git a/qemu/vl.c b/qemu/vl.c
index 5627862..6a240bf 100644
--- a/qemu/vl.c
+++ b/qemu/vl.c
@@ -8739,11 +8739,13 @@ void *alloc_mem_area(unsigned long memory, const char *path)
 
     memory = (memory+hpagesize-1) & ~(hpagesize-1);
 
-    if (ftruncate(fd, memory) == -1) {
-	perror("ftruncate");
-	close(fd);
-	return NULL;
-    }
+    /*
+     * ftruncate is not supported by hugetlbfs in older 
+     * hosts, so don't bother checking for errors.
+     * If anything goes wrong with it under other filesystems,
+     * mmap will fail.
+     */
+    ftruncate(fd, memory);
 
     area = mmap(0, memory, PROT_READ|PROT_WRITE, MAP_PRIVATE, fd, 0);
     if (area == MAP_FAILED) {


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace

Re: hugetlbfs not working

[Avi Kivity]

Marcelo Tosatti wrote:
> QEMU/KVM: ftruncate() is not supported by hugetlbfs on older hosts
>
> Signed-off-by: Marcelo Tosatti <mtosatti@redhat.com>
>   

Applied, thanks.

-- 
error compiling committee.c: too many arguments to function


-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It's the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace