Hugepage Support

Hugepage Support

[keshav darak]

[-- Attachment #1.1: Type: text/plain, Size: 4077 bytes --]

hi,

I tried to make huge page request in Fedora x86_64 PV guest using xen 4.1
unstable and it crashed(crash info given below)
I had enabled superpages in config file
I had also set hugepages parameter at boot time for the PV Dom U
By excuting
# cat /proc/mem_info | grep Huge
gave me that there are 10 free huge pages available , still the domain
crashed.


[   86.403654] BUG: unable to handle kernel paging request at
ffff880002b35cb0
[   86.403695] IP: [<ffffffff8100505a>] xen_set_pte+0x47/0x52
[   86.403725] PGD 1a43067 PUD 1a47067 PMD 3e58067 PTE 8010000002b35065
[   86.403755] Oops: 0003 [#1] SMP
[   86.403773] last sysfs file: /sys/devices/virtual/vc/vcsa6/
uevent
[   86.403791] CPU 0
[   86.403800] Modules linked in: ip6t_REJECT nf_conntrack_ipv6
ip6table_filter ip6_tables ipv6 xen_netfront xen_blkfront [last unloaded:
scsi_wait_scan]
[   86.403859]
[   86.403872] Pid: 759, comm: sohuge Not tainted 2.6.35.6-45.fc14.x86_64 #1
/
[   86.403888] RIP: e030:[<ffffffff8100505a>]  [<ffffffff8100505a>]
xen_set_pte+0x47/0x52
[   86.403913] RSP: e02b:ffff88001cf43d38  EFLAGS: 00010296
[   86.403928] RAX: 0000000000000000 RBX: ffff880002b35cb0 RCX:
8000000140f750e7
[   86.403945] RDX: 0000000000000000 RSI: 8000000140f750e7 RDI:
ffff880002b35cb0
[   86.403964] RBP: ffff88001cf43d58 R08: 0000000000000200 R09:
ffff880002642128
[   86.403982] R10: 0000000000000022 R11: 0000000000000246 R12:
8000000140f750e7
[   86.404000] R13: 000000000001e74d R14: ffff88001ec8e680 R15:
ffff88001e50dd00
[   86.404026] FS:  00007f0672be1720(0000) GS:ffff880003e99000(0000)
knlGS:0000000000000000
[   86.404048] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[   86.404068] CR2: ffff880002b35cb0 CR3: 00000000025cd000 CR4:
0000000000002660
[   86.404071] DR0: 0000000000000000 DR1: 0000000000000000 DR2:
0000000000000000
[   86.404071] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7:
0000000000000400
[   86.404071] Process sohuge (pid: 759, threadinfo ffff88001cf42000, task
ffff88001e50dd00)
[   86.404071] Stack:
[   86.404071]  ffff8800025c9a10 ffff880002b35cb0 8000000140f750e7
00007f0672c00000
[   86.404071] <0> ffff88001cf43db8 ffffffff8100514a 0000000000000022
ffff880002642128
[   86.404071] <0> 0000000000000200 8000000002800027 8000000000000027
ffff8800025c9a10
[   86.404071] Call Trace:
[   86.404071]  [<ffffffff8100514a>] xen_set_pte_at+0xe5/0xf4
[   86.404071]  [<ffffffff810fc62e>] set_huge_pte_at+0xe/0x12
[   86.404071]  [<ffffffff810fec42>] hugetlb_fault+0x380/0x510
[   86.404071]  [<ffffffff810eba12>] handle_mm_fault+0xbd/0x84d
[   86.404071]  [<ffffffff8100670f>] ? xen_restore_fl_direct_end+0x0/0x1
[   86.404071]  [<ffffffff812aad54>] ? raw_local_irq_restore+0x10/0x12
[   86.404071]  [<ffffffff8146c468>] do_page_fault+0x250/0x265
[   86.404071]  [<ffffffff814697f5>] page_fault+0x25/0x30
[   86.404071] Code: ff ff 05 be 00 ca 00 e8 aa ff ff ff 44 8b 2d ba 00 ca
00 e8 48 6f 02 00 ff c8 0f 94 c0 0f b6 c0 46 8d 2c 28 44 89 2d a2 00 ca 00
<4c> 89 23 5a 5b 41 5c 41 5d c9 c3 55 48 89 e5 41 57 41 56 41 55
[   86.404071] RIP  [<ffffffff8100505a>] xen_set_pte+0x47/0x52
[   86.404071]  RSP <ffff88001cf43d38>
[   86.404071] CR2: ffff880002b35cb0
[   86.404071] ---[ end trace b6fd430c22cc2dfc ]---

Message from syslogd@localhost at Jan  8 23:05:21 ...
 kernel:[   86.403755] Oops: 0003 [#1] SMP

Message from syslogd@localhost at Jan  8 23:05:21 ...
 kernel:[   86.403773] last sysfs file: /sys/devices/virtual/vc/vcsa6/uevent

Message from syslogd@localhost at Jan  8 23:05:21 ...
 kernel:[   86.404071] Stack:

Message from syslogd@localhost at Jan  8 23:05:21 ...
 kernel:[   86.404071] Call Trace:

Message from syslogd@localhost at Jan  8 23:05:21 ...
 kernel:[   86.404071] Code: ff ff 05 be 00 ca 00 e8 aa ff ff ff 44 8b 2d ba
00 ca 00 e8 48 6f 02 00 ff c8 0f 94 c0 0f b6 c0 46 8d 2c 28 44 89 2d a2 00
ca 00 <4c> 89 23 5a 5b 41 5c 41 5d c9 c3 55 48 89 e5 41 57 41 56 41 55

Message from syslogd@localhost at Jan  8 23:05:21 ...
 kernel:[   86.404071] CR2: ffff880002b35cb0




can anyone tell the way problem can be solved ?



-
Keshav Darak.

[-- Attachment #1.2: Type: text/html, Size: 4703 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Hugepage support

[Keshav Darak]

[-- Attachment #1.1: Type: text/plain, Size: 4280 bytes --]

hi,

I tried to make huge page request in Fedora x86_64 PV guest using xen 4.1 unstable and it crashed(crash info given below)
I had enabled superpages in config file
I had also set hugepages parameter at boot time for the PV Dom U



By excuting
# cat /proc/mem_info | grep Huge
gave me that there are 10 free huge pages available , still the domain crashed.


[   86.403654] BUG: unable to handle kernel paging request at ffff880002b35cb0


[   86.403695] IP: [<ffffffff8100505a>] xen_set_pte+0x47/0x52
[   86.403725] PGD 1a43067 PUD 1a47067 PMD 3e58067 PTE 8010000002b35065

[   86.403755] Oops: 0003 [#1] SMP 
[   86.403773] last sysfs file: /sys/devices/virtual/vc/vcsa6/uevent
[   86.403791] CPU 0 
[  
 86.403800] Modules linked in: ip6t_REJECT nf_conntrack_ipv6 
ip6table_filter ip6_tables ipv6 xen_netfront xen_blkfront [last 
unloaded: scsi_wait_scan]

[   86.403859] 
[   86.403872] Pid: 759, comm: sohuge Not tainted 2.6.35.6-45.fc14.x86_64 #1 /
[   86.403888] RIP: e030:[<ffffffff8100505a>]  [<ffffffff8100505a>] xen_set_pte+0x47/0x52
[   86.403913] RSP: e02b:ffff88001cf43d38  EFLAGS: 00010296



[   86.403928] RAX: 0000000000000000 RBX: ffff880002b35cb0 RCX: 8000000140f750e7
[   86.403945] RDX: 0000000000000000 RSI: 8000000140f750e7 RDI: ffff880002b35cb0
[   86.403964] RBP: ffff88001cf43d58 R08: 0000000000000200 R09: ffff880002642128



[   86.403982] R10: 0000000000000022 R11: 0000000000000246 R12: 8000000140f750e7
[   86.404000] R13: 000000000001e74d R14: ffff88001ec8e680 R15: ffff88001e50dd00
[   86.404026] FS:  00007f0672be1720(0000) GS:ffff880003e99000(0000) knlGS:0000000000000000



[   86.404048] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[   86.404068] CR2: ffff880002b35cb0 CR3: 00000000025cd000 CR4: 0000000000002660
[   86.404071] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000



[   86.404071] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[   86.404071] Process sohuge (pid: 759, threadinfo ffff88001cf42000, task ffff88001e50dd00)
[   86.404071] Stack:
[   86.404071]  ffff8800025c9a10 ffff880002b35cb0 8000000140f750e7 00007f0672c00000



[   86.404071] <0> ffff88001cf43db8 ffffffff8100514a 0000000000000022 ffff880002642128
[   86.404071] <0> 0000000000000200 8000000002800027 8000000000000027 ffff8800025c9a10
[   86.404071] Call Trace:



[   86.404071]  [<ffffffff8100514a>] xen_set_pte_at+0xe5/0xf4
[   86.404071]  [<ffffffff810fc62e>] set_huge_pte_at+0xe/0x12
[   86.404071]  [<ffffffff810fec42>] hugetlb_fault+0x380/0x510
[   86.404071]  [<ffffffff810eba12>] handle_mm_fault+0xbd/0x84d



[   86.404071]  [<ffffffff8100670f>] ? xen_restore_fl_direct_end+0x0/0x1
[   86.404071]  [<ffffffff812aad54>] ? raw_local_irq_restore+0x10/0x12
[   86.404071]  [<ffffffff8146c468>] do_page_fault+0x250/0x265



[   86.404071]  [<ffffffff814697f5>] page_fault+0x25/0x30
[   
86.404071] Code: ff ff 05 be 00 ca 00 e8 aa ff ff ff 44 8b 2d ba 00 ca 
00 e8 48 6f 02 00 ff c8 0f 94 c0 0f b6 c0 46 8d 2c 28 44 89 2d a2 00 ca 
00 <4c> 89 23 5a 5b 41 5c 41 5d c9 c3 55 48 89 e5 41 57 41 56 41 
55 

[   86.404071] RIP  [<ffffffff8100505a>] xen_set_pte+0x47/0x52
[   86.404071]  RSP <ffff88001cf43d38>
[   86.404071] CR2: ffff880002b35cb0
[   86.404071] ---[ end trace b6fd430c22cc2dfc ]---

Message from syslogd@localhost at Jan  8 23:05:21 ...



 kernel:[   86.403755] Oops: 0003 [#1] SMP 

Message from syslogd@localhost at Jan  8 23:05:21 ...
 kernel:[   86.403773] last sysfs file: /sys/devices/virtual/vc/vcsa6/uevent

Message from syslogd@localhost at Jan  8 23:05:21 ...



 kernel:[   86.404071] Stack:

Message from syslogd@localhost at Jan  8 23:05:21 ...
 kernel:[   86.404071] Call Trace:

Message from syslogd@localhost at Jan  8 23:05:21 ...
 kernel:[  
 86.404071] Code: ff ff 05 be 00 ca 00 e8 aa ff ff ff 44 8b 2d ba 00 ca 
00 e8 48 6f 02 00 ff c8 0f 94 c0 0f b6 c0 46 8d 2c 28 44 89 2d a2 00 ca 
00 <4c> 89 23 5a 5b 41 5c 41 5d c9 c3 55 48 89 e5 41 57 41 56 41 
55 


Message from syslogd@localhost at Jan  8 23:05:21 ...
 kernel:[   86.404071] CR2: ffff880002b35cb0




can anyone tell the way problem can be solved ?








-
Keshav Darak.


      

[-- Attachment #1.2: Type: text/html, Size: 5342 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Hugepage support

[Konrad Rzeszutek Wilk]

On Mon, Jan 10, 2011 at 08:33:41AM -0800, Keshav Darak wrote:
> hi,
> 
> I tried to make huge page request in Fedora x86_64 PV guest using xen 4.1 unstable and it crashed(crash info given below)

Had you tried this with the supported kernels (which actually is just
one right now). That is the 2.6.32 kernels in Jeremy's git tree?

Re: Hugepage support

[Keshav Darak]

[-- Attachment #1.1: Type: text/plain, Size: 2915 bytes --]

I am using DOM0 as 2.6.32.27 jermey kernel. Fedora x86_64 (2.6.33.3-85.fc13.x86_64) as PV guest.

In DOM0 kernel  when I requested hugepage, kernel didnt crashed but terminal got hanged.
Initially this was the status for hugepage
#cat /proc/meminfo | grep Huge
HugePages_Total:         10
HugePages_Free:         10
HugePages_Rsvd:         0
HugePages_Surp:         2048

After running C-Code (given at end of this mail),I got following result but unfortunately terminal got hanged.
#cat /proc/meminfo | grep Huge

HugePages_Total:         10

HugePages_Free:         9

HugePages_Rsvd:         3

HugePages_Surp:         2048


I even checked dmesg but there was no error related to page fault for DOM0.

Me and my team would like to enable support of hugepage in PV-DOMU if 
its not present. Please let me know the status or any development 
related to this project. 

C-Code :
#include <stdio.h>
#include <sys/ipc.h>
#include <sys/shm.h>
#include <stdlib.h>

#define MB_1 (1024*1024)
#define MB_8 (8*MB_1)

char  *a;
int shmid1;

void init_hugetlb_seg()

{
  shmid1 = shmget(2, MB_8, SHM_HUGETLB
         | IPC_CREAT | SHM_R
         | SHM_W);
  if ( shmid1 < 0 ) {
    perror("shmget");
    exit(1);
  }
  printf("HugeTLB shmid: 0x%x\n", shmid1);

  a = shmat(shmid1, 0, 0);
  if (a == (char *)-1) {
    perror("Shared memory attach failure");
    shmctl(shmid1, IPC_RMID, NULL);
    exit(2);
  }
}

void wr_to_array()
{
  int i;

  for( i=0 ; i<MB_8 ; i++) {
    a[i] = 'A';
  }
}

void rd_from_array()
{
  int i, count = 0;
  for( i=0 ; i<MB_8 ; i++)
    if (a[i] == 'A') count++;
  if (count==i)

    printf("HugeTLB read success :-)\n");
  else
    printf("HugeTLB read failed :-(\n");
}

int main(int argc, char *argv[])
{
  init_hugetlb_seg();
  printf("HugeTLB memory segment initialized !\n");

  printf("Press any key to write to memory area\n");
  getchar();
  wr_to_array();
  printf("Press any key to rd from memory area\n");
  getchar();
  rd_from_array();
  shmctl(shmid1, IPC_RMID, NULL);

  return 0;
}


--- On Mon, 1/10/11, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Subject: Re: [Xen-devel] Hugepage support
To: "Keshav Darak" <keshav_darak@yahoo.com>
Cc: xen-devel@lists.xensource.com
Date: Monday, January 10, 2011, 4:46 PM

On Mon, Jan 10, 2011 at 08:33:41AM -0800, Keshav Darak wrote:
> hi,
> 
> I tried to make huge page request in Fedora x86_64 PV guest using xen 4.1 unstable and it crashed(crash info given below)

Had you tried this with the supported kernels (which actually is just
one right now). That is the 2.6.32 kernels in Jeremy's git tree?

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 4274 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Hugepage support

[Konrad Rzeszutek Wilk]

On Mon, Jan 10, 2011 at 10:12:01AM -0800, Keshav Darak wrote:
> I am using DOM0 as 2.6.32.27 jermey kernel. Fedora x86_64 (2.6.33.3-85.fc13.x86_64) as PV guest.
> 
> In DOM0 kernel  when I requested hugepage, kernel didnt crashed but terminal got hanged.

Was there any output in the serial log? Is the Xen hypervisor still running? Were
you able to use Alt-SysRq to see what hte Linux kernel was doing? Did you test this
on baremetal and did it work there?
> Initially this was the status for hugepage
> #cat /proc/meminfo | grep Huge
> HugePages_Total:         10
> HugePages_Free:         10
> HugePages_Rsvd:         0
> HugePages_Surp:         2048

Re: Hugepage support

[Keshav Darak]

[-- Attachment #1.1: Type: text/plain, Size: 1264 bytes --]



--- On Mon, 1/10/11, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Subject: Re: [Xen-devel] Hugepage support
To: "Keshav Darak" <keshav_darak@yahoo.com>
Cc: xen-devel@lists.xensource.com, "Ashwin Vasani" <vasani.ashwin@gmail.com>
Date: Monday, January 10, 2011, 6:36 PM

On Mon, Jan 10, 2011 at 10:12:01AM -0800, Keshav Darak wrote:
> I am using DOM0 as 2.6.32.27 jermey kernel. Fedora x86_64 (2.6.33.3-85.fc13.x86_64) as PV guest.
> 
> In DOM0 kernel  when I requested hugepage, kernel didnt crashed but terminal got hanged.

Was there any output in the serial log? Is the Xen hypervisor still running? Were
you able to use Alt-SysRq to see what hte Linux kernel was doing? Did you test this
on baremetal and did it work there?

I just copied the output from the DOMU terminal and yes hypervisor was still running. DOMU crashed when written something on hugepages from C-Code (attached with the previous mail) but for the same code in DOM0, terminal hangs up. 


> Initially this was the status for hugepage
> #cat /proc/meminfo | grep Huge
> HugePages_Total:         10
> HugePages_Free:         10
> HugePages_Rsvd:         0
> HugePages_Surp:         2048



      

[-- Attachment #1.2: Type: text/html, Size: 1797 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Hugepage support

[Konrad Rzeszutek Wilk]

> Was there any output in the serial log? Is the Xen hypervisor still running? Were
> you able to use Alt-SysRq to see what hte Linux kernel was doing? Did you test this
> on baremetal and did it work there?
> 
> I just copied the output from the DOMU terminal and yes hypervisor was still running. DOMU crashed when written something on hugepages from C-Code (attached with the previous mail) but for the same code in DOM0, terminal hangs up. 

So Alt-SysRQ did not work then (you did test Alt-SysRq before trying to play with
hugepages  just to see if you could pass it in, right?)

And can you invoke the Xen hypervisor help by hitting three times Ctrl-A on the serial console?
Can you the hit '*' and send the output?

And lastly, did this work on baremetal?

Re: Hugepage support

[Keshav Darak]

[-- Attachment #1.1: Type: text/plain, Size: 552 bytes --]


nothing happened when i tried Ctrl+a 3 times

--- On Mon, 1/10/11, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

>And can you invoke the Xen hypervisor help by hitting three times Ctrl-A on the serial console?
>Can you the hit '*' and send the output?

nothing happened when i tried Ctrl+a  3 times


>And lastly, did this work on baremetal?

Ya, it worked on baremetal


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 1052 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel

Re: Hugepage support

[Konrad Rzeszutek Wilk]

On Wed, Jan 12, 2011 at 03:03:18AM -0800, Keshav Darak wrote:
> 
> nothing happened when i tried Ctrl+a 3 times
> 
> --- On Mon, 1/10/11, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> 
> >And can you invoke the Xen hypervisor help by hitting three times Ctrl-A on the serial console?
> >Can you the hit '*' and send the output?
> 
> nothing happened when i tried Ctrl+a  3 times

Look at the PVOPS Wiki and also build the Xen hypervisor with debug enabled.
> 
> 
> >And lastly, did this work on baremetal?
> 
> Ya, it worked on baremetal
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 
>       

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: Hugepage support

[Keshav Darak]

[-- Attachment #1.1: Type: text/plain, Size: 2171 bytes --]

Well today, I compiled jeremy kernel 2.6.32.27 on DOMU and followed the steps given in the previous mail. First allocated hugepages and then written on them using C-Code from previous mail. But this time DomU didn't crashed but the terminal got handed.
I again checked dmesg but still there was no change at all. Even I put printk at few part of code which I found from the crashdump of the crashed kernel but still these printks were not reachable.

I think that jermey kernel must have handled this case but still its buggy as terminal is getting hanged.  
 
I am very much interested in doing this project  for supporting hugepages in PV-DOMU
.I found few of the work done by Dave McCracken. Lets see,waiting for his reply. 


--- On Wed, 1/12/11, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:

From: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Subject: Re: [Xen-devel] Hugepage support
To: "Keshav Darak" <keshav_darak@yahoo.com>
Cc: xen-devel@lists.xensource.com, "Ashwin Vasani" <vasani.ashwin@gmail.com>
Date: Wednesday, January 12, 2011, 2:16 PM

On Wed, Jan 12, 2011 at 03:03:18AM -0800, Keshav Darak wrote:
> 
> nothing
 happened when i tried Ctrl+a 3 times
> 
> --- On Mon, 1/10/11, Konrad Rzeszutek Wilk <konrad.wilk@oracle.com> wrote:
> 
> >And can you invoke the Xen hypervisor help by hitting three times Ctrl-A on the serial console?
> >Can you the hit '*' and send the output?
> 
> nothing happened when i tried Ctrl+a  3 times

Look at the PVOPS Wiki and also build the Xen hypervisor with debug enabled.

Soon will send the results.
> 
> 
> >And lastly, did this work on baremetal?
> 
> Ya, it worked on baremetal
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel
> 
> 
> 
>       

> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel



      

[-- Attachment #1.2: Type: text/html, Size: 3402 bytes --]

[-- Attachment #2: Type: text/plain, Size: 138 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xensource.com
http://lists.xensource.com/xen-devel