Pygrub on ARM64

Pygrub on ARM64

[Sanjeev Pandita]

[-- Attachment #1.1: Type: text/plain, Size: 1355 bytes --]

All,

Does pygrub work with ARM64 ?
I am trying to boot the CentOS-7-aarch64.img (raw) using pygrub.I am using
the below config file and it is not working.

[root@localhost xen]# cat vm1
name = "vm1"
uuid = "3fb78ba6-8182-484c-acf7-8faba9773f68"
disk = [ 'tap:raw:/mnt/xen/CentOS-7-aarch64.img,xvda,w' ]
memory = 512
bootloader="/usr/local/lib/xen/bin/pygrub"
extra = "root=/dev/xvda4 rw console=hvc0 earlyprintk=xen"
vif=[ 'mac=00:16:3e:01:01:01,bridge=br0' ]

In /var/log/xen the bootloader log is empty.

[root@localhost xen]# ls -al bootloader.24.log
-rw-r--r-- 1 root root 0 Feb 24 15:22 bootloader.24.log
[root@localhost xen]#

I got the image from
http://mirror.centos.org/altarch/7/isos/aarch64/CentOS-7-aarch64.img.xz
xz -d CentOS-7-aarch64.img.xz will create a CentOS-7-aarch64.img


Thanks,
Sanjeev

-- 
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is 
for the sole use of the intended recipient(s) and contains information that 
is confidential and proprietary to Applied Micro Circuits Corporation or 
its subsidiaries. It is to be used solely for the purpose of furthering the 
parties' business relationship. All unauthorized review, use, disclosure or 
distribution is prohibited. If you are not the intended recipient, please 
contact the sender by reply e-mail and destroy all copies of the original 
message.

[-- Attachment #1.2: Type: text/html, Size: 1798 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Pygrub on ARM64

[Ian Campbell]

On Wed, 2016-02-24 at 15:52 +0530, Sanjeev Pandita wrote:
> All,
> 
> Does pygrub work with ARM64 ?

It's not arch specific, so it should.

> I am trying to boot the CentOS-7-aarch64.img (raw) using pygrub.I am
> using the below config file and it is not working.

Please post the full "xl -vvv create vm1" output.

> [root@localhost xen]# cat vm1
> name = "vm1"
> uuid = "3fb78ba6-8182-484c-acf7-8faba9773f68"
> disk = [ 'tap:raw:/mnt/xen/CentOS-7-aarch64.img,xvda,w' ]
> memory = 512
> bootloader="/usr/local/lib/xen/bin/pygrub"

Can just be "pygrub" and xl will find the one which corresponds to the
install.

> extra = "root=/dev/xvda4 rw console=hvc0 earlyprintk=xen"
> vif=[ 'mac=00:16:3e:01:01:01,bridge=br0' ]
> In /var/log/xen the bootloader log is empty.
> [root@localhost xen]# ls -al bootloader.24.log
> -rw-r--r-- 1 root root 0 Feb 24 15:22 bootloader.24.log

Are you sure domain 24 is the correct one corresponding to a failed
attempt?

In the xl -vvv output you should see it logging the exact pygrub command
line it is going to use. If you run that manually what output does it
produce?

Ian.

Re: Pygrub on ARM64

[Sanjeev Pandita]

Hi Ian/All,

Here is the -vvv output

[root@localhost xen]# xl -vvv create vm1
Parsing config from vm1
libxl: debug: libxl_create.c:1557:do_domain_create: ao 0xa175050:
create: how=(nil) callback=(nil) poller=0xa1750e0
libxl: debug: libxl_arm.c:59:libxl__arch_domain_prepare_config:
Configure the domain
libxl: debug: libxl_arm.c:62:libxl__arch_domain_prepare_config:  -
Allocate 0 SPIs
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
vdev=xvda, using backend phy
libxl: debug: libxl_create.c:945:initiate_domain_create: running bootloader
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
vdev=(null) spec.backend=phy
libxl: debug: libxl.c:3097:libxl__device_disk_local_initiate_attach:
locally attaching RAW disk /mnt/xen/CentOS-7-aarch64.img
libxl: debug: libxl_bootloader.c:412:bootloader_disk_attached_cb:
Config bootloader value: pygrub
libxl: debug: libxl_bootloader.c:428:bootloader_disk_attached_cb:
Checking for bootloader in libexec path: /usr/local/lib/xen/bin/pygrub
libxl: debug: libxl_create.c:1580:do_domain_create: ao 0xa175050:
inprogress: poller=0xa1750e0, flags=i
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
w=0xa16c4d0 wpath=/local/domain/25 token=3/0: register slotnum=3
libxl: debug: libxl_event.c:2183:libxl__ao_progress_report: ao
0xa175050: progress report: ignored
libxl: debug: libxl_bootloader.c:540:bootloader_gotptys: executing
bootloader: /usr/local/lib/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader
arg: /usr/local/lib/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader
arg: --args=root=/dev/xvda4 rw console=hvc0 earlyprintk=xen
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader
arg: --output=/var/run/xen/bootloader.25.out
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader
arg: --output-format=simple0
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader
arg: --output-directory=/var/run/xen/bootloader.25.d
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader
arg: /mnt/xen/CentOS-7-aarch64.img
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0xa16c4d0
wpath=/local/domain/25 token=3/0: event epath=/local/domain/25


(gets stuck here. no prints after this)


[root@localhost ~]# xl list
Name                                        ID   Mem VCPUs      State   Time(s)
Domain-0                                     0  2048     4     r-----    4358.2
vm1                                         25     0     0     --p---       0.0
[root@localhost ~]#


If I run the command manually it says OSError: no such file or directory .

[root@localhost ~]# /usr/local/lib/xen/bin/pygrub
--args=root=/dev/xvda4 rw console=hvc0 earlyprintk=xen
--output=/var/run/xen/bootloader.25.out --output-format=simple0
--output-directory=/var/run/xen/bootloader.25.d
/mnt/xen/CentOS-7-aarch64.img
Traceback (most recent call last):
  File "/usr/local/lib/xen/bin/pygrub", line 888, in <module>
    part_offs = get_partition_offsets(file)
  File "/usr/local/lib/xen/bin/pygrub", line 113, in get_partition_offsets
    image_type = identify_disk_image(file)
  File "/usr/local/lib/xen/bin/pygrub", line 56, in identify_disk_image
    fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory: 'rw'

[root@localhost ~]#  ls -al /mnt/xen/CentOS-7-aarch64.img
-rw-r--r-- 1 root root 12884901888 Feb 24 14:54 /mnt/xen/CentOS-7-aarch64.img
[root@localhost ~]#


Tried setting rw permissions to the file as well. Still no luck.


[root@localhost ~]#ls -al /mnt/xen/CentOS-7-aarch64.img
-rwxrwxrwx 1 root root 12884901888 Feb 24 14:54 /mnt/xen/CentOS-7-aarch64.img

[root@localhost ~]# /usr/local/lib/xen/bin/pygrub
--args=root=/dev/xvda4 rw console=hvc0 earlyprintk=xen
--output=/var/run/xen/bootloader.25.out --output-format=simple0
--output-directory=/var/run/xen/bootloader.25.d
/mnt/xen/CentOS-7-aarch64.img
Traceback (most recent call last):
  File "/usr/local/lib/xen/bin/pygrub", line 888, in <module>
    part_offs = get_partition_offsets(file)
  File "/usr/local/lib/xen/bin/pygrub", line 113, in get_partition_offsets
    image_type = identify_disk_image(file)
  File "/usr/local/lib/xen/bin/pygrub", line 56, in identify_disk_image
    fd = os.open(file, os.O_RDONLY)
OSError: [Errno 2] No such file or directory: 'rw'

Thanks,
Sanjeev



On Wed, Feb 24, 2016 at 3:57 PM, Ian Campbell <ian.campbell@citrix.com> wrote:
> On Wed, 2016-02-24 at 15:52 +0530, Sanjeev Pandita wrote:
>> All,
>>
>> Does pygrub work with ARM64 ?
>
> It's not arch specific, so it should.
>
>> I am trying to boot the CentOS-7-aarch64.img (raw) using pygrub.I am
>> using the below config file and it is not working.
>
> Please post the full "xl -vvv create vm1" output.
>
>> [root@localhost xen]# cat vm1
>> name = "vm1"
>> uuid = "3fb78ba6-8182-484c-acf7-8faba9773f68"
>> disk = [ 'tap:raw:/mnt/xen/CentOS-7-aarch64.img,xvda,w' ]
>> memory = 512
>> bootloader="/usr/local/lib/xen/bin/pygrub"
>
> Can just be "pygrub" and xl will find the one which corresponds to the
> install.
>
>> extra = "root=/dev/xvda4 rw console=hvc0 earlyprintk=xen"
>> vif=[ 'mac=00:16:3e:01:01:01,bridge=br0' ]
>> In /var/log/xen the bootloader log is empty.
>> [root@localhost xen]# ls -al bootloader.24.log
>> -rw-r--r-- 1 root root 0 Feb 24 15:22 bootloader.24.log
>
> Are you sure domain 24 is the correct one corresponding to a failed
> attempt?
>
> In the xl -vvv output you should see it logging the exact pygrub command
> line it is going to use. If you run that manually what output does it
> produce?
>
> Ian.

-- 
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is 
for the sole use of the intended recipient(s) and contains information that 
is confidential and proprietary to Applied Micro Circuits Corporation or 
its subsidiaries. It is to be used solely for the purpose of furthering the 
parties' business relationship. All unauthorized review, use, disclosure or 
distribution is prohibited. If you are not the intended recipient, please 
contact the sender by reply e-mail and destroy all copies of the original 
message.

Re: Pygrub on ARM64

[Ian Campbell]

On Wed, 2016-02-24 at 16:35 +0530, Sanjeev Pandita wrote:
> Hi Ian/All,

Please don't top post.

> If I run the command manually it says OSError: no such file or directory
> .
> 
> [root@localhost ~]# /usr/local/lib/xen/bin/pygrub
> --args=root=/dev/xvda4 rw console=hvc0 earlyprintk=xen
> --output=/var/run/xen/bootloader.25.out --output-format=simple0
> --output-directory=/var/run/xen/bootloader.25.d
> /mnt/xen/CentOS-7-aarch64.img

[...]

> OSError: [Errno 2] No such file or directory: 'rw'

This is because you haven't quoted the "--args", everything from
"root=/dev/xvda4" up to "earlyprintk=xen" should be quoted in the shell,
that "rw" is supposed to be part of the eventual kernel command line[*].

You are using a local raw image, so it seems to be doing the local attach
thing and then passing the raw image to pygrub, which should work I think.

As well as trying to run pygrub by hand again with proper quoting you could
also try "xl console vm1" after you run the create, it could be that pygrub
is actually working and waiting for user input on the console.

Ian.

[*] normally this information would come from your grub.cfg in the guest,
so you might want to remove the extra = in your guest cfg, or maybe you are
deliberately overriding grub.cfg, in any case you are not yet at the point
where this setting would make any difference.

Ian.

Re: Pygrub on ARM64

[Sanjeev Pandita]

[-- Attachment #1.1: Type: text/plain, Size: 6901 bytes --]

On Wed, Feb 24, 2016 at 5:38 PM, Ian Campbell <ian.campbell@citrix.com>
wrote:

> On Wed, 2016-02-24 at 16:35 +0530, Sanjeev Pandita wrote:
> > Hi Ian/All,
>
> Please don't top post.
>
> > If I run the command manually it says OSError: no such file or directory
> > .
> >
> > [root@localhost ~]# /usr/local/lib/xen/bin/pygrub
> > --args=root=/dev/xvda4 rw console=hvc0 earlyprintk=xen
> > --output=/var/run/xen/bootloader.25.out --output-format=simple0
> > --output-directory=/var/run/xen/bootloader.25.d
> > /mnt/xen/CentOS-7-aarch64.img
>
> [...]
>
> > OSError: [Errno 2] No such file or directory: 'rw'
>
> This is because you haven't quoted the "--args", everything from
> "root=/dev/xvda4" up to "earlyprintk=xen" should be quoted in the shell,
> that "rw" is supposed to be part of the eventual kernel command line[*].
>
> You are using a local raw image, so it seems to be doing the local attach
> thing and then passing the raw image to pygrub, which should work I think.
>
> As well as trying to run pygrub by hand again with proper quoting you could
> also try "xl console vm1" after you run the create, it could be that pygrub
> is actually working and waiting for user input on the console.
>
>
Hi Ian,

I started fresh today.
Installed CentOS on a disk and compiled released xen 4.6.0 with
"./configure  --prefix=/usr ", followed by make tools and  make install.
Used xen.efi to boot the xen , vanilla kernel 4.2.4 (from kernel.org) and
booted the Dom0.

I ran the "xl console vm1" command in another terminal. No prints are seen
on the console. I have run some more commands in different terminals as
well. Here are the commands and results.

Original issue:-


Terminal 1: (xl create output)
[root@dhcp-194 xen]# xl -vvv create vm1
Parsing config from vm1
libxl: debug: libxl_create.c:1557:do_domain_create: ao 0x15d48050: create:
how=(nil) callback=(nil) poller=0x15d480e0
libxl: debug: libxl_arm.c:59:libxl__arch_domain_prepare_config: Configure
the domain
libxl: debug: libxl_arm.c:62:libxl__arch_domain_prepare_config:  - Allocate
0 SPIs
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
vdev=xvda spec.backend=unknown
libxl: debug: libxl_device.c:298:libxl__device_disk_set_backend: Disk
vdev=xvda, using backend phy
libxl: debug: libxl_create.c:945:initiate_domain_create: running bootloader
libxl: debug: libxl_device.c:269:libxl__device_disk_set_backend: Disk
vdev=(null) spec.backend=phy
libxl: debug: libxl.c:3097:libxl__device_disk_local_initiate_attach:
locally attaching RAW disk /mnt/xen/CentOS-7-aarch64.img
libxl: debug: libxl_bootloader.c:412:bootloader_disk_attached_cb: Config
bootloader value: pygrub
libxl: debug: libxl_bootloader.c:428:bootloader_disk_attached_cb: Checking
for bootloader in libexec path: /usr/lib/xen/bin/pygrub
libxl: debug: libxl_create.c:1580:do_domain_create: ao 0x15d48050:
inprogress: poller=0x15d480e0, flags=i
libxl: debug: libxl_event.c:639:libxl__ev_xswatch_register: watch
w=0x15d3f4d0 wpath=/local/domain/1 token=3/0: register slotnum=3
libxl: debug: libxl_event.c:2183:libxl__ao_progress_report: ao 0x15d48050:
progress report: ignored
libxl: debug: libxl_bootloader.c:540:bootloader_gotptys: executing
bootloader: /usr/lib/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader arg:
/usr/lib/xen/bin/pygrub
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader arg:
--args=root=/dev/xvda4 rw console=hvc0 earlyprintk=xen
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader arg:
--output=/var/run/xen/bootloader.1.out
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader arg:
--output-format=simple0
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader arg:
--output-directory=/var/run/xen/bootloader.1.d
libxl: debug: libxl_bootloader.c:544:bootloader_gotptys:   bootloader arg:
/mnt/xen/CentOS-7-aarch64.img
libxl: debug: libxl_event.c:576:watchfd_callback: watch w=0x15d3f4d0
wpath=/local/domain/1 token=3/0: event epath=/local/domain/1


Terminal 2: [root@dhcp-194 ~]# xl console vm1
                                                   (nothing comes on
console after this)



Terminal 3: (manually running the pygrub command)
[root@dhcp-194 xen]# export LD_LIBRARY_PATH=/usr/lib
[root@dhcp-194 xen]# mkdir -p /var/run/xen/bootloader.2.d
[root@dhcp-194 xen]# touch /var/run/xen/bootloader.2.out
[root@dhcp-194 xen]# /usr/lib/xen/bin/pygrub --args="root=/dev/xvda4 rw
console=hvc0 earlyprintk=xen" --output=/var/run/xen/bootloader.2.out
--output-format=simple0 --output-directory=/var/run/xen/bootloader.2.d
/mnt/xen/CentOS-7-aarch64.img

(nothing comes on console after this)

Terminal4: (ls of the logs directory and files. All files are empty)
[root@dhcp-194 ~]# cd /var/run/xen/
[root@dhcp-194 xen]# ls
bootloader.1.d  bootloader.1.out  bootloader.2.d  bootloader.2.out
[root@dhcp-194 xen]# ls -al
total 0
drwxr-xr-x  4 root root  120 Feb 25 12:50 .
drwxr-xr-x 35 root root 1100 Feb 25 10:00 ..
drw-------  2 root root   40 Feb 25 12:07 bootloader.1.d
-rw-------  1 root root    0 Feb 25 12:07 bootloader.1.out
drwxr-xr-x  2 root root   40 Feb 25 12:50 bootloader.2.d
-rw-r--r--  1 root root    0 Feb 25 12:50 bootloader.2.out
[root@dhcp-194 xen]# ls -al bootloader.*
-rw------- 1 root root  0 Feb 25 12:07 bootloader.1.out
-rw-r--r-- 1 root root  0 Feb 25 12:50 bootloader.2.out
bootloader.1.d:
total 0
drw------- 2 root root  40 Feb 25 12:07 .
drwxr-xr-x 4 root root 120 Feb 25 12:50 ..
bootloader.2.d:
total 0
drwxr-xr-x 2 root root  40 Feb 25 12:50 .
drwxr-xr-x 4 root root 120 Feb 25 12:50 ..
[root@dhcp-194 xen]#


Issue #2:

While above things are in dangling state if I try to create another VM
then a new DomU does not get create.

[root@dhcp-194 xen]# xl -vvv create vm8
Parsing config from vm8
<Nothing comes after the above line>


If I kill pygrub from another terminal , press ctrl C in all other blocking
terminals of vm1 and then start the vm8 , the vm8 boots fine.

Please let me know if am I missing anything in this sequence ?

Thanks,
Sanjeev


> Ian.
>
> [*] normally this information would come from your grub.cfg in the guest,
> so you might want to remove the extra = in your guest cfg, or maybe you are
> deliberately overriding grub.cfg, in any case you are not yet at the point
> where this setting would make any difference.
>
> Ian.
>

-- 
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is 
for the sole use of the intended recipient(s) and contains information that 
is confidential and proprietary to Applied Micro Circuits Corporation or 
its subsidiaries. It is to be used solely for the purpose of furthering the 
parties' business relationship. All unauthorized review, use, disclosure or 
distribution is prohibited. If you are not the intended recipient, please 
contact the sender by reply e-mail and destroy all copies of the original 
message.

[-- Attachment #1.2: Type: text/html, Size: 8728 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Pygrub on ARM64

[Ian Campbell]

On Thu, 2016-02-25 at 13:20 +0530, Sanjeev Pandita wrote:
> 
> Terminal 3: (manually running the pygrub command)
> [root@dhcp-194 xen]# export LD_LIBRARY_PATH=/usr/lib
> [root@dhcp-194 xen]# mkdir -p /var/run/xen/bootloader.2.d
> [root@dhcp-194 xen]# touch /var/run/xen/bootloader.2.out
> [root@dhcp-194 xen]# /usr/lib/xen/bin/pygrub --args="root=/dev/xvda4 rw
> console=hvc0 earlyprintk=xen" --output=/var/run/xen/bootloader.2.out --
> output-format=simple0 --output-directory=/var/run/xen/bootloader.2.d
> /mnt/xen/CentOS-7-aarch64.img
> 
> (nothing comes on console after this) 

I don't remember ever seeing pygrub fail silently in this way.

I think at this point I would be trying a few different things, firstly
using strace(1) on the pygrub invocation to see if I could see where it was
blocked. I think you can drop all of the arguments except for the image,
e.g.

    pygrub /mnt/xen/CentOS-7-aarch64.img

leading to

    strace -o pygrub.strace pygrub /mnt/xen/CentOS-7-aarch64.img

(maybe add -fff if it looks to be using threads)

Secondly manually mounting CentOS-7-aarch64.img (e.g. "mount -o loop etc",
or maybe kpartx -a first if the image has a partition table) to check it
really is some sort of sensible/readable image.

Ian.

> 
> Terminal4: (ls of the logs directory and files. All files are empty)
> [root@dhcp-194 ~]# cd /var/run/xen/
> [root@dhcp-194 xen]# ls
> bootloader.1.d  bootloader.1.out  bootloader.2.d  bootloader.2.out
> [root@dhcp-194 xen]# ls -al
> total 0
> drwxr-xr-x  4 root root  120 Feb 25 12:50 .
> drwxr-xr-x 35 root root 1100 Feb 25 10:00 ..
> drw-------  2 root root   40 Feb 25 12:07 bootloader.1.d
> -rw-------  1 root root    0 Feb 25 12:07 bootloader.1.out
> drwxr-xr-x  2 root root   40 Feb 25 12:50 bootloader.2.d
> -rw-r--r--  1 root root    0 Feb 25 12:50 bootloader.2.out
> [root@dhcp-194 xen]# ls -al bootloader.*
> -rw------- 1 root root  0 Feb 25 12:07 bootloader.1.out
> -rw-r--r-- 1 root root  0 Feb 25 12:50 bootloader.2.out
> bootloader.1.d:
> total 0
> drw------- 2 root root  40 Feb 25 12:07 .
> drwxr-xr-x 4 root root 120 Feb 25 12:50 ..
> bootloader.2.d:
> total 0
> drwxr-xr-x 2 root root  40 Feb 25 12:50 .
> drwxr-xr-x 4 root root 120 Feb 25 12:50 ..
> [root@dhcp-194 xen]#
> 
> 
> Issue #2:
> 
> While above things are in dangling state if I try to create another VM
> then a new DomU does not get create.
> 
> [root@dhcp-194 xen]# xl -vvv create vm8
> Parsing config from vm8
> <Nothing comes after the above line>
> 
> 
> If I kill pygrub from another terminal , press ctrl C in all other
> blocking terminals of vm1 and then start the vm8 , the vm8 boots fine.
> 
> Please let me know if am I missing anything in this sequence ?
> 
> Thanks,
> Sanjeev
>  
> >  Ian.
> > 
> > [*] normally this information would come from your grub.cfg in the
> > guest,
> > so you might want to remove the extra = in your guest cfg, or maybe you
> > are
> > deliberately overriding grub.cfg, in any case you are not yet at the
> > point
> > where this setting would make any difference.
> > 
> > Ian.
> > 
> 
> CONFIDENTIALITY NOTICE: This e-mail message, including any attachments,
> is for the sole use of the intended recipient(s) and contains information
> that is confidential and proprietary to Applied Micro Circuits
> Corporation or its subsidiaries. It is to be used solely for the purpose
> of furthering the parties' business relationship. All unauthorized
> review, use, disclosure or distribution is prohibited. If you are not the
> intended recipient, please contact the sender by reply e-mail and destroy
> all copies of the original message.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Pygrub on ARM64

[Stefano Stabellini]

[-- Attachment #1: Type: text/plain, Size: 1920 bytes --]

On Thu, 25 Feb 2016, Ian Campbell wrote:
> On Thu, 2016-02-25 at 13:20 +0530, Sanjeev Pandita wrote:
> > 
> > Terminal 3: (manually running the pygrub command)
> > [root@dhcp-194 xen]# export LD_LIBRARY_PATH=/usr/lib
> > [root@dhcp-194 xen]# mkdir -p /var/run/xen/bootloader.2.d
> > [root@dhcp-194 xen]# touch /var/run/xen/bootloader.2.out
> > [root@dhcp-194 xen]# /usr/lib/xen/bin/pygrub --args="root=/dev/xvda4 rw
> > console=hvc0 earlyprintk=xen" --output=/var/run/xen/bootloader.2.out --
> > output-format=simple0 --output-directory=/var/run/xen/bootloader.2.d
> > /mnt/xen/CentOS-7-aarch64.img
> > 
> > (nothing comes on console after this) 
> 
> I don't remember ever seeing pygrub fail silently in this way.
> 
> I think at this point I would be trying a few different things, firstly
> using strace(1) on the pygrub invocation to see if I could see where it was
> blocked. I think you can drop all of the arguments except for the image,
> e.g.
> 
>     pygrub /mnt/xen/CentOS-7-aarch64.img
> 
> leading to
> 
>     strace -o pygrub.strace pygrub /mnt/xen/CentOS-7-aarch64.img
> 
> (maybe add -fff if it looks to be using threads)
> 
> Secondly manually mounting CentOS-7-aarch64.img (e.g. "mount -o loop etc",
> or maybe kpartx -a first if the image has a partition table) to check it
> really is some sort of sensible/readable image.

CentOS's first partition is fat to boot on efi systems, but it only
contains the grub efi binary. All the other partitions are xfs. Does
pygrub support it?



I know that's know what you asked, but alternatively you could use
Tianocore to boot the the guest, which usually works pretty well. You
need to specify:

kernel="/path/to/XEN_EFI.fd"

in your VM config, where XEN_EFI.fd is your tianocore arm64 binary.
You can build Tianocore on arm64 with Xen support with the following
runes:

source edksetup.sh
build -a AARCH64 -t GCC48 -p ArmVirtPkg/ArmVirtXen.dsc -b RELEASE

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Pygrub on ARM64

[Ian Campbell]

On Thu, 2016-02-25 at 10:35 +0000, Stefano Stabellini wrote:
> On Thu, 25 Feb 2016, Ian Campbell wrote:
> > On Thu, 2016-02-25 at 13:20 +0530, Sanjeev Pandita wrote:
> > >  
> > > Terminal 3: (manually running the pygrub command)
> > > [root@dhcp-194 xen]# export LD_LIBRARY_PATH=/usr/lib
> > > [root@dhcp-194 xen]# mkdir -p /var/run/xen/bootloader.2.d
> > > [root@dhcp-194 xen]# touch /var/run/xen/bootloader.2.out
> > > [root@dhcp-194 xen]# /usr/lib/xen/bin/pygrub --args="root=/dev/xvda4
> > > rw
> > > console=hvc0 earlyprintk=xen" --output=/var/run/xen/bootloader.2.out
> > > --
> > > output-format=simple0 --output-directory=/var/run/xen/bootloader.2.d
> > > /mnt/xen/CentOS-7-aarch64.img
> > > 
> > > (nothing comes on console after this) 
> > 
> > I don't remember ever seeing pygrub fail silently in this way.
> > 
> > I think at this point I would be trying a few different things, firstly
> > using strace(1) on the pygrub invocation to see if I could see where it
> > was
> > blocked. I think you can drop all of the arguments except for the
> > image,
> > e.g.
> > 
> >     pygrub /mnt/xen/CentOS-7-aarch64.img
> > 
> > leading to
> > 
> >     strace -o pygrub.strace pygrub /mnt/xen/CentOS-7-aarch64.img
> > 
> > (maybe add -fff if it looks to be using threads)
> > 
> > Secondly manually mounting CentOS-7-aarch64.img (e.g. "mount -o loop
> > etc",
> > or maybe kpartx -a first if the image has a partition table) to check
> > it
> > really is some sort of sensible/readable image.
> 
> CentOS's first partition is fat to boot on efi systems, but it only
> contains the grub efi binary. All the other partitions are xfs. Does
> pygrub support it?

I don't know, but pygrub is normally pretty vocal in these situations if it
doesn't ("cannot find partition" style message), here it is apparently just
silently hanging, which is pretty odd.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Pygrub on ARM64

[Ian Campbell]

On Thu, 2016-02-25 at 10:47 +0000, Ian Campbell wrote:
> 
> > 
> > CentOS's first partition is fat to boot on efi systems, but it only
> > contains the grub efi binary. All the other partitions are xfs. Does
> > pygrub support it?
> 
> I don't know, but pygrub is normally pretty vocal in these situations if it
> doesn't ("cannot find partition" style message), here it is apparently just
> silently hanging, which is pretty odd.

Here is what I get though:

# wget http://mirror.centos.org/altarch/7/isos/aarch64/CentOS-7-aarch64.img.xz
# xz -d CentOS-7-aarch64.img.xz
# pygrub CentOS-7-aarch64.img 
Traceback (most recent call last):
  File "/usr/local/bin/pygrub", line 926, in <module>
    raise RuntimeError, "Unable to find partition containing kernel"
RuntimeError: Unable to find partition containing kernel

Which is much more like what I would normally expect (and matches your
description of the image layout  think).

That was on a random x86_64 test box because that was all I had to hand,
but I'd be pretty surprised if something like this behaved very differently
on arm64. 

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Pygrub on ARM64

[Ian Campbell]

On Thu, 2016-02-25 at 10:53 +0000, Ian Campbell wrote:
> On Thu, 2016-02-25 at 10:47 +0000, Ian Campbell wrote:
> > 
> > > 
> > > CentOS's first partition is fat to boot on efi systems, but it only
> > > contains the grub efi binary. All the other partitions are xfs. Does
> > > pygrub support it?
> > 
> > I don't know, but pygrub is normally pretty vocal in these situations
> > if it
> > doesn't ("cannot find partition" style message), here it is apparently
> > just
> > silently hanging, which is pretty odd.
> 
> Here is what I get though:
> 
> # wget http://mirror.centos.org/altarch/7/isos/aarch64/CentOS-7-aarch64.i
> mg.xz
> # xz -d CentOS-7-aarch64.img.xz
> # pygrub CentOS-7-aarch64.img 
> Traceback (most recent call last):
>   File "/usr/local/bin/pygrub", line 926, in <module>
>     raise RuntimeError, "Unable to find partition containing kernel"
> RuntimeError: Unable to find partition containing kernel
> 
> Which is much more like what I would normally expect (and matches your
> description of the image layout  think).
> 
> That was on a random x86_64 test box because that was all I had to hand,
> but I'd be pretty surprised if something like this behaved very
> differently on arm64. 

But running on both arm32 arm64 it does indeed spin taking 100% of the CPU.

My guess would be a bug in the file system parsing code, perhaps something
to do with alignment, which results in an infinite loop. The libfsimage
backends originally came from grub (1/legacy) and is therefore only really
has history on x86.

We certainly test pygrub on arm32 regularly, and I am sure it must have
worked on arm64 when I was adding arm64 support to osstest (which is
_still_ pending h/w to go into the colo!), so the basics (i..e simple
partition tables with ext* partitions) must be ok, so I would hazzard a
stab in the dark that the issue is in one of the things specific to these
images, i.e. either the FAT or the XFS backend (if there even is an XFS
backend, I didn't check).

Ian.


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Pygrub on ARM64

[Ian Campbell]

On Thu, 2016-02-25 at 11:12 +0000, Ian Campbell wrote:
> On Thu, 2016-02-25 at 10:53 +0000, Ian Campbell wrote:
> > On Thu, 2016-02-25 at 10:47 +0000, Ian Campbell wrote:
> > > 
> > > > 
> > > > CentOS's first partition is fat to boot on efi systems, but it only
> > > > contains the grub efi binary. All the other partitions are xfs.
> > > > Does
> > > > pygrub support it?
> > > 
> > > I don't know, but pygrub is normally pretty vocal in these situations
> > > if it
> > > doesn't ("cannot find partition" style message), here it is
> > > apparently
> > > just
> > > silently hanging, which is pretty odd.
> > 
> > Here is what I get though:
> > 
> > # wget http://mirror.centos.org/altarch/7/isos/aarch64/CentOS-7-aarch64
> > .i
> > mg.xz
> > # xz -d CentOS-7-aarch64.img.xz
> > # pygrub CentOS-7-aarch64.img 
> > Traceback (most recent call last):
> >   File "/usr/local/bin/pygrub", line 926, in <module>
> >     raise RuntimeError, "Unable to find partition containing kernel"
> > RuntimeError: Unable to find partition containing kernel
> > 
> > Which is much more like what I would normally expect (and matches your
> > description of the image layout  think).
> > 
> > That was on a random x86_64 test box because that was all I had to
> > hand,
> > but I'd be pretty surprised if something like this behaved very
> > differently on arm64. 
> 
> But running on both arm32 arm64 it does indeed spin taking 100% of the
> CPU.
> 
> My guess would be a bug in the file system parsing code, perhaps
> something
> to do with alignment, which results in an infinite loop. The libfsimage
> backends originally came from grub (1/legacy) and is therefore only
> really
> has history on x86.
> 
> We certainly test pygrub on arm32 regularly, and I am sure it must have
> worked on arm64 when I was adding arm64 support to osstest (which is
> _still_ pending h/w to go into the colo!), so the basics (i..e simple
> partition tables with ext* partitions) must be ok, so I would hazzard a
> stab in the dark that the issue is in one of the things specific to these
> images, i.e. either the FAT or the XFS backend (if there even is an XFS
> backend, I didn't check).

Given:

# fdisk -l ./CentOS-7-aarch64.img 
WARNING: fdisk GPT support is currently new, and therefore in an experimental phase. Use at your own discretion.

Disk ./CentOS-7-aarch64.img: 12.9 GB, 12884901888 bytes, 25165824 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: gpt


#         Start          End    Size  Type            Name
 1         2048       206847    100M  EFI System      EFI System Partition
 2       206848      1026047    400M  Linux filesyste Linux filesystem
 3      1026048      5122047      2G  Linux swap      Linux swap
 4      5122048     24373247    9.2G  Linux filesyste Linux filesystem

I see

    # python
    Python 2.7.5 (default, Nov 26 2015, 23:01:14) 
    [GCC 4.8.5 20150623 (Red Hat 4.8.5-4)] on linux2
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import fsimage
    >>> N = <xxxx>
    >>> fs = fsimage.open("CentOS-7-aarch64.img", N*512, "")

succeed for N = 2048 (ESP), fail (as expected) with "Operation not
supported" for N=1026048 (Swap) and hang for both N=206848 and N=5122048,
i.e. the two XFS filesystems in the image hang.

I'm afraid someone is probably going to have to dig
into tools/libfsimage/xfs/fsys_xfs.c to figure out what is going wrong. It
would probably be easiest (IMHO) to take Python out of the picture by
writing a simple main.c which uses libfsimage directly to open one of the
partitions (hardcoded offsets etc) and then use gdb on it to see why/where
it hangs.

Ian.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel

Re: Pygrub on ARM64

[Sanjeev Pandita]

[-- Attachment #1: Type: text/plain, Size: 5481 bytes --]

On Thu, Feb 25, 2016 at 3:09 PM, Ian Campbell <ian.campbell@citrix.com> wrote:
>
> On Thu, 2016-02-25 at 13:20 +0530, Sanjeev Pandita wrote:
> >
> > Terminal 3: (manually running the pygrub command)
> > [root@dhcp-194 xen]# export LD_LIBRARY_PATH=/usr/lib
> > [root@dhcp-194 xen]# mkdir -p /var/run/xen/bootloader.2.d
> > [root@dhcp-194 xen]# touch /var/run/xen/bootloader.2.out
> > [root@dhcp-194 xen]# /usr/lib/xen/bin/pygrub --args="root=/dev/xvda4 rw
> > console=hvc0 earlyprintk=xen" --output=/var/run/xen/bootloader.2.out --
> > output-format=simple0 --output-directory=/var/run/xen/bootloader.2.d
> > /mnt/xen/CentOS-7-aarch64.img
> >
> > (nothing comes on console after this)
>
> I don't remember ever seeing pygrub fail silently in this way.
>
> I think at this point I would be trying a few different things, firstly
> using strace(1) on the pygrub invocation to see if I could see where it was
> blocked. I think you can drop all of the arguments except for the image,
> e.g.
>
>     pygrub /mnt/xen/CentOS-7-aarch64.img
>
> leading to
>
>     strace -o pygrub.strace pygrub /mnt/xen/CentOS-7-aarch64.img

Hi Ian,

I ran the plain command "trace -o pygrub.strace pygrub
/mnt/xen/CentOS-7-aarch64.img".

In the "pygrub_beforehang_ctrlc_kill9.zip"  file I have three files.
First strace file "pygrub.strace_before_CTRLC" is file at hang.
Second  strace file "pygrub.strace_after_CTRLC" is file after I press
CTRL+c. (pygrub does not exit after CTRL+C).
Third strace file "pygrub.strace_after_KILL-9" is file after I run kill -9 PID.

You can skip second and third files in the zip file as they do not
have much info except the below lines.

After CTRL+C
--- SIGINT {si_signo=SIGINT, si_code=SI_KERNEL, si_value={int=0,
ptr=0x7f00000000
0}} ---
rt_sigreturn()                          = 63404881008

After Kill-9 gets executed

+++ killed by SIGKILL +++

> (maybe add -fff if it looks to be using threads)

I have attached the "pygrub_strace_fff_2process_exit.zip" file which
contains the strace files by running "strace -o pygrub.strace -fff
pygrub /mnt/xen/CentOS-7-aarch64.img" command.
It contains three files out of which two have exit(0) in the end.
pygrub.strace.18269 has more calls than other two files.

Thanks,
Sanjeev

> Secondly manually mounting CentOS-7-aarch64.img (e.g. "mount -o loop etc",
> or maybe kpartx -a first if the image has a partition table) to check it
> really is some sort of sensible/readable image.
>
> Ian.
>
> >
> > Terminal4: (ls of the logs directory and files. All files are empty)
> > [root@dhcp-194 ~]# cd /var/run/xen/
> > [root@dhcp-194 xen]# ls
> > bootloader.1.d  bootloader.1.out  bootloader.2.d  bootloader.2.out
> > [root@dhcp-194 xen]# ls -al
> > total 0
> > drwxr-xr-x  4 root root  120 Feb 25 12:50 .
> > drwxr-xr-x 35 root root 1100 Feb 25 10:00 ..
> > drw-------  2 root root   40 Feb 25 12:07 bootloader.1.d
> > -rw-------  1 root root    0 Feb 25 12:07 bootloader.1.out
> > drwxr-xr-x  2 root root   40 Feb 25 12:50 bootloader.2.d
> > -rw-r--r--  1 root root    0 Feb 25 12:50 bootloader.2.out
> > [root@dhcp-194 xen]# ls -al bootloader.*
> > -rw------- 1 root root  0 Feb 25 12:07 bootloader.1.out
> > -rw-r--r-- 1 root root  0 Feb 25 12:50 bootloader.2.out
> > bootloader.1.d:
> > total 0
> > drw------- 2 root root  40 Feb 25 12:07 .
> > drwxr-xr-x 4 root root 120 Feb 25 12:50 ..
> > bootloader.2.d:
> > total 0
> > drwxr-xr-x 2 root root  40 Feb 25 12:50 .
> > drwxr-xr-x 4 root root 120 Feb 25 12:50 ..
> > [root@dhcp-194 xen]#
> >
> >
> > Issue #2:
> >
> > While above things are in dangling state if I try to create another VM
> > then a new DomU does not get create.
> >
> > [root@dhcp-194 xen]# xl -vvv create vm8
> > Parsing config from vm8
> > <Nothing comes after the above line>
> >
> >
> > If I kill pygrub from another terminal , press ctrl C in all other
> > blocking terminals of vm1 and then start the vm8 , the vm8 boots fine.
> >
> > Please let me know if am I missing anything in this sequence ?
> >
> > Thanks,
> > Sanjeev
> >
> > >  Ian.
> > >
> > > [*] normally this information would come from your grub.cfg in the
> > > guest,
> > > so you might want to remove the extra = in your guest cfg, or maybe you
> > > are
> > > deliberately overriding grub.cfg, in any case you are not yet at the
> > > point
> > > where this setting would make any difference.
> > >
> > > Ian.
> > >
> >
> > CONFIDENTIALITY NOTICE: This e-mail message, including any attachments,
> > is for the sole use of the intended recipient(s) and contains information
> > that is confidential and proprietary to Applied Micro Circuits
> > Corporation or its subsidiaries. It is to be used solely for the purpose
> > of furthering the parties' business relationship. All unauthorized
> > review, use, disclosure or distribution is prohibited. If you are not the
> > intended recipient, please contact the sender by reply e-mail and destroy
> > all copies of the original message.

-- 
CONFIDENTIALITY NOTICE: This e-mail message, including any attachments, is 
for the sole use of the intended recipient(s) and contains information that 
is confidential and proprietary to Applied Micro Circuits Corporation or 
its subsidiaries. It is to be used solely for the purpose of furthering the 
parties' business relationship. All unauthorized review, use, disclosure or 
distribution is prohibited. If you are not the intended recipient, please 
contact the sender by reply e-mail and destroy all copies of the original 
message.

[-- Attachment #2: pygrub_beforehang_ctrlc_kill9.zip --]
[-- Type: application/zip, Size: 49448 bytes --]

[-- Attachment #3: pygrub_strace_fff_2process_exit.zip --]
[-- Type: application/zip, Size: 19449 bytes --]

[-- Attachment #4: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel